Skip to content

You are viewing documentation for Immuta version 2024.1.

For the latest version, view our documentation for Immuta SaaS or the latest self-hosted version.

cancel

Use Project UDFs (Databricks)

Audience: Data Users

Content Summary: This page outlines the available functions to view and switch your projects using UDFs in Databricks. A tutorial is also provided to illustrate how to switch project contexts.

Project UDFs Overview

Use Project UDFs in Databricks

Currently, caches are not all invalidated outside of Databricks because Immuta caches information pertaining to a user's current project in the NameNode plugin and in Vulcan. Consequently, this feature should only be used in Databricks.

You can switch project contexts and view a list of your current project or available projects through UDFs in Spark.

Available Functions

UDF Description
immuta.set_current_project(id) Sets the user's current project to the project ID denoted by the id parameter. This UDF must be called in its own notebook cell to ensure the changes take effect.
immuta.set_current_project() (no parameters) Sets the user's current project to None.
immuta.clear_caches() Clears all client caches for the current user's ImmutaClient instance. This can be used when a user would like to invalidate cached items, like data source subscription information or if the state of Immuta has changed and the cache is outdated. For backward compatibility, this UDF is also available at default.immuta_clear_caches()
default.immuta_clear_metastore_cache() Clears the cluster-wide Metastore cache. This UDF can only be run by a privileged user.

Virtual Tables

To view a list of your current project or available projects in a Spark job, you can query these virtual tables.

Virtual Table Query Return
immuta.get_current_project select * from immuta.get_current_project This virtual table returns a single row with "name" and "id" columns that show your currently selected project.
immuta.list_projects select * from immuta.list_projects This virtual table returns rows with "name," "id," and "current_project" columns. Each row is a different project to which you are subscribed (and can use as your current project). The "current_project" row will be true for the row defining the project that you have set as your current project.

Switch Your Project Using UDFs

  1. View your available projects by running the following query in Spark: select * from immuta.list_projects. In the resulting table, note the values listed in the id column; this value will be used at the parameter in the following step.
  2. Run select immuta.set_current_project(<id>). This UDF must be called in its own notebook cell to ensure the changes take effect.

Your project context will be switched, and that project's data sources and workspaces will now be visible. To set your project context to None, run select immuta.set_current_project() with no parameters.

Note: Since the UDFs are not actually registered with the FunctionRegistry, if you call DESCRIBE FUNCTION immuta.set_current_project, you won't get back the documentation for the UDF.