Compute Libraries and Access Permissions

Install dependencies at the right scope

Notebooks and jobs regularly need packages that the runtime does not ship. The first decision is not which package, but how widely the installation should reach.

A compute-scoped library installs on the cluster and becomes available to every notebook and job that runs on that cluster. Azure Databricks reinstalls it automatically each time the cluster starts, so the environment survives restarts and nobody has to reinstall anything by hand. That is what makes a shared cluster consistent for everyone attached to it.

A notebook-scoped library, by contrast, applies only to the session of the notebook that installed it. Nothing about it changes the cluster or affects anyone else.

Neither scope reaches further than that: an installation never propagates to every cluster in the workspace, and it never follows a user account across workspaces.

The width of compute scope is also its limitation. Because the library affects every workload on the cluster, two teams that need incompatible versions of the same package cannot both be served by one compute-scoped installation.

Installing both versions cluster-wide does not resolve the conflict, it stages it on the shared cluster. Storing both files in a volume and installing both cluster-wide has exactly the same effect: the scope is what matters, not where the files sit.

Where libraries come from

Compute-scoped libraries accept three artifact kinds: Python wheels, Java JAR files, and R packages. They can be installed from a package repository or from files you control.

Repositories resolve dependencies for you, which is convenient and slightly dangerous: convenience includes picking a version.

For production, specify an exact version, for example pymssql==2.3.9. Without a version the latest available package is installed, and "latest" can change between two cluster starts and break code that worked yesterday. Pinning is about reproducible installations and nothing else: it does not change a package's type, it does not bypass any approval, and it grants nobody extra permission.

Maven libraries are identified by coordinates in the format groupId:artifactId:version, such as com.microsoft.sqlserver:mssql-jdbc:13.2.1.jre11. Maven can also exclude a transitive dependency that would conflict with something already installed.

CRAN behaves differently: it always pulls the latest version from the configured mirror. To pin an R package, store the package file in workspace files or a volume instead of installing from CRAN.

Libraries kept as files

Some libraries are not in a public repository: internally built packages, or a specific version no longer published. Store those files in workspace files or a Unity Catalog volume, and reference the file when installing.

Both keep library management centralized instead of scattering ad-hoc installations through notebooks. A Python requirements file works from either location on recent runtimes, installing every package it lists in one step.

A volume adds one requirement of its own. The identity performing the installation must hold READ VOLUME on the volume that stores the library file. It is a read of a governed file, so the privilege needed is the one that permits reading that volume.

Notice what it is not:

• Not write access, and not across every volume in the catalog: reading one file needs read on that volume.
• Not a compute permission. Cluster permission levels govern the cluster, not the contents of a volume.
• Not a metastore-level privilege such as creating catalogs.

Approval on shared clusters

A cluster in Standard access mode is shared by several users, so the platform will not let arbitrary code onto it. Maven coordinates, JAR paths, library file paths, and init script paths require allowlist approval, added by a metastore admin, before they can be installed.

Two details are worth remembering. Approval entries use prefix matching, so approving a directory permits the files beneath it, and an init script needs its own entry even when it sits beside an approved JAR.

The most common failure here comes from assuming approval is enough. An allowlist entry only permits the use of that coordinate or path; it does not grant access to the underlying data. The two checks are independent and both must pass.

So a Standard-mode installation of a JAR stored in a volume needs three things to be true at once:

When an operator who manages the cluster still hits an access failure, the missing piece is one of the first two, and checking only one of them leaves the case half-diagnosed.

Init scripts are the last resort

An init script runs shell commands during cluster startup, before the driver and executors come up. It is the tool for system-level configuration that a library cannot provide: installing system packages, setting environment variables, or preparing a monitoring agent.

Databricks does not recommend init scripts as the normal way to install libraries. Compute-scoped libraries are the better path for ordinary packages, and they are simpler to reason about.

Two consequences follow from running at startup. Scripts execute in the order you specify, and a script returning a non-zero exit code prevents the cluster from starting at all, which is a deliberate protection against running with an incomplete configuration.

An init script also solves nothing about permissions: someone who cannot modify the cluster cannot add one, and a dependency needed by a single notebook session belongs in a notebook-scoped installation.

Four levels of compute permission

Compute permissions are workspace-level and are separate from Azure subscription permissions. They build up in four tiers, each adding to the one below it.

Two boundaries in that ladder decide most questions.

The first: CAN ATTACH TO cannot control the lifecycle. It is the usage tier — run work, watch it execute — and the person who has it cannot stop or restart anything, resize the cluster, install a shared library, or change anyone's permissions.

The second: installing a compute-scoped library requires CAN MANAGE. Adding a library modifies cluster configuration, which is why the library installation interface is unavailable below that level, and why CAN RESTART is not enough even though it sounds close. Restarting is lifecycle; installing is configuration.

Workspace admins hold CAN MANAGE on every compute resource, and whoever creates a compute resource owns it with the same level.

Grant access with least privilege

Permissions are granted in the Azure Databricks workspace interface, not in the Azure portal, and changes take effect immediately — no cluster restart is required to add or revoke access.

Prefer grants to groups rather than to individuals. When someone joins or leaves a team, membership does the work, and no one has to revisit every compute resource hunting for stale access.

Then apply least privilege by mapping each role to the lowest level that lets it work:

Reading that table upwards is the sanity check: an analyst who can restart production compute, or a lead who can rewrite its configuration, has more authority than the job requires.

Key takeaways

• Compute-scoped libraries serve every notebook and job on the cluster and are reinstalled at each start; conflicting versions belong in notebook-scoped installations or separate clusters.
• Pin an exact version when reproducibility matters, remember Maven's groupId:artifactId:version coordinates, and store files in workspace files or volumes when a repository cannot supply the version you need.
• Treat the allowlist and file-access privileges as separate checks: approval permits the path, READ VOLUME permits reading the file.
• CAN ATTACH TO permits use and monitoring, CAN RESTART adds lifecycle control, and CAN MANAGE adds configuration, library, and permission control.
• Installing a compute-scoped library requires CAN MANAGE; grant to groups, apply least privilege, and expect permission changes to take effect immediately.