Selecting Azure Databricks Compute Types

Start from the work, not from the cluster

Every compute decision begins with the work you have to execute and the capabilities that work requires. Pick the wrong surface and you pay for idle machines, wait minutes for a cluster that should have started in seconds, or discover halfway through that a language or an API is simply not available.

Serverless compute is managed by Azure Databricks. You do not provision or configure infrastructure: the platform automatically allocates and scales resources to match the demand of the workload, and it scales down when the work stops. Those resources run in the Databricks Azure subscription, not yours, so no virtual machines or networking components appear in your own subscription. Startup typically takes seconds rather than minutes.

Serverless requires Unity Catalog, and it is available for notebooks, jobs, Lakeflow Spark Declarative Pipelines, and SQL warehouses.

Serverless buys speed and simplicity by taking choices away from you. Its runtime is versionless: Azure Databricks applies upgrades for you, so you always run recent features without a migration project, but you do not pin or schedule a version.

The trade-off shows up as unsupported capabilities. In notebooks, serverless does not support:

Choose serverless when fast startup and minimal infrastructure management matter more than those capabilities. Exploratory analysis, ETL pipelines, and business intelligence workloads are its natural home.

Classic compute and runtime control

Classic compute is the opposite bargain: you create, size, and manage clusters that run in your own Azure subscription, with visibility over the underlying infrastructure. You control instance types, autoscaling rules, and the Databricks Runtime version. Startup typically ranges from a few minutes rather than seconds.

Runtime management is the cleanest way to tell the two apart.

Operating-system and security updates can be automated on classic compute, but moving from one runtime version to another remains a deliberate choice you make. Use classic compute when the workload needs infrastructure control, custom networking, a specific instance type, or a capability that serverless does not offer.

Access modes and cluster shape

Classic compute has two access modes, and they answer one question: who is allowed on this cluster at the same time.

Standard access mode lets multiple users share a single cluster concurrently. Lakeguard keeps user code isolated, so one person's work cannot interfere with another's. This is the mode for collaborative data engineering and shared analytics, and it lowers cost because resources are pooled instead of duplicated per person. Compute in Standard access mode is what people mean by shared compute.

Dedicated access mode assigns the cluster to a single user or group. Because that user or group owns the machine, dedicated access grants machine-level privileges, and those privileges are what unlock RDD APIs, GPU workloads, R language support, and custom container environments.

Access mode is not the same dial as cluster mode, which describes the architecture of the cluster itself.

A multi-node cluster has one driver and one or more workers: the driver coordinates and the workers compute in parallel, so the cluster scales horizontally for large volumes. A single-node cluster has a driver and no workers, so nothing distributes; it suits light exploration, small datasets, and machine-learning frameworks that do not distribute on their own. Single-node clusters cannot scale out, so distributed processing needs a multi-node cluster.

SQL warehouses for analytics

A SQL warehouse is compute optimized specifically for SQL queries, analytics, and business intelligence. When the interface to the work is SQL and the audience is dashboards and reports, this is the surface to reach for, not a notebook cluster.

There are three types, and they are not interchangeable.

The custom-networking clause is the one that decides real cases. A pro SQL warehouse is the warehouse type that supports custom network configurations, such as reaching an on-premises database through federation or integrating with services inside your own virtual network.

Serverless warehouses cannot provide that integration, because serverless compute operates in the Databricks-managed subscription and does not join your network. So a SQL workload that must reach an on-premises system through a custom network lands on pro, even though serverless would otherwise be the faster and more automatic option.

Classic warehouses stay at the bottom of the list: Photon and basic autoscaling, useful when nothing better is available.

Job compute for automated runs

Job compute is compute for automated workflows rather than interactive development. Its defining behavior is the lifecycle: a job cluster terminates automatically once its tasks finish, so no idle compute survives between runs. That is the difference from an interactive cluster, which stays up waiting for the next person to type something.

When you configure a job you still choose between two flavors:

• Serverless job compute: faster startup, automatic infrastructure management, and lower cost for most automated work.
• Classic job compute: more configuration options, for workloads that need something serverless does not support.

Classic job compute keeps the optimization levers: autoscaling, instance choice, and spot instances, which use spare Azure capacity at a steep discount. Azure can reclaim a spot machine on short notice, and Spark's fault tolerance retries the affected tasks elsewhere, which makes spot a reasonable saving for batch and ETL work.

The Job Compute policy provides a template for production-ready job clusters with sensible defaults, including a recent long-term-support runtime.

Put the two decisions in order: first job compute rather than an interactive cluster, then serverless unless a specific requirement forces classic.

Instance pools keep machines warm

An instance pool holds a set of idle virtual machines ready for immediate use. When a cluster starts from a pool, Databricks takes machines that already exist instead of requesting new ones from Azure, which can cut startup from minutes to under a minute. Released machines return to the pool for the next cluster.

The bill explains the trade-off: you pay the virtual-machine cost of idle instances in the pool, but not Databricks compute units for them. Pools pay off only when workloads run often enough for the saved startup time to be worth that idle capacity.

Note what a pool is not: it supplies machines to classic clusters, it does not execute queries or tasks itself. With serverless available, pools matter less, since serverless starts faster without any idle capacity to maintain. They remain useful when you need classic features and want faster starts.

Serverless performance modes

Serverless compute for jobs and pipelines exposes two performance modes, a cost-versus-startup dial that exists only inside serverless. It is not an access mode and it has nothing to do with sharing a classic cluster.

You set the mode with the Performance optimized toggle on the job details page. Standard mode is not offered for continuous pipelines or for one-time runs submitted through the submit-run endpoint, and notebooks only run in performance-optimized mode.

Put the decision together

Three questions, in this order, settle almost every case.

1. What is the interface of the work? SQL analytics and BI point to a SQL warehouse. An automated workflow points to job compute. Interactive notebook development points to a notebook cluster.
2. Does anything block serverless? RDD APIs, R, GPUs, custom containers, custom networking, or a specific instance type move the choice to classic compute, and machine-level features move it specifically to dedicated access mode.
3. What about overhead and startup? With nothing blocking it, serverless wins on management and startup; frequent classic workloads may justify a pool.

A worked case: a production notebook trains a neural network on GPU machines, uses R, needs a chosen runtime version, and must keep resources inside the company's Azure subscription. GPUs and R require machine-level privileges, the runtime choice requires classic, and the subscription requirement rules serverless out. The answer is classic compute in dedicated access mode.

Key takeaways

• Job compute is for automated workflows that should leave nothing running: the cluster terminates when its tasks finish.
• Serverless compute manages, allocates, and scales itself for supported workloads, at the price of Scala, R, JAR libraries, RDD APIs, and free-form Spark configuration.
• A SQL warehouse is the surface for SQL analytics and business intelligence, with pro reserved for custom networking and classic for when nothing better is available.
• Classic compute is the answer whenever the workload needs configurable infrastructure, a chosen runtime version, or a capability serverless lacks.
• Shared compute means classic compute in Standard access mode: several users, one cluster, isolated code; dedicated access is for a single user or group that needs machine-level features.