Configuring Compute Features

Features decide capability, capacity decides size

Performance settings allocate resources: how many cores, how much memory, how many workers. Feature settings answer a different question — which execution technology and which runtime environment the workload receives. A cluster can be enormous and still be unable to run your code, or perfectly sized and slower than it needs to be.

Three decisions belong to this family:

1. Photon acceleration, the execution engine for SQL and DataFrame work.
2. The Databricks Runtime and, with it, the Spark version.
3. The machine learning environment, when the workload needs prepared libraries or GPUs.

Make each one from how the workload behaves and what it is compatible with, not by treating every feature as a general speed switch.

Photon is a query execution engine that replaces traditional Spark execution components with optimized native code. Enable it and the compute resource runs SQL queries and DataFrame operations through that accelerated engine, with no change to the code you already wrote.

Its scope is precise, and knowing the edges matters:

Photon is an execution feature, not an access-control feature and not a machine learning environment. It changes how supported operators run; it does not change who can run them.

When Photon pays off, and when it cannot be used

Photon rewards work that has something substantial to optimize. Complex transformations, joins, aggregations, and scans across large tables benefit most, and so do workloads that hit disk often, process wide tables, or transform the same data repeatedly. An hourly aggregation across millions of rows is the archetype.

The opposite case is just as clear. A simple batch job over a small dataset, or a calculation that finishes in a couple of seconds, has almost nothing to accelerate, and the extra compute cost is not repaid.

Photon is enabled by default on Databricks Runtime 9.1 LTS and later, and the setting can be verified or changed when the compute resource is created.

There is one hard incompatibility: Photon is not supported on GPU-enabled clusters. A workload that requires GPU instances for machine learning or deep learning must run with Photon disabled.

That turns into a straightforward choice rather than a dilemma:

The two features cannot be combined on the same cluster, so the workload's real requirement decides which one survives.

The runtime carries Spark with it

Databricks Runtime supplies the core components that run on the compute resource: Apache Spark, preloaded packages, and Databricks optimizations. The version you select decides which features are available and how your code meets those packages.

The consequence that catches people out is this: selecting a Databricks Runtime also selects its Apache Spark version. Spark is not a separate dropdown. Newer runtimes carry more recent Spark releases with additional features and fixes; older runtimes carry older ones, stable but without recent Spark functionality.

That is why a missing Spark feature is a runtime question. No access mode, subscription, or catalog binding can add it — those are independent controls that never change which Spark release is running.

Current versus long-term support

The right runtime depends on what the compute is for.

A Long Term Support (LTS) release is chosen precisely for that stability posture. It does not convert anything to serverless, it does not make Photon work on GPUs, and it does not excuse testing.

Whatever the choice, test the workload against the new runtime in a development environment before you change production compute. That step is the difference between an upgrade and an incident.

Stability and a needed feature are not in conflict as often as they seem. When a daily pipeline requires a recent Spark capability but must stay dependable, the approach is to pick an LTS runtime that already contains the feature, then validate it in development before production.

Compare that with the alternatives:

• Taking the newest non-LTS release straight to production buys the feature and gives up the stability that was a requirement.
• Staying on the old runtime and changing something else — an access mode, for example — cannot add Spark functionality at all.

The feature comes from the runtime; the safety comes from choosing a supported release and testing it first.

Who controls the version

Runtime selection is not available everywhere, and that fact often decides the compute type.

On classic compute, the version is explicit: you choose it, and you decide when to move to another. On serverless compute, the runtime is versionless and Azure Databricks upgrades it automatically — convenient, but there is no version for you to pin.

So when a workload has to sit on a specific runtime version — for a certified dependency, a validated pipeline, or a compatibility requirement — the requirement points at classic compute. Nothing about an access mode or a specialized runtime restores version control on serverless.

Databricks Runtime ML and GPU work

Databricks Runtime ML is the runtime variant prepared for machine learning. It ships with preinstalled machine learning libraries, GPU drivers, and frameworks such as CUDA, so a team does not spend its first day resolving package conflicts.

Select it instead of the standard runtime whenever the workload needs that prepared environment. A standard runtime does not contain the bundle, and neither a SQL warehouse nor a runtime preloaded onto pool machines is a substitute for it: preloading only copies a chosen runtime onto idle machines in advance.

The hardware underneath follows the stage of the work.

• For initial experimentation, a single-node resource with a large instance type is often enough, especially for frameworks that do not distribute across workers: there is no shuffle overhead to pay for. What a single-node resource cannot do is scale out with more workers, and it offers no worker redundancy if the driver is lost.
• For computationally intensive deep learning — image recognition, natural language processing, neural network training — GPU instances turn work that would take hours on processors into minutes.

Keep the combination compatible

Feature settings have to agree with each other. Two configurations are worth memorizing because they cover most real cases.

Read the two rows together and the invalid combinations become obvious. A standard runtime on GPU compute is missing the drivers and CUDA the work depends on. Runtime ML on GPU compute with Photon enabled is not a valid cluster at all. And Photon is never the GPU framework: it accelerates SQL and DataFrame operators, it does not drive graphics processors.

Key takeaways

• Photon accelerates SQL queries and DataFrame operations, and pays off most on large scans, joins, aggregations, and repeated transformations; tiny, fast jobs gain little.
• Photon is enabled by default from Databricks Runtime 9.1 LTS onward, and it is not supported on GPU-enabled clusters.
• Selecting a Databricks Runtime also selects its Apache Spark version; a missing Spark feature is a runtime decision, not a permissions one.
• Prefer a Long Term Support runtime for operational job compute and the current release for interactive development, and test any upgrade in development first.
• Use Databricks Runtime ML for its prepared libraries, GPU drivers, and CUDA; a large single-node resource can carry early experiments, while GPU instances carry deep learning.