Spark Workspace Settings

Where Spark settings live and who can change them

Every notebook and Spark job definition in a workspace lands on compute that somebody configured in advance. That configuration is the Spark part of the workspace settings, and the exam expects you to know which setting exists, what its default is, and who is allowed to change it.

The path is always the same: open Workspace settings, expand Data Engineering/Science, and select Spark settings. The page groups the options by purpose:

One rule covers the whole page: to make changes to the Spark settings in a workspace, you need the Admin role for that workspace. Members and contributors run Spark jobs, but they work inside the limits that the admin has set here.

This lesson is about configuration. How these features affect performance at run time is the subject of the Spark Platform Acceleration lesson.

The starter pool: fast start, few choices

The starter pool exists so that nobody has to design a cluster before running a first notebook. When you create a workspace, a starter pool that is associated with that workspace is automatically created. You do not choose node or machine sizes: they are handled for you.

Three facts define it:

• Starter pools support only the Medium node size. A Medium node has 8 vCores and 64 GB of memory.
• With no extra libraries and no custom Spark properties, a starter pool session typically starts in 5 to 10 seconds, because the cluster is already running.
• Autoscale and Dynamically allocate executors are both enabled by default.

The fast start is not a guarantee. Starter pools are a best-effort optimization that uses pre-warmed capacity. When that capacity is not available, the session starts on on-demand capacity and takes longer. There is also one case where the starter pool cannot be used at all: when the workspace uses Private Links or Managed VNets, starter pools are not supported, so Fabric creates a cluster on demand, which adds 2 to 5 minutes to the session start.

What an admin can tune in the starter pool is the scale, not the node. In the pool details, sliders set the limits for autoscale and for executors. You can keep the default values, reduce the limits for smaller workloads, or increase them up to the maximum allowed for the capacity SKU.

In words: the smallest capacities, F2 and F4, run a starter pool of a single node. An F64 capacity starts with a maximum of 10 nodes and lets the admin raise it to 16. From F64 upward the default stays at 10 and only the allowed maximum grows, up to 200 nodes. Some related material describes the size of a pool by the burst cores of the capacity instead, so treat this table as the reference for starter pools.

Custom pools: you choose the nodes

A custom pool is the answer when a workload needs something the starter pool does not offer, most often a different node size. It has two prerequisites, at two different levels:

• You need the Admin role in the workspace.
• A capacity admin must enable Customized workspace pools in the Spark Compute settings for the capacity. This capacity-level option is enabled by default, so it only blocks you if somebody turned it off.

In Spark settings, the Default pool for workspace dropdown has a New Pool option. You can create multiple custom pools and select any of them as the default pool for your workspace. For each pool you choose a node family, a node size, the autoscale range, and dynamic executor allocation.

Each step doubles the one before: Small has 4 vCores and 32 GB, Medium 8 and 64, Large 16 and 128, X-Large 32 and 256, and XX-Large 64 and 512. X-Large and XX-Large nodes are only allowed for non-trial Fabric SKUs.

A custom pool has its own limits and costs:

• Custom pools support a maximum node limit of 200.
• The capacity sets a second ceiling. An F64 capacity has 384 Spark VCores with burst, and the total Spark VCores of the pool cannot exceed that number. That allows 96 Small nodes, 48 Medium, 24 Large, 12 X-Large, or 6 XX-Large.
• Without a live pool configuration, custom pools take about three minutes to start, because Spark must get the nodes first. A custom live pool with a Full mode environment keeps clusters warm and starts sessions in approximately 5 seconds.
• Custom pools have a default autopause duration of 2 minutes after inactivity. When autopause is reached, the cluster deallocates. Billing applies only while compute is actively used.

A pool can also be very small. With a minimum of one node, the driver and the executor run in a single node, which suits workloads with small compute requirements.

Autoscale and dynamic allocation

These two switches appear in both pool types, and they answer different questions. Autoscale is about nodes. Dynamic allocation is about executors inside the nodes that the pool has.

Autoscale. When you enable it, you set the minimum and maximum number of nodes, and the pool scales up and down between them based on activity. When you disable autoscale, the number of nodes remains fixed. A fixed pool is the choice when a job must get the same compute on every run.

Dynamically allocate executors. With this option, a Spark application requests more executors when the tasks exceed the load that the current executors can bear, and releases them when the jobs are completed. It removes the need to tune executor counts by hand for every stage of a job.

Settings can change after the pool exists. Node sizes can be altered after pool creation, although the active session has to be restarted.

Let items customize compute, or keep one standard

By default, all Spark jobs in a workspace share the same pool and resource settings. A light transformation and a large training job do not need the same driver memory, so Fabric lets an admin decide whether single items may differ.

The switch is on the Pool tab: Customize compute configurations for items.

Said in full: when this toggle is on, members and contributors can change session-level compute configurations in a Fabric environment. They can select the starter pool or a custom pool that the workspace admin created, and set Spark driver cores, driver memory, executor cores, and executor memory. When it is off, the Compute section in environment items is disabled and all Spark jobs use the workspace default pool.

Two details matter. The values offered for cores and memory depend on the node size of the pool, so an environment tunes sessions inside the nodes that the pool has and cannot change the node size. And only the admin creates pools: the toggle lets other roles pick and tune, not build.

Default environment or runtime version

The Environment tab decides what every notebook starts with when its author chooses nothing. There are two ways to set that baseline.

A default environment. An environment is an item that holds compute properties, a Spark runtime, and library package dependencies. A workspace admin can select an Environment as the workspace default Environment, or create a new one from the dropdown. Notebooks and Spark job definitions in the workspace then start from that runtime and those libraries.

A runtime version only. If you disable the option to have a default environment, you select the Fabric runtime version from the available runtime versions in the dropdown. This sets the Spark version for the workspace and nothing else.

The decision rule follows from what each one carries. When the workspace needs shared libraries or shared compute properties, use a default environment. When only the Spark version matters, the runtime dropdown is enough.

Remember that a change of runtime inside an environment does not take effect until you save and publish the environment.

Job admission: optimistic or reserved

The Jobs tab controls the admission logic for all Spark jobs in the workspace, which is the rule that decides whether a job may start now.

By default, all workspaces are enabled with optimistic job admission. A job is admitted on its minimum core requirement, compared with the Spark VCores currently available in the capacity. It starts with its minimum node setting and can scale toward its configured maximum if capacity remains available.

The alternative is one switch: enabling Reserve maximum cores for active Spark jobs turns off optimistic job admission and reserves the maximum cores for each job. Every admitted job then holds everything it may ever need, so fewer jobs fit.

A worked example makes the trade clear. An F32 capacity can use up to 192 Spark VCores with burst.

In words: if each job is admitted at a full pool maximum of 64 Spark VCores, the F32 capacity runs three concurrent jobs. With optimistic job admission, the same capacity can admit up to 24 jobs if each starts with a minimum of 8 Spark VCores.

Admission also decides what happens when the capacity is full. If there are no cores available for the minimum requirement of a new job, the result depends on the type of job:

• Interactive notebook jobs and lakehouse operations are blocked with a capacity error.
• Batch jobs are queued and run when cores become available.

What the error looks like, how the queue works, and how to react are covered in the Spark Job Failure Diagnosis lesson. Here the point is the setting: reserving maximum cores makes the capacity fill up with fewer jobs, so both effects arrive sooner.

Session timeout and maximum job lifetime

Two more settings on the Jobs tab protect the capacity from compute that nobody is using well. They look similar and act on different things.

Spark session timeout. The default session expiry is set to 20 minutes for interactive Spark sessions. The setting customizes that expiry for all the notebook interactive sessions of the workspace.

Set maximum job lifetime. This is a guardrail against runaway jobs. When you turn it on and specify a duration, Fabric automatically cancels any eligible Spark job that runs longer than the configured lifetime. Set a value that leaves room for your longest legitimate job.

Which jobs are eligible is the part to learn:

The rule behind the table: the maximum job lifetime applies only to user-submitted jobs, which a user starts and which run with the user's identity. System-managed jobs are not affected by this setting and continue to run to completion.

The high concurrency setting

High concurrency mode lets notebooks share one Spark session, so a notebook can attach to a session that is already running. The rules for sharing belong to the Spark Platform Acceleration lesson. Here the point is the workspace switch.

• By default, all Fabric workspaces are enabled with high concurrency mode.
• The switch is in Spark settings, in the High concurrency section, as the For notebooks setting.
• Enabling it allows users to start a high concurrency session in their notebooks or attach to an existing one.
• Disabling it hides the option to start a new high concurrency session from the notebook menu, and also hides the section that configures the period of inactivity.

One limit is set outside the workspace page. By default a high concurrency session supports up to five notebooks. The spark.highConcurrency.max property, set in the Spark properties of an environment, can raise that limit to up to 50.

Key takeaways

• Custom pools need two things: the Admin role in the workspace, and the capacity option Customized workspace pools, which is enabled by default.
• Starter pools support only Medium nodes. A job that needs another node size goes to a custom pool, with autoscale when its node count should follow the load.
• In a starter pool, Autoscale and Dynamically allocate executors are both enabled by default.
• Only a workspace admin creates pools. Members and contributors select a pool and tune compute in an environment, and only when item customization is on.
• Customize compute configurations for items off: the Compute section of environments is disabled and every job uses the workspace default pool.
• A default environment gives the workspace a shared runtime and shared libraries. The runtime dropdown sets a version only.
• Reserve maximum cores for active Spark jobs turns off optimistic job admission. On F32, three jobs fit at 64 Spark VCores each.
• The maximum job lifetime cancels user-submitted jobs only. Table maintenance and other system-managed jobs run to completion.
• Disabling For notebooks under High concurrency removes the option to start a new high concurrency session.