Data Engineer Associate · Databricks
Databricks real-time mode is here. Is it on the Data Engineer Associate exam?
Databricks has a new way to run streaming pipelines with very low latency. What real-time mode is, how it differs from the streaming you study, and whether it belongs in your exam notes.
Databricks made real-time mode in Lakeflow pipelines generally available on October 29, 2026, on both classic and serverless compute. It targets operational workloads, such as fraud detection, that need end-to-end latency as low as five milliseconds. It is not named in the Data Engineer Associate exam guide today. The guide's ingestion objectives still ask you to know batch, streaming and incremental loading, streaming tables and trigger types, so that is where your study time goes. Real-time mode builds directly on those ideas, which makes it easy to place once the basics are solid.
What is Databricks real-time mode?
Real-time mode is a specialized continuous trigger for streaming flows in Lakeflow pipelines. Instead of processing data in a series of small batches, it runs long-running batches (five minutes by default), schedules all query stages at the same time, and uses a streaming shuffle that passes data between stages as soon as it is produced. The result is much lower latency than the standard streaming modes.
Turning it on takes three settings:
- Run the pipeline in continuous mode.
- Set the Spark configuration
spark.databricks.streaming.realTimeMode.enabledtotrue. - Define the flow with
pipelines.triggerset toRealTime. The optionalpipelines.trigger.intervalcontrols the batch length.
Real-time mode vs the streaming you study for the exam
For the exam, the core distinction is still triggered vs continuous pipelines: a triggered pipeline processes what is available and stops, while a continuous pipeline keeps running and processes new data as it arrives. Real-time mode sits on top of continuous mode. It is not a third pipeline mode, it is a faster way to run a continuous flow.
The bigger surprise is what it reads and writes:
- Sources and sinks: Apache Kafka, Azure Event Hubs through the Kafka connector and Amazon MSK work in both directions. Amazon Kinesis is supported as a source only, and ForEach as a sink only.
- Delta is not supported as a source or a sink. That matters, because most of the pipelines in the exam guide read from and write to Delta tables. Real-time mode is for event streams, not for building your Bronze, Silver and Gold layers.
Limits worth knowing
- Stream-to-stream joins are not supported. Broadcast joins are.
- Session windows,
dropDuplicatesWithinWatermark,flatMapGroupsWithStateandtransformWithStateInPandasare not supported. - Databricks recommends one real-time flow per pipeline, because several flows compete for task slots and add latency. Running several flows without that competition is in Beta (since October 28, 2026), and only on serverless pipelines in the preview channel.
Should you study it for the Data Engineer Associate exam?
Not as an exam topic yet. The guide has not added it, and its August 2026 revision kept every objective the same. If Databricks adds real-time mode to the guide, the change will appear on our update calendar.
For now, treat it as useful context. Learn how streaming tables, triggers and continuous pipelines work first. Then real-time mode reads as what it is: the same continuous pipeline, tuned for event streams that cannot wait.
