Platform Architecture and Core Components

One platform, several cooperating responsibilities

Most architecture questions are really questions about ownership: which part of the platform is responsible for the thing being described. Start by refusing to collapse the parts into one blob.

Databricks is a unified, open analytics platform for building, deploying, sharing, and maintaining data, analytics, and AI solutions at scale. The Databricks Data Intelligence Platform integrates with cloud storage and security in the customer's cloud account while managing and deploying cloud infrastructure.

That framing explains how storage, processing, and governance cooperate without any one of them absorbing the others. Your cloud account keeps the data and the security perimeter; the platform deploys and runs the infrastructure that works on it.

The architectural pattern underneath is the lakehouse. A data lakehouse combines the benefits of data lakes and data warehouses. It provides one architectural pattern for workloads such as data engineering, machine learning, business intelligence, and reporting.

Notice what a lakehouse is not. It is not a governance product, not a processing engine, and not a storage layer — those are components that operate inside it.

Apache Spark: processing, kept apart from storage

Apache Spark provides the massively scalable processing engine, and its compute resources are decoupled from storage. Both halves of that sentence carry weight.

Massively scalable is the reason Spark is there at all: transformations run across a cluster instead of one machine. Decoupled from storage is the design decision that makes the rest of the architecture possible. Separating those responsibilities lets the processing layer operate over lakehouse data without making the engine itself the storage layer.

So Spark processes; it does not store, and it does not govern. An arrangement that couples compute permanently to storage is describing something other than this architecture.

Delta Lake: reliability where the tables live

Delta Lake is an optimized storage layer that supports ACID transactions and schema enforcement. These capabilities add table reliability to the lakehouse architecture — the guarantee that a half-finished write does not become a half-correct table.

When files are converted to Delta tables, Delta Lake schema enforcement can check for missing or unexpected data. That check happens at the moment of conversion, which is exactly when a surprise field or an absent column is cheapest to notice.

Be precise about what kind of responsibility this is. Schema enforcement is a storage and data-quality responsibility, not an access-control responsibility. It answers is this data shaped as the table expects?, never is this person allowed to read it?

Delta Lake along the data path

Storage reliability is not a one-off event at conversion time; it accompanies the data as it moves. Apache Spark and Delta Lake together provide speed, scalability, and reliability for data transformations.

Structured Streaming integrates tightly with Delta Lake, and together they provide foundations for Lakeflow pipelines and Auto Loader. That is why streaming ingestion and declarative pipelines feel like part of the same system rather than bolted-on extras: they rest on the same storage layer.

Delta Lake therefore supports the data path from ingestion through transformation while retaining its distinct role as the optimized storage layer. It never becomes the processing engine, and the streaming integration does not turn Unity Catalog into an ingestion component.

Unity Catalog: one governance model

Unity Catalog is a unified, fine-grained governance solution for data and AI. It governs lakehouse objects rather than replacing the processing engine or the table storage layer.

Its work covers four recognizable jobs:

• Registration. Unity Catalog can register tables according to a governance model and required data-isolation boundaries.
• Lineage. Unity Catalog can track lineage as data is transformed and refined.
• Privacy. Unity Catalog can also apply a unified governance model to keep sensitive data private and secure.
• Access management. Unity Catalog privileges are managed through access control lists by using user interfaces or SQL syntax.

That last point is where the boundary becomes obvious: the decision about who may use governed data belongs to Unity Catalog, and it is a different decision from how reliably Delta Lake stores the table.

Lineage deserves its own moment, because it is the capability people reach for without naming it. When someone asks to follow data as it is transformed and refined — from a served figure back through each refinement to its source — that is lineage tracking, and Unity Catalog owns it.

The neighbouring capabilities answer different questions:

Watch the three roles through one flow

The Databricks lakehouse uses Apache Spark as its processing engine, Delta Lake as its optimized storage layer, and Unity Catalog as its unified governance solution. The roles stay distinguishable at every step, including ingestion.

Batch or streaming data first arrives from multiple sources and formats in a raw logical layer. After files are converted to Delta tables, Unity Catalog can register those tables and track their lineage while Delta Lake enforces the table schema.

Note that the two control requirements at conversion time are met by two different components — one checks the structure, the other records the history. Dropping either one leaves a real gap.

The same separation holds when the data is served. A unified governance model lets users track lineage from served data back to a single source of truth. Optimized data layouts can serve machine learning, data engineering, business intelligence, and reporting from the same lakehouse.

That is the shape of the answer for an organization that wants one source of truth across workloads and traceability from a served number back through its refinements: one lakehouse, optimized layouts, and a unified governance model. Splitting governance per workload would break the traceability; splitting the lakehouse per workload would break the single source of truth.

The decision rule is simple. Identify processing, storage reliability, and governance as cooperating responsibilities, then select the component that owns the responsibility named in the question.

Key takeaways

• Apache Spark supplies scalable processing on compute decoupled from storage. Compute and data scale and live independently.
• Delta Lake supplies ACID transactions and schema enforcement as the optimized storage layer, and its schema check catches missing or unexpected data when files become tables.
• Unity Catalog supplies unified, fine-grained governance and lineage for data and AI, with privileges managed through access control lists in the interface or in SQL.
• Structured Streaming integrates tightly with Delta Lake, and together they underpin Lakeflow pipelines and Auto Loader.
• Together, Spark, Delta Lake, and Unity Catalog provide processing, reliable table storage, and governance in the Databricks lakehouse — three responsibilities, never merged into one.