Data Engineering End-to-End Workflow

Playback speed

Swipe, scroll or use ← →

Start with ingestion

Every later feature makes more sense once you can place it on one journey. That journey starts by bringing source data into the lakehouse.

At the ingestion layer, batch or streaming data arrives from multiple sources and formats and lands in a raw logical layer. Landing it raw is deliberate: the first goal is to preserve an accessible starting point before applying business transformations. If a rule is wrong three months from now, the untouched landing data is what lets you re-derive everything downstream.

1 / 20

Start with ingestion

Lakeflow Connect provides connectors for ingesting data from databases, enterprise applications, files, and streaming sources. That is one named component with one job: reaching out to the source and bringing its data in.

At ingestion	Detail
What arrives	Batch or streaming data, in many formats
Where it lands	A raw logical layer
What brings it in	Lakeflow Connect, with its connectors
Sources covered	Databases, enterprise applications, files, streaming sources

2 / 20

Quick check

Which component supplies the connectors for ingesting data from databases, enterprise applications, files, and streaming sources?
1. ALakeflow Connect
  
  Right. Lakeflow Connect is the component that provides the connectors used at the ingestion layer.
2. BLakeflow Jobs
  
  Jobs orchestrates production tasks; it coordinates work rather than reaching out to the source systems.
3. CLakeflow pipelines
  
  Pipelines can ingest as well as transform, reading files with Auto Loader or events straight from a message bus, but the named catalogue of source connectors is Lakeflow Connect.
3 / 20

Reliability and governance travel with the data

A common beginner assumption is that quality and governance come last, after the interesting transformations. In this workflow they arrive with the data.

When raw files are converted to Delta tables, Delta Lake schema enforcement can detect missing or unexpected data. That check compares each write against the table schema, so it is a first line of defence rather than a complete quality system: decide which fields and values a table requires, add explicit checks for them, and choose what happens when one of those checks fails.

For example, if age must be an integer, check a file value such as twenty during ingestion. With Auto Loader and a rescued data column, twenty can enter _rescued_data without failing the write.

Unity Catalog can register the tables, apply governance boundaries, and track lineage as the data is transformed and refined. Lineage is the part that pays off later: when a business figure looks wrong, the traceable path back to the source is already recorded.

Concern	What handles it	When
Unexpected or missing structure	Delta Lake schema enforcement	As raw files become Delta tables
Who may use the table	Unity Catalog governance boundaries	At registration
Where a value came from	Unity Catalog lineage	Continuously, through refinement

4 / 20

Quick check

What happens as raw files become governed Delta tables early in the workflow?
1. AUnity Catalog runs the transformations while Delta Lake assigns the workspace identities
  
  Unity Catalog governs and traces data; it does not execute transformations, and Delta Lake does not manage identities.
2. BSchema enforcement detects missing or unexpected data, and Unity Catalog registers it
  
  Right. Schema enforcement checks writes against the table schema as raw files become Delta tables, and Unity Catalog registers those tables, applies governance boundaries, and tracks lineage.
3. CJobs turns the tables into dashboards while notebooks take the place of the original source files
  
  Dashboards present results at the end of the journey, and notebooks are a development surface rather than a replacement for source files.
5 / 20
Curate and refine what landed

Raw data is a starting point, not a product. After verification, engineers curate and refine data by cleansing it, combining it, and creating useful features. Thoroughly cleansed data can then be integrated and reorganized into tables designed for business needs.

Note the usual order inside that stage. Cleansing normally comes before combining, because joining two dirty sources multiplies the mess rather than resolving it. In practice cleaning and combining can take several passes, and features are created once the joined result can be trusted.

6 / 20
Curate and refine what landed

A common mental picture is movement from raw data toward validated and enriched data, even when the implementation uses several pipelines or jobs. This is the stage where a technical dataset turns into something a business team recognizes: shaped around the questions it answers, not around the system it arrived from.

Landing source data is ingestion. Presenting the finished tables is serving. Cleansing, combining, and reorganizing sit between them, and that is the curation stage.

7 / 20
Quick check

Which activity belongs to curation and refinement rather than to ingestion or serving?
1. ALanding batch or streaming source data in the raw logical layer
  
  Landing source data in the raw layer is the ingestion stage that comes before any curation.
2. BPresenting the final enriched tables to BI and machine-learning consumers
  
  Presenting finished tables to consumers is the serving stage at the end of the journey.
3. CCleansing verified data, combining it, and reorganizing it into tables designed for business needs
  
  Right. Engineers curate and refine by cleansing, combining, and creating features, then integrate the result into tables designed for business needs.
8 / 20
Keep your progress in the app

That’s 3 of 8 quick checks. In the app they stay answered, and every lesson remembers where you left off.

Opens in your browser. Nothing to install.
Start studying

On your phone, with the same account

Keep reading here

Pipelines: declare how data should change

Transformation logic needs somewhere to live. Lakeflow pipelines provide a declarative framework for building and managing batch and streaming data pipelines in SQL and Python.

Declarative is the operative word. You describe the result you want rather than the exact order of steps, and a pipeline automatically orchestrates flows and targets such as streaming tables, materialized views, and sinks. For example, instead of writing a script that loops through files, you declare: "Create this streaming table from that source folder," and the pipeline decides when and how to compute it.

Pipeline concept	What it means
Flow	The movement of data from a source into a target
Streaming table	A target kept up to date as new data arrives
Materialized view	A target holding precomputed results of a query
Sink	A destination the pipeline writes out to

Because dependencies between those targets are worked out for you, the engineer's job is to define how data should move and change, then let the pipeline coordinate the dependent data-processing work.

9 / 20

Quick check

What does a Lakeflow pipeline give an engineer who defines batch and streaming logic in SQL or Python?
1. AA declarative framework that orchestrates the flows and targets: streaming tables, materialized views, sinks
  
  Right. Pipelines are declarative, and the pipeline itself orchestrates flows and targets such as streaming tables, materialized views, and sinks.
2. BA governance layer that registers tables and records their lineage across every workspace in the account
  
  Registration and lineage are Unity Catalog's responsibilities, not what the pipeline framework provides.
3. CA connector catalogue that reaches into source databases, enterprise applications, files, and streaming systems
  
  Reaching into source systems is Lakeflow Connect's role at the ingestion layer.
10 / 20

Orchestrate the production work

Development logic becomes operational when it runs reliably, at the right time and in the right order. Lakeflow Jobs provides orchestration and production monitoring for data and AI workloads.

A job contains one or more tasks that can run notebooks, pipelines, managed connectors, SQL queries, machine learning training, or model deployment and inference. That list is worth reading twice: a pipeline can be a task inside a job, which is why the two are not competing choices.

	Lakeflow pipelines	Lakeflow Jobs
Coordinates	Data transformations and their data dependencies	Units of work, in sequence
Typical contents	Flows, streaming tables, materialized views, sinks	Notebooks, pipelines, connectors, SQL queries, ML training and inference
Question it answers	How should this data change?	What should run, and in what order?

11 / 20

Orchestrate the production work

That division settles most design arguments. Suppose a declarative data flow must maintain streaming tables and materialized views, while a wider production process must also run a downstream notebook and a SQL query once those tables are ready.

The data targets belong to a pipeline: it maintains the streaming tables and materialized views and resolves their data dependencies. The broader sequence belongs to a job: it runs the pipeline as one task, then the notebook, then the SQL query.

Nothing else in the platform takes that role. Volumes and schemas organize governed data, dashboards and views present or define queries, and libraries package code: none of them coordinates a sequence of production tasks.

12 / 20
Quick check

A declarative flow must maintain streaming tables and materialized views, and a production process must then run a notebook and a SQL query. How should the responsibilities split?
1. AA dashboard maintains the data targets, and a view runs the wider task sequence
  
  A dashboard presents results and a view saves a query definition; neither maintains streaming targets nor runs tasks.
2. BA pipeline maintains the data targets, and a job runs the wider production task sequence
  
  Right. Pipelines orchestrate flows and targets such as streaming tables and materialized views, while a job coordinates tasks including notebooks and SQL queries.
3. CA volume maintains the data targets, and a schema runs the wider task sequence
  
  Volumes and schemas are governed data containers, so they neither maintain pipeline targets nor execute a sequence.
13 / 20
Serve data that is fit for its purpose

The final lakehouse layer serves clean, enriched data to end users. The work is not finished when a transformation succeeds; it is finished when someone can use the result.

Final tables should be designed for their use cases, including machine learning, data engineering, business intelligence, and reporting. A table shaped for a training run and a table shaped for an executive report are rarely the same table, and pretending otherwise pushes the difference onto every consumer.

14 / 20
Serve data that is fit for its purpose

Governance does not stop here either. Unity Catalog governance and lineage remain relevant because consumers need authorized access and a traceable path back to trusted sources. A number nobody can trace is a number nobody will act on.

So the last layer is defined by two obligations: the data is clean and enriched, and its shape matches the use case it serves. Leaving files unverified, copying governed tables into per-user notebooks, or scheduling ingestion that produces nothing consumable all fail that test.

15 / 20
Quick check

What is the purpose of the final lakehouse layer?
1. ATo keep incoming source files permanently unverified so no transformation can alter them
  
  Unverified raw files belong to the landing layer, and the workflow exists precisely to refine them.
2. BTo serve clean, enriched data in tables designed for their downstream use cases
  
  Right. The final layer serves clean, enriched data, and final tables should be designed for use cases such as ML, engineering, BI, and reporting.
3. CTo replace the governed tables with a private copy inside each user's own notebook
  
  Per-user copies discard the governance and lineage that consumers depend on for authorized, traceable access.
16 / 20

Read the journey in order

One typical route, five recognizable stages. The stages are not strictly sequential, and projects vary in tools and cadence, but this route gives every later feature a place in the overall system.

#	Stage	What it produces
1	Ingest	Source data landed in a raw logical layer
2	Govern and store	Delta tables registered, checked, and traceable
3	Transform and refine	Validated, enriched, business-shaped data
4	Orchestrate	Repeatable production execution
5	Serve	Fit-for-purpose data for its consumers

The order is not decorative. Serving before refining exposes untrusted numbers; refining before governing loses the traceability; discarding the raw landing layer removes the simplest way to reprocess source data after a mistake.

17 / 20

Quick check

A team needs raw landing, detection of unexpected table structure, traceability through refinement, and clean BI-ready output. Which ordered approach meets that?
1. ABuild the BI views first, drop the source lineage, and only then ingest copies of the tables
  
  Starting at the serving end leaves the BI views resting on data that was never verified or traced.
2. BServe the raw files directly, strip their metadata, and copy them into per-user notebooks
  
  Serving unverified files with their metadata removed defeats both the structure check and the traceability requirement.
3. CLand the raw data, convert it to Delta, register it with Unity Catalog, refine it, and then serve it
  
  Right. Landing preserves the starting point, the Delta conversion detects unexpected structure, Unity Catalog registration keeps lineage through refinement, and serving delivers the BI-ready tables.
18 / 20
Key takeaways
- The typical path runs from ingestion through governed storage and refinement to serving, with orchestration coordinating the tasks along it. Governance and orchestration are not layers waiting their turn; they apply across the whole path.
- Lakeflow Connect handles ingestion, Lakeflow pipelines manage data transformations, and Lakeflow Jobs orchestrates production tasks. A pipeline can itself be one task inside a job.
- Reliability rides along: Delta Lake schema enforcement detects missing or unexpected data, and Unity Catalog registers tables, applies governance boundaries, and tracks lineage as data is refined.
- Curation is its own stage: cleanse, combine, create features, then integrate and reorganize into tables designed for business needs.
- The served result is clean, enriched data designed for downstream analytics, AI, engineering, or reporting: still governed, still traceable.
19 / 20
Quick check

A project ingests database and streaming sources, transforms both with managed batch and streaming logic, and schedules a notebook once the prepared tables are ready. Which mapping fits?
1. ALakeflow Connect ingests the sources, Lakeflow pipelines transform them, and Jobs orchestrates the scheduled notebook
  
  Right. Connect provides the source connectors, pipelines manage the batch and streaming transformations, and Jobs orchestrates the notebook task.
2. BLakeflow Jobs supplies the connectors, Unity Catalog performs the transformations, and volumes handle the scheduling
  
  Jobs orchestrates rather than connecting, Unity Catalog governs rather than transforming, and a volume stores non-tabular data rather than scheduling.
3. CLakeflow pipelines reach into the source systems, notebooks provide the governance, and tables carry out the scheduling
  
  A pipeline can ingest and transform, but the connector catalogue is Connect; notebooks are a development surface rather than a governance layer, and a table stores data instead of scheduling it.
20 / 20
8 quick checks · then the test

In the app, finishing the quick checks opens this lesson’s 10-question test, and the ones you miss come back exactly when you’re about to forget them.

Take this lesson’s test → Next lesson: Data Engineer Associate Exam Overview

Opens in your browser. Nothing to install.
Start studying

On your phone, with the same account