Prepstellar
Get the app

DEA-C01 · Getting started

20 cards

Data Engineering End-to-End Workflow

Swipe, scroll or use ← →

  1. Start with ingestion

    Every later feature makes more sense once you can place it on one journey. That journey starts by bringing source data into the lakehouse.

    At the ingestion layer, batch or streaming data arrives from multiple sources and formats and lands in a raw logical layer. Landing it raw is deliberate: the first goal is to preserve an accessible starting point before applying business transformations. If a rule is wrong three months from now, the untouched landing data is what lets you re-derive everything downstream.

    1 / 20

  2. Start with ingestion

    Lakeflow Connect provides connectors for ingesting data from databases, enterprise applications, files, and streaming sources. That is one named component with one job — reaching out to the source and bringing its data in.

    At ingestion Detail
    What arrives Batch or streaming data, in many formats
    Where it lands A raw logical layer
    What brings it in Lakeflow Connect, with its connectors
    Sources covered Databases, enterprise applications, files, streaming sources
    2 / 20

  3. Quick check

    Which component supplies the connectors for ingesting data from databases, enterprise applications, files, and streaming sources?

    1. ALakeflow Connect

      Right. Lakeflow Connect is the component that provides the connectors used at the ingestion layer.

    2. BLakeflow Jobs

      Jobs orchestrates production tasks; it coordinates work rather than reaching out to the source systems.

    3. CLakeflow pipelines

      Pipelines transform data that has already arrived, so they are not what establishes the source connection.

    3 / 20

  4. Reliability and governance travel with the data

    A common beginner assumption is that quality and governance come last, after the interesting transformations. In this workflow they arrive with the data.

    When raw files are converted to Delta tables, Delta Lake schema enforcement can detect missing or unexpected data. A file with a renamed column or an absent field stops being a silent problem and becomes a visible one at the moment of conversion.

    Unity Catalog can register the tables, apply governance boundaries, and track lineage as the data is transformed and refined. Lineage is the part that pays off later: when a business figure looks wrong, the traceable path back to the source is already recorded.

    Concern What handles it When
    Unexpected or missing structure Delta Lake schema enforcement As raw files become Delta tables
    Who may use the table Unity Catalog governance boundaries At registration
    Where a value came from Unity Catalog lineage Continuously, through refinement
    4 / 20

  5. Quick check

    What happens as raw files become governed Delta tables early in the workflow?

    1. AUnity Catalog runs the transformations while Delta Lake assigns the workspace identities

      Unity Catalog governs and traces data; it does not execute transformations, and Delta Lake does not manage identities.

    2. BSchema enforcement detects missing or unexpected data, and Unity Catalog registers it

      Right. Schema enforcement catches missing or unexpected data at conversion, and Unity Catalog registers the tables, applies governance boundaries, and tracks lineage.

    3. CJobs turns the tables into dashboards while notebooks take the place of the original source files

      Dashboards present results at the end of the journey, and notebooks are a development surface rather than a replacement for source files.

    5 / 20

  6. Curate and refine what landed

    Raw data is a starting point, not a product. After verification, engineers curate and refine data by cleansing it, combining it, and creating useful features. Thoroughly cleansed data can then be integrated and reorganized into tables designed for business needs.

    Note the order inside that stage. Cleansing comes before combining, because joining two dirty sources multiplies the mess rather than resolving it. Features are created once the joined result can be trusted.

    6 / 20

  7. Curate and refine what landed

    A common mental picture is movement from raw data toward validated and enriched data, even when the implementation uses several pipelines or jobs. This is the stage where a technical dataset turns into something a business team recognizes: shaped around the questions it answers, not around the system it arrived from.

    Landing source data is ingestion. Presenting the finished tables is serving. Cleansing, combining, and reorganizing sit between them, and that is the curation stage.

    7 / 20

  8. Quick check

    Which activity belongs to curation and refinement rather than to ingestion or serving?

    1. ALanding batch or streaming source data in the raw logical layer

      Landing source data in the raw layer is the ingestion stage that comes before any curation.

    2. BPresenting the final enriched tables to BI and machine-learning consumers

      Presenting finished tables to consumers is the serving stage at the end of the journey.

    3. CCleansing verified data, combining it, and reorganizing it into tables designed for business needs

      Right. Engineers curate and refine by cleansing, combining, and creating features, then integrate the result into tables designed for business needs.

    8 / 20

  9. Keep your progress in the app

    That’s 3 of 8 quick checks. In the app they stay answered, and every lesson remembers where you left off.

    Get the app Keep reading here

  10. Pipelines: declare how data should change

    Transformation logic needs somewhere to live. Lakeflow pipelines provide a declarative framework for building and managing batch and streaming data pipelines in SQL and Python.

    Declarative is the operative word. You describe the result you want rather than the exact order of steps, and a pipeline automatically orchestrates flows and targets such as streaming tables, materialized views, and sinks.

    Pipeline concept What it means
    Flow The movement of data from a source into a target
    Streaming table A target kept up to date as new data arrives
    Materialized view A target holding precomputed results of a query
    Sink A destination the pipeline writes out to

    Because dependencies between those targets are worked out for you, the engineer's job is to define how data should move and change, then let the pipeline coordinate the dependent data-processing work.

    9 / 20

  11. Quick check

    What does a Lakeflow pipeline give an engineer who defines batch and streaming logic in SQL or Python?

    1. AA declarative framework that orchestrates the flows and targets: streaming tables, materialized views, sinks

      Right. Pipelines are declarative, and the pipeline itself orchestrates flows and targets such as streaming tables, materialized views, and sinks.

    2. BA governance layer that registers tables and records their lineage across every workspace in the account

      Registration and lineage are Unity Catalog's responsibilities, not what the pipeline framework provides.

    3. CA connector catalogue that reaches into source databases, enterprise applications, files, and streaming systems

      Reaching into source systems is Lakeflow Connect's role at the ingestion layer.

    10 / 20

  12. Orchestrate the production work

    Development logic becomes operational when it runs reliably, at the right time and in the right order. Lakeflow Jobs provides orchestration and production monitoring for data and AI workloads.

    A job contains one or more tasks that can run notebooks, pipelines, managed connectors, SQL queries, machine learning training, or model deployment and inference. That list is worth reading twice: a pipeline can be a task inside a job, which is why the two are not competing choices.

    Lakeflow pipelines Lakeflow Jobs
    Coordinates Data transformations and their data dependencies Units of work, in sequence
    Typical contents Flows, streaming tables, materialized views, sinks Notebooks, pipelines, connectors, SQL queries, ML training and inference
    Question it answers How should this data change? What should run, and in what order?
    11 / 20

  13. Orchestrate the production work

    That division settles most design arguments. Suppose a declarative data flow must maintain streaming tables and materialized views, while a wider production process must also run a downstream notebook and a SQL query once those tables are ready.

    The data targets belong to a pipeline: it maintains the streaming tables and materialized views and resolves their data dependencies. The broader sequence belongs to a job: it runs the pipeline as one task, then the notebook, then the SQL query.

    Nothing else in the platform takes that role. Volumes and schemas organize governed data, dashboards and views present or define queries, and libraries package code — none of them coordinates a sequence of production tasks.

    12 / 20

  14. Quick check

    A declarative flow must maintain streaming tables and materialized views, and a production process must then run a notebook and a SQL query. How should the responsibilities split?

    1. AA dashboard maintains the data targets, and a view runs the wider task sequence

      A dashboard presents results and a view saves a query definition; neither maintains streaming targets nor runs tasks.

    2. BA pipeline maintains the data targets, and a job runs the wider production task sequence

      Right. Pipelines orchestrate flows and targets such as streaming tables and materialized views, while a job coordinates tasks including notebooks and SQL queries.

    3. CA volume maintains the data targets, and a schema runs the wider task sequence

      Volumes and schemas are governed data containers, so they neither maintain pipeline targets nor execute a sequence.

    13 / 20

  15. Serve data that is fit for its purpose

    The final lakehouse layer serves clean, enriched data to end users. The work is not finished when a transformation succeeds; it is finished when someone can use the result.

    Final tables should be designed for their use cases, including machine learning, data engineering, business intelligence, and reporting. A table shaped for a training run and a table shaped for an executive report are rarely the same table, and pretending otherwise pushes the difference onto every consumer.

    14 / 20

  16. Serve data that is fit for its purpose

    Governance does not stop here either. Unity Catalog governance and lineage remain relevant because consumers need authorized access and a traceable path back to trusted sources. A number nobody can trace is a number nobody will act on.

    So the last layer is defined by two obligations: the data is clean and enriched, and its shape matches the use case it serves. Leaving files unverified, copying governed tables into per-user notebooks, or scheduling ingestion that produces nothing consumable all fail that test.

    15 / 20

  17. Quick check

    What is the purpose of the final lakehouse layer?

    1. ATo keep incoming source files permanently unverified so no transformation can alter them

      Unverified raw files belong to the landing layer, and the workflow exists precisely to refine them.

    2. BTo serve clean, enriched data in tables designed for their downstream use cases

      Right. The final layer serves clean, enriched data, and final tables should be designed for use cases such as ML, engineering, BI, and reporting.

    3. CTo replace the governed tables with a private copy inside each user's own notebook

      Per-user copies discard the governance and lineage that consumers depend on for authorized, traceable access.

    16 / 20

  18. Read the journey in order

    Five stages, always in the same order. Different projects vary in tools and cadence, but this sequence gives every later feature a place in the overall system.

    # Stage What it produces
    1 Ingest Source data landed in a raw logical layer
    2 Govern and store Delta tables registered, checked, and traceable
    3 Transform and refine Validated, enriched, business-shaped data
    4 Orchestrate Repeatable production execution
    5 Serve Fit-for-purpose data for its consumers

    The order is not decorative. Serving before refining exposes untrusted numbers; refining before governing loses the traceability; discarding the raw landing layer removes the only place a mistake can be corrected from.

    17 / 20

  19. Quick check

    A team needs raw landing, detection of unexpected table structure, traceability through refinement, and clean BI-ready output. Which ordered approach meets that?

    1. ABuild the BI views first, drop the source lineage, and only then ingest copies of the tables

      Starting at the serving end leaves the BI views resting on data that was never verified or traced.

    2. BServe the raw files directly, strip their metadata, and copy them into per-user notebooks

      Serving unverified files with their metadata removed defeats both the structure check and the traceability requirement.

    3. CLand the raw data, convert it to Delta, register it with Unity Catalog, refine it, and then serve it

      Right. Landing preserves the starting point, the Delta conversion detects unexpected structure, Unity Catalog registration keeps lineage through refinement, and serving delivers the BI-ready tables.

    18 / 20

  20. Key takeaways

    • The typical path is ingestion, governed storage, transformation and refinement, orchestration, and serving. Each stage assumes the one before it.
    • Lakeflow Connect handles ingestion, Lakeflow pipelines manage data transformations, and Lakeflow Jobs orchestrates production tasks. A pipeline can itself be one task inside a job.
    • Reliability rides along: Delta Lake schema enforcement detects missing or unexpected data, and Unity Catalog registers tables, applies governance boundaries, and tracks lineage as data is refined.
    • Curation is its own stage: cleanse, combine, create features, then integrate and reorganize into tables designed for business needs.
    • The served result is clean, enriched data designed for downstream analytics, AI, engineering, or reporting — still governed, still traceable.
    19 / 20

  21. Quick check

    A project ingests database and streaming sources, transforms both with managed batch and streaming logic, and schedules a notebook once the prepared tables are ready. Which mapping fits?

    1. ALakeflow Connect ingests the sources, Lakeflow pipelines transform them, and Jobs orchestrates the scheduled notebook

      Right. Connect provides the source connectors, pipelines manage the batch and streaming transformations, and Jobs orchestrates the notebook task.

    2. BLakeflow Jobs supplies the connectors, Unity Catalog performs the transformations, and volumes handle the scheduling

      Jobs orchestrates rather than connecting, Unity Catalog governs rather than transforming, and a volume stores non-tabular data rather than scheduling.

    3. CLakeflow pipelines reach into the source systems, notebooks provide the governance, and tables carry out the scheduling

      Pipelines transform data that has already landed, notebooks are a development surface rather than a governance layer, and a table stores data instead of scheduling it.

    20 / 20

  22. 8 quick checks · then the test

    In the app, finishing the quick checks opens this lesson’s 10-question test, and the ones you miss come back exactly when you’re about to forget them.

    Take this lesson’s test → Next lesson: Data Engineer Associate Exam OverviewGet the app
← Previous Keep your progress · Get the app Next card →

The whole course, on your phone

Lessons you can read, audio you can listen to on the way to work, and practice that remembers what you got wrong.

Get the app