Data lineage is the record of where a piece of data came from, every system and transformation it passed through, and everything that uses it now. Good lineage answers two questions fast: if this source is wrong, what does it break, and where did this reported number come from? Banks, insurers and health systems need those answers for audits, regulatory reports and, more and more, for the AI models and agents that read their data. This guide covers how lineage is captured, how the main tools compare, and how to build it on Cloudera with Apache Atlas and on other platforms with OpenLineage. It is part of our Private AI & Data Architecture series.
Lineage is a graph. The nodes are datasets, such as tables, files, topics and reports, and the processes that read and write them, such as jobs, queries and pipelines. The edges show which process read which input and wrote which output.
Three distinctions matter when you scope a lineage program.
Table-level vs column-level. Table-level lineage says the general ledger table feeds the capital report. Column-level lineage says the "exposure_amount" column in the report comes from two specific columns in two source tables, through a named calculation. Auditors and examiners usually need column level for the figures they test.
Technical vs business lineage. Technical lineage is the graph of systems and jobs. Business lineage maps that graph to business terms and critical data elements, such as "total deposits" or "claims reserve", so a non-engineer can follow it.
Captured vs declared. Captured lineage is recorded from what actually ran. Declared lineage is drawn by hand or parsed from code that may not match production. Captured lineage is worth more as evidence, because it shows what happened, not what was intended.
Lineage is related to data provenance but not the same. Provenance is the origin and custody history of a specific record or dataset: who created it, when, and under what conditions. Lineage is the flow between systems. A complete audit trail usually needs both.
Why Lineage Matters for AI and Audit
For audit and regulatory reporting, lineage is how you prove a number. Supervisors expect banks to trace key risk figures back to source. The ECB's guide on risk data aggregation and risk reporting, for example, expects complete and up-to-date lineage at data attribute level for the risk indicators and critical data elements in scope, from capture through transformation to the report. Lineage also lets you place controls where they matter and trace a data quality incident to its cause.
For AI, lineage answers the questions a model risk reviewer now asks:
Which data trained this model, and was any of it restricted? If a source system changes, which models and features are affected? Which documents did this retrieval system index, and are they current? When an agent answered a question, which tables and definitions did it use?
Without lineage, each of those answers is a reconstruction. With it, they come from system records.
Lineage also makes change safer. Before you alter a column, drop a table or migrate a source, lineage shows every report, model and agent downstream. That turns a risky change into a planned one.
How Lineage Is Captured
Every lineage tool uses one or more of four methods. Knowing which one a tool relies on tells you where its gaps will be.
Runtime hooks and listeners. The engine reports lineage as jobs run. Atlas hooks on Cloudera and the OpenLineage Spark listener work this way. This is the most reliable method, because it records what actually executed.
Query log analysis. The tool reads the platform's query history and works out reads and writes. Databricks and Snowflake build their native lineage from what runs on the platform.
Static parsing. The tool parses SQL, stored procedures, ETL definitions or BI models without running them. This covers legacy tools that cannot emit events, but it can miss dynamic SQL and logic that differs at runtime.
Manual declaration. Someone records the flow by hand or through an API. It is the fallback for mainframes, spreadsheets and vendor systems, and it goes stale unless it has an owner.
Most regulated estates need all four. Runtime capture covers the modern platform. Static parsing covers legacy ETL and BI. Manual declaration covers what is left, with an owner and a review date.
Data Lineage Tools Compared
Apache Atlas on Cloudera. Captures lineage at runtime through hooks for HiveServer, Impala and Spark, and from NiFi through a reporting task. Column-level for supported engines. Best for Cloudera estates that need lineage under the same governance as the data.
Octopai (Cloudera). Discovers and parses lineage across many platforms, including systems outside Cloudera. Column-level. Best for mixed estates with legacy ETL and BI tools.
Databricks Unity Catalog. Captures lineage automatically from queries run on Databricks, and accepts external lineage by API. Column-level, with some exceptions such as path-only references. Best for Databricks-centered estates.
Snowflake lineage. Built from activity in Snowflake, and accepts OpenLineage events from external jobs. Column-level on Enterprise Edition or higher. Best for Snowflake-centered estates.
OpenLineage with Marquez. Collects runtime events from Airflow, Spark, dbt, Flink and other tools. Column-level where the integration supports it. Best for open, vendor-neutral capture across tools.
Catalog products such as Collibra, Alation and Microsoft Purview. Assemble lineage from connectors, parsing and imported metadata, with column-level detail that varies by connector. Best for business lineage and stewardship across the whole estate.
Two patterns stand out. First, the platforms are converging on OpenLineage as a common format: Snowflake accepts OpenLineage events, and Databricks and others accept external lineage by API. Second, no single tool sees everything. Plan for a platform-native capture layer plus one place where the full graph is assembled and reviewed.
Data Lineage with Apache Atlas on Cloudera
On Cloudera, Apache Atlas is the lineage and metadata service, and it works with the platform's security and governance layer. Lineage is captured, not drawn.
By default, Atlas collects lineage from HiveServer, Impala and Spark. Each query that reads and writes data creates a process entity linked to its input and output datasets. Spark jobs appear as a spark_application with a spark_process for each query that produces lineage. NiFi flows can report lineage to Atlas through a reporting task, which brings ingestion into the same graph. For supported engines, Atlas shows how the data in a given column was produced.
Atlas also carries classifications, such as PII or confidential, and can propagate them along lineage. A column tagged as personal data in a source table passes that tag to the columns derived from it. That is the link between lineage and access policy: the tag follows the data, and policy follows the tag.
For estates that run more than Cloudera, Cloudera announced in November 2024 that it would acquire Octopai's lineage and catalog platform, which discovers lineage across systems outside Cloudera as well. That matters for banks and insurers whose critical data passes through mainframe extracts, legacy ETL and BI tools before and after the data platform.
Keeping lineage inside the platform is also part of keeping AI inside the perimeter. When models are trained and served on governed data, lineage shows exactly what each one used. That is the core of private AI on Cloudera.
How to Build Data Lineage: Six Steps
Start from critical data elements, not from tools. List the 50 to 100 data elements behind your regulatory reports, risk metrics and highest-risk AI models. Lineage for those comes first. Turn on runtime capture where the work runs. On Cloudera, confirm the Atlas hooks are active for HiveServer, Impala and Spark, and configure the NiFi reporting task. On Spark outside Cloudera, add the OpenLineage listener. A Spark job sends events to a lineage backend with settings like these:
Add the OpenLineage Spark listener (spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener) and set the transport type to http, the transport URL to your lineage backend, and a namespace for the environment.
Check the package coordinates for your Spark and Scala versions in the OpenLineage documentation.
Fill the gaps. Use static parsing for legacy ETL and BI, and declare the rest by API with a named owner. Mainframe extracts, vendor feeds and spreadsheets are the usual gaps. Make lineage queryable, not just viewable. Reviews and impact checks should run from scripts, not screenshots. The Atlas REST API returns the lineage graph for any entity. This call returns everything upstream of a Hive table, five levels deep:
Call GET /api/atlas/v2/lineage/uniqueAttribute/type/hive_table with attr:qualifiedName set to the table, direction=INPUT and depth=5. On a Kerberized cluster, authenticate with SPNEGO.
The direction parameter takes INPUT, OUTPUT or BOTH. Use OUTPUT for impact analysis before a change.
Map lineage to business terms. Link each critical data element to its columns in the lineage graph and to its definition. A semantic layer helps here: when reports and AI agents query named metrics, the metric definition becomes the bridge between business meaning and technical lineage. Test lineage like any other control. Pick a reported figure and trace it to source using only the lineage tool. Change a test column and confirm the impact report finds every consumer. Measure coverage: the share of critical data elements with complete, captured lineage.
Lineage for AI Models and Agents
AI adds new nodes to the lineage graph, and they need the same treatment as reports.
For trained models, record the training and evaluation datasets, their versions and the feature pipelines that built them. When a source changes, impact analysis should reach the model, not stop at the feature table.
For retrieval systems, record which document collections were indexed, when, and from which sources. A retrieval system that cites a superseded policy is a lineage failure as much as a content failure.
For agents, record each query an agent runs and the datasets or metrics it read. Agents that query governed definitions, rather than raw tables, produce lineage that is far easier to follow, because each answer traces to a named metric with its own lineage behind it.
What Auditors and Examiners Check
Reviewers test lineage the way they test any control: they pick a number and follow it. Expect requests like these:
A list of critical data elements in scope, each with an owner and a definition. Column-level lineage from source to report for each critical data element, captured from production where possible. The capture method for each segment of the flow, with gaps and manual segments named and owned. Evidence that lineage is current: last capture time, and alerts when a job stops reporting. Controls placed along the lineage, such as reconciliations and data quality checks, with their results. Impact analysis records for recent changes to in-scope sources or transformations. Classification tags, such as PII, and evidence that they follow the data downstream. For AI models, the training and evaluation data lineage, with dataset versions. For agents and retrieval systems, logs that tie each answer to the data or documents used. A lineage walk-through performed by someone outside the data team, using only the tool.
If you can trace any in-scope figure to source in minutes, without asking an engineer, the lineage program is ready for review.
Turning on lineage capture is quick. Getting complete, column-level lineage for the figures that matter, across legacy ETL, the data platform and AI, takes more work. So does keeping it current. Duczer East provides data governance consulting for banks, insurers and health systems, including lineage on Cloudera with Apache Atlas, OpenLineage across other platforms, and the evidence reviewers ask for.
