Data Lineage: The Link Between Trust, Compliance, and AI

Lineage has moved from nice-to-have to evidence

Data lineage answers three questions: where did this number come from, what happened to it on the way, and what else breaks if I change it. For years it was treated as documentation, useful for architects and usually out of date.

That has changed because three groups now ask for it as evidence.

• Banking supervisors. The Basel Committee's 2013 risk data principles, BCBS 239, never use the word "lineage". They ask for integrated data taxonomies with metadata, a dictionary of concepts, reconciliation to source and documentation of every aggregation process. Supervisors have since made lineage the test of whether those principles are met. The Committee's November 2023 progress report found that only two of 31 global systemically important banks were fully compliant, and named the lack of a common taxonomy and complete data lineage as a persistent obstacle. The European Central Bank's May 2024 guide is explicit: it expects complete, up-to-date lineage at data attribute level, from data capture through extraction, transformation and loading, for the risk indicators and critical data elements in scope.
• AI regulation. Article 10 of the EU AI Act requires providers of high-risk AI systems to apply data governance to training, validation and testing data, covering the origin of the data and, for personal data, the original purpose of collection, and the preparation steps such as labelling, cleaning, enrichment and aggregation. That is a lineage requirement in all but name.
• National data standards. Saudi Arabia's NDMO standards ask for lineage in several places, starting with Priority 1 specifications, as set out below.

Our thesis: lineage is only useful as evidence if it is captured from execution, at the granularity your obligations require, and scoped to the data that matters. Programs that try to draw the whole estate by hand produce diagrams no one trusts. Programs that instrument their pipelines and start from critical data elements produce lineage that answers auditors' questions on demand.

Four kinds of lineage, four different audiences

"Lineage" is used for several different things. Being precise avoids buying the wrong tool or promising the wrong thing to a regulator.

Business lineage describes how a business concept, such as "net exposure" or "active customer", flows from source processes to reports, in terms a risk officer or data owner understands. It links to the glossary and is mostly curated.

Technical lineage at table or dataset level records which jobs read which datasets and wrote which outputs. It is the basis for impact analysis and can be captured automatically.

Column-level lineage records which input columns produced each output column and how: copied, transformed, aggregated, or used only to filter or join. This is the level the ECB means by attribute level, and the level needed to trace a single wrong figure back to its source.

Operational lineage records each run: when a job executed, which version of the code ran, which data it read, and whether it succeeded. It is what lets you say which version of the data a model was trained on, or which load introduced an error.

Most regulatory questions need column-level and operational lineage together, presented in business terms.

What regulators and standards actually ask for

The expectations differ in wording but converge on the same capability.

Two details deserve attention.

The ECB's definition is functional. It describes lineage as information about the movement and transformation of data from capture to end use, and lists what it should enable: checking that quality controls are well placed in the flow, confirming that transformed fields still match reporting definitions, finding the data points needed for ad hoc requests, tracing incidents back to their source quickly, and supporting external validation. That list is a good acceptance test for any lineage program, regulated or not.

AI Act timing has moved, the content has not. The Digital Omnibus on AI, agreed by the Council and Parliament in May 2026 and adopted by the Council on 29 June 2026, defers the high-risk obligations to 2 December 2027 for stand-alone high-risk systems and 2 August 2028 for high-risk AI embedded in regulated products. Organizations building systems that will still be in service then should design for Article 10 now, because reconstructing the origin and preparation history of a training set after the fact is often impossible.

NDMO: lineage starts in Priority 1

For Saudi public entities and their partners, lineage is not a later-stage capability. The NDMO Data Management and Personal Data Protection Standards include:

• DSI.1.1 (Priority 1): the initial data integration assessment must document a high-level data lineage, including the rules by which data is changed and how often.
• DSI.3.2 (Priority 1): each integration solution design must include a data flow diagram and a source-to-target mapping that specifies formats and the transformations at every intermediate staging point.
• DAM.3.1 (Priority 1) and DAM.3.2 (Priority 2): current-state and target-state data architecture must include visualized data flows and lineage across key processes and systems.
• MCM.1.3 (Priority 1): the target metadata architecture must define metadata flows from sources into the data catalog.
• DAM.4.2 (Priority 2): the entity's toolset must capture and maintain data flows between systems to enable impact analysis.
• OD.3.6 (Priority 3): published open datasets must maintain traceability through documented provenance and version history.

The pattern matches the one we describe in our NDMO implementation guide: Priority 1 asks for documented designs and mappings, Priority 2 for the tooling that keeps them current. Entities that write source-to-target mappings in a form a lineage tool can ingest avoid doing the work twice.

Capture lineage from execution, not from interviews

There are three ways to produce lineage, and they differ sharply in cost and reliability.

Manual documentation captures business meaning that no parser can infer, but decays as soon as code changes. It should be limited to business lineage and to linking technical lineage to business terms.

Static analysis parses SQL, ETL definitions and code repositories to infer lineage. It covers a lot of ground quickly but misses what happens only at runtime: dynamic SQL, parameters, and code paths that depend on the data.

Runtime capture records lineage as jobs actually run. This is where open standards matter. OpenLineage defines a common model of runs, jobs and datasets, extended with facets that carry schema, data quality results and column-level lineage. It joined the LF AI & Data Foundation in 2021 and graduated there in 2023. Integrations exist for Apache Airflow, Apache Spark, Apache Flink and dbt, and several catalogs consume its events. Its column lineage facet distinguishes direct relationships (identity, transformation, aggregation) from indirect ones (join, filter, group by, sort, window, conditional), and flags where values were masked.

Platforms increasingly capture lineage natively. Databricks Unity Catalog, for example, records lineage down to column level for queries run through Spark DataFrames or Databricks SQL, including links to notebooks, jobs, dashboards and model serving. Its documentation also lists the limits, which are typical of runtime capture: column lineage is lost when tables are referenced by storage path rather than name, and user-defined functions can hide the mapping between source and target columns.

Two design rules follow:

1. Prefer an open event format at the boundary. Native lineage from each platform is valuable, but an enterprise that runs more than one platform needs to stitch lineage across them. Emitting or converting to an open standard keeps that option open and avoids lineage being locked into one catalog.
2. Treat lineage gaps as defects. Path-based reads, opaque functions, spreadsheets and manual uploads break the chain. Inventory them and either change the pattern or document the gap explicitly, as BCBS 239 already expects for manual workarounds.

Lineage for AI: from answer back to source

AI systems add hops that traditional lineage tools were not designed for. A training set is assembled from curated tables, filtered and labelled. A retrieval-augmented generation (RAG) system splits documents into chunks, embeds them, and stores them in an index. An answer cites chunks, not systems of record.

Lineage for AI should let you answer, for any model version or any answer:

• Which data was used, down to the dataset version or document version, and under what purpose and legal basis it was collected.
• What was done to it: filtering, labelling, de-duplication, redaction and enrichment, with the code version that did it.
• What depends on it: which models, indexes and applications must be retrained or re-indexed if a source is corrected, withdrawn or subject to a deletion request.

For RAG specifically, record the source document identifier, version and access permissions on every chunk at indexing time, and log which chunks were retrieved for each answer. This one design choice supports three needs at once: citations users can check, access control that follows the source, and the ability to remove content from the index when the source is deleted or reclassified. For the wider design choice between retrieval and model adaptation, see RAG vs Fine-Tuning vs Custom LLM.

A practical scoping grid

Lineage effort should follow risk and use, not coverage targets.

What to do in the next 90 days

1. Name the critical data elements behind your regulatory reports, executive metrics and planned high-risk AI systems. Lineage scope starts here.
2. Write down the questions lineage must answer, using the ECB's six purposes as a template, and test any tool against them.
3. Inventory what already emits lineage: orchestration, transformation and data platforms often capture it natively or through OpenLineage integrations that are not switched on.
4. Instrument one end-to-end flow for a single critical report, at column level, and measure where the chain breaks.
5. List the lineage breakers such as path-based reads, opaque functions, spreadsheets and manual uploads, and assign owners to remove or document each.
6. Add source ID, version and permissions to every chunk in your RAG indexing pipelines, and log retrieved chunks per answer.
7. For NDMO entities, write DSI.3.2 source-to-target mappings in a structured form your catalog can ingest, so Priority 1 work feeds Priority 2 tooling.

Frequently asked questions

What is data lineage?

Data lineage is the record of where data comes from, how it moves between systems and how it is transformed on the way to reports, applications and AI models. It ranges from business-level flows to column-level and run-level technical detail.

What is column-level lineage?

Column-level lineage records which source columns produce each output column and how, for example copied, calculated, aggregated, or used to filter or join. The ECB expects lineage at this attribute level for risk data in scope of BCBS 239.

What is OpenLineage?

OpenLineage is an open standard for collecting lineage metadata as jobs run, modelling runs, jobs and datasets with extensible facets. It is a graduated project of the LF AI & Data Foundation, with integrations for tools such as Apache Airflow, Apache Spark and dbt.

Does the EU AI Act require data lineage?

Article 10 does not use the word, but for high-risk AI systems it requires governance of training, validation and testing data covering data origin, the original purpose of personal data collection and data preparation steps. Meeting that in practice requires lineage and dataset versioning. The Digital Omnibus on AI moved high-risk application dates to December 2027 and August 2028.

Sources and further reading

• Principles for effective risk data aggregation and risk reporting (BCBS 239), Basel Committee on Banking Supervision, January 2013. Principles 2 and 3.
• Progress in adopting the Principles for effective risk data aggregation and risk reporting, Basel Committee on Banking Supervision, 28 November 2023.
• Guide on effective risk data aggregation and risk reporting, European Central Bank, May 2024. Section 3.4 and footnote 23 on data lineage.
• EU AI Act, Article 10: Data and data governance, Regulation (EU) 2024/1689.
• Artificial intelligence: Council and Parliament agree to simplify and streamline rules, Council of the European Union, 7 May 2026.
• Digital Omnibus on AI: adoption of the legislative act, Council of the European Union, 29 June 2026.
• Data Management and Personal Data Protection Standards, Version 1.5, NDMO, SDAIA. Specifications DSI.1.1, DSI.3.2, DAM.3.1, DAM.3.2, DAM.4.2, MCM.1.3 and OD.3.6.
• OpenLineage graduation announcement, LF AI & Data Foundation, 20 September 2023.
• OpenLineage column lineage facet, OpenLineage project.
• Capture and view data lineage using Unity Catalog, Databricks documentation.

This article is general guidance, not legal or regulatory advice. Confirm obligations with your supervisor and current regulatory texts.