Why governance programs stall
Most data governance programs start the same way. A council is formed, a policy framework is written, data owners are named in a spreadsheet, and a review step is added before new data products go live. Eighteen months later the policies are approved, the spreadsheet is out of date, and delivery teams describe governance as the queue they wait in.
The evidence that this pattern fails is unusually consistent.
- Gartner predicted in February 2024 that 80% of data and analytics governance initiatives will fail by 2027. Its analysts tied failure to programs that do not enable prioritized business outcomes, and argued against "center-out, command-and-control" governance in favor of approaches that are tied to outcomes and adapt as the organization changes. It is a prediction, not a measurement, but the diagnosis matches what practitioners see.
- Banking supervisors have measured it. The Basel Committee's November 2023 progress report on its risk data principles (BCBS 239) found that only two of 31 global systemically important banks were fully compliant, nearly ten years after the principles were published. Bank programs were often underfunded, limited in scope and lacking board attention.
- The European Central Bank, in its May 2024 guide on risk data aggregation, recalls that none of the 25 significant institutions in its 2016 thematic review had fully followed those principles, and that the review and later on-site inspections revealed shortcomings in the effectiveness of data governance frameworks.
These are some of the best-funded, most heavily supervised data governance programs anywhere. If they struggle, the problem is not effort. It is design.
Our thesis is simple: governance fails when it is a separate layer that reviews work after it is done, and works when its rules run inside the systems that move data. The goal is not less governance. It is fewer meetings per governed dataset.
Figure 1
Move governance out of the meeting and into the pipeline
The same decisions, made in two different places
Governance as a committee
- Main control
- Review before go-live
- Scales with
- Reviewer capacity and meeting cadence
- Source of truth
- Policy documents and spreadsheets
- Evidence
- Compiled before audits
- People spend time on
- Routine approvals
Governance in the pipeline
- Main control
- Rules applied automatically as data moves
- Scales with
- Platform coverage
- Source of truth
- Versioned policies, tags and contracts
- Evidence
- Produced by every run
- People spend time on
- Rules, exceptions and disputed definitions
Where the bureaucracy comes from
Bureaucracy in governance is rarely the result of bad intentions. It comes from three design choices that look sensible on paper.
Governance as a gate. When the main control is a review before go-live, every dataset waits for the same reviewers, whatever its risk. Teams learn to bundle requests, avoid the process, or build in tools the reviewers cannot see. A review step that cannot keep pace with delivery does not stop risky work. It moves it out of sight.
Governance as documentation. Policies, glossaries and ownership matrices are written once and drift from reality the day after approval. A data owner listed in a spreadsheet nobody reads has no effect on how data is used.
Governance measured by activity. Programs report policies approved, stewards appointed and council meetings held. None of these tell leadership whether data is more trusted, better protected or faster to use. When activity is the measure, adding process looks like progress.
The fix for each is the same move: take the decision the committee was making and turn the routine part into a default that the platform applies automatically, leaving people to handle the exceptions.
Four mechanisms that replace meetings
1. Classify at ingestion, automatically
Every protection rule depends on knowing what the data is. Manual classification campaigns cannot keep up with data that is created daily. Classification should happen when data lands, using pattern matching and trained classifiers, with people reviewing the uncertain cases rather than labeling everything.
The tooling exists in mainstream platforms. Microsoft Purview, for example, can apply sensitivity labels automatically, both in the client as users write documents and in the service for content already stored in SharePoint, OneDrive and Exchange, and it supports a simulation mode so the rules can be tested before they apply. Data platforms offer similar scanning for structured data. The governance decision that remains for people is the classification scheme itself and the threshold at which a machine's guess needs a human check.
In Saudi Arabia the scheme is not optional. The NDMO standards prescribe a national classification and impact assessment, covered in our guide to implementing the NDMO standards. Automation changes how fast you apply the scheme, not what the scheme is.
2. Bind policies to classifications, not to tables
Once data carries a classification tag, protection can follow the tag. Snowflake's tag-based masking, for example, lets an administrator assign a masking policy to a tag once, after which any column carrying that tag is protected, including columns added later (it is an Enterprise Edition feature). Most major platforms offer an equivalent.
This is policy as code in its most useful form. The policy is written once, reviewed like code, versioned, tested and applied everywhere the tag appears. For rules that span systems, general-purpose policy engines such as Open Policy Agent, a graduated project of the Cloud Native Computing Foundation since January 2021, let teams express access rules in code and evaluate them consistently across services.
The governance council stops approving individual access requests and starts approving the rules that decide them.
Figure 2
Write the rule once, let the tag carry it
How classification and policy as code replace case-by-case approvals
- Step 1LandData arrives in the platformPlatform
- Step 2ClassifyClassifiers tag columns and documents; low-confidence results go to a stewardAutomated + steward
- Step 3BindTag triggers the policy written once: masking, row filters, retentionCentral rule, automated
- Step 4EnforceEvery query and service account gets the policy resultPlatform
- Step 5EvidencePolicy decisions, tag history and access logs kept as audit recordsAutomated
People approve step 3, the rule, once. They no longer approve each request that the rule decides.
3. Put a contract at every important handover
Most data quality incidents start when a producer changes something a consumer depended on: a renamed column, a changed type, a field that starts arriving empty. A data contract makes that dependency explicit and machine-checkable.
Two layers are now common:
- Build-time contracts. dbt model contracts, for example, declare the columns, data types and constraints a model must return. With enforcement on, dbt checks the model's output against the contract and the model fails to build if it does not match. Removing a column or changing its type is flagged as a breaking change.
- Exchange contracts. The Open Data Contract Standard, maintained by the Bitol project under the LF AI & Data Foundation, defines a YAML format covering schema, data quality rules, service levels, ownership, roles and support channels. It gives producers and consumers in different tools a shared description of what was promised.
A contract replaces a recurring conversation ("why did the report break?") with a check that runs on every change. Start with the handovers that feed regulatory reports, executive dashboards and AI systems. Not every table needs one.
4. Federate stewardship, centralize the rules
Zhamak Dehghani's data mesh principles describe federated computational governance: decisions about a domain's own data model stay with the domain, decisions that must be consistent across domains are made globally, and those global rules are embedded in the platform and applied automatically rather than by manual review.
You do not need to adopt data mesh to use the idea. The practical split is:
- Central: the classification scheme, the policy rules, identity and access standards, interoperability standards such as shared identifiers, and the platform that enforces them.
- Domain: ownership of data products, business definitions, quality thresholds that reflect business use, and the decision to publish.
- Automated: classification, masking, contract checks, quality monitoring and evidence capture.
Figure 3
Centralize the rules, federate the judgement, automate the routine
Where common governance decisions belong in a federated model
- Central
Classification scheme and thresholds
National scheme where mandated, such as NDMO
- Central
Access and masking rules
Written as code, versioned and reviewed
- Central
Shared identifiers and interoperability standards
Customer, product, legal entity
- Domain
Business definitions and data product design
Closest to how the data is used
- Domain
Quality thresholds and publication
Set by the owner, checked by contract
- Automated
Applying tags, masking and contract checks
On every load and change
- Automated
Collecting audit evidence
By-product of policy and pipeline runs
Stewardship works when it is part of named people's jobs in the domain, with time allocated. A central team of stewards trying to understand every domain's data recreates the bottleneck. For how this decision-rights approach applies to AI programs more broadly, see The Enterprise AI Operating Model.
Measure governance by outcomes
If governance is measured by activity, it will produce activity. The measures that matter describe the state of the data and the experience of the people using it.
Figure 4
Stop counting governance activity. Measure what it changes.
Activity measures and the outcome measures that should replace them
- Policies approvedShare of critical data elements with owner, classification and contract
- Stewards appointedQuality incidents reaching consumers per month, and time to resolve
- Council meetings heldMedian time to grant access to a governed dataset
- Datasets cataloguedRequests and pipelines bypassing the governed route
- Training sessions deliveredShare of audit evidence generated automatically
Two points make these measures work. First, measure on the critical data, not the whole estate: the datasets behind regulatory reports, core services and AI systems. A program that fully governs 50 critical data elements is worth more than one that has catalogued 50,000 tables without owners. Second, report time-to-access and time-to-approve alongside risk measures. If governance makes data slower to use without making it safer, the numbers will show it, and teams will route around it long before the numbers do.
Weak versus strong governance in practice
| Area | Bureaucratic version | Embedded version |
|---|---|---|
| Classification | Annual labeling campaign; spreadsheet of sensitive tables | Classifiers run at ingestion; humans review low-confidence results |
| Access | Ticket to a data owner for every request | Requests decided by policy on tags and roles; owners handle exceptions |
| Quality | Quality issues raised in council meetings | Contract and quality checks fail the pipeline; incidents tracked with service levels |
| Ownership | Owners named in a RACI document | Owner and steward recorded on every catalog asset, reviewed when people move |
| Change | Architecture review board approves schema changes | Breaking changes detected automatically against contracts |
| Evidence | Compiled manually before audits | Generated as a by-product of policy, contract and quality runs |
| Reporting | Policies approved, stewards appointed | Critical data under contract, incidents, time-to-access |
A worked example (illustrative)
Consider a retail bank building a customer complaints data product that feeds a regulatory report and a generative AI assistant for complaint handlers.
In the bureaucratic version, the product waits six weeks for a governance review. The review checks a classification spreadsheet, approves access for the assistant team, and records the owner. Three months later an upstream system renames a field, the report breaks, and nobody notices that the assistant is now retrieving free-text complaints containing card numbers.
In the embedded version, classifiers tag the free-text column as containing payment card data when it lands. The tag triggers a masking policy, so the assistant's service account sees redacted text by default. A contract on the upstream feed flags the rename as a breaking change before it is deployed. The steward in the complaints team gets an alert, not a meeting invitation. The evidence an auditor would ask for, who could see what and when, already exists in the policy and pipeline logs.
The second version involves less governance effort, and more governance.
What to do in the next 90 days
- List your 20 to 50 critical data elements and the reports, services and AI systems that depend on them. Scope everything below to these first.
- Measure today's friction: median time to grant access, time from request to a new dataset, and how many requests bypass the process. These are your baseline.
- Turn on automated classification for one major platform in simulation mode, and compare its results with your current manual labels.
- Convert your top five access rules into policies bound to classification tags, versioned in a repository and reviewed like code.
- Add contracts to the three handovers that break most often, starting with feeds into regulatory reporting and AI systems.
- Move one steward role into a domain team with allocated time and a quality objective tied to that domain's critical data.
- Replace the activity dashboard with outcome measures and report it to the same executive forum that sponsors the program.
Governance designed this way also gives AI programs the classified, contracted and owned data they need. For how that connects to value, see From AI Experimentation to Enterprise Value.
Frequently asked questions
Why do data governance programs fail?
Most fail because governance is designed as a separate review layer, documented rather than enforced, and measured by activity instead of outcomes. Gartner predicts that 80% of data and analytics governance initiatives will fail by 2027, linking failure to programs that do not enable business outcomes.
What is policy as code in data governance?
It means expressing governance rules, such as who can see which classification of data, as versioned code that the platform evaluates automatically. Examples include tag-based masking policies in data platforms and general-purpose policy engines such as Open Policy Agent.
What is a data contract?
A data contract is a machine-checkable agreement between a data producer and its consumers that defines schema, quality rules, service levels and ownership. Tools such as dbt can fail a build when a model breaks its contract, and the Open Data Contract Standard provides a common format.
How should data governance be measured?
By the state of critical data and the experience of its users: the share of critical data with an owner, classification and contract, the number of quality incidents reaching consumers, time to grant access, and how much evidence is produced automatically. Counts of policies and meetings are not outcomes.
Sources and further reading
- Gartner Predicts 80% of D&A Governance Initiatives Will Fail by 2027, Gartner press release, February 2024, republished by BizTechReports.
- Progress in adopting the Principles for effective risk data aggregation and risk reporting, Basel Committee on Banking Supervision, 28 November 2023.
- Guide on effective risk data aggregation and risk reporting, European Central Bank, May 2024.
- Data Mesh Principles and Logical Architecture, Zhamak Dehghani, martinfowler.com, 3 December 2020.
- Open Policy Agent, Cloud Native Computing Foundation. Graduated January 2021.
- Open Data Contract Standard, Bitol, LF AI & Data Foundation.
- Model contracts, dbt Labs documentation.
- Tag-based masking policies, Snowflake documentation.
- Automatically apply a sensitivity label to Microsoft 365 data, Microsoft Learn.
Product capabilities are cited as examples of a pattern, not recommendations. Confirm current features and editions with each vendor.