In this chapter
- Lineage is cheapest at derivation. Recorded at column grain as the work happens, a trace becomes a walk.
- Governance lives on the business attribute. Every mapped column inherits it, and gates name what they block.
- Rules earn trust or lose it. Keep them in a tunable catalog, and design against noise and stale findings.
- The model belongs under version control. Branch, diff, review and a gated commit, with tests generated from the model.
Every go-live review asks three questions. Where did this number come from? Who is accountable for it? What changed, and who agreed to it?
Answer them while you build and the review is a formality. Leave them until an incident or an auditor asks, and it’s the same work at several times the price.
Is data lineage a by-product, or a project?
Answering the first question means walking back from a number to every column that fed it. Record which upstream column produced which downstream column at the moment of derivation, and that walk is already written down. It costs one field, filled in by the step that creates the column anyway.
That record is data lineage.
Reconstruct it later from SQL, query logs and notebooks nobody has opened since March, and it’s the expensive project every team eventually pays for. And you get a weaker result. It’s an observation of what already ran, usually at table grain, stale from the first pipeline edit.
Here’s lineage recorded at derivation, on one column of Willibald, the garden webshop of Maluga, the outdoor-living retailer the book’s examples run on. The customer’s email address travels through the family of copies a source table arrives as: landing, staging, and a persistent staging copy that keeps every version of every row.
Figure 9.1: one column, four layers, four records that nobody drew by hand.Three of the fifteen columns in that staging table name no parent: the hash difference, the load timestamp and the record source. They’re system columns, which every layer adds by a convention declared once, so “nothing upstream” is their correct record.
What you can’t afford is a column whose record is silent because nobody captured anything.
A real lineage record needs column grain with expressions resolved. A computed column records every column its expression reads, or the trace goes dark where the logic gets interesting. The record also needs the mappings, each a recorded decision that this column carries that business attribute.
Because the mappings end in the conceptual model, a governed attribute traces back to every column that feeds it, in every source. So asking what a change will break, which is impact analysis, becomes a query you run before you change anything.
Governance flows through the mappings
Who’s accountable for that email address? It’s personal data in every landing table, staging copy and downstream structure it reaches. Tag each column and you record the same fact once per column, in places that will disagree by the second quarter.
So hold data governance where the meaning lives, on the business entity and its attributes. Classify the Email Address attribute once as personal data, set its sensitivity, and name its owner and its steward, the person who can say which definition is authoritative. Every column mapped to it inherits all of that, because the mapping is how meaning reaches the tables and columns.
Measurement turns that inheritance into governance. Declare requirements per property and asset type: a table must carry an owner and a description, an entity an approved definition and a classification, an attribute its definition. Each is present or absent, so completeness becomes a number for a subject area instead of a feeling. Most teams have the policy document, not the number, so the honest answer to “are we governed?” is usually a pause.
A requirement that stops nothing is only tracking. It becomes a gate when it names the action it blocks: generating a build, approving a definition, accepting a mapping, finishing an import. Two rules keep gates from becoming an obstacle course.
Gate the release, not the thinking. Block generation, approval and promotion to the main line. But never block committing metadata to a branch. If a half-governed draft can’t be saved, the work moves into spreadsheets where nothing is measured.
You can’t require a fact about something that doesn’t exist yet. A gate that fails an import because its new tables carry no descriptions blocks the very step that creates them. Let new assets pass, then govern them. And check the unattended paths. A gate built around a person accepting a proposal does nothing when nobody’s there.
Rules with teeth, aimed carefully
Requirements ask whether a fact was captured. Validation asks a harder question: does it make sense? Think of a business key (how the business tells one occurrence from another) that no column implements, a system column that has drifted from its convention, or a transformation calling something the target platform doesn’t have.
Keep the rules as a catalog, not logic buried in code. Each rule carries plain-word criteria, remediation guidance, a severity and an enforcement level, all adjustable without a release. The check stays code, because a check is a program. Everything about it is data, because tuning it is a governance decision, not an engineering task.
Calling a naming advisory a warning rather than an error decides how much noise the team will tolerate. So stewards own the catalog. Rules nobody can see or tune get ignored the first week they turn noisy.
Two failure modes are fatal to trust.
Rules that fire on things nobody can act on. Show every advisory as loudly as a genuine defect and the only message left is “everything is wrong”. The sharpest version is the false positive. Check expressions against a hand-curated list rather than what the platform documents, and every name the list lacks becomes a problem in a correct expression. The cost isn’t the wrong warning. It’s the engineer who stops reading the warnings, including the one that mattered.
Findings that outlive what they complained about. A finding records what a rule said when it last ran, so it goes stale unless the rule runs again when the metadata changes. The convention moves, the column is fixed, and the finding still says what it said in June. So let anyone ask for a fresh evaluation, clear out the findings a fresh run didn’t rewrite, and make every fix report what it did. A fix that claims success without changing anything is worse than no fix.
A model you can diff
What changed, and who agreed to it? Software engineering settled that fifty years ago. Michael Fagan’s 1976 paper on design and code inspections made code review a measured practice rather than a courtesy. And the branch-and-pull-request loop around version control made it a reflex. But data teams inherited none of it for their metadata, which is the part everyone has to agree on.
The alternative is metadata kept only in application tables, updated in place. It can’t say what it looked like in June or show a reviewer a change before it lands. Governance built on that metadata reinvents half the loop in change-request forms. Five design choices make the loop real.
Figure 9.2: the loop software solved fifty years ago, with metadata files where code usually sits.- Edits land on a branch, never on the main line. Two people on separate branches never see each other’s half-finished thinking, and the main line gets only reviewed work.
- Deterministic file paths. The repository’s layout mirrors the catalog at the grain people edit, so a diff shows exactly what changed.
- Identity that survives a rename. References carry a stable identifier beside the readable name, so they keep resolving.
- A review that shows the exact difference. You see before and after per entity, revert one change and keep the rest, and get a named reviewer, since an unowned review is a stamp nobody read.
- A validated commit and a history that survives deletion. The history records who changed what, when, and who accepted it, and keeps a deleted item as a suppression so its audit trail stays.
Trust is also tests
One more question sits behind the other three: does what you built do what you said, in production, on a Tuesday? It takes the same metadata, pointed at the data.
Assertions generated from the model cover more ground than hand-written checks usually do. Reconcile row counts between source and target within a tolerance you set, and a numeric column’s totals, grouped so a total can’t hide a regional gap. Add a row-count band for loads that are suspiciously small or implausibly large. Check a declared key for uniqueness, and write a custom assertion for the invariant only your business knows.
Each assertion declares what a failure means: stop the workflow so nothing downstream runs on bad data, or record a warning. Define them once on the source table and every workflow step that loads it inherits them. That’s classify-once applied to tests.
The cheapest time is now
Record all of it while the work happens, before go-live.
| The people part | Name a reviewer for every metadata change, let stewards own the rule catalog, and decide who answers the auditor, from a history rather than a memory. |
| Metadata to capture | Capture column-grain provenance, classification and ownership on attributes, gates with the actions they block, rule severities, and who approved each change. |
| Regardless of stack | Version control, review gates and lineage written at creation are practices no vendor grants or forbids, and text files in a repository, with the discipline to record provenance at derivation, give better lineage than a catalog harvesting query logs nightly. |
In DeltaVault today
DeltaVault, our metadata catalog, records each derived column’s source columns and transform, and a table’s Lineage tab draws them column to column. A mapped column shows its business attribute’s classification as derived. Four governance gates (Generate build, Approve definition, Accept mapping and Complete import) can each be set to Block until complete; none blocks a branch commit. With GitHub connected, Review and Commit shows you each pending change on your branch before and after. A blocking validation rule stops the commit unless someone acknowledges it.
Every load step that reads a table inherits the data tests on the table’s Tests tab. Each test’s On Failure setting is Fail or Warn, and on Databricks a failing test set to Fail stops the workflow step it runs in.
