In this chapter
- Lineage recorded at creation, and what reconstructing it later costs.
- Governance as measurable completeness with gates, not a policy document.
- Rules with teeth, and the two ways rules go wrong.
- The model under version control: branch, diff, review, commit.
Chapter 8 closed on a trade: records written while the work happens cost you nothing, while the same records rebuilt afterwards become a project with a budget. This chapter takes that trade in its three most expensive forms, which happen to be the three questions every go-live review asks. Where did this number come from? Who is accountable for it? Can you show me what changed and who agreed to it?
Answer them while you build and the review is a formality. Answer them afterwards and you answer under time pressure, in an incident or in front of an auditor, which is the same work at several times the price.
Is data lineage a by-product, or a project?
Record which upstream column produced which downstream column at the moment of derivation and data lineage costs almost nothing: one field, filled in by the step that was creating the column anyway. Reconstruct it later, from SQL and query logs and notebooks nobody has opened since March, and you are paying the expensive alternative every team eventually pays for. You also get a weaker result: an observation of what already ran, usually at table grain, stale from the first pipeline edit.
Lineage written at birth is unglamorous, and that is the argument for it. Here it is, on one column of the demo dataset.
Figure 9.1: one column, four layers, four records that nobody drew by hand.Four files and four one-line records are all it takes, and “where did this email address come from” becomes a walk rather than an investigation.
Three of the fifteen columns in that staging table name no parent: the hash difference, the load timestamp, and the record source. Those are chapter 7’s system columns, and their provenance is a declared convention, so “nothing upstream” is the correct record. What you cannot afford is a column whose record is silent because nobody captured anything.
Two demands separate a real lineage record from a diagram. The first is column grain with expressions resolved: a computed column has to record every column its expression reads, or the trace goes dark exactly where the logic gets interesting. The second follows from chapter 5. Because the mappings terminate in the conceptual model, a governed attribute traces back to every column that populates it, across every system that feeds it. Impact analysis stops being archaeology and becomes a query you run before you change anything.
Governance flows through the mappings
Who is accountable has the same shape as where it came from: capture it once, where the meaning lives, and let it travel. Data governance belongs on the business entity and its attributes, not on the tables and columns. Classify an attribute once as personal data, set its sensitivity, name its owner and its steward, and every column mapped to it inherits all of that, across every landing table, staging copy, and satellite the attribute reaches. Tag the columns instead and you record the same fact several hundred times, in places that will disagree with each other by the second quarter.
Inheritance gives you propagation, and measurement turns that propagation into governance. Declare requirements per property and per asset type: a table must carry an owner and a description, an entity an approved definition and a classification, an attribute its definition. Each one is either present or absent, so completeness becomes a number for a subject area instead of a feeling about one. Most teams have the policy document and no measurement, which is why the honest answer to “are we governed” is usually a pause.
A requirement that stops nothing is tracking rather than enforcement. Turning it into a gate means naming the action it blocks: generating a build, approving a definition, accepting a mapping, finishing an import. Two rules keep those gates from becoming an obstacle course.
Gate the release, not the thinking. Committing metadata to a branch should never be blocked by a governance requirement, because if a half-governed draft cannot be saved, the work moves into spreadsheets where nothing is measured at all. Block the generation, the approval stamp, and the promotion to the main line, but never the act of writing down what somebody just learned.
You cannot require a fact about something that does not exist yet. A gate that fails an import because the tables it is about to create carry no descriptions blocks the very step that creates them. Let new assets pass, then govern them once there is something to govern. Check the unattended paths as well, since a gate built around a person clicking Accept does not stand in an automatic one.
Rules with teeth, aimed carefully
Requirements answer whether a fact was captured. Validation answers a harder question: whether what was captured makes sense. A business key that no column implements, a system column that has drifted from the convention, a transformation referencing something the target platform does not have.
Treat the rules as a catalog rather than as logic buried in code. Each rule gets a title, criteria in plain words, remediation guidance, a severity, an enforcement level, and a switch, all adjustable without a release. The check itself stays code, because a check is a program. Everything about it is data, because tuning a severity is a governance decision rather than an engineering task. Rules nobody can see or tune get ignored the first week they turn noisy.
Two failure modes are worth designing against, because both are fatal to trust.
Rules that fire on things nobody can action. Render every advisory convention at the same volume as a genuine defect and you get a canvas full of amber icons, which communicates “everything is wrong” rather than anything actionable. The sharpest version of this is the false positive, and its usual cause is a hand-curated reference list. A checker holding a couple of dozen function names, judging expressions written against a platform that documents several hundred, reports every name it lacks as a problem when the expression is correct. The cost is not the wrong warning. It is the engineer who learns the warnings are wrong and stops reading them, including the one that mattered.
Findings that outlive the thing they complained about. A finding is a stored row, and if nothing re-evaluates the rule when the metadata changes, that row becomes the only statement left on a question it can no longer answer. The convention moved, the column was fixed, the severity was retuned, and the finding still says what it said in June. So let anyone ask for a fresh evaluation on demand, sweep the findings a fresh run did not rewrite, and make every remediation report what it actually did. A repair button that claims success without changing anything is worse than no button.
A model you can diff
Software engineering settled the third question fifty years ago. Michael Fagan’s 1976 paper on design and code inspections turned code review into a measured practice rather than a courtesy, and the branch-and-pull-request loop that grew up around version control turned it into a reflex. Data teams inherited none of it for their metadata, which is the part everyone has to agree on.
Five things make that loop real, and every one of them is a design choice you can make on any stack.
Figure 9.2: the loop software solved fifty years ago, with metadata files
where code usually sits.- Edits land on a branch, never on the main line directly. Two people working on different branches never see each other’s half-finished thinking, and the main line receives only work that has already been through the loop.
- Deterministic file paths, at the grain people edit. The layout mirrors the catalog, one file per column, so a pull request reads like a map of what changed. Write a 200-column table as a single file instead and every one-column edit rewrites the whole thing, leaving a diff nobody reads.
- Identity that survives a rename. References carry a stable identifier alongside the readable name, so renaming a table moves every descendant file in one commit while the references keep resolving. Identity carried by name alone breaks the first time a name changes.
- A review that shows the exact difference. You get before-and-after per entity, one change revertable without abandoning the rest, and preview paths that match the paths that land.
- A commit gated by validation, and a history that answers questions. The history records who changed what, when, and who accepted it, and it has to survive deletion: record a delete as a suppression, or the audit trail leaves with the object.
The alternative is a database nobody can diff, where metadata lives only in application tables, gets updated in place, and cannot say what it looked like in June or show a reviewer a change before it lands. Governance programs built on one end up reinventing half this loop by hand, in change-request forms.
Trust is also tests
The last question a go-live review asks is whether what you built does what you said it does, in production, on a Tuesday. Answering it takes the same metadata again, this time pointed at the data.
Assertions generated from the model cover more ground than hand-written checks usually do. You get a row-count reconciliation between source and target with a tolerance you set, an aggregate reconciliation on a numeric column, grouped so that a total cannot hide a regional gap, a row-count band that catches loads which are suspiciously small or implausibly large, a uniqueness check on a declared key, and a custom assertion for the invariant only your business knows about. Each one declares what a failure means: stop the workflow so nothing downstream runs on bad data, or record a warning worth knowing about but not worth stopping for.
Define them once on the source table, and every workflow step that loads it inherits them. The assertion then travels with the source instead of being copied into every pipeline that touches it, which is the classify-once argument applied to tests, and it leaves the model asserting itself in production.
The people part. Metadata changes need a named reviewer rather than whoever happens to notice the pull request, because a review nobody owns becomes an approval stamp with no reading behind it. Stewards own the rule catalog, since deciding that a naming advisory is a warning rather than an error is really a call about how much noise the team will tolerate. And somebody has to answer the auditor, which is far easier when the answer is a history rather than a memory.
Regardless of stack. Version control, review gates, and lineage written at creation are practices, not platform features: no vendor grants them and none forbids them. A team with text files, a repository, and the discipline to record provenance at derivation has better lineage than a team with an expensive catalog that harvests query logs once a night.
The cheapest time is now
Every item in this chapter is cheap while the work happens and expensive afterwards, which is why it belongs before go-live rather than after it. Provenance is one field at derivation, and a parsing project once you are in the middle of an incident. Classification is one decision on an attribute, or a hundred decisions on columns. History is a by-product of a loop you were going to run anyway, and an impossibility once the edits went straight into a database.
Metadata to capture. Write provenance pointers at creation, at column grain, with expressions resolved. Hold classification and ownership on the business attributes and let them inherit through the mappings. Set governance requirement levels per property and asset type, naming the gates you want and the actions they block. Keep the rule catalog so that each rule carries its severity, enforcement, and remediation guidance. Then keep the audit trail: who changed what, who approved it, and when.
You now have the whole of what chapters 2 through 9 asked you to capture, and a reason to capture all of it. What is missing is a plan for Monday morning, in an order that produces something usable long before it produces everything. That is chapter 10.