Choose an integration shape without a religion

In this chapter

• Data Vault, third normal form, dimensional-first, and one big table, each surveyed fairly enough that its advocates would nod.
• The decision factors that actually separate the shapes.
• What the lakehouse changed about the old arguments.
• The obligations no shape excuses.

Chapter 5 left you holding the record that proves integration: the mapping. The next question is what shape the integrated data should take, and the industry treats that question as a matter of faith. This chapter treats it as a decision, because that is all it is.

Consider the entire official silver-layer modeling guidance from the largest lakehouse platform: “From a data modeling perspective, the Silver Layer has more 3rd-Normal Form like data models. Data Vault-like, write-performant data models can be used in this layer.” That is Databricks’ medallion glossary quoted in full, two methods named in two sentences, neither chosen and neither taught. So the vendors will not decide this for you, and the religions will decide it for the wrong reasons. The truth sits in between: the choice is real, it is smaller than the arguments make it, and it excuses you from nothing.

Which integration shape should you choose?

Every shape in this survey works, has production estates behind it, and has advocates who are not fools. So the honest question is never which shape is correct. It is what each one optimizes for, and what it charges you in return. The Data Vault posts on the blog go deeper into the shape this chapter can only give a paragraph.

Dimensional-first optimizes for query usability now. Ralph Kimball’s stars put facts and dimensions in front of analysts in a shape they can read without a translator, and Lawrence Corr’s BEAM showed how to design them with business people in the room rather than after they leave. The cost is hidden in where the integration went: it is deferred into conformed dimensions, so the hardest cross-system agreement arrives late, mart by mart, under delivery pressure. A star estate that never does the conformance work becomes a collection of departmental answers that agree with nobody.

Third normal form optimizes for integrity and one version of the truth. Bill Inmon’s atomic warehouse stores every fact once, non-redundantly, integrated before anyone consumes it. That integrity creates friction at both ends. Analysts pay a join tax to answer ordinary questions, and change amplifies, because a new source or a revised relationship reshapes structures that everything downstream already depends on. It also demands enterprise modeling stamina up front, which is exactly the appetite most organizations claim and few fund.

Data Vault optimizes for multi-source agility and audit. Dan Linstedt’s hubs, links, and satellites separate business keys from relationships from context, so a second source lands beside the first without reshaping what already exists, and full history is kept by default. Here the cost is scale of a different kind: every source table becomes several vault tables, the pattern discipline is unforgiving, and the people who hold it are the scarcest skill pool of the four.

One big table optimizes for simplicity now. Wide, denormalized tables are cheap to store on columnar formats and cheap to query, and there is no modeling gate between landing data and asking questions of it. The cost arrives on a delay. The semantics live in transformation code instead of in a model, every consumer re-derives them slightly differently, and governance drifts because there is no agreed structure for it to attach to. It is the fastest route to a first answer and the slowest route to a shared one.

Figure 6.1: the same subject in four shapes. Every one of them works; every one of them charges you differently.

The factors decide, not the faith

Strip the advocacy away and a small set of factors does the actual separating. Weigh them for your estate, in writing, and the decision largely makes itself.

• Change tolerance. How often do new sources arrive, and how often do the existing ones change shape? Frequent arrivals favor the shape built for addition, while a stable estate can afford a shape that integrates harder up front.
• Audit and history obligations. If regulation or dispute resolution requires you to show what you knew and when, a shape that keeps history by default beats one where history is a per-table design decision someone might skip.
• Team skills. The best shape your team cannot staff is worse than the second-best shape it can. That constraint is real, and it is also the factor most often allowed to decide alone, which is precisely what it should not do.
• Query and compute economics. Who pays, the load or the read? Join-heavy shapes tax every consumer, wide shapes tax change and governance, and your platform’s pricing decides how much each tax costs.
• Latency. The shorter the path from landing to consumption has to be, the fewer transformation stages the shape can afford.
• Appetite for modeling. Some organizations will fund the agreement work and others will not, whatever they said at kickoff. Choose a shape whose demands match the appetite you have actually observed, not the one in the slide deck.

Weigh those factors across the practitioner discourse of the last few years and a pattern emerges: comparisons that start as arguments keep ending as hybrids, with an integration shape in silver (Data Vault or third normal form) and dimensional serving in gold. The books teach each shape alone, so the hybrid that practitioners keep converging on is a process nobody’s book walks through. That process is what chapters 3 through 9 of this one are.

Figure 6.2: the decision table. Four shapes weighed on the factors that actually differ, so the choice is argued rather than inherited: the decision is yours, and the obligations are not optional.

Factor	Dimensional-first	Third normal form	Data Vault	One big table
Optimizes for	Query usability now	Integrity, one version	Multi-source agility, audit	Simplicity, first answer fast
New source arrives	Conform another dimension	Reshape affected structures	Add tables beside existing ones	Rebuild the wide table
History	Per-dimension design choice	Design choice	Kept by default	Whatever the pipeline kept
Skills market	Widely held	Classical, thinning	Scarcest of the four	None required at first
Read-side cost	Lowest	Join tax	Join tax, absorbed by views	Lowest until drift
Modeling appetite required	Moderate, rising at conformance	High, up front	High, sustained	None, which is the trap

The people part. The architect proposes the shape, and the proposal should read like this chapter: factors weighed, trade-offs named, no religion. The rulings underneath the shape, though, which business keys the estate counts by and what history depth the business owes, belong to the stewards and business experts who can answer for meaning, exactly as chapter 3’s rulings and chapter 5’s mappings did. Watch, too, for the quiet anti-pattern of a shape chosen purely by hiring history. “We know stars, so stars” is a real factor wearing the costume of a decision. Put team skills in the table as one row, and make the other rows show up to the same meeting.

The lakehouse moved the goalposts

Whole shelves of shape advocacy were written on assumptions a columnar lakehouse quietly deleted. Storage width was expensive on row stores, so shapes split tables narrow. On columnar formats a repeated value collapses to nearly nothing and a query reads only the columns it asks for, so width became cheap. Joins were the counter-argument, and generated read views absorbed them: current views, end-dated views, and point-in-time reconstructions, all produced from metadata rather than hand-written, so consumers stopped seeing the physical layout. Even change detection, the load mechanic that justified so much structural cleverness, moved into the platform as declarative change data capture that keys, sequences, and versions rows from metadata you already hold.

What survives of the old physics is row count, which remains a real cost worth designing for, along with one hazard that got sharper. In a vault, every satellite boundary is a hash boundary, so moving a column across it makes the histories on both sides incomparable. That makes a physical split the least reversible decision in the model, and the conceptual model is the thing most likely to change. Take those two facts together and you should split less, and never casually.

Every split and structure in your current shape that was justified by row-store physics therefore needs re-deciding, because its justification no longer exists. What remains as a split criterion is business meaning. Sensitivity is a boundary, because a regulatory erasure should touch a small table rather than rewrite years of history. Governance status is a boundary too: one working modeling standard keeps governed, mapped columns in a separate satellite from ungoverned, unmapped ones, so the table boundary itself records a governance fact. A change-rate difference of orders of magnitude is a boundary when profiling measures it, not when a rule of thumb merely feels it. “We always split by source system” is not a boundary. It is an inherited habit whose reasons retired.

One vocabulary correction rescues an entire misreading of the most argued-about shape: raw describes the data, never the structure. A raw vault holds data exactly as the sources delivered it, integrated by business keys the business ruled on. Mechanically decomposing source tables into vault shapes skips the ruling and keeps the ritual.

The obligations no shape excuses

Here is what the religions get wrong, and they get it wrong in both directions: advocates imply their shape provides these things automatically, while skeptics imply that skipping the shape skips the work. Both are false. Every shape in the table carries the same five obligations, and each one is a decision a person makes rather than a property a structure has.

• Business keys someone ruled on. A hub is a business key, and a business key is a decision about what the business counts: one Customer, however many systems store one. A conformed dimension needs the identical ruling, and so does a third normal form entity, and so does the grain of a wide table. No shape makes the decision for you; some shapes only make its absence visible sooner.
• Identity that survives renames. Chapter 4’s inventory of stable identifiers has to hold all the way through the integration layer, because otherwise every physical rename silently orphans your history and your lineage.
• History with a declared depth. How far back, at what grain, decided by whom. “The platform keeps versions” is a mechanism, not a policy.
• The mapping. Chapter 5’s record of what means what is the substance of integration in every shape. A vault without those rulings is source-shaped storage wearing vault vocabulary, and a star without conformance is a dashboard cache.
• Lineage. Where each value came from, kept as metadata rather than reconstructed by archaeology. Chapter 9 takes this up.

Once the obligations are met, write the decision down: the shape you chose, the factors you weighed, the date, and the people who decided. One page, kept where the next engineer will look. Estates outlive their architects, and an integration shape with no recorded rationale gets relitigated by every new hire who arrives fluent in a different one.

Metadata to capture. Start with the decision record itself, covering shape, factors weighed, date, and deciders. Underneath it sit the business-key rulings and the declared history depth. Capture every split boundary along with the reason it exists, so the next modeler can tell a meaning boundary from a retired habit.

Regardless of stack. The factor list survives every platform, and only the economics row changes its weights. A different engine reprices joins, width, and history, and touches nothing else in the table. That is exactly why choosing a shape is not a platform decision, and why no platform migration will ever do it for you.

Whichever column of the table you choose, this book supports the choice, because the point was never the pattern. The point is what happens after the choice: an integration layer generated from the model you have been building since chapter 3, rather than hand-written until it drifts away from it. A shape you picked with a written rationale, expressed as structures no human hand-drifts, is the difference between architecture and archaeology. Generating it is chapter 7.