Choose an integration shape without a religion

In this chapter

• Every shape works, and every shape charges you. Dimensional-first, third normal form, Data Vault and one big table.
• A few factors decide, not faith. Weighed in writing.
• The lakehouse retired old arguments. Meaning is what’s left to split on.
• Five obligations come with every shape. Keys, identity, history, the mapping and lineage.

Your integration layer, the silver layer built against the business’s agreed model, still needs a physical shape. Which one? The industry treats that as a matter of faith. Treat it as a decision, weighed in writing, because that’s all it is.

Vendor guidance won’t decide it for you. Databricks’ medallion glossary names third-normal-form-like and Data Vault-like models for silver without choosing between them. The religions will decide it for the wrong reasons.

The choice is real, it’s smaller than the arguments make it, and it excuses you from nothing.

Which integration shape should you choose?

Every shape here works, has production estates behind it and has advocates who aren’t fools. So don’t ask which one is correct. Ask what each optimizes for, and what it charges you in return.

Analysts need answers they can read. Put each measurable event in a fact table, surround it with the dimensions that describe it, and an analyst reads the result without a translator. That’s the dimensional-first shape of Ralph Kimball’s star schema. Lawrence Corr’s BEAM showed how to design it with business people in the room. That shape optimizes for query usability now.

The cost hides in where the integration went. It’s deferred into conformed dimensions, the dimensions every report slices by the same way. So the hardest cross-system agreement arrives late, mart by mart, under delivery pressure. Skip the conformance work and a star estate becomes departmental answers that agree with nobody.

You want every fact stored once. Integrate everything before anyone consumes it, non-redundantly. Bill Inmon’s atomic warehouse does exactly that, in third normal form. It optimizes for integrity and one version of the truth.

But that integrity creates friction at both ends. Analysts pay a join tax on ordinary questions, and a new source or a revised relationship reshapes structures everything downstream depends on. It also demands enterprise modeling stamina up front, which most organizations claim and few fund.

Sources keep arriving, and audit wants history. You want a second source to land beside the first without reshaping what’s there. So you separate what a source table mixes together. One kind of table holds the business keys, how the business tells one occurrence from another. Another holds the relationships between those keys, and a third holds the descriptive context and its history. Those are the hubs, links and satellites of Dan Linstedt’s Data Vault. It’s built for multi-source agility and audit, keeping full history by default.

The cost is scale of a different kind. Every source table becomes several vault tables. The pattern discipline is unforgiving, and the people who hold it are the scarcest skill pool of the four. The blog’s Data Vault posts go deeper.

The first answer is wanted fast. Land the data in wide, denormalized tables, cheap on columnar formats, and query it straight away, with no modeling gate. That’s one big table, and it optimizes for simplicity now.

Its cost arrives on a delay. The semantics live in transformation code instead of a model. Every consumer re-derives them slightly differently, and governance drifts with no agreed structure to attach to. It’s the fastest route to a first answer and the slowest to a shared one.

Figure 6.1: the same subject in four shapes. Every one of them works; every one of them charges you differently.

The factors decide, not the faith

Strip the advocacy away and a few factors do the separating. Weigh them in writing and the decision largely makes itself.

• Change tolerance. How often do new sources arrive, or old ones change shape? Frequent arrivals favor the shape built for addition. A stable estate can afford one that integrates harder up front.
• Audit and history obligations. If you must show what you knew and when, a shape that keeps history by default beats one where history is a per-table choice someone might skip.
• Team skills. The best shape your team can’t staff is worse than the second-best one it can.
• Query and compute economics. Who pays, the load or the read? Join-heavy shapes tax every consumer, wide shapes tax change and governance, and pricing sets each tax.
• Latency. The shorter the path from landing to consumption, the fewer transformation stages the shape can afford.
• Appetite for modeling. Some organizations fund the agreement work and others won’t. Match the appetite you’ve observed, not the one in the slide deck.

Team skills is also the factor most often allowed to decide alone, and “we know stars, so stars” is a real factor wearing the costume of a decision. The other rows belong in the same meeting.

In recent practitioner discourse, comparisons that start as arguments keep ending as hybrids: an integration shape in silver (Data Vault or third normal form) and dimensional serving in gold. The books teach each shape alone, so nobody’s book walks through that hybrid. This one walks its silver half and leaves gold outside its scope.

Figure 6.2: the decision table. Four shapes weighed on the factors that actually differ, so the choice is argued rather than inherited: the decision is yours, and the obligations are not optional.

Factor	Dimensional-first	Third normal form	Data Vault	One big table
Optimizes for	Query usability now	Integrity, one version	Multi-source agility, audit	Simplicity, first answer fast
New source arrives	Conform another dimension	Reshape affected structures	Add tables beside existing ones	Rebuild the wide table
History	Per-dimension design choice	Design choice	Kept by default	Whatever the pipeline kept
Skills market	Widely held	Classical, thinning	Scarcest of the four	None required at first
Read-side cost	Lowest	Join tax	Join tax, absorbed by views	Lowest until drift
Modeling appetite required	Moderate, rising at conformance	High, up front	High, sustained	None, which is the trap

The lakehouse moved the goalposts

Whole shelves of shape advocacy were written on assumptions a columnar lakehouse quietly deleted. Width was expensive on row stores, so shapes split tables narrow. On columnar formats a repeated value collapses to nearly nothing, and a query reads only the columns it asks for. So width became cheap.

Joins were the counter-argument, and generated read views absorbed them: current views, end-dated views and point-in-time reconstructions. They’re built from metadata, so consumers no longer see the physical layout. Even change detection moved into the platform, as declarative change data capture that keys, sequences and versions rows from metadata you already hold.

What survives of the old physics? Row count still costs you, and one hazard got sharper. A vault detects change by hashing each satellite’s columns, so every satellite boundary is a hash boundary. Move a column across one and the histories on both sides become incomparable. A physical split is the least reversible decision in the model. The business’s agreed model is the thing most likely to change.

So split less, and never casually.

Every split that row-store physics justified needs re-deciding, and business meaning is what’s left to split on. Sensitivity is a boundary, because a regulatory erasure should touch a small table rather than rewrite years of history. Governance status is one too. One working modeling standard keeps governed, mapped columns in a separate satellite from ungoverned, unmapped ones, so the boundary itself records a governance fact. A change-rate difference of orders of magnitude is a boundary when profiling measures it, not when a rule of thumb feels it.

“We always split by source system” isn’t a boundary. It’s an inherited habit whose reasons retired.

The most argued-about shape also suffers a misreading. Read “raw vault” as a vault shaped like its sources and you’ll decompose each source table mechanically into hubs, links and satellites. That skips the business key ruling and keeps the ritual. But raw describes the data, never the structure. A raw vault holds data exactly as the sources delivered it, integrated by business keys the business ruled on.

The obligations no shape excuses

Here’s what the religions get wrong, in both directions. Advocates imply their shape delivers the integration work automatically, and skeptics imply that skipping the shape skips that work. Both are false.

Every shape carries the same five obligations, each a decision a person makes, not a property a structure has.

• Business keys someone ruled on. A business key is a decision about what the business counts: one Customer, however many systems store one.
• Identity that survives renames. Key the source inventory on stable identifiers, not names, and carry them through the integration layer, or every rename silently orphans your history and lineage.
• History with a declared depth. How far back, at what grain (what one row records), decided by whom. “The platform keeps versions” is a mechanism, not a policy.
• The mapping. Each source table and column paired with what it means, with who decided and why, plus a per-attribute ruling on whose value survives when matched records disagree.
• Lineage. Where each value came from, kept as metadata rather than reconstructed by archaeology.

No shape makes those decisions for you.

A hub is a business key, and a conformed dimension, a third normal form entity and a wide table’s grain need the same ruling. Some shapes only make a missing one visible sooner.

The mapping is the substance of integration in every shape. A vault without its rulings is source-shaped storage wearing vault vocabulary, and a star without conformance is a dashboard cache.

Once the obligations are met, write the decision down: the shape you chose, the factors you weighed, the date and the people who decided. Keep it to one page, where the next engineer will look. Why write it down? Estates outlive their architects. And a shape with no recorded rationale gets relitigated by every new hire fluent in a different one.

The people part	The architect proposes the shape with its factors weighed, and the stewards and business experts who answer for meaning rule on the business keys and the history depth.
Metadata to capture	Keep the decision record (shape, factors weighed, date and deciders), each business key ruling, the declared history depth and each split boundary with its reason.
Regardless of stack	The factors hold on every platform and a different engine reprices only query and compute economics, so choosing a shape is not a platform decision.

In DeltaVault today

DeltaVault, our metadata catalog, generates Data Vault loads and star schema definitions for Databricks. Where your methodology models both a vault and a mart, right-clicking a staging table on a workspace’s Canvas tab offers Data Vault and Data Mart side by side, so a mart needn’t wait for a vault. In a generated star schema, a conformed dimension has identical column definitions wherever it appears.

The Data Vault action turns business keys into hubs, references into links and descriptive columns into satellites, and it won’t build a hub until you designate the table’s business key. Nothing physical exists until you choose Materialize. From a materialized hub, Create Point in Time and Create Bridge add tables for past dates and multi-hop paths, so nobody hand-writes those joins.