Most data teams’ weeks have a shape, and it isn’t the shape any of us would choose. The decisions about what the data actually means, who stewards Customer, what counts as Revenue, which columns hold personal information, land in the scraps of Friday afternoon, because everything before that went to plumbing. AI hasn’t changed that ordering. You can vibe code an ingest notebook from a chat prompt in minutes, which makes the plumbing cheaper to produce without making it any cheaper to own. What you get is more generated code, the same undocumented columns, and the same lineage nobody captured.

The wiring you maintain by hand

Count what a data team keeps alive by hand. It starts with the per-table load notebooks, where notebook twelve is a copy of notebook eleven with two edits, one of which nobody remembered to make. Then comes the catalog somebody bootstrapped by hand-typing column lists, so six weeks later it describes the database as it was rather than as it is. Alongside it sits a spreadsheet tracking which columns hold personal data, and next to that a wiki glossary that was accurate eighteen months ago.

Data lineage gets the worst deal of all, because most of the time it only gets reconstructed mid-incident. A column changed three pipelines upstream, a report broke, and the afternoon disappears into grepping notebooks to find out why. None of that is data work; it is plumbing. What keeps it unfixable is that every artifact decays at its own rate, so the spreadsheet, the wiki, and the catalog are never in phase with each other. Pointing AI at the problem does not fix that either, for the same reason AI does not fix bad data models: generating the notebook faster is not the same as owning it. The bottleneck was never typing speed. It was that the code carried no record of what the data means.

How does metadata-driven pipeline generation work?

DeltaVault is a data warehouse automation platform: you declare what a table is, and it generates the warehouse code you would otherwise hand-write and hand-maintain. You describe a table once, and the engine renders the platform-correct code for Databricks, Snowflake, or Microsoft Fabric from the metadata already sitting in your catalog: create-table statements, staging notebooks, merge procedures, and whole workflow definitions. The twelfth table then costs you one more declaration, not a copy of notebook eleven with two edits. Metadata goes in, pipelines come out, and when the metadata changes, the pipelines follow.

Pipeline orchestration falls out the same way. Pick a Databricks project, a topology policy, and a parallelism degree, and the ingest and transform workflows are generated with one step per table and dependencies wired from the lineage the catalog already holds. Lineage stops being archaeology for the same reason: it is captured when you declare the mapping rather than parsed out of query logs afterwards.

One metadata layer, so the catalog can’t drift from the code

“All in one place” is the line every vendor uses, us included, and it’s usually thin enough to mean nothing. The claim worth arguing about is narrower: when the catalog and the code disagree, the code wins. In a suite of modules held together by sync jobs, nothing mechanically ties the two together, so data catalog drift sets in, and the catalog loses that argument every time.

The alternative is a single metadata spine that discovery, business modeling, data governance, lineage, and generation all read from and write to. In practice, that means you attach classification and ownership to a business entity once, and the physical columns inherit both through accepted mappings. Naming and modeling standards then run as checks when you commit, and the strict ones block the commit outright. The AI works that same spine rather than sitting beside it (AI-native, not AI-added), so its writes pass the same role checks and land in the same audit trail as a human edit, and every proposal waits for your review before it applies.

The catalog cannot drift from the code, because the code came from it. The cost is real: generation inherits whatever the catalog holds, so the catalog is something you have to get right up front.

What your week looks like instead

Once the plumbing stops taking its cut, what’s left is the set of decisions that were always yours to make. Someone has to decide who stewards Customer, whether date of birth is Restricted, which validation rules block a commit, and which quality tests apply. Whether the AI’s proposed domain structure actually fits the business is your judgment too, made item by item before any of it is approved. When a source changes, you review a handful of Changed rows instead of reconciling duplicates by hand. Before you rename a column, you click it and watch the blast radius light up across the whole pipeline. None of that is the part of the week anyone brags about, and all of it is the part that decides whether the output gets trusted.

So what is the job, if not plumbing?

The industry spent a decade training data engineers to be excellent plumbers, then wondered why nobody trusted the data. But plumbing was never the job; metadata management was. Put meaning before machinery: declare what the data is, who needs it, and what standards it has to meet, and the code turns into a derived artifact, closer to a compiled binary than to something you maintain. So where would you draw the line between the logic that belongs in declared metadata and the logic that will always be worth hand-writing? Odd source systems and one-off reconciliation code look like the obvious holdouts. For everything else, a better wrench does not get you there, and a catalog that builds does:

  1. Connect a source.
  2. Declare one table.
  3. Let the platform write the pipeline.