We built AI into every corner of DeltaVault, and we still recorded it losing to a data engineer. It did not lose on a hard problem. It lost on a single view over two staging tables, the kind of thing you build before lunch.

Every AI demo rests on the same premise, that the engineer is optional. You type what you want, and the panel does the rest. We spent months building that panel. This post is about the day we watched it lose, because that recording is the best argument for your job we have.

Will AI replace data engineers? Watch it try to build one view

The task was ordinary. A Data Vault satellite needs to carry a customer’s delivery address, and the source for that satellite is a view over the persistent staging layer: the customer address table joined to the address table, full history, so that every version of the address lands on its own row.

Ask AI opened well. It took the request “create a full history”, found the staging tables, listed the customer and address tables, and asked what came next. Then the conversation started. The business key is defined on the staging table rather than on the view, so we asked it to pick the key up from there, and it could not see it. We pointed at the table, and it still saw nothing. We told it the table was on the canvas. Let me check, it said, and then: found it.

You have had this conversation. Everyone doing AI and data engineering together has had it, and every turn of it spends credits. To be fair, what came back at the end was decent:

So we accepted, added a couple of columns, confirmed the proposed joins, and read the SQL.

That is where it fell apart. One join was wrong, and one join should never have been there at all. The customer address and the address were joined, which we wanted, but as a plain join rather than a temporal overlap, so the history the view exists to keep would have been flattened. A third table had also worked its way into the query: an inner join to the staging customer table, which crosses two staging layers and filters rows out of a full-history view that must not lose any. A data engineer spots both in a second.

Note: We have run this request many ways over many sessions, and each run returns a different answer, with the joins wrong in a different place each time.

Watch this in action:

How DeltaVault lets a data engineer build the same view by hand

Here is the other take, with no AI in it.

Start in business modeling, where the intent lives. The model already says a customer is the customer, a home address, and a delivery address, so the entity the view serves is settled before anyone touches a table.

From there, move to the workspace canvas. The persistent staging table already has an end-dated view generated for it, and that is the timeline a full history needs. Right-click the table and choose Create View. Name it for the delivery address entity, set the layer to persistent staging, and pull in the address table. The designer proposes the join and marks it as a temporal overlap, because that is how two end-dated tables join and the tool knows it. Add the customer identifier and the columns the satellite needs, then rename the projected columns so the generated code reads well.

Now open the preview. The full CREATE VIEW statement is on screen, left join and all, carrying the overlap condition the designer promised. It is the view we wanted, so save it. On the saved view, we then designated the business key derived from the customer identifier, set the record source, and added the relationship to the customer table so the hub has something to key on.

That took less time than the conversation did, and nothing had to be undone.

Why the human was faster

The AI answered the question it was asked. The data engineer knew which question to ask, and brought three things to it that nobody had written into the request:

  • Two staging layers must not be inner-joined in a history view.
  • Two end-dated tables join on overlap.
  • This view serves the delivery address entity and will feed a satellite hanging off the customer hub.

None of that is prompt engineering. It is data engineering, and it is the job.

Without that mindset behind it, an AI panel never converges on the right answer. It converges on a different plausible answer each time, and you pay for every one of them. Put a data engineer in front of the same panel, someone who knows the model, knows the layers, and knows when not to press the button, and the panel turns into a real accelerator. That is why human in the loop AI is the only kind we build.

Ask AI earns its keep elsewhere every day:

  • discovering business entities from an interview
  • matching source tables and columns to the business model
  • drafting the descriptions nobody wants to write

That is where AI drafts the tedious part and you keep the decision. When the work is judgment over a model you already hold in your head, the right-click is faster.

We need data engineers now more than ever

The industry story is that AI makes the engineer optional. Our recording says the opposite. The more AI a platform carries, the more it needs someone who can tell a wrong join from a right one before it ships, and who can judge in a second that this one is faster by hand.

So keep doing what you are doing.

  1. Open the canvas, right-click, and build the view.
  2. Ask the AI when it can see something you cannot.

If you are the data engineer who knows what they are doing, the panel has not made you replaceable. It has made you worth more than you were before it arrived.