Change data capture with AUTO CDC

Databricks Lakeflow applies change data capture declaratively: you describe the target table and a flow that keeps it up to date, and the platform works out the inserts, updates, and deletes. DeltaVault writes these flows for you from your table metadata, so the streaming tables in your data mart stay correct without hand-written merge logic.

This applies to the data mart. Data Vault objects keep their history a different way, with generated load notebooks rather than declarative flows; see Data Vault on Lakeflow. The derivation rules below are shared, so what DeltaVault works out from your metadata is the same either way. Only the statement it feeds differs: an AUTO CDC INTO flow for a data mart dimension, a MERGE in a load notebook for a Data Vault object.

What an AUTO CDC flow looks like

For each data mart dimension that tracks changes, DeltaVault emits a flow in the current Lakeflow syntax:

CREATE FLOW orders_cdc AS AUTO CDC INTO orders
FROM STREAM(orders_clean)
KEYS (order_id)
SEQUENCE BY _commit_timestamp
STORED AS SCD TYPE 2;

This AUTO CDC INTO form is the current Lakeflow change-data-capture syntax. It replaces the older APPLY CHANGES INTO form, so the generated code stays aligned with current Databricks guidance.

What DeltaVault derives

DeltaVault fills in each part of the flow from the metadata you have already captured:

• Keys identify a row across changes. DeltaVault uses the business key columns you have marked on the table; if there are none, it falls back to the primary key columns.
• Sequence by orders the changes so the latest version wins. DeltaVault uses the sequencing column you set, and defaults to the commit timestamp when you have not chosen one.
• Delete handling decides what an incoming delete does. DeltaVault reads your table’s delete handling, or, for file-based change feeds, the operation column and the value that means delete.
• History type decides whether the table keeps history. A type 1 target overwrites in place; a type 2 target keeps the full history of each row with validity windows.

Derived versus overridden

DeltaVault resolves every parameter with a clear order of precedence, so you can let the catalog drive the result and step in only where you need to:

1. An explicit override on the object always wins.
2. Otherwise, the object’s own configuration is used: the transformation settings for a medallion table, or the Data Vault configuration for a vault object.
3. Otherwise, the column flags are used: business-key columns first, then primary-key columns.
4. If none of these yield a key, DeltaVault reports a configuration error rather than guessing, so a missing key is caught before deployment.

Tombstone and delete handling

How deletes are handled depends on your ingestion pattern:

• For database and streaming change feeds, your table’s delete handling controls whether a delete removes the row, marks it as deleted, or is ignored.
• For file-based change data capture, the change feed carries an operation column. DeltaVault reads the operation column and the value that means delete, and writes an APPLY AS DELETE WHEN clause so deletes in the feed are applied to the target.

Related guides

• Generate Databricks Lakeflow pipelines
• Data Vault on Lakeflow