Orchestration workflows

DeltaVault generates the orchestration for your project as workflows: one workflow per layer, plus a master workflow that chains them into a single run. Generating, customizing, and regenerating workflows all happen as rows in the app; nothing runs against Databricks at this point. The build turns those rows into deployable job definitions in your bundle, described in Build and deploy your project, and even then nothing runs until you commit the bundle and the generated continuous integration workflow deploys it.

Per-layer workflows

Generate creates one workflow for every layer that has tables to run, and skips any layer that is empty:

Ingest loads your landing tables.
Transform runs your staging and persistent staging tables together, in one workflow.
Data Vault runs in three stages: your reference tables load first, then one step per staging source’s control notebook, then the point-in-time views, bridge views, and business vault objects that depend on them.
Data Mart runs your dimensions before the facts that reference them, so a fact always finds its dimensions’ surrogate keys already loaded.

The master workflow

The master workflow chains your layer workflows into a single run, in canonical order: ingest, transform, Data Vault, then data mart. Each step in the master workflow simply runs one layer workflow to completion before the next one starts. A layer with no tables in scope has no workflow of its own, so the master workflow skips straight past it: a project with no data mart tables ends its chain at Data Vault.

Customize steps in the app

Open a workflow from Operations, Workflows and it opens on its Canvas, a diagram of the steps and what each one waits for. The List tab shows the same steps as a table, which is the quicker way to find one in a large workflow, and Code shows the generated definition.

Selecting a step, on the canvas or in the list, opens it in the panel docked to the side of the page. That panel is where every change is made, and each one ends with Save.

Rename a step. Select it, edit Name, and select Save.
Reorder steps. Change Solve Order to move a step earlier or later, or use the Depends on checkboxes to name exactly which sibling steps must finish first.
Change a step’s thread count. Set Thread Count (override) to give that one step its own concurrency, independent of the workflow’s overall setting.
Insert a custom step. Select Add task, then change its Type from a table load to a custom SQL statement, a call to another workflow, or a notification, and fill in the fields that appear.
Delete a step. Right click its card on the canvas, choose Delete task, and confirm when prompted.

Regenerate safely

Regenerating a project’s workflows never throws away work you have already customized. Select Generate from project on the Workflows page, choose your project and topology, and select Preview before anything changes.

Edited steps are never overwritten. A step you renamed, reordered, or otherwise changed shows as preserved in the preview, alongside what the regenerate would otherwise have changed, so you can see exactly what your edit is protecting.
Deleted steps stay deleted. If the plan would still generate a step you deleted, the preview marks it stays deleted; applying leaves it gone rather than bringing it back.
Steps whose source disappeared are kept, not removed. If you edited a step and its underlying table later left the project, the preview marks it source missing; the step stays exactly as you left it.

The preview summarizes every one of these changes before you select Apply, so nothing in your workflows changes that you have not already reviewed.

Troubleshooting

Adding or renaming a step fails, naming a deleted step. A step’s name stays reserved after you delete it, so DeltaVault will not let a new or renamed step reuse it. Choose a different name and save again.
A staging step’s code generation fails, naming its landing and staging family. DeltaVault could not resolve the staging table’s related landing table. Create the landing sibling for that staging table first, then generate the step’s code again.