Orchestration families

Six families sequence and ship the generated loads: the control notebooks on all three platforms, and the five Azure Data Factory shapes.

workflow

Templates: databricks/orchestration/control.jinja, fabric/orchestration/control.jinja, snowflake/orchestration/master.jinja

Roots: workflow

The control notebooks that sequence a workflow’s steps. Steps arrive sorted by solve order (the template regroups by it at render time), and each step carries only the resolution its type needs.

Field	Type	Meaning
workflow.ulid	string	The workflow’s identifier.
workflow.name	string	Workflow name.
workflow.threadCount	number?	Parallelism hint the control notebook reads.
workflow.steps[]	array	Steps sorted ascending by solve order.
workflow.steps[].name	string	Step name.
workflow.steps[].stepType	string	The literal the template branches on: table_load, sub_workflow, custom_sql, or notification.
workflow.steps[].solveOrder	number	Solve-order group; also the sort key.
workflow.steps[].dependsOn	string[]	Upstream step identifiers, for the ordering comment.
workflow.steps[].table	object?	Conditional: for a table_load step: schema, name, and custom attribute values; absent when unresolvable (a preview degrades silently).
workflow.steps[].subWorkflow.name	string?	Conditional: for a sub_workflow step.
workflow.steps[].sqlBody / .sqlBodyJson	string?	Conditional: the custom SQL body verbatim (macros are not expanded on the preview path), and the same as the Python string literal the notebook emits.
workflow.steps[].notificationTarget	string?	Conditional: the notification step’s target.
workflow.steps[].dependencyActivityNames	string[]	One Azure Data Factory activity dependency name per dependsOn entry (read by the Azure Data Factory master, not these notebooks).

adf-orchestration

Templates: adf/orchestration/master.jinja

Roots: pipeline, workflow

The Azure Data Factory master pipeline that sequences a workflow’s steps as activities. The workflow root is the same step model as the workflow family above; the pipeline root adds the master pipeline’s own name. Steps carry precomputed dependencyActivityNames, one per upstream dependency.

Field	Type	Meaning
pipeline.name	string	The master pipeline’s name.
workflow.name	string	The workflow being orchestrated.
workflow.steps[]	array	Steps with ulid, name, stepType, dependsOn (upstream step identifiers), dependencyActivityNames, and the per-type members (table, subWorkflow, notificationTarget).

adf-pipeline

Templates: adf/staging/copy_to_staging.jinja, adf/staging/copy_to_staging_with_lookup.jinja, adf/staging/delete_detection.jinja

Roots: pipeline, table, connection

The Azure Data Factory staging copy pipelines. Values that land inside emitted JSON travel in two forms: the raw value and a serialized literal (the Literal suffix), so the template emits the finished token without re-quoting. High-water-mark lookups on this path are self-targeted: the Lookup reads the maximum from the source table itself, because a direct copy has no landing hop.

Field	Type	Meaning
pipeline.name / .nameSafe	string	The pipeline’s raw and sanitized names.
pipeline.sourceDataset / .sourceDatasetSafe	string	The Copy activity’s source dataset, raw and sanitized.
pipeline.sinkDataset / .sinkDatasetSafe	string	The Copy activity’s sink dataset, raw and sanitized.
pipeline.sinkLinkedService	string?	Sink linked service, read only inside the post-copy branch.
pipeline.timeout / .retryCount / .retryInterval	string / number	Activity policy; defaults 7.00:00:00, 0, 30.
pipeline.sourceQuery / .sourceQueryLiteral	string / string?	The source query. With composable extraction filters it is SELECT * FROM schema.name WHERE the composed predicate; empty otherwise. The literal is the JSON-serialized form.
pipeline.deltaColumn	string	The high-water-mark column: a filter’s target column outranks the change-type scan; final fallback ModifiedDate.
pipeline.lookupQuery / .lookupQueryLiteral	string / string?	The composed high-water-mark query, SELECT MAX(...) AS LastLoadDate over the source table, raw and serialized.
pipeline.writeBehavior	string	Sink write behavior; default insert.
pipeline.folder	string	The folder the pipeline lands in; default Extract.
pipeline.loadType	string?	The project’s load mode (delta selects the Lookup variant); carried as a visible input.
pipeline.postCopyStoredProcedure / Literal, .deleteQuery / Literal	string?	Branch-only values, carried raw: absent stays absent so the branch does not flip.
pipeline.keyColumns / .keyColumnLiterals	string[]	Upsert key columns, raw and serialized per entry, in order.
table.name / .nameSafe / .schema	string	The source table being copied; the schema travels into composed queries unsanitized.
connection.type / .sinkType	string	The source and sink system connection types, raw.
connection.copySourceType / .copySinkType	string	The Copy activity’s type literals for each.
connection.lookupSourceType	string	The Lookup activity’s source literal: the sink type through the source map (the Lookup reads from the sink dataset, and a source type is still required there).

adf-dataset

Templates: adf/_platform/datasets/*.jinja

Roots: dataset

The dataset definitions (tabular and file). The builder is keyed by template path for one purpose: the schema default (dbo for the SQL datasets, PUBLIC for Snowflake). Branch-only members stay raw: an absent value must stay absent, or the emitted JSON gains a property it never had.

Field	Type	Meaning
dataset.name / .nameSafe	string	The dataset’s raw and sanitized names.
dataset.linkedService / .linkedServiceSafe	string	The linked service the dataset binds to; the sanitized form is the emitted reference.
dataset.columns[]	array	Schema-block columns: name, dataType, and their serialized literals.
dataset.schema	string?	Tabular datasets: resolved against the target template’s default; carried as given elsewhere.
dataset.table	string?	Tabular datasets, read with no default.
dataset.locationType / .container / .folderPath / .fileName	string	File datasets: location type (default AzureBlobFSLocation), container, and the two branch-only path members carried raw.
dataset.columnDelimiter / .rowDelimiter / .quoteChar / .escapeChar (+ Literals)	string	Delimited text facets with their defaults (comma, newline, double quote, backslash) and serialized forms.
dataset.firstRowAsHeader	boolean	Delimited text; defaults to true, and an explicit false survives (defaulting substitutes for strictly undefined only).
dataset.compressionCodec	string	Parquet; default snappy.

adf-linked-service

Templates: adf/_platform/linked_services/*.jinja

Roots: connection

The linked-service definitions. This is the wide connector shape: these templates read endpoint, principal, and cluster members that the narrow connection identity does not carry. The family previews live from a connection you pick, through a projection built from an explicit allowlist of connection columns rather than from that narrow identity, so a column added to the table later is excluded until it is allowlisted. Any root the allowlist cannot fill stays unset and carries its own finding, so a blank field is visibly blank rather than silently so. Secrets never appear: password is a Key Vault secret name the service resolves at run time, never a value.

Field	Type	Meaning
connection.name / .nameSafe	string	The linked service’s raw and sanitized names (the artifact path uses the raw one).
connection.connectionString	string	Connection string; default empty.
connection.integrationRuntime	string	Integration runtime; default AutoResolveIntegrationRuntime.
connection.url / .serviceEndpoint / .baseUrl / .domain	string	Endpoint members; default empty.
connection.servicePrincipalId / .servicePrincipalKey / .tenant	string	Service principal members; default empty.
connection.accountKind	string	Storage account kind; default StorageV2.
connection.keyVaultLinkedService	string	The Key Vault linked service secrets resolve through; default AzureKeyVault.
connection.enableServerCertificateValidation	boolean	Default true; an explicit false survives.
connection.accountIdentifier / .database / .warehouse	string	Snowflake members; default empty.
connection.authenticationType	string?	Resolved per target template (Anonymous for the HTTP and REST services, Basic for Snowflake); carried as given elsewhere.
connection.newClusterNumOfWorkers	string	Databricks new-cluster workers; default 1.
connection.accountKey / .accessToken / .newClusterNodeType / .user / .password	string?	Branch-only members carried raw: absent stays absent.
connection.newClusterSparkConf / Literal	object? / string?	The Spark configuration blob and its structural serialization (the emitted JSON is exactly that quoting).

adf-trigger

Templates: adf/_platform/triggers/*.jinja

Roots: trigger

The trigger definitions (schedule, event, tumbling window). The trigger root carries the trigger’s name, its schedule or event facets, and the pipeline references it starts; the family previews from its curated sample.