Landing families
Five families write the landing layer. Three of them are the per-table shapes a
landing table routes to, and this page documents those. The other two are
ingest-workflow (databricks/landing/ingest_workflow.jinja, roots
workflow and dv), which renders at workflow grain rather than per table, and
ingest-piece (databricks/landing/pieces/cloud_files_table.jinja and
jdbc_table.jinja), the include-only per-table piece those workflow-grain
bodies compose. Both are indexed on the
overview page
and their field-by-field sections are still to be written.
A landing table renders through one of three shapes, decided in a fixed order. First the ingest runtime is routed from the table’s ingestion settings, the source connector type, and whether the source connection carries Lakeflow Connect settings. Then the file change-data-capture mode is rejected outright (the routed runtime would be change-blind for it). What remains routes: the Lakeflow Connect runtime renders through the run-scoped pipeline resource (never a per-table file), the declarative pipeline runtime takes the ingest-pipeline shape, and every other runtime takes the ingest-notebook shape with the runtime naming which notebook.
ingest-pipeline
Section titled “ingest-pipeline”Templates: databricks/landing/ingest_pipeline.jinja
Roots: workflow, this, connection (conditional), source
(conditional), sources, ingest, dv
The declarative landing pipeline: one pipeline per source connection, replicating source tables verbatim into the landing (bronze) catalog. Because a landing table has no audit columns of its own, there is no methodology root here.
Root: workflow
Section titled “Root: workflow”| Field | Type | Meaning |
|---|---|---|
workflow.name | string | landing_ plus the source connection’s raw name; the literal landing_unlinked in a preview with no resolved source connection (a Build throws first). |
Root: this
Section titled “Root: this”The landing table being written: a table reference with the resolved ingestion block and a Python-safe function name on top.
| Field | Type | Meaning |
|---|---|---|
this.name | string | Landing table name, raw. |
this.fqn | string | The stored fully qualified name, else composed from the landing catalog, schema, and name. |
this.columns[] | array | The landing table’s own columns, ordinal-sorted, system columns included. Elements carry ulid (when present), name, dataType, isPrimaryKey, isNullable. |
this.parts.catalog | string | Destination catalog assigned to the landing layer (sample_catalog in an unpinned preview). |
this.parts.schema | string | Destination schema (sample_schema in an unpinned preview). |
this.parts.name | string | Destination table name. |
this.ingestion.ingestMode | string | Resolved ingest mode; defaults to snapshot. |
this.ingestion.watermarkColumn | string | The landing column the high-water bound is read from; empty when none. A resolvable high-water-mark filter’s target column outranks the stored setting. |
this.ingestion.sourceClause | string | The restriction appended unconditionally to the source read, complete with its leading WHERE; empty when nothing restricts. Python-string escaped. Exactly one of sourceClause and watermarkClause is ever populated. |
this.ingestion.watermarkClause | string | The restriction that applies only once a previous run has landed rows, complete with its leading WHERE, carrying the {lower_bound} placeholder, brace-escaped for the formatted string it lands in. |
this.ingestion.cdcKeyColumns | string[] | Change-data-capture key columns; empty by default. |
this.ingestion.cdcSequenceColumn | string | Change-data-capture sequence column; defaults to __$start_lsn. |
this.ingestion.cdcTombstoneHandling | string | Tombstone policy; defaults to delete_row. |
this.pythonFunctionName | string | The table name as a Python identifier: sanitized, with every hyphen replaced by an underscore. |
Root: connection (conditional)
Section titled “Root: connection (conditional)”Present only when the source connection resolved. Always the source connector, never the destination.
| Field | Type | Meaning |
|---|---|---|
connection.name | string | Source connection’s raw name. |
connection.connectionType | string | Source connector type (for example SQL Server). |
Roots: source and sources
Section titled “Roots: source and sources”source is present only when the source connection, source table, and source
schema all resolved; it is omitted rather than emitted empty, so a template’s
{% if source %} guard takes the branch a real Build would. sources is
always present: a one-element list holding the identical object, or empty.
Beyond the
table reference
fields, source.schema carries the source schema name; source.parts is
deliberately absent (a source-system table has no honest catalog).
ingest-notebook
Section titled “ingest-notebook”Templates: databricks/landing/auto_loader.jinja,
databricks/landing/pyspark_jdbc.jinja, databricks/landing/pyspark_rest.jinja
Roots: this, connection, source (conditional), sources,
environment, extract, dv
The per-table landing notebooks. Unlike the pipeline shape there is no workflow root, the connection root is always present (empty strings when unresolved), and the ingestion block is passed raw rather than resolved: the notebooks read change-data-capture settings with no defaults, and resolving them would change rendered bytes.
Root: this
Section titled “Root: this”| Field | Type | Meaning |
|---|---|---|
this.ulid | string | Landing table identifier (supplied on this shape, unlike the pipeline shape). |
this.name | string | Landing table name. |
this.fqn | string | The stored fully qualified name, else composed from catalog, schema, and name. |
this.columns[] | array | The landing table’s own columns, ordinal-sorted, system columns included. |
this.parts.catalog / .schema / .name | string | Destination (bronze) catalog, schema, and table name. |
this.ingestion | object | The raw stored ingestion block, empty object when absent. Deliberately unresolved: fields may be missing and templates read them with no defaults. |
this.ingestion.ingestMode | string? | Ingest mode, raw. |
this.ingestion.watermarkColumn | string? | Watermark column, raw. |
this.ingestion.runtimeOverride | string? | The stored runtime override that routed this table to a notebook. |
this.ingestion.cdcKeyColumns | string[]? | Raw change-data-capture key columns. |
this.ingestion.cdcSequenceColumn | string? | Raw sequence column. |
this.ingestion.cdcTombstoneHandling | string? | Raw tombstone policy. |
Root: extract
Section titled “Root: extract”Always present: what the notebook actually extracts from, resolved ahead of render. When a source table is linked, its schema and name; when none is, the landing table’s own (and the builder reports an unlinked-source-self-reference diagnostic so the fallback stays visible). A linked source wins even when its fields are empty strings. Both the raw and the snake-cased forms travel: raw feeds the notebook’s headers and the JDBC table option, snake-cased feeds the physical bronze table name and the quarantine path.
| Field | Type | Meaning |
|---|---|---|
extract.schema | string | Source schema when linked; the landing table’s own schema when not. |
extract.name | string | Source table name when linked; the landing table’s own name when not. |
extract.schemaSnakeCase | string | The schema through the snake-case naming model. |
extract.nameSnakeCase | string | The name through the snake-case naming model. |
Roots: source, sources, connection, environment
Section titled “Roots: source, sources, connection, environment”| Field | Type | Meaning |
|---|---|---|
source | table ref? | Conditional: present only when a source table resolved (no connection or schema check on this shape). Identical object to sources[0]. Carries the extra source.schema; parts is absent by design. |
sources | array | Always present: one element or empty. |
connection.name | string | Source connection name; empty string when unresolved (always present on this shape). |
connection.connectionType | string | Source connector type; empty string when unresolved. |
environment.catalogs.bronze | string | The landing (bronze) catalog the notebook writes into. |
environment.schemaPolicy | string | Target connection’s schema policy; defaults to permissive. |
environment.landingRoot | string | Target connection’s landing storage root; empty when unset. |
lakeflow-entry
Section titled “lakeflow-entry”Templates: databricks/landing/lakeflow_ingest_object.jinja
Roots: this, source, sources, connection
One Lakeflow Connect ingestion entry. All four roots are unconditional: a table
with no source link refuses to render rather than emitting a partial context.
Entries carry no column data by design, so this.columns and source.columns
are always empty; the entry names tables, it does not enumerate them.
| Field | Type | Meaning |
|---|---|---|
this.name | string | Destination table name. |
this.fqn | string | Destination catalog.schema.table. |
this.columns | array | Always empty by design. |
this.parts.catalog / .schema / .name | string | The destination split into parts. |
this.key | string | The pipeline resource key: lfc_ plus the snake-cased connection, source schema, and source table. |
this.ingestion.ingestMode | string | Resolved ingest mode; snapshot default. |
this.ingestion.watermarkColumn | string | The source column a high-water-mark filter restricts (name only, never composed SQL), else the stored setting, else empty. The filter outranks the stored setting. |
source.name | string | Source table name. |
source.fqn | string | Source schema.table. |
source.schema | string | The source schema name (family extra; parts is absent by design). |
sources | array | One element, identical object to source. |
connection.name | string | Source connection name. |
connection.connectionType | string | Source connector type as DeltaVault stores it (for example Azure SQL Database). |
connection.connector | string | Precomputed Lakeflow Connect connector key (sqlserver, oracle, postgres, mysql, salesforce, workday, servicenow, netsuite, googleanalytics); an unmapped type lowercases through. |
The bundle’s bundle-lakeflow family loops enriched entries carrying these same four roots.