Pipelines as code

A pipeline can live in your Git repository as a file and move between environments — development, staging, production — as a reviewed change.

The file

Export (pipeline editor → Export, or the API below) writes the pipeline as JSON that is:

• Deterministic — the same pipeline always exports byte-identical, so Git shows exactly what changed.
• Free of ids, data and credentials — every connection the pipeline uses (the source it extracts from, a database or store it loads into, a watched folder) is written as $source:0, $source:1…, and listed by name under references. A pipeline it is chained to becomes $pipeline:0. It is safe to commit.

Promoting it

Importing a pipeline file is a promotion: it is matched by name in the workspace you import into.

1. Check (a dry run) returns the plan: create (no pipeline has that name yet), update — with every change listed, field by field (steps."extract".config.table: public.orders → public.orders_v2) — or already up to date.

2. Connections are mapped by name: when the target has one connection with the same name (Shop DB in development and Shop DB in production), it is used; otherwise you pick it.

3. Promote applies the plan with the same checks as saving in the editor (the step definitions, the trigger, a schedule that can run). The target’s run history, cursors and datasets stay where they are.

In the app: Pipelines → Import, choose the file, Check, then Promote (or Create).

Note

Importing needs permission to create pipelines, or to update them when the plan is an update; a Check needs only view. Pipelines must be enabled for the workspace.

From Git and CI

Use an API key with the pipeline:read scope to export and plan, and pipeline:write to apply. A typical flow:

• Export from development and commit the file:

  curl -s -H "X-API-Key: $DS_DEV_KEY" \
    https://<your-host>/api/alm/export/pipeline/<pipeline-id> > pipelines/orders-nightly.json

• Plan on a pull request — fail the check when the plan has issues:

  jq -n --slurpfile a pipelines/orders-nightly.json '{artifact: $a[0], dryRun: true}' \
    | curl -s -X POST -H "X-API-Key: $DS_PROD_KEY" -H "content-type: application/json" \
        --data @- https://<your-host>/api/alm/import \
    | tee plan.json | jq '.plan'
  jq -e '(.plan.issues | length == 0) and (.requiredMappings | length == 0)' plan.json

• Apply on merge — the same call without dryRun.

When a connection’s name differs between environments, pass "mappings": {"$source:0": "<connection id>"} with the request.

Related

• Building a pipeline
• Runs, backfill & monitoring