API: Pipelines

Pipeline definitions, runs, backfill, schedules, CDC streams, and flowlets. All endpoints require authentication unless noted. 35 endpoints.

`GET /api/pipelines`

List pipelines (with latest-run summary)

Responses: 200 · 400 · 401

`POST /api/pipelines`

Create a pipeline (DAG validated via the shared contract)

Request body (JSON)

Field	Type	Required
`name`	string	yes
`dag`	object	yes
`schedule`	object	no
`concurrency`	object	no
`status`	`"active"` \| `"paused"`	no

Responses: 201 · 400 · 401 · 409 · 422

`GET /api/pipelines/{id}`

Get a pipeline

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`PATCH /api/pipelines/{id}`

Update a pipeline (optimistic concurrency via expectedUpdatedAt)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required
`name`	string	no
`dag`	object	no
`schedule`	object	no
`concurrency`	object	no
`compute`	object	no
`status`	`"active"` \| `"paused"`	no
`expectedUpdatedAt`	string	no

Responses: 200 · 400 · 401 · 404 · 409 · 422

`DELETE /api/pipelines/{id}`

Delete a pipeline (soft; refused while a run is queued/running)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 204 — No content · 401 · 404 · 409

`POST /api/pipelines/{id}/backfill`

Backfill a window: one run per date (inclusive, max 366 days)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`from`	string	yes
`to`	string	yes

Responses: 201 · 401 · 404 · 409 · 422 · 429 · 503

`GET /api/pipelines/{id}/lineage`

Column-level lineage of this pipeline (static + last-run observed)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/{id}/quality`

A pipeline’s data-quality gate history (newest first)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Query parameters

Name	Type	Required	Description
`limit`	integer	no

Responses: 200 · 400 · 401 · 404

`POST /api/pipelines/{id}/run`

Trigger a pipeline run (worker handshake enforced)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 201 · 400 · 401 · 409 · 429 · 503

`GET /api/pipelines/{id}/runs`

List a pipeline’s runs (newest first, paged)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Query parameters

Name	Type	Required	Description
`limit`	integer	no	Maximum number of items to return
`offset`	integer	no	Number of items to skip

Responses: 200 · 400 · 401

`PUT /api/pipelines/{id}/schedule`

Set or clear the pipeline schedule (cron or every-N-seconds; fires via the schedule poller)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required
`cron`	string	no
`everySeconds`	integer	no
`timezone`	string	no
`enabled`	boolean	no

Responses: 200 · 400 · 401 · 404 · 422

`POST /api/pipelines/{id}/suggest-ops`

Describe a transform in plain language; get a validated visual-recipe op chain back

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required
`description`	string	yes
`upstreamIds`	array of string	yes
`inputs`	array of object	no
`currentOps`	array of object	no

Responses: 200 · 400 · 401 · 404 · 422 · 503

`POST /api/pipelines/{id}/transform-preview`

Preview a transform step: its compiled SQL, and rows from the last run’s data

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required
`stepId`	string	yes
`dag`	object	no
`limit`	integer	no

Responses: 200 · 400 · 401 · 404 · 422

`POST /api/pipelines/{id}/validate`

Validate a DAG (the given one, or the stored one) without saving

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`dag`	object	no

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/{id}/webhook`

The webhook trigger URL for this pipeline

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404 · 503

`POST /api/pipelines/{id}/webhook`

Trigger a pipeline from an external system (token-guarded)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Query parameters

Name	Type	Required	Description
`token`	string	no

Responses: 201 · 400 · 401 · 404 · 409 · 429 · 503

`GET /api/pipelines/cdc-streams`

List CDC streams

Responses: 200 · 400 · 401

`POST /api/pipelines/cdc-streams`

Create a CDC stream (postgres-family sources, v1)

Request body (JSON)

Field	Type	Required
`sourceId`	string	yes
`streamId`	string	yes
`tables`	array of string	yes

Responses: 201

`PATCH /api/pipelines/cdc-streams/{id}`

Enable or disable a CDC stream

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`enabled`	boolean	yes

Responses: 200

`DELETE /api/pipelines/cdc-streams/{id}`

Delete a CDC stream (its capture container is removed on the next reconcile)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 204 — No content · 401 · 404

`GET /api/pipelines/flowlets`

The flowlet library (latest version of each)

Responses: 200 · 400 · 401 · 404

`POST /api/pipelines/flowlets`

Save a flowlet — a new version if the name already exists

Responses: 201 · 400 · 401 · 409 · 422

`GET /api/pipelines/flowlets/{id}`

One flowlet version by id

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`DELETE /api/pipelines/flowlets/{id}/all`

Retire a flowlet (all versions) from the library

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 204 · 404

`POST /api/pipelines/flowlets/{id}/instantiate`

Instantiate a flowlet version with arguments → concrete ops

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`params`	object	no

Responses: 200 · 400 · 401 · 404 · 422

`GET /api/pipelines/flowlets/{id}/versions`

Version history for the flowlet’s name

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/quality/summary`

Per-pipeline data-quality gate health for the workspace

Query parameters

Name	Type	Required	Description
`days`	integer	no

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/runs/{runId}`

Get one run with its step runs

Path parameters

Name	Type	Required	Description
`runId`	string	yes

Responses: 200 · 400 · 401 · 404

`POST /api/pipelines/runs/{runId}/cancel`

Cancel a queued/running run (cooperative via the orchestration plane)

Path parameters

Name	Type	Required	Description
`runId`	string	yes

Responses: 200 · 400 · 401 · 404 · 409

`GET /api/pipelines/runs/{runId}/events`

Live run/step events (SSE; ends when the run reaches a terminal status)

Path parameters

Name	Type	Required	Description
`runId`	string	yes

Responses: 200

`POST /api/pipelines/runs/{runId}/retry`

Retry a failed run (re-queues the same run row; worker owns attempt semantics)

Path parameters

Name	Type	Required	Description
`runId`	string	yes

Responses: 200 · 400 · 401 · 404 · 409 · 503

`GET /api/pipelines/runs/{runId}/steps/{stepId}/logs`

The log tail a step attempt produced

Path parameters

Name	Type	Required	Description
`runId`	string	yes
`stepId`	string	yes

Query parameters

Name	Type	Required	Description
`attempt`	integer	no

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/usage`

Current-month pipeline usage per workspace (admin)

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/usage/breakdown`

Per-pipeline and per-sink usage for a month (admin)

Query parameters

Name	Type	Required	Description
`month`	string	no

Responses: 200 · 400 · 401 · 403 · 404

`GET /api/pipelines/workers/status`

Execution-plane availability (Run-now preflight)

Responses: 200 · 400 · 401 · 404