REST API

API: Pipelines

Pipeline definitions, runs, backfill, schedules, CDC streams, and flowlets. All endpoints require authentication unless noted. 47 endpoints, 1 of them app only: an API key gets 403 there.

`GET /api/pipelines`

List pipelines (with latest-run summary)

Responses: 200 · 400 · 401

`POST /api/pipelines`

Create a pipeline (DAG validated via the shared contract)

Request body (JSON)

Field	Type	Required
`name`	string	yes
`dag`	object	yes
`schedule`	object \| null	no
`concurrency`	object	no
`status`	`"active"` \| `"paused"`	no

Schema: PipelineDag (the dag field of the request body)

Responses: 201 · 400 · 401 · 409 · 422

`GET /api/pipelines/{id}`

Get a pipeline

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`PATCH /api/pipelines/{id}`

Update a pipeline (optimistic concurrency via expectedUpdatedAt)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required
`name`	string	no
`dag`	object	no
`schedule`	object \| null	no
`concurrency`	object	no
`compute`	object	no
`status`	`"active"` \| `"paused"`	no
`expectedUpdatedAt`	string	no

Schema: PipelineDag (the dag field of the request body)

Responses: 200 · 400 · 401 · 404 · 409 · 422

`DELETE /api/pipelines/{id}`

Delete a pipeline (soft; refused while a run is queued/running)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 204 — No content · 401 · 404 · 409

`GET /api/pipelines/{id}/advice`

Cost estimate and backend / performance advice for this pipeline

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`POST /api/pipelines/{id}/backfill`

Backfill a window: one run per date (inclusive, max 366 days)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`from`	string	yes
`to`	string	yes

Responses: 201 · 401 · 404 · 409 · 422 · 429 · 503

`GET /api/pipelines/{id}/impact-notices`

Whether this pipeline’s failed runs tell the owners of the dashboards it feeds

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`PUT /api/pipelines/{id}/impact-notices`

Turn on or off the impact notices of this pipeline’s failed runs (off by default)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`enabled`	boolean	yes

Responses: 200 · 400 · 401 · 404 · 409

`GET /api/pipelines/{id}/lineage`

Column-level lineage of this pipeline (static + last-run observed)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/{id}/quality`

A pipeline’s data-quality gate history (newest first)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Query parameters

Name	Type	Required	Description
`limit`	integer	no

Responses: 200 · 400 · 401 · 404

`POST /api/pipelines/{id}/run`

Trigger a pipeline run (worker handshake enforced; optional parameters)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`parameters`	object	no

Responses: 201 · 400 · 401 · 409 · 422 · 429 · 503

`GET /api/pipelines/{id}/runs`

List a pipeline’s runs (newest first, paged)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Query parameters

Name	Type	Required	Description
`limit`	integer	no	Maximum number of items to return
`offset`	integer	no	Number of items to skip

Responses: 200 · 400 · 401

`PUT /api/pipelines/{id}/schedule`

Set or clear the pipeline schedule (cron or every-N-seconds; fires via the schedule poller)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required
`cron`	string	no
`everySeconds`	integer	no
`timezone`	string	no
`enabled`	boolean	no

Responses: 200 · 400 · 401 · 404 · 422

`POST /api/pipelines/{id}/schedule/resume`

Resume a schedule the poller paused after repeated permanent refusals

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404 · 409 · 422

`POST /api/pipelines/{id}/suggest-ops`

Describe a transform in plain language; get a validated visual-recipe op chain back

App only. An API key gets 403: AI features are available in the DataSquares app; API keys can’t use them.

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required
`description`	string	yes
`upstreamIds`	array of string	yes
`fromPorts`	object	no
`currentOps`	array of object	no

Responses: 200 · 400 · 401 · 403 · 404 · 422 · 429 · 503

`POST /api/pipelines/{id}/transform-preview`

Preview a transform step: its compiled SQL, and rows from the last run’s data

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required
`stepId`	string	yes
`dag`	object	no
`limit`	integer	no

Responses: 200 · 400 · 401 · 404 · 422

`POST /api/pipelines/{id}/validate`

Validate a DAG (the given one, or the stored one) without saving

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`dag`	object	no

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/{id}/webhook`

The webhook trigger URL for this pipeline

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404 · 503

`POST /api/pipelines/{id}/webhook`

Trigger a pipeline from an external system (token-guarded; optional parameters)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Query parameters

Name	Type	Required	Description
`token`	string	no

Request body (JSON)

Field	Type	Required	Description
`parameters`	object	no

Responses: 201 · 400 · 401 · 404 · 409 · 422 · 429 · 503

`GET /api/pipelines/cdc-streams`

List CDC streams

Responses: 200 · 400 · 401

`POST /api/pipelines/cdc-streams`

Create a CDC stream (PostgreSQL- and MySQL-family sources)

Request body (JSON)

Field	Type	Required
`sourceId`	string	yes
`streamId`	string	yes
`tables`	array of string	yes

Responses: 201

`PATCH /api/pipelines/cdc-streams/{id}`

Enable or disable a CDC stream

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`enabled`	boolean	yes

Responses: 200

`DELETE /api/pipelines/cdc-streams/{id}`

Delete a CDC stream (its capture container is removed on the next reconcile)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 204 — No content · 401 · 404

`POST /api/pipelines/cdc-streams/preflight`

Check a source’s readiness for change capture

Request body (JSON)

Field	Type	Required	Description
`sourceId`	string	yes
`tables`	array of string	no

Responses: 200

`GET /api/pipelines/cdc-streams/source-cleanup`

List deleted CDC streams whose source-side cleanup is unfinished

Responses: 200 · 400 · 401

`POST /api/pipelines/cdc-streams/source-cleanup/{id}/resolve`

Mark a parked CDC source cleanup resolved (the slot was handled by hand)

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 204 — No content · 401 · 404

`POST /api/pipelines/cdc-streams/source-cleanup/{id}/retry`

Send a parked CDC source cleanup back to the worker to check the source again

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 204 — No content · 401 · 404

`GET /api/pipelines/failures/cdc/{id}`

Diagnose a CDC stream (buffer, slot, cleanup, source checks), with what its pipelines feed

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/failures/refresh/{id}`

Diagnose a dataset’s last refresh, with what it feeds that the caller may open

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/failures/run/{id}`

Diagnose a pipeline run, with the dashboards, reports and shares downstream that the caller may open

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/flowlets`

The flowlet library (latest version of each)

Responses: 200 · 400 · 401 · 404

`POST /api/pipelines/flowlets`

Save a flowlet — a new version if the name already exists

Responses: 201 · 400 · 401 · 409 · 422

`GET /api/pipelines/flowlets/{id}`

One flowlet version by id

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`DELETE /api/pipelines/flowlets/{id}/all`

Retire a flowlet (all versions) from the library

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 204 · 404

`POST /api/pipelines/flowlets/{id}/instantiate`

Instantiate a flowlet version with arguments → concrete ops

Path parameters

Name	Type	Required	Description
`id`	string	yes

Request body (JSON)

Field	Type	Required	Description
`params`	object	no

Responses: 200 · 400 · 401 · 404 · 422

`GET /api/pipelines/flowlets/{id}/versions`

Version history for the flowlet’s name

Path parameters

Name	Type	Required	Description
`id`	string	yes

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/quality/summary`

Per-pipeline data-quality gate health for the workspace

Query parameters

Name	Type	Required	Description
`days`	integer	no

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/runs/{runId}`

Get one run with its step runs

Path parameters

Name	Type	Required	Description
`runId`	string	yes

Responses: 200 · 400 · 401 · 404

`POST /api/pipelines/runs/{runId}/cancel`

Cancel a queued/running run (cooperative via the orchestration plane)

Path parameters

Name	Type	Required	Description
`runId`	string	yes

Responses: 200 · 400 · 401 · 404 · 409

`GET /api/pipelines/runs/{runId}/diagnosis`

Diagnose a run: the failing step, the cause and a concrete fix

Path parameters

Name	Type	Required	Description
`runId`	string	yes

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/runs/{runId}/events`

Live run/step events (SSE; ends when the run reaches a terminal status)

Path parameters

Name	Type	Required	Description
`runId`	string	yes

Responses: 200

`POST /api/pipelines/runs/{runId}/retry`

Retry a failed run (re-queues the same run row; worker owns attempt semantics)

Path parameters

Name	Type	Required	Description
`runId`	string	yes

Responses: 200 · 400 · 401 · 404 · 409 · 503

`GET /api/pipelines/runs/{runId}/steps/{stepId}/logs`

The log tail a step attempt produced

Path parameters

Name	Type	Required	Description
`runId`	string	yes
`stepId`	string	yes

Query parameters

Name	Type	Required	Description
`attempt`	integer	no

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/usage`

Current-month pipeline usage per workspace (admin)

Responses: 200 · 400 · 401 · 404

`GET /api/pipelines/usage/breakdown`

Per-pipeline and per-sink usage for a month (admin)

Query parameters

Name	Type	Required	Description
`month`	string	no

Responses: 200 · 400 · 401 · 403 · 404

`GET /api/pipelines/workers/status`

Execution-plane availability (Run-now preflight)

Responses: 200 · 400 · 401 · 404