Pipelines

Runs, backfill & monitoring

Every pipeline has a runs page — “every execution with its per-step timeline” — with Backfill, Lineage, Edit pipeline, and Run now in the toolbar.

Runs execute on a pipeline worker. If none is connected, a run waits in the queue and says why; it starts by itself once a worker is online, and you can cancel it at any time.

The runs table

Newest first, paged: Status · Trigger · Started · Duration · Rows · Size. Backfill runs display their date next to the trigger indicator.

Statuses

Run	Step
`queued` · `running` · `succeeded` · `failed` · `cancelled`	the same five, plus `skipped`

A step is skipped when an upstream step failed — the pipeline short-circuits cleanly rather than executing steps whose inputs never arrived.

Why a run exists

Every run identifies its origin:

Value	Shown as	The run started because…
`manual`	manual	someone clicked Run now
`schedule`	schedule	an automated schedule came due
`backfill`	backfill	a backfill window created it (one run per date)
`file_arrival`	file arrival	new files landed in a watched folder
`pipeline`	upstream pipeline	an upstream pipeline completed
`webhook`	webhook	an external system called the signed webhook

The run detail panel

Select any run to inspect:

Header summary — status badge, trigger source, duration, total rows processed, and bytes transferred.
What went wrong — on a failed run: the step that failed (and the steps it kept from running), the cause in plain words, the fix, and a link to where you make it — the connection’s settings for refused credentials or an unreachable host, the pipeline for a missing table or column. The raw error and the step’s log are one click away. See Diagnosing a failed run.
Step timeline — each step’s status, attempt count, wall-clock duration, rows and bytes processed, and Data Quality gate chips.

Real-time streaming — Changes are pushed to the page the moment the worker records them (a status change arrives in a few tens of milliseconds), over Server-Sent Events. If the live channel is interrupted the page keeps up by polling, so nothing is missed. An open step log follows a running step, refreshing every few seconds as the worker writes it.

Diagnosing a failed run

DataSquares reads the run’s root cause and the failing step’s log and matches them against the failures it knows:

Cause	Typical fix
Credentials refused	Update them on the connection, Test connection, retry
Host unreachable	Check host, port and firewall; test; retry
Private or internal address	Use the host’s public address
Missing permission on a table or schema	Grant the connection’s user the privilege
SFTP folder the login may not list	Give the login read access, or narrow the step’s path
SFTP host key not the pinned one	Confirm the server’s new key with its owner, then update the fingerprint
Table or column no longer there	Pick the table again, or fix the step that uses the column
Schema drift refused (strict)	Switch the step to Allow, or update the pipeline
Quality checks failed	Fix the data, or switch the check to quarantine
API errors (404, 429, 5xx, unexpected shape)	Fix the path, slow the extract, retry later, or adjust records path / pagination
No rows from the step before	Check the extract’s table, filters and watermark
Time or memory limit	Make the extract incremental, or partition / narrow the step

A queued run says why it hasn’t started (no worker online, all busy, waiting for the previous run…). An error DataSquares doesn’t recognise is shown as such, with the evidence — never a guessed cause. The diagnosis works without AI; in AI chat, “why did my pipeline fail?” explains the same diagnosis and can propose the change.

It is also available at GET /api/pipelines/runs/{runId}/diagnosis.

What it reaches

Under the fix, What it reaches lists what the failure leaves out of date downstream: the dashboards, paginated reports and shares that read what the pipeline feeds, followed through lineage. You see by name, with a link, only what you may open yourself (through your role, or because it was shared with you); the rest is counted (“and 3 more you can’t open”), never named.

Refreshes and change streams

The same diagnosis, with the same What it reaches, covers two more things:

A dataset’s refresh. When a managed dataset’s last refresh failed, its page says why: the refresh’s last run, diagnosed as above, with its links pointing at the dataset. People who may read the dataset see it; the step log stays on the pipeline pages.
A CDC stream. Diagnose on a stream checks it against the stream rules: a full change buffer, a replication slot lost or past its safe size, a cleanup parked for an administrator, a source that fails its checks.

The three answer in one shape at GET /api/pipelines/failures/run/{runId}, GET /api/pipelines/failures/refresh/{datasetId} and GET /api/pipelines/failures/cdc/{id}, where {id} is the stream’s id from the streams list, not its name (stream_id). They are rules, not AI: they cost no AI credits, and they name columns, tables and connections, never the data in them.

Cost & performance

Above the runs list, Cost & performance estimates what the pipeline costs in execution time — the meter your plan’s included hours count — from its own successful runs: a typical-to-slow band per run (p50–p95), and per month at its schedule, next to this month’s remaining hours. A pipeline that hasn’t succeeded yet shows no numbers.

It also says where each transform runs — in-process on DuckDB, or inside the warehouse (ClickHouse) once a declarative transform’s input passes the pushdown size — and suggests concrete changes with their effect:

a large raw-SQL transform rewritten as operations runs in the warehouse instead of on the worker;
a slow transform made only of row-by-row operations can be split into partitions;
an extract that re-reads a whole large table can read only new rows with a cursor column, or read in parallel ranges;
bigger batches for an API destination, and merge instead of replace for a large table in your database;
a warning when the schedule would use more hours than remain this month.

The run approval in AI chat states the expected run time too. API: GET /api/pipelines/{id}/advice.

Data quality step gates

When steps configure data quality rules or expectations:

Steps display inline Quality chips indicating passed vs. failed row counts.
Steps with quality rule violations flag warnings or fail according to your configured quality threshold.

Per-step logs

Expand any step row to read the narrative worker logs:

Up to ~16,000 characters per step attempt.
Multi-attempt steps keep logs per attempt so you can compare the failure cause with the recovery run.
Sensitive credentials and tokens are scrubbed automatically before logs are stored.
Built-in one-click Copy log button for quick troubleshooting.

Lineage tracking

Click Lineage in the toolbar to open the Column lineage: for each step, where its columns came from and, for a load, where they went.

Cancel & retry

Cancel — Safely stops an active run. Cancellation is cooperative: the worker stops between batches rather than killing queries mid-transaction, ensuring no partial or corrupted tables remain.
Retry — Available on failed runs. Re-queues the run and executes from the beginning. A failed dataset refresh is retried by refreshing the dataset.

Backfill

Click Backfill to re-process historical data across a date window:

Select a From and To date range (up to 366 days).
The engine creates one run per date in the window, passing the date as the backfill parameter to your extract steps.
Ideal for initial data onboarding or repairing upstream historical anomalies.

Concurrency & overlap control

Set maxConcurrentRuns (1–10, default 1) and how the pipeline handles new runs while an execution is in progress (onOverlap) in the pipeline’s concurrency setting, in its JSON or through the API; the editor has no control for it:

onOverlap	Behaviour
`skip` (default)	The new trigger is refused — current run continues
`queue`	The new run enters the queue and executes once the current run completes
`cancel_previous`	The running execution is cancelled and the new one takes over

Trigger refusals

All triggers climb the same execution ladder. When a run cannot start, an appropriate refusal code is returned:

Code	Error	Meaning
`409`	`RUN_OVERLAP`	Another run is active and onOverlap is set to `skip`
`403`	`PRODUCT_NOT_LICENSED`	The Pipelines product isn’t licensed
`429`	`ALLOWANCE_EXHAUSTED`	The workspace’s monthly Pipelines execution-hours allowance is used
`429`	`USAGE_LIMIT_REACHED`	The hosted usage limit is reached
`429`	`STORAGE_FULL`	The run loads into a warehouse that is full

With no worker connected, a run is not refused: it waits in the queue.

Failure notifications

A failed run sends the pipeline.run_failed event once to your Webhooks (Slack gets a readable message), and emails the pipeline’s owner.

After 3 runs in a row are refused for the same lasting reason (the connection’s credentials can’t be read, or the source is missing or not supported), the schedule pauses itself and the owner is told. The pipeline shows Schedule paused with Resume schedule.

Telling the people downstream

A failed run always tells the pipeline’s owner, and a failed dataset refresh tells the dataset’s owner. To tell the people whose dashboards it leaves out of date as well, turn on Tell dashboard owners when a run fails on the pipeline’s runs page (it is off until you do; you need permission to edit the pipeline). Then, when a run fails:

the owners of the dashboards it feeds, and of the active report schedules on those dashboards or on the reports it feeds, get a notice in their notifications feed;
each of them gets at most one a day for this pipeline, however often it fails;
the notice names the dashboards and reports they may open and counts the rest; someone who may open none of them gets no notice;
it names the pipeline, and links the failed run, only for people who may view pipelines. Everyone else reads that “a pipeline that feeds them failed”, with a link to their dashboard or report.

The notices are rules, not AI: they cost no AI credits and carry no data values. A dataset’s own refresh schedule can’t send them; its refreshes tell the dataset’s owner.

Building a pipeline — Step types, schedules, and triggers.
Where pipelines can write — Managed datasets, external databases, and object stores.
Change data capture (CDC) — Streaming change feeds.
Flowlets — Reusable transform modules.
Pipelines API — Programmatic run control and metrics.