Runs, backfill & monitoring

Every pipeline has a runs page — “every execution with its per-step timeline” — with Backfill, Lineage, Edit pipeline, and Run now in the toolbar.

Runs execute on a pipeline worker. If none is connected, actions respond with “No pipeline worker is connected — runs will start once one is online” — nothing is lost; try again when the worker is up.

The runs table

Newest first, paged: Status · Trigger · Started · Duration · Rows · Size. Backfill runs display their date next to the trigger indicator.

Statuses

Run	Step
queued · running · succeeded · failed · cancelled	the same five, plus skipped

A step is skipped when an upstream step failed — the pipeline short-circuits cleanly rather than executing steps whose inputs never arrived.

Why a run exists

Every run identifies its origin:

Value	Shown as	The run started because…
manual	manual	someone clicked Run now
schedule	schedule	an automated schedule came due
backfill	backfill	a backfill window created it (one run per date)
file_arrival	file arrival	new files landed in a watched folder
pipeline	upstream pipeline	an upstream pipeline completed
webhook	webhook	an external system called the signed webhook

The run detail panel

Select any run to inspect:

• Header summary — status badge, trigger source, duration, total rows processed, and bytes transferred.
• Error diagnostics — structured error messages and failure causes when a run fails.
• Step timeline — each step’s status, attempt count, wall-clock duration, rows and bytes processed, and Data Quality gate chips.

Real-time streaming — The page streams live progress over Server-Sent Events (SSE) with an automatic 5-second polling fallback, allowing you to monitor multi-step extracts and transforms in real time.

Data quality step gates

When steps configure data quality rules or expectations:

• Steps display inline Quality chips indicating passed vs. failed row counts.
• Steps with quality rule violations flag warnings or fail according to your configured quality threshold.

Per-step logs

Expand any step row to read the narrative worker logs:

• Up to ~16,000 characters per step attempt.
• Multi-attempt steps keep logs per attempt so you can compare the failure cause with the recovery run.
• Sensitive credentials and tokens are scrubbed automatically before logs are stored.
• Built-in one-click Copy log button for quick troubleshooting.

Lineage tracking

Click Lineage in the toolbar to open the pipeline lineage drawer. This shows all upstream data sources, intermediate staging tables, and downstream datasets or dashboards fed by this pipeline.

Cancel & retry

• Cancel — Safely stops an active run. Cancellation is cooperative: the worker stops between batches rather than killing queries mid-transaction, ensuring no partial or corrupted tables remain.
• Retry — Available on failed runs. Re-queues the run and executes from the beginning.

Retry restarts the whole run

There is no resume-from-failed-step. If step 4 of 5 failed, retry re-runs steps 1–5. For expensive extracts, use an incremental cursor so the re-run fetches only the current bounded delta.

Backfill

Click Backfill to re-process historical data across a date window:

1. Select a From and To date range (up to 366 days).
2. The engine creates one run per date in the window, passing the date as the backfill parameter to your extract steps.
3. Ideal for initial data onboarding or repairing upstream historical anomalies.

Concurrency & overlap control

Configure maxConcurrentRuns (1–10, default 1) and choose how the pipeline handles new runs while an execution is in progress:

onOverlap	Behaviour
skip (default)	The new trigger is refused — current run continues
queue	The new run enters the queue and executes once the current run completes
cancel_previous	The running execution is cancelled and the new one takes over

Trigger refusals

All triggers climb the same execution ladder. When a run cannot start, an appropriate refusal code is returned:

Code	Error	Meaning
409	RUN_OVERLAP	Another run is active and onOverlap is set to skip
429	ALLOWANCE_EXHAUSTED	Workspace pipeline run allowance for the billing period is reached
503	WORKER_UNAVAILABLE	No pipeline worker is currently connected

A refusal creates no run record

A refused trigger creates no run entry in the database — neither failed nor cancelled. The refusal is returned directly to the caller.

Failure notifications

Failed runs dispatch a single event through the notification plane. Configure a Webhook to send alerts to Slack, Microsoft Teams, or custom endpoints.

Related

• Building a pipeline — Step types, schedules, and triggers.
• Where pipelines can write — Managed datasets, external databases, and object stores.
• Change data capture (CDC) — Streaming change feeds.
• Flowlets — Reusable transform modules.
• Pipelines API — Programmatic run control and metrics.