Skip to content
DataSquaresDocs

Pipelines

DataSquares Pipelines moves data between your systems. A pipeline extracts from your connected sources, transforms and routes the rows, and loads the result into a managed dataset — or straight back into your own database, warehouse, or object store.

A pipeline is a visual DAG: steps joined by dependency edges, where an arrow means runs after. It runs on a schedule, on an event trigger, or on demand, and every run leaves a per-step timeline with rows, bytes, and logs.

Pipelines vs. source refresh

Section titled “Pipelines vs. source refresh”

Both keep data current — pick by the shape of the problem:

Refresh & sync Pipelines
What moves an API-pull source’s own tables, re-pulled in place data extracted from any source, transformed, and written to datasets or your own systems
Structure one source, all its tables a DAG: many extracts, transforms, routes and loads
Cadence hourly / daily / weekly cron, every-N-seconds (15s floor), file arrival, an upstream pipeline, or a webhook
History per-source run list per-run, per-step timeline with rows, bytes and logs
Extras signed inbound trigger backfill, retry, cancel, live progress, change data capture

Rule of thumb: refreshing a Google Sheet is a refresh; assembling nightly datasets from three databases — or copying a filtered slice of Postgres into your warehouse — is a pipeline.

The five step types

Section titled “The five step types”
Step What it does
Extract pull from a database, a warehouse, an object store, a REST API, or a CDC stream
Transform (SQL) reshape rows with 15 visual ops or raw SQL — fully enabled
Code (Python) anything SQL can’t express, in a sandboxed container
Route (split) send rows down named ports by condition — first match wins
Load write to a managed dataset, your own database, or files

A DAG holds up to 100 steps, and each step carries its own retry policy.

The app pages

Section titled “The app pages”
  • Pipelines — the list: every pipeline with its latest run, last run time, schedule, trigger, and lineage tracking. Hover a row for Run now / Runs / Schedule / Edit / Delete (deleting keeps run history).
  • Editor — the drag-connect canvas and per-step config drawer.
  • Runs — every execution with its per-step timeline, logs, and row metrics.
  • Flowlets — the library of reusable transform fragments.
  • CDC streams — your change-capture streams.

Where to go next

Section titled “Where to go next”
Building a pipelineThe editor: every step type, schedules, and the three event triggers.
Where pipelines can writeManaged datasets, your own databases and warehouses, and partitioned files.
Runs, backfill & monitoringLive timelines, per-step logs, retry, cancel, and date-window backfills.
Change data captureManaged Debezium: stream inserts, updates and deletes out of Postgres.
FlowletsReusable, parameterized, versioned transform fragments.

Before your first run

Section titled “Before your first run”
  1. The Pipelines product is licensed — see the note above.
  2. A pipeline worker must be connected. Without one, runs are refused with “No pipeline worker is connected — runs will start once one is online” — nothing is lost; try again when the worker is up.
  3. The lake object store is configured server-side by whoever runs your instance. It’s where steps hand rows to each other.
  4. Python code steps additionally need the sandbox executor enabled server-side; if it isn’t, the step refuses cleanly rather than half-running.

Current limits (honest list)

Section titled “Current limits (honest list)”
  • Fan-out (one load step → many destinations) and partitioned extract are configurable through the API / DAG JSON only.
  • Partitioned extract is Postgres-wire only (refused on MySQL and SQL Server).
  • Incremental extract is refused on Snowflake, BigQuery and staged-Redshift sources — those unload in bulk.
  • CDC is Postgres-family only. MySQL and SQL Server are refused, with the roadmap named.
  • CDC is micro-batch, not true streaming — there’s no event backbone, and we publish no end-to-end latency number. Sub-minute cadence is a supported configuration (15-second floor).
  • The Snowflake / BigQuery / Redshift COPY paths are statement-pinned, not live-verified against real accounts yet.
  • Retry re-runs the whole run from the top — there’s no resume-from-failed-step.
  • Route-step preview is refused, and per-port row counts aren’t on the run UI.
  • File destinations write Parquet, CSV or JSON — spreadsheet formats and SFTP delivery aren’t in the editor’s destination tabs today.
  • Decimals land as double — values past 15 significant digits are not exact.
  • The file-arrival trigger is a poll, not a push — its latency is the poll interval (seconds), not milliseconds.
Section titled “Related”
© 2026 DataSquares