Change data capture (CDC)

A Change Data Capture (CDC) stream continuously captures row-level inserts, updates, and deletes from your source database and streams them directly into your data pipelines.

Pair a CDC stream with a Merge load step to maintain an exact replica of your operational database without re-extracting entire tables on every schedule.

Manage CDC streams under Pipelines → CDC streams.

Micro-batch architecture

DataSquares CDC operates as a high-frequency micro-batch engine. While changes are captured continuously from database transaction logs, pipelines consume changes on schedules with a 15-second minimum interval.

Supported databases

DataSquares CDC supports all PostgreSQL-wire engines:

• PostgreSQL & PostgreSQL-compatible databases
• TimescaleDB
• YugabyteDB
• Neon & Supabase
• Google Cloud SQL, Amazon RDS & Azure Database for PostgreSQL
• AlloyDB

(MySQL and SQL Server CDC engines are currently in development).

Source prerequisites

Your source PostgreSQL instance must enable logical replication:

1. Logical WAL level: Set wal_level = logical in postgresql.conf.
2. Replication user permissions: The connected database user must possess the REPLICATION role attribute and privilege to create publications on the target tables:

   ALTER USER datasquares_user WITH REPLICATION;
   GRANT SELECT ON ALL TABLES IN SCHEMA public TO datasquares_user;

3. Dedicated replication slots: DataSquares creates a dedicated replication slot and publication per stream (ds_cdc_<streamId> and ds_cdc_<streamId>_pub) to ensure multi-stream isolation.

---

Creating a CDC stream

1. Navigate to Pipelines → CDC streams and click New CDC stream.
2. Source: Select any connected PostgreSQL-family data source.
3. Stream Name (Stream ID): Enter a unique identifier (up to 64 alphanumeric characters, underscores, or hyphens, e.g., orders_realtime).
4. Tables: Specify a comma-separated list of fully qualified tables to capture (e.g. public.orders, public.order_items). You can capture up to 100 tables in a single stream.
5. Click Create stream. DataSquares provisions the managed Debezium capture worker, takes an initial consistent snapshot, and tails the write-ahead log (WAL).

---

Using CDC in a pipeline

1. In the Pipeline Editor, add an Extract step.
2. Select the CDC tab and choose your Stream Name. No database credentials are required on the extract step — the supervisor manages secure ingest tokens internally.
3. (Optional) Add Transform steps to clean, filter, or mask fields.
4. Add a Load step configured with:
   • Mode: Merge (upsert by key)
   • Merge Keys: The primary key column(s) of the source table.
5. Set the pipeline schedule to Micro-batch (every N seconds) (e.g. 15s or 60s).

---

CDC metadata columns

Every extracted CDC record is enriched with system audit metadata:

Column	Data type	Description
_ds_op	string	The change operation: 'c' (create/insert), 'u' (update), or 'd' (delete)
_ds_deleted	boolean	true if the record represents a deletion in the source table
_ds_version	bigint	Microsecond timestamp ensuring deterministic out-of-order resolution

When loaded into a database or managed dataset in Merge mode, deleted source rows are automatically removed or flagged according to your destination settings.

---

Resilient offset tracking

CDC state and replication offsets are persisted to durable volumes. Container restarts, worker migrations, or platform upgrades resume seamlessly from the last committed LSN (Log Sequence Number) without re-snapshotting source tables.

Replication slot hygiene

Paused or disabled CDC streams continue to hold their PostgreSQL replication slot, causing the database to retain WAL files on disk. If a stream is no longer needed, Delete it to drop the replication slot and release disk space.

Limits

• PostgreSQL-family sources only in v1.
• Up to 100 tables per stream.
• CDC extract steps cannot combine with partitioned or incremental cursor settings.

Related

• Building a pipeline — Extract step configuration and micro-batch schedules.
• Where pipelines can write — Merge and delete handling.
• Pipelines API — CDC stream management endpoints.