Change data capture (CDC)

A Change Data Capture (CDC) stream continuously captures row-level inserts, updates, and deletes from your source database and streams them directly into your data pipelines.

Pair a CDC stream with a Merge load step to maintain an exact replica of your operational database without re-extracting entire tables on every schedule.

Manage CDC streams under Pipelines → CDC streams.

Micro-batch architecture

DataSquares CDC operates as a high-frequency micro-batch engine. While changes are captured continuously from database transaction logs, pipelines consume changes on schedules with a 15-second minimum interval.

Supported databases

DataSquares CDC captures these PostgreSQL sources:

• PostgreSQL
• TimescaleDB
• ParadeDB
• Neon & Supabase
• Google Cloud SQL, Amazon RDS & Azure Database for PostgreSQL
• AlloyDB

And these MySQL sources:

• MySQL
• Google Cloud SQL, Amazon RDS & Azure Database for MySQL

Only the databases listed here can be captured. A stream on any other source is refused when you create it, including SQL Server and databases that speak the PostgreSQL or MySQL protocol but are not listed, such as YugabyteDB, CockroachDB and MariaDB.

MySQL capture is new

MySQL capture is available, but it has not yet been verified end to end against a live MySQL server. Check what it delivers against your source before you depend on it.

Source prerequisites

PostgreSQL

Your source PostgreSQL instance must enable logical replication:

1. Logical WAL level: Set wal_level = logical in postgresql.conf.
2. Replication user permissions: The connected database user must possess the REPLICATION role attribute and privilege to create publications on the target tables:

   ALTER USER datasquares_user WITH REPLICATION;
   GRANT SELECT ON ALL TABLES IN SCHEMA public TO datasquares_user;

3. Dedicated replication slots: DataSquares creates a dedicated replication slot and publication per stream (ds_cdc_<stream name> and ds_cdc_<stream name>_pub, lower-cased, with - and other characters turned into _) to ensure multi-stream isolation.

MySQL

Your MySQL server must keep a row-based binary log, and the connected user needs replication privileges. For a MySQL source, the New CDC stream dialog lists these requirements and Check my server tests them; creating the stream is refused until the required ones pass.

1. Binary log: log_bin = ON, binlog_format = ROW and binlog_row_image = FULL.
2. Replication user permissions: SELECT on the captured tables, plus the global REPLICATION SLAVE and REPLICATION CLIENT privileges. Grant RELOAD too (on Amazon RDS and Aurora, LOCK TABLES), or the first snapshot may fail:

   GRANT SELECT ON shop.* TO 'datasquares_user'@'%';
   GRANT REPLICATION SLAVE, REPLICATION CLIENT, RELOAD ON *.* TO 'datasquares_user'@'%';

3. Recommended: GTIDs on (gtid_mode = ON), so capture can follow a failover to another server in the cluster, and binary logs kept for at least a day.
4. No replication slot: nothing is held on a MySQL source. If the server purges its binary log past a stopped stream’s position, the captured tables are snapshotted again when the stream restarts.

---

Creating a CDC stream

1. Navigate to Pipelines → CDC streams and click New CDC stream.
2. Source: Select a connected source from the supported list. For a MySQL source, Check my server tests the requirements first.
3. Stream Name (Stream ID): Enter a name unique across your company’s live streams (up to 64 alphanumeric characters, underscores, or hyphens, e.g., orders_realtime).
4. Tables: Specify a comma-separated list of fully qualified tables to capture: schema.table on PostgreSQL (e.g. public.orders, public.order_items), database.table on MySQL (e.g. shop.orders). You can capture up to 100 tables in a single stream.
5. Click Create stream. DataSquares provisions the managed Debezium capture worker, takes an initial consistent snapshot, and tails the write-ahead log (WAL) on PostgreSQL, or the binary log on MySQL.

---

Using CDC in a pipeline

1. In the Pipeline Editor, add an Extract step.
2. Select the CDC tab and type the stream’s name in Stream id. No database credentials are required on the extract step — the supervisor manages secure ingest tokens internally.
3. (Optional) Add Transform steps to clean, filter, or mask fields.
4. Add a Load step configured with:
   • Mode: Merge (upsert by key)
   • Merge Keys: The primary key column(s) of the source table.
5. Set the pipeline schedule to Micro-batch (every N seconds) (e.g. 15s or 60s).

---

CDC metadata columns

Every extracted CDC record is enriched with system audit metadata:

Column	Data type	Description
_ds_op	string	The change operation: 'c' (create/insert), 'u' (update), or 'd' (delete)
_ds_deleted	boolean	true if the record represents a deletion in the source table
_ds_version	bigint	Microsecond timestamp ensuring deterministic out-of-order resolution

In a Merge load into a database, a deleted source row is deleted from your table (see CDC deletes).

---

Resilient offset tracking

CDC state and replication offsets are persisted to durable volumes. Container restarts, worker migrations, or platform upgrades resume seamlessly from the last committed LSN (Log Sequence Number) without re-snapshotting source tables, unless the stream was paused long enough for its slot to be dropped (below). A MySQL stream resumes from its saved binary-log position the same way, as long as the server still keeps that log; otherwise its tables are snapshotted again.

Replication slot hygiene

A paused PostgreSQL stream keeps its replication slot, so the database retains WAL files for it, but not forever: DataSquares drops the slot after 72 hours paused, or as soon as it holds more than 10 GB of WAL, and the stream says on which date. Enabling the stream after that takes a fresh snapshot (Enable and re-snapshot). If a stream is no longer needed, Delete it: DataSquares then drops its replication slot and publication on the source database and the disk is released. A MySQL stream holds no slot, so pausing or deleting it leaves nothing on the source.

When a slot cannot be dropped

Deleting a stream queues the drop of its replication slot and publication on the source database. Until the slot is dropped, the source database keeps write-ahead log for it. Pipelines → CDC streams lists every unfinished drop under Source cleanup: the slot, the publication, the source they live on, the attempts so far and the last reason.

DataSquares keeps retrying a drop that has not finished: while the capture connector is still stopping, while the source cannot be reached, or while it rejects the saved credentials. Such a drop stays listed as Retrying automatically, with its attempts and last reason, but nobody is notified.

A drop is parked for an administrator only when the stream’s saved source connection cannot be used for it at all: the source stores only a connection string instead of host, port, database, user and password, or its saved connection can no longer be decrypted. Then:

• Workspace administrators get a notification as soon as the drop is parked.

• The row shows the SQL to run on the source database, as a user allowed to drop replication slots. It drops the slot and publication named for that stream, and does nothing if they are already gone:

  SELECT pg_drop_replication_slot(slot_name) FROM pg_replication_slots WHERE slot_name = 'ds_cdc_orders';
  DROP PUBLICATION IF EXISTS "ds_cdc_orders_pub";

• The names come from the stream name, so a stream created later under the same name, or one that differs only in case or in - for _, uses the same slot and publication names. When a live stream in your company does, the row shows no SQL and the notification says what to check instead: if that stream reads from the same database, the slot and publication are its own, and dropping them would break its capture. If it reads from a different database, drop them by hand on the deleted stream’s source, if they are there, and then mark the cleanup resolved.

• Check again sends the row back to DataSquares, which tries again through the stream’s saved connection. It helps once that connection works again, for example after re-saving the source with host, port, database, user and password: DataSquares then drops the slot, or finds it gone, and clears the row. While the connection still cannot be used, the row is parked again.

• If you dropped the slot and publication by hand, found that they belong to a live stream, or know the database no longer exists, an administrator can select Mark resolved. DataSquares does not check the source when you do, and records the action in the audit log.

Diagnosing a stream

Diagnose on a stream’s row says how it is and, when something is wrong, what to change and where. It checks, in this order, and stops at the first that applies:

What it finds	What it means	What to do
The source fails its checks	The source was deleted, its saved connection can’t be read, or holds a value change capture can’t pass to its connector (a line break, a backslash before ${, a PostgreSQL database name with ?, /, %, +, $ or \). Capture can’t start at all.	Fix the data source (the link opens it), check the source again, and enable the stream if it is paused
The replication slot was lost	PostgreSQL removed write-ahead log the stream had not read, so the changes in that gap can’t be captured	Delete the stream and create it again for the same tables (a fresh snapshot; a merge load upserts what changed), and raise max_slot_wal_keep_size on the source
A cleanup is parked	DataSquares could not drop the stream’s slot on the source, which keeps retaining WAL for it	Use Source cleanup: run its SQL, or fix the connection and Check again
The slot is past its safe size	The source holds more WAL for the slot than it keeps safely, so the slot can be lost at the source’s next checkpoint; or a paused stream’s slot holds more than DataSquares lets it, so DataSquares drops the slot at its next check (enabling the stream later takes a fresh snapshot)	Run the pipeline that reads the stream so capture catches up, and give the slot room on the source; for a paused stream, enable it or delete it
The change buffer is full	Capture is holding changes back until the pipeline reading the stream drains them: nothing is lost, but the source keeps WAL meanwhile	Run or unpause that pipeline (the link opens it); if no pipeline reads the stream, create one with a CDC stream extract

A MySQL stream holds no slot, so only the source checks and the change buffer apply to it. A stream with nothing wrong says it is capturing, or paused.

Like a failed run’s diagnosis, it lists what it reaches: the dashboards, reports and shares the pipelines reading the stream feed, by name when you may open them. It is rules, not AI, and costs no AI credits.

Limits

• PostgreSQL and MySQL sources only, from the supported list.
• Up to 100 tables per stream.
• CDC extract steps cannot combine with partitioned or incremental cursor settings.

Related

• Building a pipeline — Extract step configuration and micro-batch schedules.
• Where pipelines can write — Merge and delete handling.
• Pipelines API — CDC stream management endpoints.