Refreshing data & schemas

Two different actions on a data source card serve distinct purposes:

Action	What it does	For
Refresh	Re-pulls and updates the data rows themselves	Staged API-pull sources (REST API, Google Sheets)
Sync	Re-discovers the schema (tables, columns, and data types)	Direct-query databases, warehouses, and staging

Refreshing data (API pulls)

For REST API and Google Sheets sources, clicking Refresh opens the Refresh dialog:

Refresh now — Triggers an immediate re-fetch. The UI displays the outcome in real-time (e.g., “Refreshed 3 tables, 14,250 rows”).
Atomic transaction swap — Incoming data is staged in temporary tables alongside existing data. Once validation completes successfully, the tables are swapped in a single atomic database transaction. Dashboards and reports never observe half-refreshed states, and removed upstream rows or tables are pruned cleanly.
File uploads (CSV, Excel, SQLite, Parquet, JSON) are point-in-time snapshots — re-upload the updated file in the Data Sources page to refresh staged tables.

Automated refresh schedules

Inside the Refresh dialog, set the Schedule dropdown to automate ingestion:

No schedule (manual only)
Hourly
Daily
Weekly

Schedules run as resilient relative intervals evaluated server-side. The scheduler persists across server restarts and multi-instance deployments; a transient network failure will report an error in history but will not disable the recurring schedule.

Run history & auditing

The dialog’s Run history table maintains an audit log of the last 20 sync executions:

Status — Success, Running, or Error with full stack error diagnostics.
Trigger — Identifies whether the run was triggered by manual user action, an automated schedule, or an inbound webhook.
Metrics — Total tables processed, row count ingested, start timestamp, and run duration.

Failure notifications & webhooks

When a scheduled or webhook refresh fails, DataSquares dispatches a source.sync_failed event to your configured Webhooks. You can route this to Slack or alert channels to catch upstream API issues instantly.

Event-driven refresh (inbound webhook trigger)

When upstream systems know exactly when new data is ready, you can push updates instead of waiting for the next polling interval:

Open the source’s Refresh dialog.
Enable the Inbound trigger toggle. DataSquares generates a secure signed endpoint URL and secret token (the secret is displayed once upon creation).
Upstream pipelines or cron jobs send an authenticated POST request to:

POST /api/sources/{sourceId}/sync-trigger
Content-Type: application/json
X-DataSquares-Timestamp: 1724587200
X-DataSquares-Delivery-Id: 550e8400-e29b-41d4-a716-446655440000
X-DataSquares-Signature: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Signature verification contract

HMAC SHA-256 — The signature is calculated as the HMAC-SHA256 hex digest of "{timestamp}.{deliveryId}.{sourceId}" using your secret key.
Replay attack protection — Each X-DataSquares-Delivery-Id is accepted strictly once.
Timestamp window — Requests with timestamps older than 5 minutes are rejected.
The resulting execution appears in Run History tagged with the webhook trigger indicator.

Schema sync (Discovery)

Click Sync on any database, warehouse, or object store source card to trigger Schema Discovery:

Detects structure changes — Identifies newly created tables, added columns, modified column data types, and dropped columns.
Preserves semantic metadata — Keeps existing descriptions, tags, and SquareX calculated measures attached to untouched fields.
No data movement — Runs lightweight metadata catalog queries (e.g. information_schema or driver reflection) without querying row data.

Connection flows explained
Connector catalog
Webhooks & outbound events
Data Pipelines — Multi-step transformation, change data capture (CDC), and reverse ETL.