Refreshing datasets

A managed dataset is a copy, so it is only as fresh as its last refresh. Refresh it yourself with Refresh now, or hand it to a cron schedule.

Every refresh is a full re-import: DataSquares re-reads the source table or saved query in its entirety and rebuilds the copy. There is no incremental or partition refresh, so the same row ceiling that applies to the first import applies to every refresh.

Refresh now

Refresh now, on the dataset’s page, re-imports it immediately. It is safe to run at any time, because the swap is atomic:

1. The new copy is built in a staging table next to the live one.
2. Once it’s complete, the two tables are exchanged in a single operation.
3. The old table is dropped.

The live table is never updated in place. Queries keep working throughout — a reader sees either the whole old copy or the whole new one, never a mix, and never an empty table mid-refresh.

Start a refresh while one is already running and it’s refused with “A refresh is already running” — refreshes don’t queue up or run in parallel.

What error really means

If a refresh fails — the source is unreachable, the query no longer runs, the result outgrew the row ceiling — the swap never happens. The staging copy is discarded and the live table is left exactly as it was.

So a dataset badged error is still serving data. The badge means “the last refresh didn’t land”, not “there’s nothing here”. Dashboards keep answering from the previous copy; what you have lost is freshness, and the badge is telling you precisely that. Fix the cause and refresh again.

Schedules

Schedule on the dataset’s page opens Refresh schedule. You can also set one up front, in the import dialog.

Option	Cron
No schedule (refresh on demand)	— clears the schedule; refresh manually
Hourly	0 * * * *
Daily (02:00)	0 2 * * *
Weekly (Mon 02:00)	0 2 * * 1
Custom (cron)	your own expression, e.g. 30 6 * * 1-5

Custom (cron) reveals a Cron expression field and a Timezone (default UTC). An Enabled checkbox arms or pauses a schedule without discarding it — a paused schedule reads (paused) in the Warehouse list, and a dataset with no schedule at all reads Manual.

A cron expression or timezone that could never fire is refused when you save it, rather than accepted and quietly ignored.

Scheduled refreshes do fire

A scheduler polls for armed schedules and runs them. If your server was down when an occurrence came due, it fires once on the way back up — for the latest missed occurrence, not once for every occurrence missed — and then re-arms on the normal cadence. Each scheduled run is recorded in refresh history.

Cron only — micro-batch is not supported on datasets

The frequency picker is shared with Pipelines, so it also lists Micro-batch (every N seconds). That option does not work on managed datasets today — a dataset scheduled that way will not refresh. Use one of the cron options above; Hourly is the shortest preset, and a custom cron can go down to the minute.

Refresh history

The dataset’s Refresh history lists past scheduled refreshes, newest first — 20 per page, paged with Newer / Older.

Column	Shows
Status	running, succeeded or failed
Started	when the run began
Duration	how long it took
Rows	rows written by the run
Size	bytes moved by the run

Two honest notes, so the table doesn’t mislead you:

• “Refresh now” runs are not recorded here. A manual refresh moves the freshness stamp at the top of the page (Last refreshed, Rows, Size) immediately but writes no history row. An empty history therefore doesn’t mean the dataset has never been refreshed — only that it has never been refreshed on a schedule.
• Scheduled runs record duration, rows, and bytes moved. The history table tracks the throughput of the underlying system pipeline runs that execute your schedule.

When a scheduled refresh fails

The dataset’s owner — whoever imported it — is notified once per failure (not once per poll): by email, and as an outbound webhook event you can route into Slack or your own alerting. The dataset is badged error and, as described above, keeps serving its previous copy until a refresh lands.

Related

• Warehouse (managed datasets) — importing, using and deleting datasets.
• Datasets API — POST /api/datasets/{id}/refresh, PUT /api/datasets/{id}/schedule, GET /api/datasets/{id}/refreshes.
• Pipelines: runs, backfill & monitoring — for multi-step loads.
• Webhooks — where the failure event lands.