Refreshing datasets

A managed dataset is a copy, so it is only as fresh as its last refresh. Refresh it yourself with Refresh now, or hand it to a cron schedule.

Every refresh is a full re-import: DataSquares re-reads the source table or saved query in its entirety and rebuilds the copy. There is no incremental or partition refresh, so the same row ceiling that applies to the first import applies to every refresh.

Refresh now

Refresh now, on the dataset’s page, re-imports it immediately. It is safe to run at any time, because the swap is atomic:

1. The new copy is built in a staging table next to the live one.
2. Once it’s complete, the two tables are exchanged in a single operation.
3. The old table is dropped.

The live table is never updated in place. Queries keep working throughout — a reader sees either the whole old copy or the whole new one, never a mix, and never an empty table mid-refresh.

Start a refresh while one is already running and it’s refused with “A refresh is already running” — refreshes don’t queue up or run in parallel.

What error really means

If a refresh fails — the source is unreachable, the query no longer runs or no longer passes the safety check, the result outgrew the row ceiling — the swap never happens. The staging copy is discarded and the live table is left exactly as it was. When the source can’t be read, the dataset also keeps its status, and shows as overdue once its schedule passes; it is badged error when the copy itself fails to load.

So a dataset badged error is still serving data. The badge means “the last refresh didn’t land”, not “there’s nothing here”. Dashboards keep answering from the previous copy; what you have lost is freshness, and the badge is telling you precisely that. Fix the cause and refresh again.

Schedules

Schedule on the dataset’s page opens Refresh schedule. You can also set one up front, in the import dialog.

Option	Cron
No schedule (manual only)	— clears the schedule; refresh manually
Hourly	0 * * * *
Daily (02:00)	0 2 * * *
Weekly (Mon 02:00)	0 2 * * 1
Custom (cron)	your own expression, e.g. 30 6 * * 1-5

Custom (cron) reveals a Cron expression field; every option has a Timezone (default UTC). An Enabled checkbox arms or pauses a schedule without discarding it — a paused schedule reads (paused) in the Warehouse list, and a dataset with no schedule at all reads Manual.

A cron expression or timezone that could never fire is refused when you save it, rather than accepted and quietly ignored.

Scheduled refreshes do fire

A scheduler polls for armed schedules and runs them. If your server was down when an occurrence came due, it fires once on the way back up — for the latest missed occurrence, not once for every occurrence missed — and then re-arms on the normal cadence. Each scheduled run is recorded in refresh history.

Cron only

Dataset schedules are cron only; Hourly is the shortest preset, and a custom cron can go down to the minute.

Refresh history

The dataset’s Refresh history lists past refreshes, scheduled and manual, newest first — 20 per page by default (50 or 100 if you choose), with page numbers and Previous / Next.

Column	Shows
Status	running, succeeded or failed
Started	when the run began
Duration	how long it took
Rows	rows written by the run
Size	not recorded for refreshes, so it shows —

Two honest notes, so the table doesn’t mislead you:

• “Refresh now” is recorded once it starts. A manual refresh refused before it begins — one is already running, the warehouse is full, or this month’s refresh hours are used up — shows its message where you clicked and adds no row. A scheduled refresh refused by the first two is recorded as a failed run; one past the refresh hours is skipped. Manual refreshes before October 2026 were not recorded.
• Refreshes record duration and rows only. Bytes moved are not recorded, so Size stays empty.

Refresh hours

Each refresh, scheduled or manual, counts toward the Warehouse size’s refresh hours — 100, 300 or 1,000 a month for S, M and L (seats and billing) — by the time it runs, per second with a one-minute minimum, and never toward Pipelines execution hours. A refresh refused before it starts counts nothing. Administration → Usage shows this month’s refresh hours.

Past them, refreshes are execution hours: a month uses the workspace’s execution-hour packs (what its pipeline runs leave of them), then go on the next invoice at the execution-hour rate, under the workspace’s usage limit. At the workspace’s usage limit, new refreshes wait: a scheduled refresh is skipped and tried again at its next time, and Refresh now is refused with “This workspace has reached its usage limit for this month”, with no amount. A refresh already running finishes, and the dataset keeps serving its last copy. A workspace whose administrator had turned overage off keeps it off until DataSquares changes it, and its refreshes wait at the refresh hours instead: Refresh now says “This workspace has used its monthly Warehouse refresh hours”.

When a scheduled refresh fails

The dataset’s owner — whoever imported it — is notified once per failure (not once per poll): by email, in their notifications, and as an outbound webhook event (pipeline.run_failed, for the hidden refresh pipeline) you can route into Slack or your own alerting. A manual refresh that fails after it started is notified the same way. As described above, the dataset keeps serving its previous copy until a refresh lands.

A scheduled refresh reads its table or saved query as the dataset’s owner, as they are when it runs. When the owner is no longer an active member of the workspace, their role no longer allows the import, or the read is refused for them (row-level security, a labelled column, or a saved query’s safety check), the refresh stops: one failed run in the refresh history says why, and the schedule is paused, its cron kept. Put the cause right, then switch the schedule back on. A source that no longer exists pauses the schedule the same way.

If the source is an app connector (GitLab, Zendesk, Slack, …) and its key has expired or been replaced, open its refresh dialog and choose Update credentials. The connector’s fields start from the saved values; type the new key (leave a secret empty to keep the current one, and empty any other field to remove its value) and Test and save — the new values are tried on one read before they replace the saved ones, and the source keeps its tables, schedules and the models and dashboards built on them. Then Refresh now. A connector you signed in to (rather than pasting a key) reconnects by signing in again.

A saved secret is only ever sent to the address it was saved for. Changing an address field — a self-managed GitLab URL, a WooCommerce store URL, an OData service root or token URL — needs every secret typed again; the dialog asks for them as soon as the address changes.

Freshness on dashboards

Every card that reads a dataset shows a Dataset badge — green when the dataset refreshed on schedule, amber when it missed a scheduled refresh (including when the source couldn’t be read), red when its last copy failed to load, blue while it refreshes, and grey when it hasn’t refreshed yet or has no schedule. A missed refresh is counted from the schedule itself: the dataset is overdue once an occurrence has passed by one schedule period (at least 2 minutes, at most 15) without a refresh — for example when the source it copies can no longer be read. See Stay current. The Data Quality Freshness tab uses the same verdict for tables on a dataset. A dataset loaded by a pipeline takes its freshness from that pipeline’s runs and schedule.

Related

• Warehouse (managed datasets) — importing, using and deleting datasets.
• Datasets API — POST /api/datasets/{id}/refresh, PUT /api/datasets/{id}/schedule, GET /api/datasets/{id}/refreshes.
• Pipelines: runs, backfill & monitoring — for multi-step loads.
• Webhooks — where the failure event lands.