REST API

You connect REST API from the DataSquares connection wizard. Pull a JSON endpoint into a table — headers supported for auth.

1. In DataSquares, open Data Sources and choose Connect Source. The Connect a data source page opens.

2. Pick REST API on that page (search or browse by category).

3. Fill in the connection details below, then test the connection and save.

Connection details

Field	What to enter
Endpoint URL	The HTTP(S) endpoint that returns JSON.
Headers	Optional request headers — use these for auth (e.g. Authorization: Bearer …).
Records path	Where the array of records lives in the response (e.g. data.items).

The response is pulled into a staged table; re-sync the source to refresh it.

Authentication

Auth goes in the headers field. The common patterns:

Authorization: Bearer <token>
X-API-Key: <key>

Basic auth works as a header too: Authorization: Basic <base64 user:pass>.

Pointing at the records

If the endpoint returns the array directly ([ {...}, {...} ]), leave the records path empty. If the array is nested, give its dot path:

{ "data": { "items": [ … ] } }   →   records path: data.items

Each record is flattened into a row; nested objects become dotted columns.

Refresh model

The endpoint is pulled when you create the source and again on each re-sync — it’s a snapshot between syncs, not a live proxy. Every page is fetched before anything is replaced, so a refresh either lands the whole result or changes nothing.

Pagination

If the API returns its records a page at a time, choose how it pages under Pagination in the wizard. Leave it on None for an endpoint that returns everything in one response.

Mode	What it sends / reads	Fields	Stops when
Offset and limit	?offset=0&limit=100, then offset=100, …	Page size (100), offset parameter (offset), limit parameter (limit)	a page comes back short or empty
Page number	?page=1, then page=2, …	Page parameter (page), first page 1 or 0, optional page-size parameter + page size	a page comes back empty (or short, when a page size is sent)
Cursor	reads the next cursor from each response and sends it back	Cursor path (required, e.g. $.meta.next_cursor or next), cursor parameter	the cursor is missing, null or empty
Link header	follows Link: <…>; rel="next"	—	there is no rel="next"

In cursor mode, if the value at the cursor path is a full https://… URL (the “next page URL” style), it is followed as-is and the cursor parameter can stay blank. Otherwise it is sent back as ?<cursor parameter>=<value>.

Blank fields use the defaults shown in brackets. The records on each page are found with the same records path and appended in page order.

A field a record does not have, and a JSON null, are NULL in the staged table; a real empty string ("") stays an empty string in a text column and becomes NULL in a number, boolean or date column.

Limits — the import fails rather than truncates. A REST source is published as one complete table, so every limit ends the import with a message naming it, and nothing is staged:

• Max pages (default 1,000) — the endpoint still had pages after that many. Raise it in the wizard, or narrow the request with URL filters.
• Total size — all pages together over 256 MiB (the operator’s REST_MAX_TOTAL_BYTES); each single response is also capped at 100 MiB (REST_MAX_RESPONSE_BYTES).
• Rows — more records than the import row limit (1,000,000 by default).

For a larger or incremental pull, use a pipeline REST extract, which spools pages to the lake and can carry a watermark.

Rules that refuse a configuration:

• Next-page links (a Link header or a URL cursor) must stay on the endpoint URL’s own origin — same scheme, host and port. A link to another site is refused, because it would send your request headers somewhere you did not configure.
• A query parameter the mode sets on every page (for example limit in offset mode) must not already be in the endpoint URL — remove it from the URL.
• An API that repeats a cursor is stopped instead of looping.

Rate limits are handled per page: a 429/503 with Retry-After is waited out and retried a few times.

With pagination left on None, a response that says more pages exist (Link: rel="next", has_more: true, or a non-empty next / next_page) is refused — importing page one as if it were everything would silently lose data. The error points you to the Pagination setting.

Test outside first

curl the exact URL with the same headers before wiring it in — it separates API-side problems (auth, path) from configuration typos immediately.

Related

• Connector catalog — everything DataSquares connects to.
• Quickstart — from connection to shared dashboard.