Skip to content
DataSquaresDocs

Azure Data Lake Gen2

Azure Data Lake Gen2 is generally available in the DataSquares connection wizard. Same fields as Azure Blob — directories are skipped when listing.

  1. In DataSquares, open Data Sources and choose Add data source.

  2. Pick Azure Data Lake Gen2 in the connector picker (search or browse by category).

  3. Fill in the connection details below, then test the connection and save.

Connection details

Section titled “Connection details”
Field What to enter
Storage account The storage account name (e.g. mycompanydata).
Container The container to read from.
SAS token A SAS token with list and read permission — or an account key below. Either one satisfies the connection.
Account key The storage account key, if you are not using a SAS token.

What you can do with it

Section titled “What you can do with it”

An object store is not a SQL database — there are no schemas or tables to SELECT from. Testing the connection performs a bounded listing, which proves the credentials and the container, nothing more. From there:

  • Browse the bucket or container from the source’s detail page.
  • Import an object (Parquet, JSON/NDJSON or CSV) as a staged dataset — it is decoded and landed in your workspace, and from that point behaves like any other table: models, dashboards, SQL, SquareX.
  • Import a whole prefix as one dataset. A partitioned export (exports/2026-08-09/part-00000 … part-00031) is many objects that are logically one table, so every importable object under the prefix is read and stacked into a single staged dataset.
  • Read and write in bulk from a pipeline — extract objects under a prefix or glob, load results back (optionally hive-partitioned), and trigger runs on file arrival. See Building a pipeline and Destinations.

Format is read from the file, not the filename

Section titled “Format is read from the file, not the filename”

An object’s format comes from its bytes, so an extensionless key — what most exporters write — imports fine, and a .csv that actually holds Parquet is read as Parquet rather than failing with a parse error that blames your data. Only self-describing formats are detected this way (Parquet, JSON/NDJSON); CSV has no signature, so a CSV object still needs a .csv/.tsv name.

When a prefix import refuses

Section titled “When a prefix import refuses”

Everything answerable from the listing is checked before anything is downloaded, so you find out immediately rather than after a partial fetch:

Refusal Why
The objects total more than the import limit This path reads objects into memory. A dataset that large belongs in a pipeline.
More objects than a single import reads It refuses rather than quietly importing the first N — a table that looks complete and is not is the worse outcome.
The prefix holds more than one format Two formats under one prefix is two datasets. Narrow the prefix, or import them separately.
An object shares no columns with the ones before it Same reason — that is a different dataset, and null-filling it would produce a mostly-empty table that looks successful.

Parts whose columns differ are unioned by name, not by position: a partition that gained a column mid-backfill is null-filled for the earlier parts, one that dropped a column keeps it, and a writer that reorders columns is realigned rather than silently transposed.

Importing stages a copy: re-import (or re-run the pipeline) to pick up changes to the object. Objects are fetched over the platform’s validated outbound path and read locally — DataSquares never queries your bucket in place.

Gen2 or plain Blob?

Section titled “Gen2 or plain Blob?”

Gen2 containers speak the same Blob REST API as Azure Blob Storage, so the connection takes the same fields and either credential works. Use this type when the storage account has hierarchical namespaces enabled — with a namespace, directories are real entries rather than key prefixes, and DataSquares skips them when listing so a folder never shows up as an importable object.

Hive-partitioned lake layouts (region=eu/date=2026-07-01/…) are the usual reason to be here; a pipeline can both read and write that shape. See Destinations.

Creating the connection through the API

Section titled “Creating the connection through the API”
Terminal window
curl -X POST https://your-instance/api/data-sources \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Lake — curated",
    "type": "adls_gen2",
    "config": {
      "account": "mylakeaccount",
      "container": "curated",
      "accountKey": "…"
    }
  }'

Swap accountKey for sasToken to use a shared-access signature instead.

Section titled “Related”
© 2026 DataSquares