Apache Iceberg

Apache Iceberg is in Preview in the connection wizard: it reads tables written by real engines, but has not yet been proven against a live vendor catalog, so check the imported tables before relying on them. Connect a REST catalog or point at table folders in a storage connection — each table’s current snapshot is imported as a table.

1. In DataSquares, open Data Sources and choose Connect Source. The Connect a data source page opens.

2. Pick Apache Iceberg on that page (search or browse by category).

3. Fill in the connection details below, then test the connection and save.

Before you start

An Iceberg table’s files are read through one of your storage connections — create an Amazon S3 (or S3-compatible), Azure Blob Storage / Azure Data Lake Gen2 or Google Cloud Storage connection for the bucket or container the tables live in first. Every metadata and data file is fetched through it, and a table whose files are anywhere else is refused, naming both locations.

Connection details

Field	What to enter
Source name	What the source is called in DataSquares (e.g. Lakehouse orders).
Storage connection	The bucket or container the tables live in.

Then choose where the tables are: REST catalog or Table folders.

REST catalog

Field	What to enter
Catalog URL	The Iceberg REST endpoint — the part before /v1 (e.g. https://catalog.example.com/api/catalog). It must be reachable on a public address.
Warehouse	Optional — the warehouse to ask the catalog for (in Polaris / Snowflake Open Catalog, the catalog name).
Sign-in	None, Bearer token, or OAuth2 client credentials (client ID and secret; optional scope, default catalog, and token URL, default the catalog’s /v1/oauth/tokens). Secrets are stored encrypted and never shown again.

Choose Connect — it reads the catalog’s configuration, which is the connection test, and lists its namespaces. Open a namespace to list its tables and the namespaces inside it; each list shows one page at a time with Load more, so a catalog of any size can be browsed when it pages its lists: one answer from the catalog (a page of a list, or a table’s metadata) is read up to 1 MiB and 50,000 JSON objects, arrays and keys, about 12,500 tables. Tick the tables to import.

Any catalog that speaks the Apache Iceberg REST specification works — Apache Polaris / Snowflake Open Catalog, Lakekeeper, Nessie, Unity Catalog’s Iceberg endpoint and others. A catalog that only hands out temporary storage credentials (credential vending) is not used for them: the storage connection must be able to read the files itself.

Table folders

Give each table’s folder — the one that holds metadata/ and data/ — as a location in the storage connection (s3://lake/warehouse/shop/orders) or a path inside it (warehouse/shop/orders). The version read is the one metadata/version-hint.text names, when the folder has one (Hadoop-catalog tables and DataSquares pipeline Iceberg destinations write it), and otherwise the newest metadata file (vN.metadata.json or NNNNN-….metadata.json); give a …/metadata/vN.metadata.json to read that version instead. This is also how tables a pipeline’s Iceberg destination wrote are imported.

Review

Choose Review. From each table’s metadata and manifests — without reading a data file — the form shows its data files, size, rows, columns and the snapshot that would be read, or why the table cannot be imported:

Refusal	Why
The current snapshot has delete files	Rows deleted or updated merge-on-read (position deletes, equality deletes, deletion vectors) are still in the data files, so reading them alone would show those rows. Compact the table (rewrite its data files so the deletes are applied), then import it again.
A data file is ORC or Avro	Only Parquet data files are read.
The table has no snapshot yet	There is no data to import until something is written to it.
More data than one import reads	One source reads at most 500 data files and 256 MB in total, within the import’s row and cell limits — compact small files, or import fewer tables per source.
More metadata than one import reads	One source reads at most 256 MB of metadata, a compressed manifest counted at its inflated size, and walks its manifests in at most 150 million steps (about one per value read or skipped) — expire old snapshots and rewrite the table’s manifests. Each table’s current metadata (its vN.metadata.json once inflated, or the catalog’s answer for it) is read up to 1 MiB and 50,000 JSON objects, arrays and keys, about 1,100 to 1,600 snapshots not yet expired as Spark or PyIceberg writes them, and is refused past either with the same advice. A manifest whose header holds more than 1,024 entries or 16 MiB, whose schema takes more than 1 MiB or holds more than 2,048 JSON values or 32 record types, or one of whose blocks inflates past 16 MiB is refused as not a readable Avro file; engines write theirs far smaller.
A column type this reader does not stage yet	Variant, geometry, geography and nanosecond-timestamp columns, initial defaults and encrypted tables are refused by name.

Import

Choose Import & Create (it waits for a review of the current picks). Each table’s current snapshot is imported as its own table and staged in your workspace like any other staged source: models, dashboards and SQL read it from there.

• Columns are matched to the table’s current schema by Iceberg field id, so a renamed column arrives under its new name; a column added after a file was written is empty for that file’s rows, and an identity-partition column a file does not store is filled from its partition value.
• Each column is staged with the type the table declares (a long stays a 64-bit integer, a decimal(38, 10) keeps its scale, timestamp and timestamptz stay apart); time, uuid, binary (base64) and nested values (JSON) are text. A timestamp keeps its microseconds and a timestamptz is stored to the millisecond; text is stored with leading and trailing spaces trimmed.

Refresh

Refresh the source by hand (Refresh now) or on a schedule. A refresh first reads each table’s metadata: when every table’s snapshot and schema are the ones last imported, the run succeeds without reading or replacing anything, so an hourly schedule on a quiet table costs a metadata read. When one moved, every table is read again and replaced in one step.

Edit the tables

Open the source’s Refresh panel and choose Edit tables to keep or remove tables, add tables (from the catalog, or as folders) or update the catalog sign-in — a secret left blank keeps the saved one. Each kept table keeps its table, so models and dashboards keep their bindings. Save and re-read re-reads every table; removing a table that holds rows asks first.

Writing Iceberg tables from a pipeline (Preview)

A pipeline’s Files destination writes an Apache Iceberg v2 table — Parquet data files with Iceberg field ids and Avro manifests — into a storage connection’s bucket or container, two ways:

• Catalog-less. The table is its folder (metadata/vN.metadata.json and version-hint.text); DuckDB iceberg_scan, Spark’s Hadoop catalog, Trino’s file metastore, Athena and Snowflake external tables read it by path.
• Through an Iceberg REST catalog. Add your catalog (Apache Polaris / Snowflake Open Catalog, Lakekeeper, Databricks Unity’s Iceberg endpoint …) to the store connection — its URI and an OAuth2 client-credentials or bearer sign-in — and the step publishes namespace.table there with an atomic catalog commit, so every engine that reads the catalog sees it. The namespace must already exist.

Append, replace, partitioning and the exact rules are on Destinations → Apache Iceberg tables.

Related

• Connector catalog — everything DataSquares connects to.
• Quickstart — from connection to shared dashboard.