IBM Cloud Object Storage

IBM Cloud Object Storage is in Preview in the connection wizard: it speaks the Amazon S3 API at IBM Cloud Object Storage’s own endpoint and is read and written with the same client as Amazon S3, but it has not yet been proven against a live account, so check what you import before relying on it. The bucket’s public endpoint, e.g. https://s3.us-south.cloud-object-storage.appdomain.cloud (private and direct endpoints are not reachable from DataSquares). Region: taken from the endpoint.

1. In DataSquares, open Data Sources and choose Connect Source. The Connect a data source page opens.

2. Pick IBM Cloud Object Storage on that page (search or browse by category).

3. Fill in the connection details below, then test the connection and save.

Connection details

Field	What to enter
Bucket	The bucket to read from.
Region	Not asked for: taken from the endpoint (us-south for the example).
Access key ID / Secret access key	HMAC credentials: a service credential created with HMAC on — its cos_hmac_keys access_key_id and secret_access_key. Listing and reading the bucket is enough to browse and import; a pipeline that writes files back also needs write and delete.
Endpoint	Required: https://s3.<region>.cloud-object-storage.appdomain.cloud — e.g. https://s3.us-south.cloud-object-storage.appdomain.cloud. Where IBM Cloud Object Storage shows it.
Path-style addressing	On by default: the bucket goes in the URL path. Turn it off only if your store is set up for bucket subdomains.

What you can do with it

An object store is not a SQL database — there are no schemas or tables to SELECT from. Testing the connection performs a bounded listing, which proves the credentials and the container, nothing more. From there:

• Browse the bucket or container from the source’s detail page.
• Import an object (Parquet, JSON/NDJSON or CSV) as a staged dataset — it is decoded and landed in your workspace, and from that point behaves like any other table: models, dashboards, SQL, SquareX.
• Import a whole prefix as one dataset. A partitioned export (exports/2026-08-09/part-00000 … part-00031) is many objects that are logically one table, so every importable object under the prefix is read and stacked into a single staged dataset — with key=value folders (year=2026/month=10/) read back as columns, see Folder names as columns.
• Read and write in bulk from a pipeline — extract objects under a prefix or glob (hive-partitioned folders such as year=2026/month=10/ read back as columns), load results back (optionally hive-partitioned), and trigger runs on file arrival. See Building a pipeline and Destinations.

Format is read from the file, not the filename

An object’s format comes from its bytes, so an extensionless key — what most exporters write — imports fine, and a .csv that actually holds Parquet is read as Parquet rather than failing with a parse error that blames your data. Only self-describing formats are detected this way (Parquet, JSON/NDJSON); CSV has no signature, so a CSV object still needs a .csv/.tsv name.

Folder names as columns

When every object under the listed prefix sits in the same key=value folders — the layout Spark, Hive, Glue, DuckDB and the pipeline Files load target write — the browser shows Folder names are columns, ticked: each folder becomes a column of every row of its objects (sales/year=2026/month=10/part-0.parquet gives year and month). The import reads them exactly as a pipeline’s Files extract does with folder partitions on:

• Values are text, kept exactly as the folder spells them: month=01 keeps its zero, percent-encoded characters are decoded (city=New%20York is New York), spaces and an empty value are kept. NULL and Hive’s __HIVE_DEFAULT_PARTITION__ are NULL.
• The partition columns come after the files’ own columns; a column of the same name inside the files takes the folder’s value.
• Only folders count, never the file name, and every object must sit in the same key=value folders.
• What is not data is refused, never read as rows or skipped: a non-empty marker or checksum (_SUCCESS, _committed_*, .crc), anything under a hidden folder such as an unfinished write’s _temporary/, and a Delta, Hudi or Iceberg table or a Spark streaming sink, whose folders keep files the table no longer counts — a prefix that starts inside one (tbl/region=emea/) included. The refusal names the object; since an import reads everything under its prefix, narrow the prefix, or read the folder with a pipeline, whose Files extract can take a glob such as sales/**/*.parquet.

Through the REST API, send "hivePartitioning": true with the prefix import, and hiveTypes to type a partition ({ "year": "integer" } — string, integer, double, boolean, date or timestamp); a value its type cannot read is refused.

When a prefix import refuses

Everything answerable from the listing is checked before anything is downloaded, so you find out immediately rather than after a partial fetch:

Refusal	Why
The objects total more than the import limit	This path reads objects into memory. A dataset that large belongs in a pipeline — see Too large to import.
More objects than a single import reads	It refuses rather than quietly importing the first N — a table that looks complete and is not is the worse outcome. The whole prefix is listed before anything is decided.
The prefix holds more than one format	Two formats under one prefix is two datasets. Narrow the prefix, or import them separately.
An object shares no columns with the ones before it	Same reason — that is a different dataset, and null-filling it would produce a mostly-empty table that looks successful.

Parts whose columns differ are unioned by name, not by position: a partition that gained a column mid-backfill is null-filled for the earlier parts, one that dropped a column keeps it, and a writer that reorders columns is realigned rather than silently transposed.

Too large to import

An import reads into memory, so it has limits: the objects under a prefix (500, and 256 MB together), one object’s size, and the rows one import holds. Refused for any of them, a prefix or an object offers Create a pipeline dataset: the pipeline editor opens on a new pipeline that reads the same objects with a Files extract — folder names as columns when they were on for the import, and enough Max files per run for the folder — and loads them into a dataset of the name you gave. A pipeline reads in the background with no such limit. Nothing runs until you save and run it.

Importing stages a copy: re-import (or re-run the pipeline) to pick up changes to the object. Objects are fetched over the platform’s validated outbound path and read locally — DataSquares never queries your bucket in place.

Related

• Connector catalog — everything DataSquares connects to.
• Quickstart — from connection to shared dashboard.