Compute & workers

Pipeline runs and heavy data transformations execute on dedicated execution workers. The Compute tab under Admin (/admin/compute) provides centralized visibility into worker fleet health, VM infrastructure, queue wait times, and autoscaling policies.

Primary-box-first architecture

By default, pipeline executions run directly on the primary host. The Compute & workers configuration allows administrators to register external VMs and worker nodes when offloading compute-intensive runs or scaling horizontally.

The worker fleet

The compute overview lists all registered worker instances with live resource telemetry:

• Resource Gauges: Live CPU utilization %, Memory (used/total), Disk storage, and active execution slots.
• Sparklines: 1-hour rolling historical resource utilization sparklines per worker.
• Health Indicators: Nodes with expired heartbeats transition into a distinct Stale state to highlight network or infrastructure disruptions.

VM node management

Admins can register and manage virtual machines directly in the VMs Section:

• Register VM: Define host address, SSH/API connection credentials, tags, and label metadata ({"env": "high-memory", "region": "us-east-1"}).
• Worker Verification: Automated health probe (/api/admin/compute/vms/:id/verify) that validates Docker/container runtimes, DuckDB execution engines, and network connectivity.

Worker lifecycle actions

Per-worker control options allow graceful node maintenance without interrupting active jobs:

• Drain (/drain): Prevents new pipeline runs from dispatching to this worker while allowing active in-flight tasks to finish completely before shutting down.
• Resume (/resume): Restores an idle or drained worker back into the active dispatch pool.
• Stop (/stop): Immediately terminates worker execution activities.

Run queue & dispatch reasons

The Queue section shows pending pipeline runs awaiting execution, annotating each with explicit wait reasons:

Wait reason	Description
No capacity	All eligible workers in the pool have reached their concurrency slot limits.
No eligible worker	No active workers match the pipeline’s placement constraints or required node labels.
Contract mismatch	The pipeline requires a newer contract schema than what the current worker fleet version supports (parking runs safely during zero-downtime rolling upgrades).

Placement policies

Pipelines can configure execution placement rules:

Placement mode	Behavior
auto (default)	Dynamically dispatches to any available worker with free capacity.
primary	Directs execution strictly to the primary node (useful when accessing local data files or private network endpoints).
extended	Directs execution to dedicated worker VMs to isolate heavy compute from the web application server.

• Node Label Matching: Pipelines can require specific key-value label tags (e.g. gpu: true or warehouse: ClickHouse).
• Safe Fallback: If no worker satisfies the required labels, the orchestrator logs the placement fallback reason on the run rather than deadlocking indefinitely.

Scale timeline & dispatch SLOs

The Timeline & Activity section visualizes scheduling performance:

• Inflow vs. Outflow: Hourly charts comparing queued runs against dispatched runs.
• Autoscaling Actions: Visual markers showing automated scale-up and scale-in events.
• Dispatch SLO Percentiles: Tracks p50 and p95 time-to-start latencies to diagnose queue bottlenecks before they impact scheduled deliveries.

Related

• Pipelines overview — architecture and DAG capabilities.
• Runs, backfill & monitoring — pipeline run history and step logs.
• Workspace settings & operations — tenant administration and API keys.
• Audit log — audit trail of compute worker lifecycle events.