Flowlets

A flowlet is a saved chain of transform ops with typed parameter holes — “dedupe by key”, “standardize an address”, “normalize currency”, or “calculate tax” — authored once and instantiated into multiple pipelines with customized arguments.

Manage and build flowlets under Pipelines → Flowlets.

Why use flowlets?

Copying transform chains across pipelines causes maintenance drift:

• When business logic changes, copied chains must be found and edited manually in every pipeline.
• A flowlet provides a single centralized definition.
• Every instantiation maintains provenance metadata (“from flowlet Dedupe by key v2”), allowing you to audit which pipelines use which versions.

Anatomy of a flowlet

A flowlet definition consists of:

• Name & Description — Identifies the reusable module.
• Parameters (up to 24) — Typed inputs with optional default values and descriptions.
• Transform Ops (up to 50) — Standard declarative operations with ${parameterName} placeholders.

{
  "name": "Dedupe by key",
  "description": "Keep the newest row per primary key.",
  "params": [
    { "name": "key_col", "type": "column", "description": "The identity column to deduplicate on" },
    { "name": "sort_col", "type": "column", "description": "Timestamp column for ordering" }
  ],
  "ops": [
    {
      "op": "dedupe",
      "columns": ["${key_col}"],
      "orderBy": [{ "column": "${sort_col}", "direction": "desc" }]
    }
  ]
}

Parameter types

Type	Input control	Description
column	Autocomplete input	Bound to incoming pipeline dataset column names with datalist matching
string	Text field	Arbitrary string values and literal expressions
number	Numeric field	Concrete numbers (rendered as typed numeric values in expressions)
boolean	Dropdown	true or false boolean flags

Substitution is type-aware: when a ${param} placeholder represents the entire value, it is replaced with the concrete typed value rather than a string literal.

Authoring flowlets

1. Go to Pipelines → Flowlets and click New flowlet.
2. Define your parameters and assign their types and defaults.
3. Build the transform sequence using the visual Ops Builder. Declared parameters automatically appear in autocomplete suggestions as ${paramName} wherever column references or values are expected.
4. Click Save flowlet. The platform validates the template by synthesizing test inputs to ensure all referenced ops and parameters are valid.

Instantiating into a pipeline

1. In the pipeline editor, select or add a Transform (SQL) step.
2. Click Insert from flowlet in the transform toolbar.
3. Select the desired flowlet from your library.
4. Choose version: The dialog displays the active version and allows you to inspect Version history (v1, v2, …) if you need to instantiate a specific release.
5. Provide the concrete column arguments and parameter values.
6. Click Insert. The flowlet expands into concrete pipeline ops and marks the step with a provenance badge (from <Name> v<Version>).

Provenance and editing

Instantiated ops are copied into the step recipe. If you subsequently customize or modify the generated ops manually, the step drops the strict provenance badge to reflect that the step logic has diverged from the canonical flowlet version.

Immutable versioning

Every save to an existing flowlet mints a new immutable version (v1 → v2 → v3).

• No surprise regressions: Publishing a new flowlet version never modifies existing pipelines. Running pipelines continue using their frozen op snapshots.
• Audit trail: When a new version is published, pipeline authors can choose when to open the flowlet dialog and re-instantiate with the updated logic.
• Retirement safety: Deleting or archiving a flowlet from the library does not break existing pipelines that already expanded its operations.

Security & SQL injection safety

Parameters cannot smuggle unescaped SQL:

• Parameter values are inserted into strictly typed op schemas.
• The pipeline SQL compiler automatically quotes all identifiers and parameterizes literals before query compilation.
• Broken templates fail validation at authoring time rather than during scheduled execution.

Limits

• Flowlets support declarative ops only (SQL ops builder). Raw SQL queries and Python code steps cannot be packaged as flowlets.
• Maximum of 24 parameters and 50 ops per flowlet definition.

Related

• Building a pipeline — Transform step operations and DAG editor.
• Where pipelines can write — Output destinations.
• Pipelines API — Flowlet endpoints.