Chains

A job's config/chains file defines its pipeline. It ranges from a single image to a directed graph of steps that pipe data through one another. You write it to the job's config/chains path:

report_builder

Referencing a repo image

A step points at a repo image. There are two reference forms:

• Short form — a bare repo name resolves to your latest image: report_builder → @<your_username>/report_builder:latest. Add a label to pin one, as in report_builder:prod.
• Canonical form — name the owner and label explicitly: @novem_demo/report_builder:latest. Use this to reference an image shared with you by another user.

The label is one of the repo's image labels: latest for the default branch, prod for the newest tag (the default branch until the repo has one), or tag:<name> for one tag, as in @novem_demo/report_builder:tag:v1.0.0. Branch names are not labels, and a step naming a label the repo does not have fails the run.

A label is resolved once, when the run is submitted. The run records the exact image each step resolved to and the commit it was built from, and keeps them for its whole life, so a new build landing on latest mid-run does not change what that run executes. The next run picks up the new image. The run log names what each step resolved to.

Referencing another job

A step can run another novem job instead of a repo image. Reference it by path: /j/<job> for one of your own jobs, or /u/<user>/j/<job> for a job shared with you by someone else.

fetch   <- data_fetcher
rollup  <- /j/nightly_rollup

fetch -> rollup

At run time novem expands the referenced job into the steps it is built from and splices them into the chain, so rollup runs whatever nightly_rollup runs, fed by fetch's output. A referenced job may itself reference further jobs; novem expands the whole tree, rejecting reference cycles and nesting that runs too deep. Each referenced job records its own run, nested under this run — see runs.

Each referenced job runs with its own environment variables; a referenced job's steps do not inherit the environment of the job that composes them.

Permissions. You must have execute access to every job your chain references directly — your own jobs, or another user's job shared with you with execute permission. Whatever those jobs reference internally is their own concern and is not checked against you. Conversely, someone you grant execute on this job does not need their own access to the jobs it references.

A single step

The simplest chain is one reference. This runs that image and is exactly what the quick start uses.

report_builder

Multi-step pipelines

To pipe steps together, use the chain grammar. It has two kinds of line:

• Bindings assign a short label to a repo image with <-.
• Edges wire labels together with ->, left to right.

fetch  <- data_fetcher
render <- chart_builder

fetch -> render

This runs fetch first, then pipes its output files into render as that step's input. The labels keep the pipeline readable and let you reference the same image more than once.

Edges can be chained in a single line for longer pipelines:

fetch     <- data_fetcher
transform <- transformer
render    <- chart_builder

fetch -> transform -> render

Each step receives the previous step's output as its input; the final step's output becomes the job's result.

Fan-out

Prefix a step with a dot to fan out over the previous step's output: the step runs once per output file, in parallel. A fan-out step sits between a producer and a barrier — the step before it supplies the files to spread over, and the step after it waits for every parallel instance and collects their output — so a fan-out is never the first or last step in a chain.

split   <- splitter
render  <- chart_builder
collect <- collector

split -> .render -> collect

Here render runs once for each file split produced, and collect receives all of their outputs together once the instances finish. Fan-out currently supports the per-file mode shown above.

A chain may contain one fan-out step, and that step must have both a predecessor to spread over and a successor to collect into. Naming two fan-out steps, or putting one at either end of the chain, is a configuration error.

Holds

Put a ! on a step to stop the run there and wait for a person. The mark is positional and goes in an edge line, next to the step's name:

!a -> b     stop before a runs
a! -> b     stop after a has run
!a! -> b    both: look at a's input, then at its output

A run that reaches a hold parks the files at that point in a space, reports waiting, and continues only when someone with execute access on the job answers. Whatever is in the space then becomes the next step's input, so a reviewer can correct, add or delete files before the run goes on. See runs waiting at a hold for the review itself.

Holds need spaces, which are gated behind a per-account feature flag. On an account without them, a chain with a hold fails when it is triggered, and the run log says why. A reviewer needs spaces too, to open the parked files.

A space cannot hold every file name. It refuses < > : " | ? * \, control characters, a name ending in a dot or a space, and device names such as CON or NUL. A name may be at most 255 bytes, and a path at most 1024 bytes and 64 folders deep. It also compares names without case, so Report.csv and report.csv would be one file. If a hold would park such a file, the run fails before parking anything, and the log names the file.

extract <- data_extractor
render  <- report_builder

extract! -> render

Here render runs only after someone has looked at what extract produced.

A step that exists only to stop binds to _ instead of an image:

extract <- data_extractor
audit   <- _
render  <- report_builder

extract -> !audit -> render

A _ step runs nothing, so it must carry a hold, and either mark parks before it: audit! is the same as !audit. A _ step with no mark is a configuration error. A hold on the first step parks the trigger's own input, which makes a leading ! a manual start for a scheduled job. A hold after the last step (render!) is a sign-off: nothing runs after it, so the files the reviewer leaves in the space become the run's output.

A chain may hold at several steps, and each one waits for its own answer in turn. A hold cannot be a fan-out step: .!b and .b! are configuration errors. Because the mark lives in an edge line, a chain needs at least one -> line to carry a hold.

Step resources

A step runs with modest defaults — one CPU, 200MB of memory, a 1GB disk. Give a step that needs more room a comma-separated list of resources after its reference:

extract  <- data_extractor
crunch   <- number_cruncher,cpu=4,mem=4G
report   <- report_builder,timeout=30s

extract -> crunch -> report

Four are available:

The single-reference form takes them too: report_builder,mem=2G.

timeout is the one that only ever tightens. It exists to fail fast when you know work should be quick — a step that normally finishes in ten seconds and is still running after thirty is stuck, and waiting out the full budget tells you nothing. Asking for longer than a step already gets is a configuration error.

What your plan allows

Resources are capped per step, by plan. Asking for more than your plan allows fails the run with a message naming the step and the ceiling; nothing is silently reduced, because a step given less memory than it asked for fails in a far more confusing way.

Free jobs run at the defaults and cannot set cpu, mem or disk — a chain that sets one fails rather than ignoring it. timeout is not a plan limit, so any job may shorten a step.

The ceiling is per step. On a fan-out step it is therefore per parallel instance: mem=4G on a step that spreads over 20 files means each of those instances gets 4G, not 4G between them. How many run at once is not yours to set — a chain runs on one worker and takes its instances through the slots that worker has — but a fan-out step is the one place where raising a number multiplies rather than adds.

Shape rules

A chain is a single linear sequence of steps. Each step has at most one step before it and one after it. Once the file declares at least one -> line, every bound step must be connected to the chain — a step nothing points at is an error rather than a step that never runs. A file with no -> line at all is not validated for connectivity: its bound steps run in definition order.

Two shapes are therefore not available. Branching, where one step feeds two others:

A -> B
A -> C

And merging, where one step takes input from two others:

A -> C
B -> C

Run several independent things by fanning out over files instead, or by splitting the work across separate jobs.

There is also no comment syntax. Every non-blank line is either a binding or an edge, so a line that is neither is a configuration error.

How it's stored

novem normalises whatever you write into a canonical form — bindings first, then the edge line — so reading config/chains back may look slightly different from what you wrote:

fetch <- @you/data_fetcher:latest
render <- @you/chart_builder:latest

fetch -> render

Next steps

• Jobs overview — env vars, schedules and running.
• Repos — build the images your chain references.