Skip to content

feat(nextflow): step-only inputs and inherited requirements - #584

Merged
vjaganat90 merged 5 commits into
PolusAI:masterfrom
vjaganat90:feat/nextflow-step-only-inputs
Nov 13, 2026
Merged

vjaganat90 merged 5 commits into
PolusAI:masterfrom
vjaganat90:feat/nextflow-step-only-inputs

Conversation

@vjaganat90

Copy link
Copy Markdown
Member

Stacked on #583 (refactor/resolved-view). Merge that first; this PR's own changes are the commits above it.

What

  • The Nextflow target carries a step's step-only inputs, the ones its tool does not declare, whenever they matter: the step's when or scatter reads one, or one waits on another step's output. Each becomes a val input of the process and stages nothing. For docs/cookbook/when.wic with --target nextflow:

    • before: when.wic:9:16: step 'echo': input 'enabled': the step's when reads an input its tool does not declare, which is not supported
    • after: it compiles, runs, and gives cwltool's outputs.

    A step-only input fed only by root workflow inputs and read by nothing stays out, as before, so every pipeline that compiles today keeps its workflow.nf.

  • Tools inherit the requirements and hints written outside them, on a workflow, a nested workflow and its call, and the step, as cwltool gives them: for each class the tool's own requirement, else the innermost inherited one, else the tool's own hint, else the innermost inherited hint. A workflow's DockerRequirement hint now gives every tool without its own image one, and a step's ResourceRequirement inside a nested workflow wins over the root's. The four feature classes (ScatterFeatureRequirement, SubworkflowFeatureRequirement, MultipleInputFeatureRequirement, StepInputExpressionRequirement) only declare a feature and give a tool nothing.

  • A when is typed by what its inputs deliver, through nested workflows, and per element for a scattered input; an input that may be null is read only beside null, since CWL fails a when that gives null.

  • The model gains NfProcess.step_only_inputs and NfPort.scalar_type, both last and defaulted. A model built by hand as in 0.7.0 still builds: a scalar_type left out is filled from the port's connection, and only a given one that disagrees is refused. import_nextflow reads both.

What changes in generated files

  • Every nextflow_workflow.json says "schema_version": 21 and has a scalar_type key on each port.
  • workflow.nf and the other files are unchanged for every pipeline that compiled before. Emitted CWL is unchanged.

Deleted

  • The refusals of workflow hints, of a step's requirements and hints, and of workflow-level requirements other than the feature classes: inheritance lowers them instead.
  • The refusals of a step-only input that a when reads or a step output feeds: they are carried instead. A scatter over a step-only input stays refused, since no process port receives its element.

Why
CWL lets a step bind inputs its tool does not declare and put requirements outside the tool, and ordinary workflows such as the cookbook's when.wic do both; the Nextflow target refused them.

@darkclad darkclad left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I reviewed the five commits above #583 (0ea45e3..f3bab3d). Step-only inputs are carried as trailing val ports and round-trip through ext WorkflowStepInput and import_nextflow. That held for every case I probed: a when over one or two of them, a when over a conditional step's output read beside null, a File output that only orders the tasks, a scatter whose when reads a step-only input, and a source with a default, which lowers to first_non_null typed int. Inheritance follows cwltool's order (root, then the call step, then the nested workflow, innermost winning, any requirement before any hint), and a SoftwareRequirement hint is refused at the step. In WSL with Nextflow 26.04.2 and cwltool 3.3, the differential test and the touched Nextflow test files give 1092 passed, with only the Docker batch deselected. On Windows tests/core fails the same 123 environmental tests on head and base, and head passes 55 more. One thing for the author: the branch still sits on #583's old head ba07f58, and replaying these commits onto its current head 6bce14c conflicts in src/sophios/utils_nf.py.

A when is typed per scatter job and through calls; inherited entries are
refused at the step; the Docker cases run in the Docker step.
NfProcess.step_only_inputs, NfPort.scalar_type and schema 21; an ext
WorkflowStepInput entry carries them. A workflow fills a val port built with
no scalar_type from its connection and refuses only one given that disagrees.
A step-only input a when or scatter reads, or a step output feeds, is a val input;
root or via a call, refuses: cwltool skips on null > 1, Groovy throws on [] > 1.
The view gives each tool step what cwltool gives its tool from outside it.
The target writes it into the tool and lowers it as the tool's own; workflow
hints and step requirements and hints lower, and feature classes are inert.
A bounded property runs generated batches, and the hand-written cases of
resolved-view §8 and §9, under cwltool and Nextflow and compares the outputs.
@vjaganat90
vjaganat90 force-pushed the feat/nextflow-step-only-inputs branch from f3bab3d to 8f546e9 Compare November 13, 2026 15:53
@vjaganat90
vjaganat90 merged commit 606605c into PolusAI:master Nov 13, 2026
3 checks passed
@vjaganat90
vjaganat90 deleted the feat/nextflow-step-only-inputs branch November 13, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants