Repository navigation
feat(nextflow): step-only inputs and inherited requirements - #584
Conversation
d711127 to
f3bab3d
Compare
darkclad
left a comment
There was a problem hiding this comment.
LGTM. I reviewed the five commits above #583 (0ea45e3..f3bab3d). Step-only inputs are carried as trailing val ports and round-trip through ext WorkflowStepInput and import_nextflow. That held for every case I probed: a when over one or two of them, a when over a conditional step's output read beside null, a File output that only orders the tasks, a scatter whose when reads a step-only input, and a source with a default, which lowers to first_non_null typed int. Inheritance follows cwltool's order (root, then the call step, then the nested workflow, innermost winning, any requirement before any hint), and a SoftwareRequirement hint is refused at the step. In WSL with Nextflow 26.04.2 and cwltool 3.3, the differential test and the touched Nextflow test files give 1092 passed, with only the Docker batch deselected. On Windows tests/core fails the same 123 environmental tests on head and base, and head passes 55 more. One thing for the author: the branch still sits on #583's old head ba07f58, and replaying these commits onto its current head 6bce14c conflicts in src/sophios/utils_nf.py.
A when is typed per scatter job and through calls; inherited entries are refused at the step; the Docker cases run in the Docker step.
NfProcess.step_only_inputs, NfPort.scalar_type and schema 21; an ext WorkflowStepInput entry carries them. A workflow fills a val port built with no scalar_type from its connection and refuses only one given that disagrees.
A step-only input a when or scatter reads, or a step output feeds, is a val input; root or via a call, refuses: cwltool skips on null > 1, Groovy throws on [] > 1.
The view gives each tool step what cwltool gives its tool from outside it. The target writes it into the tool and lowers it as the tool's own; workflow hints and step requirements and hints lower, and feature classes are inert.
A bounded property runs generated batches, and the hand-written cases of resolved-view §8 and §9, under cwltool and Nextflow and compares the outputs.
f3bab3d to
8f546e9
Compare
Stacked on #583 (
refactor/resolved-view). Merge that first; this PR's own changes are the commits above it.What
The Nextflow target carries a step's step-only inputs, the ones its tool does not declare, whenever they matter: the step's
whenorscatterreads one, or one waits on another step's output. Each becomes avalinput of the process and stages nothing. Fordocs/cookbook/when.wicwith--target nextflow:when.wic:9:16: step 'echo': input 'enabled': the step's when reads an input its tool does not declare, which is not supportedA step-only input fed only by root workflow inputs and read by nothing stays out, as before, so every pipeline that compiles today keeps its
workflow.nf.Tools inherit the requirements and hints written outside them, on a workflow, a nested workflow and its call, and the step, as cwltool gives them: for each class the tool's own requirement, else the innermost inherited one, else the tool's own hint, else the innermost inherited hint. A workflow's
DockerRequirementhint now gives every tool without its own image one, and a step'sResourceRequirementinside a nested workflow wins over the root's. The four feature classes (ScatterFeatureRequirement,SubworkflowFeatureRequirement,MultipleInputFeatureRequirement,StepInputExpressionRequirement) only declare a feature and give a tool nothing.A
whenis typed by what its inputs deliver, through nested workflows, and per element for a scattered input; an input that may benullis read only besidenull, since CWL fails awhenthat givesnull.The model gains
NfProcess.step_only_inputsandNfPort.scalar_type, both last and defaulted. A model built by hand as in 0.7.0 still builds: ascalar_typeleft out is filled from the port's connection, and only a given one that disagrees is refused.import_nextflowreads both.What changes in generated files
nextflow_workflow.jsonsays"schema_version": 21and has ascalar_typekey on each port.workflow.nfand the other files are unchanged for every pipeline that compiled before. Emitted CWL is unchanged.Deleted
whenreads or a step output feeds: they are carried instead. A scatter over a step-only input stays refused, since no process port receives its element.Why
CWL lets a step bind inputs its tool does not declare and put requirements outside the tool, and ordinary workflows such as the cookbook's
when.wicdo both; the Nextflow target refused them.