Expressing workflows

Work in progress

The document form is not final: the dirac: hint schema of DX-ADR-007 is still being written, so each guide carries a placeholder listing what its document needs.

These guides show how to express common workflows as workgraphs within the DiracX transformation system. Each starts from a goal, shows the workgraph as a running model, marks where the CWL document goes, and describes what the system does with it at run time, in the terms of How the Transformation System works.

• Basic simulation

  ---

  Seeds in, simulated, reconstructed and merged events out.

• Scouting simulation

  ---

  The same workgraph run on a sample first, with approving actions and a sign-off.

• Removing intermediates

  ---

  Removal transformations for the files a downstream step has consumed.

• Replicating the output

  ---

  A replication of the workgraph's deliverable, started on approval.

• Processing data from a query

  ---

  A bookkeeping query as input, run ranges extended over time, and staging from tape.

• Adding ancestor files

  ---

  Jobs that need a second file found by lookup, the RDST and RAW case.

• Collecting histograms

  ---

  Small files from every job, merged by a step, by a finalizing action, or both.

• Splitting an output

  ---

  One output consumed by two steps, and a removal that waits for both.

• Joining two outputs

  ---

  One step consuming the outputs of two others for the same source file.