How to Write a Workflow

A workflow (or simulation block) is a self-contained Python script that produces all raw and calibration FITS files for a particular instrument mode. This guide walks you through creating one from scratch.

Overview

Every workflow does three things:

1. Lists the YAML template files that define which observations to simulate.

2. Provides a parameter dictionary controlling output location, calibration depth, parallelism, etc.

3. Calls runSimulationBlock(yamlFiles, params) to execute everything.

Quick Start

Create a file Simulations/python/myWorkflow.py:

import runSimulationBlock as rs

if __name__ == "__main__":

    params = {
        "outputDir":  "output/myWorkflow",
        "small":      False,
        "doStatic":   True,
        "doCalib":    2,
        "sequence":   True,
        "startMJD":   "2027-03-15 00:00:00",
        "calibFile":  None,
        "nCores":     8,
        "testRun":    False,
    }

    yamlFiles = [
        "YAML/scienceLM.yaml",
        "YAML/stdLM.yaml",
        "YAML/distortionLM.yaml",
        "YAML/detlinLM.yaml",
    ]

    rs.runSimulationBlock(yamlFiles, params)

Run it from the Simulations/ directory:

python python/myWorkflow.py

Parameter Reference

Parameter	Default	Description
outputDir	(required)	Directory where FITS files are written. Created automatically.
small	False	If True, generate 32x32 pixel images instead of full-size. Useful for quick testing and CI.
doStatic	False	If True, generate static calibration prototype files (pinhole table, laser table, distortion solution, etc.) alongside the raw files.
doCalib	0	Number of dark and flat frames to auto-generate per unique DIT/filter combination found in the templates. Set to 0 to skip.
sequence	True	If True, observations get sequential, incrementing timestamps (MJD-OBS). If False, each template uses the date from the YAML dateobs field.
startMJD	(required)	ISO-format start date for the observation sequence ("YYYY-MM-DD HH:MM:SS").
calibFile	None	Path to a separate YAML file defining calibration parameters. Usually left as None to use defaults from simulationDefinitions.py.
nCores	8	Number of CPU cores for parallel execution. The framework will use at most os.cpu_count() - 1.
testRun	False	If True, run through all validation and setup but skip actual ScopeSim simulation. Useful for checking YAML templates.

Additional per-mode parameters can be added to the dict and will be forwarded:

• slit_name – slit identifier for LSS modes (e.g. "C-38_1").

Writing YAML Templates

YAML files live in Simulations/YAML/ and define one or more observation templates. Each template is a YAML mapping with a unique label as key:

MY_UNIQUE_LABEL:
  do.catg: LM_IMAGE_SCI_RAW
  mode: "img_lm"
  source:
    name: star_field
    kwargs: {}
  properties:
    dit: 0.25
    ndit: 4
    filter_name: "Lp"
    catg: "SCIENCE"
    tech: "IMAGE,LM"
    type: "OBJECT"
    nObs: 5

A single YAML file can contain multiple templates. They are executed in order.

Template Fields

do.catg (required)

The DO category name that becomes the FITS file-name prefix, e.g. LM_IMAGE_SCI_RAW.

mode (required)

The ScopeSim instrument mode:

Mode	Description
img_lm	LM-band imaging
img_n	N-band imaging
lss_l	L-band long-slit spectroscopy
lss_m	M-band long-slit spectroscopy
lss_n	N-band long-slit spectroscopy
lms	IFU (integral field unit)

source (required)

The astronomical or calibration source to use. Has two sub-fields:

• name – one of the source functions defined in sources.py

• kwargs – keyword arguments passed to the source function (can be empty {})

Available sources:

Source	Description
empty_sky	Blank sky (for sky / dark frames)
flat_field	WCU lamp flat
star_field	Fixed random 20-star field
star_sky1	25-star sky field (calibration set 1)
star_sky2	25-star sky field (calibration set 2)
calib_star	Single centred point source
simple_gal	Elliptical galaxy (SED + morphology)
simple_gal1	Elliptical galaxy (variant)
pinhole_mask	WCU pinhole mask
laser_spectrum_lm	WCU laser spectrum, LM band
laser_spectrum_n	WCU laser spectrum, N band

properties (required)

Observation parameters:

dit (float)

Detector integration time in seconds.

ndit (int)

Number of integrations per exposure.

filter_name (str)

Filter wheel selection. Common values:

• LM band: Lp, short-L, Mp, L_spec, M_spec, HCI_L_short, HCI_L_long, HCI_M, Br_alpha, PAH_3.3, CO_1-0_ice, H2O-ice, IB_4.05, open, closed

• N band: N1, N2, N3, N_spec, PAH_8.6, PAH_11.25, Ne_II, S_IV, open, closed

ndfilter_name (str, optional)

Neutral density filter: open, ND_OD1 … ND_OD5.

catg (str)

DPR category: SCIENCE, CALIB, or TECHNICAL.

tech (str)

DPR technique: IMAGE,LM, IMAGE,N, LMS, LSS,LM, LSS,N, PUP,M, PUP,N, APP,LM, RAVC,LM, RAVC,IFU.

type (str)

DPR type: OBJECT, SKY, STD, DARK, FLAT,LAMP, FLAT,TWILIGHT, DETLIN, DISTORTION, WAVE, PSF,OFFAXIS, PUPIL, CHOPHOME, DARK,WCUOFF, SLITLOSS, PERSIST.

nObs (int)

How many exposures of this template to generate.

dateobs (str, optional)

Explicit date in YYYY-MM-DD HH:MM:SS format. Only used when sequence=False.

tplname (str, optional)

ESO template name string for the FITS header.

wcu (optional)

Wavefront Control Unit parameters for internal calibrations:

wcu:
  current_lamp: "bb"          # bb (blackbody) or laser
  current_fpmask: "open"      # open, grid_lm, grid_n
  bb_aperture: 0.5            # blackbody aperture size
  bb_temp: 8000               # blackbody temperature (K)
  is_temp: 300                # integrating sphere temperature
  wcu_temp: 300               # WCU enclosure temperature
  xshift: 0                   # pinhole mask X offset
  yshift: 0                   # pinhole mask Y offset

Adding a New Source

Sources are defined in Simulations/python/sources.py. Each source is a function that returns a ScopeSim-compatible source object.

Example – adding a bright point source:

def bright_star(**kwargs):
    """Single bright star at field centre."""
    return sim_tp.star(
        flux=1e4,        # photons/s/m2
        filter_curve="Ks",
        spec_type="A0V",
    )

After adding the function, you can reference it in YAML templates:

source:
  name: bright_star
  kwargs: {}

Creating a Small Variant

For CI testing, create a _small.py variant of your script that sets small=True. This generates 32x32 pixel images that run quickly:

# myWorkflow_small.py
import runSimulationBlock as rs

if __name__ == "__main__":

    params = {
        "outputDir":  "output/myWorkflow",
        "small":      True,         # <-- small images
        "doStatic":   True,
        "doCalib":    1,            # fewer calibrations
        "sequence":   True,
        "startMJD":   "2027-03-15 00:00:00",
        "calibFile":  None,
        "nCores":     4,
        "testRun":    False,
    }

    yamlFiles = [
        "YAML/scienceLM.yaml",
        "YAML/stdLM.yaml",
    ]

    rs.runSimulationBlock(yamlFiles, params)

Then add a call to it in testAll.py so CI picks it up.

Tips

• Reuse existing YAML files where possible. Most calibration templates (darks, detector linearity, distortion) are shared across workflows.

• Start with a copy of the most similar existing workflow and modify it.

• Use testRun=True to validate your YAML files without running ScopeSim.

• Use small=True during development to iterate quickly.

• The doCalib mechanism inspects your templates and auto-generates matching darks and flats – you rarely need to write dark/flat YAML by hand.

• Run python/generateSummary.py output/myWorkflow after a run to get a CSV summary of all generated FITS files.