Over the last couple of weeks there has been significant effort to look at how `PipelineTask` classes can be used to execute the broad design goals presented in DMTN-141. This has required implementing three different tasks:
1) A task to create matched catalogs that can be used by other measurement tasks
2) The actual measurement task
3) And a task to aggregate the individual measurements into a single summary number
My contribution has been in the task to produce matched catalogs and in the very simple pipeline description.
The current pipeline runs in about 1 minute on the r-band visits in the ci_hsc_gen3 repository on a single core. This can be parallelized and has been shown to run in about 20 seconds on eight cores.
The quantum graph builder identifies 25 patches as quanta for the tasks (the measurements are at the patch level for this prototype). Though only 9 patches have data since the quantum building is conservative in terms of a boundary around in put datasets for which to select patches.
The code resides in this repository. If a build ci_hsc_gen3 repository is set up, the pipeline can be run with a call similar to the following:
pipetask run -j 1 -b "$CI_HSC_GEN3_DIR"/DATA/butler.yaml --register-dataset-types -p pipelines/metrics_pipeline.yaml -d "abstract_filter = 'r'" -o pipeTest -i shared/ci_hsc_output
Note that the tasks directory of the above github repository must be put on your PYTHONPATH by hand for things to work.