(following discussion with Sophie on Weds, added myself as a reviewer) - this ticket looks great, adding nice gen3 functionality to SSI processing. I only have 1 minor comment on the pipe_tasks PR regarding the WCS pixel scale offset factor, and whether or not this factor should be configurable. If you think a factor of 2 will always be sufficient to catch this issue, then that looks fine to me, with no extra code required.
I think some amount (most?) of the obs_subaru code was already merged to master previously
, so I haven't looked at that specifically here. However, as noted above, I have been using the pipeline on this ticket branch without issue, and the outputs look good to me. I note that ticket branch DM-31491 in obs_subaru still has some additional differences to master in the pipelines/DRPFakes.yaml pipeline - should this also be in a PR?
Otherwise, this looks great. Assuming Jenkins doesn't complain, then this should be good to merge. Nicely done.
I've been testing this pipeline over the last week or so using single-Sersic extended sources supplied by the LSST:UK LSB working group, and all seems to be working as expected. For reference, prior to running this pipeline, it is necessary to ingest the input SSI catalogue (e.g., in FITS format) into the repo. The commands I've used to perform this in Python are:
# which user will be running these data?
# synthetic source input catalogue filename, injection tract, and RUN collection name
# print catalogue and run information
#SSI input catalogue: '/project/lskelvin/ssi/gen3/lsstuk_lsb_sersic_gen3_single.fits'
#SSI input RUN collection: 'u/lskelvin/ssiInputs/lsstuk_lsb_sersic_gen3_single'
# read in a FITS catalogue
# alternatively, read in a parquet catalogue, etc...
#ssi_cat = pd.read_parquet(ssi_cat_filename)
# set up writeable butler and SSI RUN collection name
# define synthetic source dataset type
# register synthetic source dataset type and RUN collection
writeable_butler.registry.registerDatasetType(ssi_dataset_type)
# put the SSI catalogue into the repo
Once this is in place, running the pipeline is simply:
pipetask --long-log run --register-dataset-types -j 12 \
-b /repo/main --instrument lsst.obs.subaru.HyperSuprimeCam \
-i $WEEKLY_INPUT_COLL,$SSI_INPUT_COLL \
-o $SSI_OUTPUT_COLL \
-p $OBS_SUBARU_DIR/pipelines/DRPFakes.yaml \
-d "$INSTSKY AND $SCIDETS AND visit IN $GAMA_I"
where:
WEEKLY_INPUT_COLL="HSC/runs/RC2/w_2021_38/DM-31795"
SSI_INPUT_COLL="u/USER/ssiInputs/lsstuk_lsb_sersic_gen3_single"
SSI_OUTPUT_COLL="u/USER/ssiOutputs/lsstuk_lsb_sersic_gen3_single"
INSTSKY="instrument='HSC' AND skymap='hsc_rings_v1'"
SCIDETS="detector!=9 and detector.purpose='SCIENCE'"
I've attached example before/after/diff images at both the single frame level and the coadd level to this ticket, for reference.