Christopher Waters added a comment - 07/May/21 8:50 PM The attached notebook (I can create a test directory on lsst-dev if you'd like to browse it live) contains the some statistics for two matched input runs.  There are statistics presented per detector for the gen2 versions, as well as for the gen2 - gen3 difference.  The difference values are far smaller (generally ~1e-3-1e-4) than the gen2 means/std values.  Looking them over, it seems like the largest differences are on the perimeter detectors.  The difference images show some structure, but they largely look like small differences in the values used for interpolation from binned to full backgrounds.

Lee Kelvin added a comment - 21/Sep/21 10:36 PM - edited This ticket is in great shape, thank you Chris for getting everything together. This ticket allows for sky frames to be produced in the gen3 framework, starting from raw data and processing through IsrTask , CpSkyImageTask , CpSkyScaleMeasureTask , CpSkySubtractBackgroundTask , and CpSkyCombineTask (graph flowchart attached above). I've tested the pipeline using this command: pipetask --long-log run --register-dataset-types -j 12 \ -b $REPO --instrument lsst.obs.subaru.HyperSuprimeCam \ -i HSC/defaults \ --output-run u/lskelvin/cpSkyTask-test \ -p $CP_PIPE_DIR/pipelines/cpSky.yaml \ -d "instrument='HSC' AND exposure=1228" Processing times for this single exposure were as follows: QuantumGraph contains 449 quanta for 5 tasks.   task time pass fail incomp -------------------------------------------------------------------------------- isr 2050.31s (~65%) 112 0 0 cpSkyImage 906.92s (~29%) 112 0 0 cpSkySubtractBackground 122.16s (~4%) 112 0 0 cpSkyCombine 81.87s (~3%) 112 0 0 cpSkyScaleMeasure 0.65s (~0%) 1 0 0 -------------------------------------------------------------------------------- total 3161.90s 449 0 0   Executed 449 quanta out of a total of 449 quanta (100%). When reading in an example final output sky frame, the following warning is printed in Python: /software/lsstsw/stack_20210813/stack/miniconda3-py38_4. 9.2 - 0.7 . 0 /Linux64/obs_base/ 22.0 . 1 - 18 -g3ad3863+00a0915085/python/lsst/obs/base/formatters/fitsExposure.py: 635 : UserWarning: Reading file: ///repo/main/u/lskelvin/cpSkyTask-test/sky/i/HSC-I/sky_HSC_i_HSC-I_1_54_u_lskelvin_cpSkyTask-test.fits with data ID {instrument: 'HSC', detector: 1, physical_filter: 'HSC-I', ...}: filter label mismatch (file is None, data ID is FilterLabel(band="i", physical="HSC-I")). This is probably a bug in the code that produced it. warnings.warn(f "Reading {self.fileDescriptor.location} with data ID {self.dataId}: " This warning doesn't prevent the sky frame from being loaded however, so is not a critical failure. Having chatted with Chris about this, it's possible that the necessary data was removed during one of the intermediate processing steps. A future ticket will address this issue. I've had some issues trying to construct a full focal plane mosaic image, with VisualizeMosaicExpTask failing with the error lsst::pex::exceptions::LengthError: 'Dimension mismatch: 32x65 v. 1x2' . I don't think this is related to issues with this ticket however, so wouldn't want to hold it up any further. This is a very nicely written ticket - thank you Chris for addressing the issues I've ran into during testing and the comments on GH. Assuming Jenkins doesn't complain, then this looks good to merge to me.