Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22534

Implement cpSkyTask

    XMLWordPrintable

Details

    • Improvement
    • Status: Done
    • Resolution: Done
    • None
    • cp_pipe
    • None

    Description

      The gen3 conversion of the sky correction generation code doesn't seem to scale things correctly. Check the algorithm, and update the code.

      Attachments

        Issue Links

          Activity

            The attached notebook (I can create a test directory on lsst-dev if you'd like to browse it live) contains the some statistics for two matched input runs.  There are statistics presented per detector for the gen2 versions, as well as for the gen2 - gen3 difference.  The difference values are far smaller (generally ~1e-3-1e-4) than the gen2 means/std values.  Looking them over, it seems like the largest differences are on the perimeter detectors.  The difference images show some structure, but they largely look like small differences in the values used for interpolation from binned to full backgrounds.

            czw Christopher Waters added a comment - The attached notebook (I can create a test directory on lsst-dev if you'd like to browse it live) contains the some statistics for two matched input runs.  There are statistics presented per detector for the gen2 versions, as well as for the gen2 - gen3 difference.  The difference values are far smaller (generally ~1e-3-1e-4) than the gen2 means/std values.  Looking them over, it seems like the largest differences are on the perimeter detectors.  The difference images show some structure, but they largely look like small differences in the values used for interpolation from binned to full backgrounds.
            lskelvin Lee Kelvin added a comment - - edited

            This ticket is in great shape, thank you Chris for getting everything together. This ticket allows for sky frames to be produced in the gen3 framework, starting from raw data and processing through IsrTask, CpSkyImageTask, CpSkyScaleMeasureTask, CpSkySubtractBackgroundTask, and CpSkyCombineTask (graph flowchart attached above).

            I've tested the pipeline using this command:

            pipetask --long-log run --register-dataset-types -j 12 \
            -b $REPO --instrument lsst.obs.subaru.HyperSuprimeCam \
            -i HSC/defaults \
            --output-run u/lskelvin/cpSkyTask-test \
            -p $CP_PIPE_DIR/pipelines/cpSky.yaml \
            -d "instrument='HSC' AND exposure=1228"
            

            Processing times for this single exposure were as follows:

            QuantumGraph contains 449 quanta for 5 tasks.
             
            task                        time                  pass     fail     incomp
            --------------------------------------------------------------------------------
            isr                         2050.31s (~65%)       112      0        0     
            cpSkyImage                  906.92s (~29%)        112      0        0     
            cpSkySubtractBackground     122.16s (~4%)         112      0        0     
            cpSkyCombine                81.87s (~3%)          112      0        0     
            cpSkyScaleMeasure           0.65s (~0%)           1        0        0     
            --------------------------------------------------------------------------------
            total                       3161.90s              449      0        0     
             
            Executed 449 quanta out of a total of 449 quanta (100%).
            

            When reading in an example final output sky frame, the following warning is printed in Python:

            /software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-0.7.0/Linux64/obs_base/22.0.1-18-g3ad3863+00a0915085/python/lsst/obs/base/formatters/fitsExposure.py:635: UserWarning: Reading file:///repo/main/u/lskelvin/cpSkyTask-test/sky/i/HSC-I/sky_HSC_i_HSC-I_1_54_u_lskelvin_cpSkyTask-test.fits with data ID {instrument: 'HSC', detector: 1, physical_filter: 'HSC-I', ...}: filter label mismatch (file is None, data ID is FilterLabel(band="i", physical="HSC-I")).  This is probably a bug in the code that produced it.
              warnings.warn(f"Reading {self.fileDescriptor.location} with data ID {self.dataId}: "
            

            This warning doesn't prevent the sky frame from being loaded however, so is not a critical failure. Having chatted with Chris about this, it's possible that the necessary data was removed during one of the intermediate processing steps. A future ticket will address this issue.

            I've had some issues trying to construct a full focal plane mosaic image, with VisualizeMosaicExpTask failing with the error lsst::pex::exceptions::LengthError: 'Dimension mismatch: 32x65 v. 1x2'. I don't think this is related to issues with this ticket however, so wouldn't want to hold it up any further.

            This is a very nicely written ticket - thank you Chris for addressing the issues I've ran into during testing and the comments on GH. Assuming Jenkins doesn't complain, then this looks good to merge to me.

            lskelvin Lee Kelvin added a comment - - edited This ticket is in great shape, thank you Chris for getting everything together. This ticket allows for sky frames to be produced in the gen3 framework, starting from raw data and processing through IsrTask , CpSkyImageTask , CpSkyScaleMeasureTask , CpSkySubtractBackgroundTask , and CpSkyCombineTask (graph flowchart attached above). I've tested the pipeline using this command: pipetask --long-log run --register-dataset-types -j 12 \ -b $REPO --instrument lsst.obs.subaru.HyperSuprimeCam \ -i HSC/defaults \ --output-run u/lskelvin/cpSkyTask-test \ -p $CP_PIPE_DIR/pipelines/cpSky.yaml \ -d "instrument='HSC' AND exposure=1228" Processing times for this single exposure were as follows: QuantumGraph contains 449 quanta for 5 tasks.   task time pass fail incomp -------------------------------------------------------------------------------- isr 2050.31s (~65%) 112 0 0 cpSkyImage 906.92s (~29%) 112 0 0 cpSkySubtractBackground 122.16s (~4%) 112 0 0 cpSkyCombine 81.87s (~3%) 112 0 0 cpSkyScaleMeasure 0.65s (~0%) 1 0 0 -------------------------------------------------------------------------------- total 3161.90s 449 0 0   Executed 449 quanta out of a total of 449 quanta (100%). When reading in an example final output sky frame, the following warning is printed in Python: /software/lsstsw/stack_20210813/stack/miniconda3-py38_4. 9.2 - 0.7 . 0 /Linux64/obs_base/ 22.0 . 1 - 18 -g3ad3863+00a0915085/python/lsst/obs/base/formatters/fitsExposure.py: 635 : UserWarning: Reading file: ///repo/main/u/lskelvin/cpSkyTask-test/sky/i/HSC-I/sky_HSC_i_HSC-I_1_54_u_lskelvin_cpSkyTask-test.fits with data ID {instrument: 'HSC', detector: 1, physical_filter: 'HSC-I', ...}: filter label mismatch (file is None, data ID is FilterLabel(band="i", physical="HSC-I")). This is probably a bug in the code that produced it. warnings.warn(f "Reading {self.fileDescriptor.location} with data ID {self.dataId}: " This warning doesn't prevent the sky frame from being loaded however, so is not a critical failure. Having chatted with Chris about this, it's possible that the necessary data was removed during one of the intermediate processing steps. A future ticket will address this issue. I've had some issues trying to construct a full focal plane mosaic image, with VisualizeMosaicExpTask failing with the error lsst::pex::exceptions::LengthError: 'Dimension mismatch: 32x65 v. 1x2' . I don't think this is related to issues with this ticket however, so wouldn't want to hold it up any further. This is a very nicely written ticket - thank you Chris for addressing the issues I've ran into during testing and the comments on GH. Assuming Jenkins doesn't complain, then this looks good to merge to me.

            People

              czw Christopher Waters
              czw Christopher Waters
              Lee Kelvin
              Christopher Waters, Lauren MacArthur, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.