Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31986

"filter label mismatch" on loading cp_pipe calibration

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe
    • Labels:
      None
    • Story Points:
      4
    • Epic Link:
    • Team:
      Data Release Production

      Description

      Flats and Sky frames warn about a "filter label mismatch" on loading the files.  This is likely a result of something not being set properly during generation, and should be fixed.

      Update (from Lee Kelvin), specifically, when trying to run a command such as this:

      sky1 = butler.get('sky', dataId={'visit': 1254, 'detector': 1}, instrument='HSC',
                        collections='u/lskelvin/sky/iAmpOffsetSky')
      

      I get messages such as this:

      /software/lsstsw/stack_20210813/stack/miniconda3-py38_4.9.2-
      0.7.0/Linux64/obs_base/22.0.1-22-
      gc07a94d+a112d84a14/python/lsst/obs/base/formatters/fitsExposure.py:635: UserWarning: 
      Reading file:///repo/main/u/lskelvin/sky/iAmpOffsetSky/sky/i/HSC-I/sky_HSC_i_HSC-
      I_1_54_u_lskelvin_sky_iAmpOffsetSky.fits with data ID {instrument: 'HSC', detector: 1, 
      physical_filter: 'HSC-I', ...}: filter label mismatch (file is None, data ID is 
      FilterLabel(band="i", physical="HSC-I")).  This is probably a bug in the code that 
      produced it.
        warnings.warn(f"Reading {self.fileDescriptor.location} with data ID {self.dataId}: "
      

        Attachments

        1. sky-ticket.log
          1.62 MB
        2. sky-ticket4.log
          239 kB

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment - - edited

            This happens when you try to read old files that were created before we supported FilterLabel. If these are newly-created files then there is clearly a problem with FilterLabel propagation.

            Show
            tjenness Tim Jenness added a comment - - edited This happens when you try to read old files that were created before we supported FilterLabel. If these are newly-created files then there is clearly a problem with FilterLabel propagation.
            Hide
            lskelvin Lee Kelvin added a comment -

            Can confirm that this is still an issue with sky frames created more recently. A recent run of $CP_PIPE_DIR/pipelines/cpSky.yaml on visit 7112 detector 1 produces this output:

            sky1 = butler.get('sky', dataId={'visit': 7112, 'detector': 1}, instrument='HSC',
                              collections='u/lskelvin/DM-32378/corrected-HSC-r-2014-09-22')
            

            /software/lsstsw/stack_20220125/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/obs_base/gcaa7f91c06+32dde6e53f/python/lsst/obs/base/formatters/fitsExposure.py:651: UserWarning: Reading file:///repo/main/u/lskelvin/DM-32378/corrected-HSC-r-2014-09-22/20220205T160340Z/sky/r/HSC-R/sky_HSC_r_HSC-R_1_54_u_lskelvin_DM-32378_corrected-HSC-r-2014-09-22_20220205T160340Z.fits with data ID {instrument: 'HSC', detector: 1, physical_filter: 'HSC-R', ...}: filter label mismatch (file is None, data ID is FilterLabel(band="r", physical="HSC-R")).  This is probably a bug in the code that produced it.
              warnings.warn(
            

            This is using w_2022_06 of lsst_distrib.

            Show
            lskelvin Lee Kelvin added a comment - Can confirm that this is still an issue with sky frames created more recently. A recent run of $CP_PIPE_DIR/pipelines/cpSky.yaml on visit 7112 detector 1 produces this output: sky1 = butler.get('sky', dataId={'visit': 7112, 'detector': 1}, instrument='HSC', collections='u/lskelvin/DM-32378/corrected-HSC-r-2014-09-22') /software/lsstsw/stack_20220125/stack/miniconda3-py38_4.9.2-1.0.0/Linux64/obs_base/gcaa7f91c06+32dde6e53f/python/lsst/obs/base/formatters/fitsExposure.py:651: UserWarning: Reading file:///repo/main/u/lskelvin/DM-32378/corrected-HSC-r-2014-09-22/20220205T160340Z/sky/r/HSC-R/sky_HSC_r_HSC-R_1_54_u_lskelvin_DM-32378_corrected-HSC-r-2014-09-22_20220205T160340Z.fits with data ID {instrument: 'HSC', detector: 1, physical_filter: 'HSC-R', ...}: filter label mismatch (file is None, data ID is FilterLabel(band="r", physical="HSC-R")). This is probably a bug in the code that produced it. warnings.warn( This is using w_2022_06 of lsst_distrib .
            Hide
            czw Christopher Waters added a comment -

            I've added a method to propagate the filterLabel, and confirmed that those warnings are removed in ci_cpp_gen3 with the new code.

            Show
            czw Christopher Waters added a comment - I've added a method to propagate the filterLabel, and confirmed that those warnings are removed in ci_cpp_gen3 with the new code.
            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks Chris. I ran a quick test today, and unfortunately I'm still seeing this error cropping up when using this ticket branch. I tried to run the $CP_PIPE_DIR/pipelines/cpSky.yaml pipeline, which contains the following tasks: isr, cpSkyImage, cpSkyScaleMeasure, cpSkySubtractBackground and cpSkyCombine. Specifically, I'm running this:

            pipetask --long-log run --register-dataset-types -j 12 \
            -b /repo/main --instrument lsst.obs.subaru.HyperSuprimeCam \
            -i HSC/defaults \
            -o u/lskelvin/scratch/DM-31986/sky-ticket \
            -p $CP_PIPE_DIR/pipelines/cpSky.yaml \
            -d "instrument='HSC' AND exposure.observation_type='science' AND detector!=9 AND exposure.exposure_time>30.0 AND exposure.target_name='SSP-Wide' AND physical_filter='HSC-I' AND exposure.day_obs>=19700101 AND exposure.day_obs<20141023 AND visit IN (1254,1258)" \
            -c isr:doAmpOffset=True
            

            After running that, and allowing the 779 quanta to complete, I have a new sky frame available to access. I access it:

            sky_tkt = butler.get('sky', dataId=dict(visit=1254, detector=1), instrument='HSC',
                                 collections='u/lskelvin/scratch/DM-31986/sky-ticket')
            

            and still get the following error message:

            /software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/obs_base/gc18b5ea8f9+4bef655d58/python/lsst/obs/base/formatters/fitsExposure.py:654: UserWarning: Reading file:///repo/main/u/lskelvin/scratch/DM-31986/sky-ticket/20220607T205812Z/sky/i/HSC-I/sky_HSC_i_HSC-I_1_54_u_lskelvin_scratch_DM-31986_sky-ticket_20220607T205812Z.fits with data ID {instrument: 'HSC', detector: 1, physical_filter: 'HSC-I', ...}: filter label mismatch (file is None, data ID is FilterLabel(band="i", physical="HSC-I")).  This is probably a bug in the code that produced it.
              warnings.warn(
            

            For reference, I'm using lsst_distrib w23 with cp_pipe checked out on branch tickets/DM-31986. I attach my output logs to this ticket, for reference: sky-ticket.log. In there, I do see a whole bunch of error messages relating to the filter not being found - is this related?

            Filter not found in FringeTaskConfig.filters. Skipping fringe correction.
            

            Can you see anything I've missed here?

            Show
            lskelvin Lee Kelvin added a comment - Thanks Chris. I ran a quick test today, and unfortunately I'm still seeing this error cropping up when using this ticket branch. I tried to run the $CP_PIPE_DIR/pipelines/cpSky.yaml pipeline, which contains the following tasks: isr , cpSkyImage , cpSkyScaleMeasure , cpSkySubtractBackground and cpSkyCombine . Specifically, I'm running this: pipetask --long-log run --register-dataset-types -j 12 \ -b /repo/main --instrument lsst.obs.subaru.HyperSuprimeCam \ -i HSC/defaults \ -o u/lskelvin/scratch/DM-31986/sky-ticket \ -p $CP_PIPE_DIR/pipelines/cpSky.yaml \ -d "instrument='HSC' AND exposure.observation_type='science' AND detector!=9 AND exposure.exposure_time>30.0 AND exposure.target_name='SSP-Wide' AND physical_filter='HSC-I' AND exposure.day_obs>=19700101 AND exposure.day_obs<20141023 AND visit IN (1254,1258)" \ -c isr:doAmpOffset=True After running that, and allowing the 779 quanta to complete, I have a new sky frame available to access. I access it: sky_tkt = butler.get( 'sky' , dataId = dict (visit = 1254 , detector = 1 ), instrument = 'HSC' , collections = 'u/lskelvin/scratch/DM-31986/sky-ticket' ) and still get the following error message: /software/lsstsw/stack_20220525/stack/miniconda3-py38_4.9.2-4.0.0/Linux64/obs_base/gc18b5ea8f9+4bef655d58/python/lsst/obs/base/formatters/fitsExposure.py:654: UserWarning: Reading file:///repo/main/u/lskelvin/scratch/DM-31986/sky-ticket/20220607T205812Z/sky/i/HSC-I/sky_HSC_i_HSC-I_1_54_u_lskelvin_scratch_DM-31986_sky-ticket_20220607T205812Z.fits with data ID {instrument: 'HSC', detector: 1, physical_filter: 'HSC-I', ...}: filter label mismatch (file is None, data ID is FilterLabel(band="i", physical="HSC-I")). This is probably a bug in the code that produced it. warnings.warn( For reference, I'm using lsst_distrib w23 with cp_pipe checked out on branch tickets/ DM-31986 . I attach my output logs to this ticket, for reference: sky-ticket.log . In there, I do see a whole bunch of error messages relating to the filter not being found - is this related? Filter not found in FringeTaskConfig.filters. Skipping fringe correction. Can you see anything I've missed here?
            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks Chris for our discussions on Slack and pushing the commits which address the above! I've re-run my test, with the output log here: sky-ticket4.log. All seems to be well, and I confirm that the sky image outputs on this ticket are identical to the prior vanilla image outputs. The output collection is here: 'u/lskelvin/scratch/DM-31986/sky-ticket4'. FYI, for testing of this ticket, I needed to set up the following:

            • lsst_distrib w23
            • cp_pipe tickets/DM-31986
            • afw main
            • ip_isr main
            • obs_base main
            • obs_lsst main
            • obs_subaru main

            Assuming Jenkins doesn't complain, then I think this is good to merge to me. Thanks Chris, this is great!

            Show
            lskelvin Lee Kelvin added a comment - Thanks Chris for our discussions on Slack and pushing the commits which address the above! I've re-run my test, with the output log here: sky-ticket4.log . All seems to be well, and I confirm that the sky image outputs on this ticket are identical to the prior vanilla image outputs. The output collection is here: 'u/lskelvin/scratch/ DM-31986 /sky-ticket4'. FYI, for testing of this ticket, I needed to set up the following: lsst_distrib w23 cp_pipe tickets/ DM-31986 afw main ip_isr main obs_base main obs_lsst main obs_subaru main Assuming Jenkins doesn't complain, then I think this is good to merge to me. Thanks Chris, this is great!
            Show
            czw Christopher Waters added a comment - Jenkins is happy:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36796/pipeline

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              czw Christopher Waters
              Reviewers:
              Lee Kelvin
              Watchers:
              Christopher Waters, Lee Kelvin, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.