Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27165

Calibration ingestion produces registry where butler cannot find matching calib product

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • butler, pipe_tasks
    • None
    • 1
    • Data Release Production
    • No

    Description

      Ingesting a gen2 repository from ap_verify_hits2015 produces a repository where IsrTask fails for certain visits. Failing visit number e.g. 412074, passing visit numbers e.g. 419802 or 412520.

      Thanks for the investigation by mrawls and krzys, the problem is thought to be overlapping validity ranges in the calibration registry, causing the butler not finding any suitable bias products for certain dates. This may be related to DM-26171 (repositories ingested earlier than this ticket seem to work fine).

       

      The topic was discussed on slack #uw-dm on 2020-10-13, here.

       

      Steps to reproduce:

       

      setup -jr ap_verify_hits2015/
       
      ingest_dataset.py --dataset HiTS2015 --gen2 --output temp/
       
      runIsr.py temp/ingested/ --calib temp/calibingested/ --id visit=412074 ccdnum=25 --rerun tempisr
      

       

       

       

       

      Attachments

        Issue Links

          Activity

            czw Christopher Waters added a comment - - edited

            If I'm remembering my tests correctly from yesterday, the issue wasn't that no calibration was available, it was that two calibrations were returned, causing a "no unique dataset" error.  Removing the +1 day validity range made them unique again.

            I've updated the dataset-level configurations as indicated.  I believe the pipeline built calibs should work the same as on other cameras, so `incrementValidEnd = True` should be correct there.

            I'm going to put this into review, and am happy to revisit this if there are unexpected problems with the pipeline built calibs.

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline

             

            czw Christopher Waters added a comment - - edited If I'm remembering my tests correctly from yesterday, the issue wasn't that no calibration was available, it was that two calibrations were returned, causing a "no unique dataset" error.  Removing the +1 day validity range made them unique again. I've updated the dataset-level configurations as indicated.  I believe the pipeline built calibs should work the same as on other cameras, so `incrementValidEnd = True` should be correct there. I'm going to put this into review, and am happy to revisit this if there are unexpected problems with the pipeline built calibs. https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline  

            This comment in DM-26171 suggests that ValidEnd_{i} == ValidStart_{i+1} is the correct approach while `incrementValidEnd == False` enforces ValidStart_{i+1} == ValidEnd_{i} + 1. DM-27018 says that even a ==1 gap should be interpreted as no gap. I could not get to a conclusion about the correct validity range convention...

            Could you please add an explanation to fixSubsetValidity docstring why and when we need to follow one or the other convention? (i.e. Why did the DM-26171 change actually give overlapping validity ranges in case of the community pipeline calibs in ap_verify_hits2015 ?) If possible, please also add a one line "instruction" to the `incrementValidEnd` config description, e.g.  "Set to True, if ... . See fixSubsetValidity docs for details."

            Of course, for ap_verify_hits2015 this fixes the problem raised in this ticket.

             

            gkovacs Gabor Kovacs [X] (Inactive) added a comment - - edited This comment in DM-26171 suggests that ValidEnd_{i} == ValidStart_{i+1} is the correct approach while `incrementValidEnd == False` enforces ValidStart_{i+1} == ValidEnd_{i} + 1. DM-27018 says that even a ==1 gap should be interpreted as no gap. I could not get to a conclusion about the correct validity range convention... Could you please add an explanation to fixSubsetValidity docstring why and when we need to follow one or the other convention? (i.e. Why did the DM-26171 change actually give overlapping validity ranges in case of the community pipeline calibs in ap_verify_hits2015 ?) If possible, please also add a one line "instruction" to the `incrementValidEnd` config description, e.g.  "Set to True, if ... . See fixSubsetValidity docs for details." Of course, for ap_verify_hits2015 this fixes the problem raised in this ticket.  

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline

            The CI link in the comment above actually points to build 32863 that failed. 32864 is OK.

            gkovacs Gabor Kovacs [X] (Inactive) added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline The CI link in the comment above actually points to build 32863 that failed. 32864 is OK.
            czw Christopher Waters added a comment - - edited

            I've updated the docstring, after investigating that seems to be related to whether the calibDate value is set (which is true for the stack generated calibrations, but community produced calibrations use the DATE-OBS value instead).

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32896/pipeline/

            czw Christopher Waters added a comment - - edited I've updated the docstring, after investigating that seems to be related to whether the calibDate value is set (which is true for the stack generated calibrations, but community produced calibrations use the DATE-OBS value instead). https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32896/pipeline/

            Everything looks good. Thanks.

            gkovacs Gabor Kovacs [X] (Inactive) added a comment - Everything looks good. Thanks.

            People

              czw Christopher Waters
              gkovacs Gabor Kovacs [X] (Inactive)
              Gabor Kovacs [X] (Inactive)
              Christopher Waters, Gabor Kovacs [X] (Inactive), Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.