Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27165

Calibration ingestion produces registry where butler cannot find matching calib product

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler, pipe_tasks
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Ingesting a gen2 repository from ap_verify_hits2015 produces a repository where IsrTask fails for certain visits. Failing visit number e.g. 412074, passing visit numbers e.g. 419802 or 412520.

      Thanks for the investigation by Meredith Rawls and Krzysztof Findeisen, the problem is thought to be overlapping validity ranges in the calibration registry, causing the butler not finding any suitable bias products for certain dates. This may be related to DM-26171 (repositories ingested earlier than this ticket seem to work fine).

       

      The topic was discussed on slack #uw-dm on 2020-10-13, here.

       

      Steps to reproduce:

       

      setup -jr ap_verify_hits2015/
       
      ingest_dataset.py --dataset HiTS2015 --gen2 --output temp/
       
      runIsr.py temp/ingested/ --calib temp/calibingested/ --id visit=412074 ccdnum=25 --rerun tempisr
      

       

       

       

       

        Attachments

          Issue Links

            Activity

            Hide
            czw Christopher Waters added a comment - - edited

            If I'm remembering my tests correctly from yesterday, the issue wasn't that no calibration was available, it was that two calibrations were returned, causing a "no unique dataset" error.  Removing the +1 day validity range made them unique again.

            I've updated the dataset-level configurations as indicated.  I believe the pipeline built calibs should work the same as on other cameras, so `incrementValidEnd = True` should be correct there.

            I'm going to put this into review, and am happy to revisit this if there are unexpected problems with the pipeline built calibs.

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline

             

            Show
            czw Christopher Waters added a comment - - edited If I'm remembering my tests correctly from yesterday, the issue wasn't that no calibration was available, it was that two calibrations were returned, causing a "no unique dataset" error.  Removing the +1 day validity range made them unique again. I've updated the dataset-level configurations as indicated.  I believe the pipeline built calibs should work the same as on other cameras, so `incrementValidEnd = True` should be correct there. I'm going to put this into review, and am happy to revisit this if there are unexpected problems with the pipeline built calibs. https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline  
            Hide
            gkovacs Gabor Kovacs added a comment - - edited

            This comment in DM-26171 suggests that ValidEnd_{i} == ValidStart_{i+1} is the correct approach while `incrementValidEnd == False` enforces ValidStart_{i+1} == ValidEnd_{i} + 1. DM-27018 says that even a ==1 gap should be interpreted as no gap. I could not get to a conclusion about the correct validity range convention...

            Could you please add an explanation to fixSubsetValidity docstring why and when we need to follow one or the other convention? (i.e. Why did the DM-26171 change actually give overlapping validity ranges in case of the community pipeline calibs in ap_verify_hits2015 ?) If possible, please also add a one line "instruction" to the `incrementValidEnd` config description, e.g.  "Set to True, if ... . See fixSubsetValidity docs for details."

            Of course, for ap_verify_hits2015 this fixes the problem raised in this ticket.

             

            Show
            gkovacs Gabor Kovacs added a comment - - edited This comment in DM-26171 suggests that ValidEnd_{i} == ValidStart_{i+1} is the correct approach while `incrementValidEnd == False` enforces ValidStart_{i+1} == ValidEnd_{i} + 1. DM-27018 says that even a ==1 gap should be interpreted as no gap. I could not get to a conclusion about the correct validity range convention... Could you please add an explanation to fixSubsetValidity docstring why and when we need to follow one or the other convention? (i.e. Why did the DM-26171 change actually give overlapping validity ranges in case of the community pipeline calibs in ap_verify_hits2015 ?) If possible, please also add a one line "instruction" to the `incrementValidEnd` config description, e.g.  "Set to True, if ... . See fixSubsetValidity docs for details." Of course, for ap_verify_hits2015 this fixes the problem raised in this ticket.  
            Hide
            gkovacs Gabor Kovacs added a comment -

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline

            The CI link in the comment above actually points to build 32863 that failed. 32864 is OK.

            Show
            gkovacs Gabor Kovacs added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline The CI link in the comment above actually points to build 32863 that failed. 32864 is OK.
            Hide
            czw Christopher Waters added a comment - - edited

            I've updated the docstring, after investigating that seems to be related to whether the calibDate value is set (which is true for the stack generated calibrations, but community produced calibrations use the DATE-OBS value instead).

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32896/pipeline/

            Show
            czw Christopher Waters added a comment - - edited I've updated the docstring, after investigating that seems to be related to whether the calibDate value is set (which is true for the stack generated calibrations, but community produced calibrations use the DATE-OBS value instead). https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32896/pipeline/
            Hide
            gkovacs Gabor Kovacs added a comment -

            Everything looks good. Thanks.

            Show
            gkovacs Gabor Kovacs added a comment - Everything looks good. Thanks.

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              gkovacs Gabor Kovacs
              Reviewers:
              Gabor Kovacs
              Watchers:
              Christopher Waters, Gabor Kovacs, Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.