# Calibration ingestion produces registry where butler cannot find matching calib product

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
Data Release Production
• Urgent?:
No

#### Description

Ingesting a gen2 repository from ap_verify_hits2015 produces a repository where IsrTask fails for certain visits. Failing visit number e.g. 412074, passing visit numbers e.g. 419802 or 412520.

Thanks for the investigation by Meredith Rawls and Krzysztof Findeisen, the problem is thought to be overlapping validity ranges in the calibration registry, causing the butler not finding any suitable bias products for certain dates. This may be related to DM-26171 (repositories ingested earlier than this ticket seem to work fine).

The topic was discussed on slack #uw-dm on 2020-10-13, here.

Steps to reproduce:

 setup -jr ap_verify_hits2015/   ingest_dataset.py --dataset HiTS2015 --gen2 --output temp/   runIsr.py temp/ingested/ --calib temp/calibingested/ --id visit=412074 ccdnum=25 --rerun tempisr 

#### Attachments

1. DM-27165_log.txt
171 kB

#### Activity

Hide
Christopher Waters added a comment - - edited

If I'm remembering my tests correctly from yesterday, the issue wasn't that no calibration was available, it was that two calibrations were returned, causing a "no unique dataset" error.  Removing the +1 day validity range made them unique again.

I've updated the dataset-level configurations as indicated.  I believe the pipeline built calibs should work the same as on other cameras, so incrementValidEnd = True should be correct there.

I'm going to put this into review, and am happy to revisit this if there are unexpected problems with the pipeline built calibs.

https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline

Show
Christopher Waters added a comment - - edited If I'm remembering my tests correctly from yesterday, the issue wasn't that no calibration was available, it was that two calibrations were returned, causing a "no unique dataset" error.  Removing the +1 day validity range made them unique again. I've updated the dataset-level configurations as indicated.  I believe the pipeline built calibs should work the same as on other cameras, so incrementValidEnd = True should be correct there. I'm going to put this into review, and am happy to revisit this if there are unexpected problems with the pipeline built calibs. https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline
Hide
Gabor Kovacs added a comment - - edited

This comment in DM-26171 suggests that ValidEnd_{i} == ValidStart_{i+1} is the correct approach while incrementValidEnd == False enforces ValidStart_{i+1} == ValidEnd_{i} + 1. DM-27018 says that even a ==1 gap should be interpreted as no gap. I could not get to a conclusion about the correct validity range convention...

Could you please add an explanation to fixSubsetValidity docstring why and when we need to follow one or the other convention? (i.e. Why did the DM-26171 change actually give overlapping validity ranges in case of the community pipeline calibs in ap_verify_hits2015 ?) If possible, please also add a one line "instruction" to the incrementValidEnd config description, e.g.  "Set to True, if ... . See fixSubsetValidity docs for details."

Of course, for ap_verify_hits2015 this fixes the problem raised in this ticket.

Show
Gabor Kovacs added a comment - - edited This comment in DM-26171 suggests that ValidEnd_{i} == ValidStart_{i+1} is the correct approach while incrementValidEnd == False enforces ValidStart_{i+1} == ValidEnd_{i} + 1. DM-27018 says that even a ==1 gap should be interpreted as no gap. I could not get to a conclusion about the correct validity range convention... Could you please add an explanation to fixSubsetValidity docstring why and when we need to follow one or the other convention? (i.e. Why did the DM-26171 change actually give overlapping validity ranges in case of the community pipeline calibs in ap_verify_hits2015 ?) If possible, please also add a one line "instruction" to the incrementValidEnd config description, e.g.  "Set to True, if ... . See fixSubsetValidity docs for details." Of course, for ap_verify_hits2015 this fixes the problem raised in this ticket.
Hide
Gabor Kovacs added a comment -

https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline

The CI link in the comment above actually points to build 32863 that failed. 32864 is OK.

Show
Gabor Kovacs added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32864/pipeline The CI link in the comment above actually points to build 32863 that failed. 32864 is OK.
Hide
Christopher Waters added a comment - - edited

I've updated the docstring, after investigating that seems to be related to whether the calibDate value is set (which is true for the stack generated calibrations, but community produced calibrations use the DATE-OBS value instead).

https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32896/pipeline/

Show
Christopher Waters added a comment - - edited I've updated the docstring, after investigating that seems to be related to whether the calibDate value is set (which is true for the stack generated calibrations, but community produced calibrations use the DATE-OBS value instead). https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/32896/pipeline/
Hide
Gabor Kovacs added a comment -

Everything looks good. Thanks.

Show
Gabor Kovacs added a comment - Everything looks good. Thanks.

#### People

Assignee:
Christopher Waters
Reporter:
Gabor Kovacs
Reviewers:
Gabor Kovacs
Watchers:
Christopher Waters, Gabor Kovacs, Krzysztof Findeisen, Meredith Rawls, Tim Jenness