Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_tasks
-
Labels:None
-
Story Points:1
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
ap_verify gen3 failed last night with the following error:
RuntimeError: 2 dataset(s) of type 'jointcal_photoCalib' was/were present in a previous query, but could not be found now.This is either a logic bug in QuantumGraph generation or the input collections have been modified since QuantumGraph generation began.
|
It seems likely that this was caused by DM-31491, which just landed and at a minimum reordered the dimensions for this dataset. https://github.com/lsst/pipe_tasks/commit/e5427992456a875016af14c907ea1f8bd786fb6f#diff-d0c181586a7ef6ce433ff7baee9b2fa03ab7de10015e6cafa57741a90a424d2dL77-R94
I believe that there is also a typo later in the same PR: https://github.com/lsst/pipe_tasks/commit/e5427992456a875016af14c907ea1f8bd786fb6f#diff-d0c181586a7ef6ce433ff7baee9b2fa03ab7de10015e6cafa57741a90a424d2dR369
Fixing this before the weekly would be highly desirable.
I found the bug and have tested ap_verify. I pulled ap_verify and ap_verify_ci_cosmos_pdr2 and then ran:
ap_verify.py --dataset ap_verify_ci_cosmos_pdr2 --gen3 --output /project/sreed/apVerifyPdr2/ -p pipelines/ApVerifyWithFakes.yaml
from the ap_verify_ci_cosmos_pdr2 repo.
I also have a jenkins run going: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35271/pipeline
I ran:
pipetask run -b /repo/main/butler.yaml -i HSC/runs/RC2/w_2021_42/
DM-32248-p /home/sr525/repos/obs_subaru/pipelines/DRPFakes.yaml -d "instrument='HSC' and skymap='hsc_rings_v1' and (band='i') and (detector=82 or detector=81 or detector=80) and (visit=1228 or visit=1230)" -i u/sr525/ssiCats -o u/sr525/fakesTest6 --register-dataset-typeswhich completed.
These were all run with:
source /project/yusra/lsst_stack/loadLSST.bash
setup lsst_distrib -t d_2021_10_25