Looking at this, I notice that the dimensions of the new fake externalSkyWcsTractCatalog https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/processCcdWithFakes.py#L73-L79
are different than the dimensions of the old externalSkyWcsTractCatalog
https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/makeCoaddTempExp.py#L617-L623

And I think this might be the problem. The original one is missing skymap as a dimension; does this mean that this is wrong?

Though Jim says this will be expanded and compare as equal so maybe a red herring.

This exception is not one that should happen from regular user error. The fact that it does was already reported in DM-32058, and was supposed to be fixed on DM-31769.

I found the bug and have tested ap_verify. I pulled ap_verify and ap_verify_ci_cosmos_pdr2 and then ran:

ap_verify.py --dataset ap_verify_ci_cosmos_pdr2 --gen3 --output /project/sreed/apVerifyPdr2/ -p pipelines/ApVerifyWithFakes.yaml

from the ap_verify_ci_cosmos_pdr2 repo.

I also have a jenkins run going: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35271/pipeline

I ran: 

pipetask run -b /repo/main/butler.yaml -i HSC/runs/RC2/w_2021_42/DM-32248 -p /home/sr525/repos/obs_subaru/pipelines/DRPFakes.yaml -d "instrument='HSC' and skymap='hsc_rings_v1' and (band='i') and (detector=82 or detector=81 or detector=80) and (visit=1228 or visit=1230)" -i u/sr525/ssiCats -o u/sr525/fakesTest6 --register-dataset-types

which completed.

These were all run with: 

source /project/yusra/lsst_stack/loadLSST.bash
setup lsst_distrib -t d_2021_10_25

Thanks Sophie, and thanks Chris for your help too. This ticket removes unnecessary wcs and photoCalib code from processCcdWithFakes.py, and corrects a problematic typo that crops up further down the line. I can confirm that I'm able to successfully run both ap_verify and DRPFakes with this ticket.

For testing, I first set up:

setup lsst_distrib -t w_2021_43

setup -j -r ap_verify

setup -j -r ap_verify_ci_cosmos_pdr2

setup -j -r pipe_tasks

setup -j -r obs_subaru

setup -j -r ci_hsc_gen3

(all on their master branches). I believe these are all the repos which were touched since DM-31491 dropped, plus the required ap_verify repos.

Next, I confirm that running on 'master':

ap_verify.py -j 8 \

--dataset ap_verify_ci_cosmos_pdr2 \

--output /project/lskelvin/apverify/run/ \

--gen3 \

--data-query "visit in (59150, 59160) and band='g'" \

-p $AP_VERIFY_CI_COSMOS_PDR2_DIR/pipelines/ApVerifyWithFakes.yaml \

2>&1 | tee log-ApVerifyWithFakes.log

fails with the above joincal error message.

Switching pipe_tasks from 'master' to 'tickets/DM-32376' and then re-running now allows for ap_verify to complete successfully, with outputs as expected.

I also ran on this ticket branch:

pipetask --long-log run --register-dataset-types \

-b /repo/main \

-i HSC/runs/RC2/w_2021_42/DM-32248,u/sr525/ssiCats \

-o u/lskelvin/ap_verify_fakes_test_ticket \

-p $OBS_SUBARU_DIR/pipelines/DRPFakes.yaml \

-d "instrument='HSC' AND skymap='hsc_rings_v1' AND band='i' AND detector=1 AND visit=1228" \

2>&1 | tee log-DRPFakes-ticket.log

which also appears to complete successfully, producing outputs as expected.

Assuming Jenkins doesn't produce any red flags, then I think this is good to merge. Thanks!

I see a backport was requested from Tim. This ticket is a bugfix for DM-31491 which was merged the night before and broke ap_verify. Backporting this without DM-31491 does not make sense.

I only added the request when we thought that ci_hsc_gen3 was broken and this branch had the fix in it. Now that we know that ci_hsc_gen3 had the tag in the wrong place it is no longer important.

Tim is correct that ci_hsc_gen3 is now able to run after the pin move, however a problem still remains. If someone uses v23 obs_subaru fakes pipeline in the way that ci_hsc_gen3 (on the newer ref) was trying to,  the system will fail. Moving the pin on ci_hsc_gen3 back only works because it reverts it to a state where it is not using the obs_subaru fakes pipeline that exists in v23, but was instead using it's own fakes pipeline.

Specifically we need this line from this change on pipe_tasks: https://github.com/lsst/pipe_tasks/commit/80aa66dc92c83c1e284b116ae07675d6559376d3#diff-7daf2a8d292dc80b5c277b573b18346f30955635071479a011e78be5e76531b3R315

There is some confusion if this was the right ticket because of a past rebase issue, but that does not matter so much as getting the change set onto v23.