# Set doApplyFinalizedPsf to False for all AP makeWarp steps, for reals

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Epic Link:
• Sprint:
AP F22-1 (June)
• Team:
Alert Production
• Urgent?:
No

#### Description

DM-35162 should have fixed this by setting doApplyFinalizedPsf to False in the "base" ApTemplate pipeline, but it didn't, because our LsstCam-imSim and HSC ApTemplate pipelines now both have lines like

file: $AP_PIPE_DIR/config/$CAMERA/makeWarp.py

in their configs. For once, DECam is unaffected and should work fine.

When configs are being read from files into pipelines, any configs from imported pipelines are apparently overwritten. Therefore, any additional configs need to be set in the file in question.

#### Activity

Hide
Meredith Rawls added a comment -

Would you be willing to review this small config update, Ken? I'll make the PR momentarily.

Show
Meredith Rawls added a comment - Would you be willing to review this small config update, Ken? I'll make the PR momentarily.
Hide
Meredith Rawls added a comment -

PR is here: https://github.com/lsst/ap_pipe/pull/116

I did not run Jenkins since I don't think it has any ability to vet pipeline configs.

Show
Meredith Rawls added a comment - PR is here: https://github.com/lsst/ap_pipe/pull/116 I did not run Jenkins since I don't think it has any ability to vet pipeline configs.
Hide
Kenneth Herner added a comment -

I'll pull the ticket branch and test myself just to make sure it will work from a fresh install.

Show
Kenneth Herner added a comment - I'll pull the ticket branch and test myself just to make sure it will work from a fresh install.
Hide
Kenneth Herner added a comment -

I ran a test on a small bit of the HSC data in the 2021 diffim sprint and it looks like it's working well.

Show
Kenneth Herner added a comment - I ran a test on a small bit of the HSC data in the 2021 diffim sprint and it looks like it's working well.
Hide
Kenneth Herner added a comment -

An interesting update on this issue: I don't think it's quite as simple as we first thought. Since I couldn't figure out why I didn't see the same failure when working on DM-33911, I did another test where I set up weekly 25 (has the fix from 35162 but not this one) and then ran exactly what Meredith Rawls tried modulo the output collection name and restricting to a single patch for speed:

 pipetask --long-log run -j 12 -b /repo/dc2 -d "skymap='DC2' AND tract=4431 AND patch IN (9)" -i 2.2i/defaults -o u/kherner/DM-35359_w25 -p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApTemplate.yaml --register-dataset-types --clobber-outputs Sure enough I could reproduce the problem. Then, without any changes, I tried doing something very similar on some of what goes into the diffim sprint datasets:  pipetask run -j8 -i HSC/runs/RC2/w_2022_22/DM-34983 -b /repo/main -d "tract=9813 AND instrument='HSC' AND skymap='hsc_rings_v1' AND patch=40" -p$AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApTemplate.yaml -o u/kherner/DM-35359_w25 --register-dataset-types --clobber-outputs

but that worked! You'll immediately see two differences:

1) Different pipeline yaml files

2) Different input collections (from different repos)

As for the first, they do indeed pull in (very) different makeWarp.py files ($AP_PIPE_DIR/config/LSSTCam-imSim/makeWarp.py and$AP_PIPE_DIR/config/HSC/makeWarp.py), but there is nothing about doApplyFinalizedPsf in either one! Maybe it's getting set indirectly by some other setting that's different between the two (I would have to look at the code again) but I'm not sure what that mechanism would be. As for the second, it's not clear why the input collection would matter.

Finally, I don't think any of that is a reason not to go ahead and merge this change in, since it certainly does what we want, but I wanted to bring it up in case we're missing something more sinister going on behind the scenes.

Show
Kenneth Herner added a comment - An interesting update on this issue: I don't think it's quite as simple as we first thought. Since I couldn't figure out why I didn't see the same failure when working on DM-33911 , I did another test where I set up weekly 25 (has the fix from 35162 but not this one) and then ran exactly what Meredith Rawls  tried modulo the output collection name and restricting to a single patch for speed:   pipetask --long-log run -j 12 -b /repo/dc2 -d "skymap='DC2' AND tract=4431 AND patch IN (9)" -i 2.2i/defaults -o u/kherner/DM-35359_w25 -p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApTemplate.yaml --register-dataset-types --clobber-outputs Sure enough I could reproduce the problem. Then, without any changes, I tried doing something very similar on some of what goes into the diffim sprint datasets: pipetask run -j8 -i HSC/runs/RC2/w_2022_22/DM-34983 -b /repo/main -d "tract=9813 AND instrument='HSC' AND skymap='hsc_rings_v1' AND patch=40" -p$AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApTemplate.yaml -o u/kherner/DM-35359_w25 --register-dataset-types --clobber-outputs   but that worked! You'll immediately see two differences: 1) Different pipeline yaml files 2) Different input collections (from different repos)  As for the first, they do indeed pull in (very) different makeWarp.py files ($AP_PIPE_DIR/config/LSSTCam-imSim/makeWarp.py and$AP_PIPE_DIR/config/HSC/makeWarp.py), but there is nothing about doApplyFinalizedPsf in either one! Maybe it's getting set indirectly by some other setting that's different between the two (I would have to look at the code again) but I'm not sure what that mechanism would be. As for the second, it's not clear why the input collection would matter. Finally, I don't think any of that is a reason not to go ahead and merge this change in, since it certainly does what we want, but I wanted to bring it up in case we're missing something more sinister going on behind the scenes.
Hide
John Parejko added a comment -

I also think it's good to merge this, but I wanted to link in two related tickets which are probably related to the trouble: DM-31063 and its followup DM-34699.

Show
John Parejko added a comment - I also think it's good to merge this, but I wanted to link in two related tickets which are probably related to the trouble: DM-31063 and its followup DM-34699 .
Hide
Meredith Rawls added a comment -

We figured out the source of the discrepancy! /repo/main has registered the finalized_psf_ap_corr_catalog dataset type thanks to myriad DRP reruns, and in fact, the "kitchen sink" input collection Ken was using included instances of those datasets. We were therefore both semi-unknowingly running with doApplyFinalizedPsf = True. I say "semi-unknowingly" because, as DM-35504 now explains, we exist in a config limbo-land in which configs set in ap_pipe/pipelines/ApTemplate.py get blown away and overwritten by configs set in ap_pipe/config/$CAMERA/whateverTask.py I'm going to add a warning at the top of the "base" ApTemplate and ApPipe warning about this state of affairs that can be removed whenever DM-35504 is done. Show Meredith Rawls added a comment - We figured out the source of the discrepancy! /repo/main has registered the finalized_psf_ap_corr_catalog dataset type thanks to myriad DRP reruns, and in fact, the "kitchen sink" input collection Ken was using included instances of those datasets. We were therefore both semi-unknowingly running with doApplyFinalizedPsf = True . I say "semi-unknowingly" because, as DM-35504 now explains, we exist in a config limbo-land in which configs set in ap_pipe/pipelines/ApTemplate.py get blown away and overwritten by configs set in ap_pipe/config/$CAMERA/whateverTask.py .  I'm going to add a warning at the top of the "base" ApTemplate and ApPipe warning about this state of affairs that can be removed whenever DM-35504 is done.

#### People

Assignee:
Meredith Rawls
Reporter:
Meredith Rawls
Reviewers:
Kenneth Herner
Watchers:
Ian Sullivan, John Parejko, Kenneth Herner, Meredith Rawls
Votes:
0 Vote for this issue
Watchers:
4 Start watching this issue

#### Dates

Created:
Updated:
Resolved:

#### Jenkins

No builds found.