Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35359

Set doApplyFinalizedPsf to False for all AP makeWarp steps, for reals

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_pipe
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      AP F22-1 (June)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      DM-35162 should have fixed this by setting doApplyFinalizedPsf to False in the "base" ApTemplate pipeline, but it didn't, because our LsstCam-imSim and HSC ApTemplate pipelines now both have lines like

      file: $AP_PIPE_DIR/config/$CAMERA/makeWarp.py

      in their configs. For once, DECam is unaffected and should work fine.

      When configs are being read from files into pipelines, any configs from imported pipelines are apparently overwritten. Therefore, any additional configs need to be set in the file in question.

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            Would you be willing to review this small config update, Ken? I'll make the PR momentarily.

            Show
            mrawls Meredith Rawls added a comment - Would you be willing to review this small config update, Ken? I'll make the PR momentarily.
            Hide
            mrawls Meredith Rawls added a comment -

            PR is here: https://github.com/lsst/ap_pipe/pull/116

            I did not run Jenkins since I don't think it has any ability to vet pipeline configs.

            Show
            mrawls Meredith Rawls added a comment - PR is here: https://github.com/lsst/ap_pipe/pull/116 I did not run Jenkins since I don't think it has any ability to vet pipeline configs.
            Hide
            kherner Kenneth Herner added a comment -

            I'll pull the ticket branch and test myself just to make sure it will work from a fresh install.

            Show
            kherner Kenneth Herner added a comment - I'll pull the ticket branch and test myself just to make sure it will work from a fresh install.
            Hide
            kherner Kenneth Herner added a comment -

            I ran a test on a small bit of the HSC data in the 2021 diffim sprint and it looks like it's working well. 

            Show
            kherner Kenneth Herner added a comment - I ran a test on a small bit of the HSC data in the 2021 diffim sprint and it looks like it's working well. 
            Hide
            kherner Kenneth Herner added a comment -

            An interesting update on this issue: I don't think it's quite as simple as we first thought. Since I couldn't figure out why I didn't see the same failure when working on DM-33911, I did another test where I set up weekly 25 (has the fix from 35162 but not this one) and then ran exactly what Meredith Rawls tried modulo the output collection name and restricting to a single patch for speed:

             

            pipetask --long-log run -j 12 -b /repo/dc2 -d "skymap='DC2' AND tract=4431 AND patch IN (9)" -i 2.2i/defaults -o u/kherner/DM-35359_w25 -p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApTemplate.yaml --register-dataset-types --clobber-outputs

             

            Sure enough I could reproduce the problem. Then, without any changes, I tried doing something very similar on some of what goes into the diffim sprint datasets:

             

            pipetask run -j8 -i HSC/runs/RC2/w_2022_22/DM-34983 -b /repo/main -d "tract=9813 AND instrument='HSC' AND skymap='hsc_rings_v1' AND patch=40" -p $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApTemplate.yaml -o u/kherner/DM-35359_w25 --register-dataset-types --clobber-outputs

             

            but that worked! You'll immediately see two differences:

            1) Different pipeline yaml files

            2) Different input collections (from different repos)

             As for the first, they do indeed pull in (very) different makeWarp.py files ($AP_PIPE_DIR/config/LSSTCam-imSim/makeWarp.py and $AP_PIPE_DIR/config/HSC/makeWarp.py), but there is nothing about doApplyFinalizedPsf in either one! Maybe it's getting set indirectly by some other setting that's different between the two (I would have to look at the code again) but I'm not sure what that mechanism would be. As for the second, it's not clear why the input collection would matter.

            Finally, I don't think any of that is a reason not to go ahead and merge this change in, since it certainly does what we want, but I wanted to bring it up in case we're missing something more sinister going on behind the scenes.

            Show
            kherner Kenneth Herner added a comment - An interesting update on this issue: I don't think it's quite as simple as we first thought. Since I couldn't figure out why I didn't see the same failure when working on DM-33911 , I did another test where I set up weekly 25 (has the fix from 35162 but not this one) and then ran exactly what Meredith Rawls  tried modulo the output collection name and restricting to a single patch for speed:   pipetask --long-log run -j 12 -b /repo/dc2 -d "skymap='DC2' AND tract=4431 AND patch IN (9)" -i 2.2i/defaults -o u/kherner/DM-35359_w25 -p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApTemplate.yaml --register-dataset-types --clobber-outputs   Sure enough I could reproduce the problem. Then, without any changes, I tried doing something very similar on some of what goes into the diffim sprint datasets:   pipetask run -j8 -i HSC/runs/RC2/w_2022_22/DM-34983 -b /repo/main -d "tract=9813 AND instrument='HSC' AND skymap='hsc_rings_v1' AND patch=40" -p $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApTemplate.yaml -o u/kherner/DM-35359_w25 --register-dataset-types --clobber-outputs   but that worked! You'll immediately see two differences: 1) Different pipeline yaml files 2) Different input collections (from different repos)  As for the first, they do indeed pull in (very) different makeWarp.py files ($AP_PIPE_DIR/config/LSSTCam-imSim/makeWarp.py and $AP_PIPE_DIR/config/HSC/makeWarp.py), but there is nothing about doApplyFinalizedPsf in either one! Maybe it's getting set indirectly by some other setting that's different between the two (I would have to look at the code again) but I'm not sure what that mechanism would be. As for the second, it's not clear why the input collection would matter. Finally, I don't think any of that is a reason not to go ahead and merge this change in, since it certainly does what we want, but I wanted to bring it up in case we're missing something more sinister going on behind the scenes.
            Hide
            Parejkoj John Parejko added a comment -

            I also think it's good to merge this, but I wanted to link in two related tickets which are probably related to the trouble: DM-31063 and its followup DM-34699.

            Show
            Parejkoj John Parejko added a comment - I also think it's good to merge this, but I wanted to link in two related tickets which are probably related to the trouble: DM-31063 and its followup DM-34699 .
            Hide
            mrawls Meredith Rawls added a comment -

            We figured out the source of the discrepancy! /repo/main has registered the finalized_psf_ap_corr_catalog dataset type thanks to myriad DRP reruns, and in fact, the "kitchen sink" input collection Ken was using included instances of those datasets. We were therefore both semi-unknowingly running with doApplyFinalizedPsf = True. I say "semi-unknowingly" because, as DM-35504 now explains, we exist in a config limbo-land in which configs set in ap_pipe/pipelines/ApTemplate.py get blown away and overwritten by configs set in ap_pipe/config/$CAMERA/whateverTask.py

            I'm going to add a warning at the top of the "base" ApTemplate and ApPipe warning about this state of affairs that can be removed whenever DM-35504 is done.

            Show
            mrawls Meredith Rawls added a comment - We figured out the source of the discrepancy! /repo/main has registered the finalized_psf_ap_corr_catalog dataset type thanks to myriad DRP reruns, and in fact, the "kitchen sink" input collection Ken was using included instances of those datasets. We were therefore both semi-unknowingly running with doApplyFinalizedPsf = True . I say "semi-unknowingly" because, as DM-35504 now explains, we exist in a config limbo-land in which configs set in ap_pipe/pipelines/ApTemplate.py get blown away and overwritten by configs set in ap_pipe/config/$CAMERA/whateverTask.py .  I'm going to add a warning at the top of the "base" ApTemplate and ApPipe warning about this state of affairs that can be removed whenever DM-35504 is done.

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              mrawls Meredith Rawls
              Reviewers:
              Kenneth Herner
              Watchers:
              Ian Sullivan, John Parejko, Kenneth Herner, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.