Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23430

Run ap_pipe on HSC G PDR2 COSMOS dataset with RC2 templates

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      4
    • Epic Link:
    • Sprint:
      AP S20-3 (February), AP S20-4 (March)
    • Team:
      Alert Production
    • Urgent?:
      No

      Attachments

        Issue Links

          Activity

          Hide
          mrawls Meredith Rawls added a comment -

          In order to successfully run ap_pipe, I had to manually edit the destination rerun repo's repositoryCfg.yaml so that it had both the template repo containing coadds and the repo containing the raw visits above as inputs and the new empty destination rerun repo as an output. Then I experimented until I got the following command to work:

          ap_pipe.py /project/mrawls/cosmos --calib /project/mrawls/cosmos/calib-20200115 --rerun ap_pipe-g-1 -c diaPipe.apdb.db_url="sqlite:////project/mrawls/cosmos/rerun/ap_pipe-g-1/association.db" -c diaPipe.apdb.isolation_level="READ_UNCOMMITTED" -c diaPipe.apdb.connection_timeout=240 --id visit=59150 ccd=11 filter=HSC-G 

          And finally, I used slurm to run the same command on all the visits listed in DM-22431 and all the HSC CCDs (0..8^10..103).

          To reiterate, the visit list is

          59150 59152 59154 59156 59158 59160 90750 90752 90754 95148 95158 95168 95178 97582 101418 101438 101450 101470 101500 103054 103076 104856 104866 106058 106068 106078 106130 107872 107882 107892 107902 113174 113176 113178 113180

          Several dataIds failed, mostly due to not being able to find a template (presumably due to dithering?), and there were additional failures that have yet to be investigated. Overall, though, at least some dataIds covering the entire tract ran successfully.

          Show
          mrawls Meredith Rawls added a comment - In order to successfully run ap_pipe, I had to manually edit the destination rerun repo's repositoryCfg.yaml  so that it had both the template repo containing coadds and the repo containing the raw visits above as inputs and the new empty destination rerun repo as an output. Then I experimented until I got the following command to work: ap_pipe.py /project/mrawls/cosmos --calib /project/mrawls/cosmos/calib- 20200115 --rerun ap_pipe-g- 1 -c diaPipe.apdb.db_url= "sqlite:////project/mrawls/cosmos/rerun/ap_pipe-g-1/association.db" -c diaPipe.apdb.isolation_level= "READ_UNCOMMITTED" -c diaPipe.apdb.connection_timeout= 240 --id visit= 59150 ccd= 11 filter=HSC-G And finally, I used slurm to run the same command on all the visits listed in DM-22431 and all the HSC CCDs (0..8^10..103). To reiterate, the visit list is 59150 59152 59154 59156 59158 59160 90750 90752 90754 95148 95158 95168 95178 97582 101418 101438 101450 101470 101500 103054 103076 104856 104866 106058 106068 106078 106130 107872 107882 107892 107902 113174 113176 113178 113180 Several dataIds failed, mostly due to not being able to find a template (presumably due to dithering?), and there were additional failures that have yet to be investigated. Overall, though, at least some dataIds covering the entire tract ran successfully.
          Hide
          mrawls Meredith Rawls added a comment -

          The result from running ap_pipe is on lsst-dev in /project/mrawls/cosmos/rerun/ap_pipe-g-1

          Show
          mrawls Meredith Rawls added a comment - The result from running ap_pipe is on lsst-dev in /project/mrawls/cosmos/rerun/ap_pipe-g-1 . 
          Hide
          swinbank John Swinbank added a comment -

          Thank you!

          Show
          swinbank John Swinbank added a comment - Thank you!

            People

            • Assignee:
              mrawls Meredith Rawls
              Reporter:
              swinbank John Swinbank
              Reviewers:
              John Swinbank
              Watchers:
              John Swinbank, Meredith Rawls
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel