# Run ap_pipe on HSC G PDR2 COSMOS dataset with RC2 templates

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
4
• Sprint:
AP S20-3 (February), AP S20-4 (March)
• Team:
• Urgent?:
No

## Activity

Hide
Meredith Rawls added a comment -

In order to successfully run ap_pipe, I had to manually edit the destination rerun repo's repositoryCfg.yaml so that it had both the template repo containing coadds and the repo containing the raw visits above as inputs and the new empty destination rerun repo as an output. Then I experimented until I got the following command to work:

 ap_pipe.py /project/mrawls/cosmos --calib /project/mrawls/cosmos/calib-20200115 --rerun ap_pipe-g-1 -c diaPipe.apdb.db_url="sqlite:////project/mrawls/cosmos/rerun/ap_pipe-g-1/association.db" -c diaPipe.apdb.isolation_level="READ_UNCOMMITTED" -c diaPipe.apdb.connection_timeout=240 --id visit=59150 ccd=11 filter=HSC-G

And finally, I used slurm to run the same command on all the visits listed in DM-22431 and all the HSC CCDs (0..8^10..103).

To reiterate, the visit list is

59150 59152 59154 59156 59158 59160 90750 90752 90754 95148 95158 95168 95178 97582 101418 101438 101450 101470 101500 103054 103076 104856 104866 106058 106068 106078 106130 107872 107882 107892 107902 113174 113176 113178 113180

Several dataIds failed, mostly due to not being able to find a template (presumably due to dithering?), and there were additional failures that have yet to be investigated. Overall, though, at least some dataIds covering the entire tract ran successfully.

Show
Meredith Rawls added a comment - In order to successfully run ap_pipe, I had to manually edit the destination rerun repo's repositoryCfg.yaml  so that it had both the template repo containing coadds and the repo containing the raw visits above as inputs and the new empty destination rerun repo as an output. Then I experimented until I got the following command to work: ap_pipe.py /project/mrawls/cosmos --calib /project/mrawls/cosmos/calib- 20200115 --rerun ap_pipe-g- 1 -c diaPipe.apdb.db_url= "sqlite:////project/mrawls/cosmos/rerun/ap_pipe-g-1/association.db" -c diaPipe.apdb.isolation_level= "READ_UNCOMMITTED" -c diaPipe.apdb.connection_timeout= 240 --id visit= 59150 ccd= 11 filter=HSC-G And finally, I used slurm to run the same command on all the visits listed in DM-22431 and all the HSC CCDs (0..8^10..103). To reiterate, the visit list is 59150 59152 59154 59156 59158 59160 90750 90752 90754 95148 95158 95168 95178 97582 101418 101438 101450 101470 101500 103054 103076 104856 104866 106058 106068 106078 106130 107872 107882 107892 107902 113174 113176 113178 113180 Several dataIds failed, mostly due to not being able to find a template (presumably due to dithering?), and there were additional failures that have yet to be investigated. Overall, though, at least some dataIds covering the entire tract ran successfully.
Hide
Meredith Rawls added a comment -

The result from running ap_pipe is on lsst-dev in /project/mrawls/cosmos/rerun/ap_pipe-g-1

Show
Meredith Rawls added a comment - The result from running ap_pipe is on lsst-dev in /project/mrawls/cosmos/rerun/ap_pipe-g-1 .
Hide
John Swinbank added a comment -

Thank you!

Show
John Swinbank added a comment - Thank you!

## People

• Assignee:
Meredith Rawls
Reporter:
John Swinbank
Reviewers:
John Swinbank
Watchers:
John Swinbank, Meredith Rawls