# Write script to ease template generation for ap_pipe reprocessing

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
4
• Sprint:
AP F19-4 (2H September), AP F19-5 (October), AP F19-6 (November), AP S20-1 (December)
• Team:

## Description

As part of our desire to have regular ~monthly processing of HiTS, we'd like to generate templates. Meredith reports this is more time-consuming than it could be. This ticket is to write a script to automate this to the extent possible

## Activity

Hide
Meredith Rawls added a comment -

Each step of generating templates from the three HiTS2014 fields which overlap the HiTS2015 dataset is now more automated. I am testing this full workflow for the first time in DM-21330. The steps to make templates are

• makeDiscreteSkyMap.py (quick, no slurm required)
• makeDiscreteSkyMap.py $repo --rerun$rerun --id object='Blind14A_04''Blind14A_09''Blind14A_10' --config skyMap.pixelScale=0.26
• makeCoaddTempExp.py (now slurm-ready for the HiTS2014 dataset assuming data in ccds 1, 3-60, and 62)
• assembleCoadd.py (now slurm-ready for the HiTS2014 dataset assuming a 0.26"/pixel scale)

Then the ap_pipe rerun can proceed as usual with ap_pipe.py via slurm. See, e.g., DM-22258 and DM-22729. I am adding these already-existing slurm scripts as well, which also assume data in ccds 1, 3-60, and 62.

Show
Meredith Rawls added a comment - Each step of generating templates from the three HiTS2014 fields which overlap the HiTS2015 dataset is now more automated. I am testing this full workflow for the first time in DM-21330 . The steps to make templates are makeDiscreteSkyMap.py (quick, no slurm required) makeDiscreteSkyMap.py $repo --rerun$rerun --id object='Blind14A_04' 'Blind14A_09' 'Blind14A_10' --config skyMap.pixelScale=0.26 makeCoaddTempExp.py (now slurm-ready for the HiTS2014 dataset assuming data in ccds 1, 3-60, and 62) assembleCoadd.py (now slurm-ready for the HiTS2014 dataset assuming a 0.26"/pixel scale) Then the ap_pipe rerun can proceed as usual with ap_pipe.py via slurm. See, e.g., DM-22258 and DM-22729 . I am adding these already-existing slurm scripts as well, which also assume data in ccds 1, 3-60, and 62.
Hide
Eric Bellm added a comment -

Looks good.

Show
Eric Bellm added a comment - Looks good.
Hide
Meredith Rawls added a comment -

Kind of you to say, but unfortunately I found a problem when running ap_pipe for DM-22729 with these templates. First, visit 289818 is utter astrometric garbage (all ccds) and must be excluded from the final assembly step. Second, and more critically, many (maybe 1/3?) of the ccds in HiTS2015 fail to successfully make it through difference imaging due to RuntimeError: Cannot find any objects suitable for KernelCandidacy. I pushed an update which addresses the first problem but am still investigating the second.

Show
Meredith Rawls added a comment - Kind of you to say, but unfortunately I found a problem when running ap_pipe for DM-22729 with these templates. First, visit 289818 is utter astrometric garbage (all ccds) and must be excluded from the final assembly step. Second, and more critically, many (maybe 1/3?) of the ccds in HiTS2015 fail to successfully make it through difference imaging due to RuntimeError: Cannot find any objects suitable for KernelCandidacy . I pushed an update which addresses the first problem but am still investigating the second.
Hide
Meredith Rawls added a comment -

I redid the processing, and the output (i.e., results for DM-22729) looks much better now. The slurm scripts now create warps without much parallelization - just two slurm processes, one for the solo field and one for the two overlapping fields. This is necessary to accommodate the current limitations of the good seeing selector. I also excluded the erroneous patches originating from visit 289818 during coadd assembly. I won't detail an analysis of the output APDB and related data products here, but the slurm scripts are now working as intended, so I think this is done now.

Show
Meredith Rawls added a comment - I redid the processing, and the output (i.e., results for DM-22729 ) looks much better now. The slurm scripts now create warps without much parallelization - just two slurm processes, one for the solo field and one for the two overlapping fields. This is necessary to accommodate the current limitations of the good seeing selector. I also excluded the erroneous patches originating from visit 289818 during coadd assembly. I won't detail an analysis of the output APDB and related data products here, but the slurm scripts are now working as intended, so I think this is done now.

## People

• Assignee:
Meredith Rawls
Reporter:
Eric Bellm
Reviewers:
Eric Bellm
Watchers:
Eric Bellm, Meredith Rawls