Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18828

Write script to ease template generation for ap_pipe reprocessing

    Details

    • Story Points:
      4
    • Sprint:
      AP F19-4 (2H September), AP F19-5 (October), AP F19-6 (November), AP S20-1 (December)
    • Team:
      Alert Production

      Description

      As part of our desire to have regular ~monthly processing of HiTS, we'd like to generate templates. Meredith reports this is more time-consuming than it could be. This ticket is to write a script to automate this to the extent possible

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            Each step of generating templates from the three HiTS2014 fields which overlap the HiTS2015 dataset is now more automated. I am testing this full workflow for the first time in DM-21330. The steps to make templates are

            • makeDiscreteSkyMap.py (quick, no slurm required)
              • makeDiscreteSkyMap.py $repo --rerun $rerun --id object='Blind14A_04''Blind14A_09''Blind14A_10' --config skyMap.pixelScale=0.26
            • makeCoaddTempExp.py (now slurm-ready for the HiTS2014 dataset assuming data in ccds 1, 3-60, and 62)
            • assembleCoadd.py (now slurm-ready for the HiTS2014 dataset assuming a 0.26"/pixel scale)

            Then the ap_pipe rerun can proceed as usual with ap_pipe.py via slurm. See, e.g., DM-22258 and DM-22729. I am adding these already-existing slurm scripts as well, which also assume data in ccds 1, 3-60, and 62.

            Show
            mrawls Meredith Rawls added a comment - Each step of generating templates from the three HiTS2014 fields which overlap the HiTS2015 dataset is now more automated. I am testing this full workflow for the first time in DM-21330 . The steps to make templates are makeDiscreteSkyMap.py (quick, no slurm required) makeDiscreteSkyMap.py $repo --rerun $rerun --id object='Blind14A_04' 'Blind14A_09' 'Blind14A_10' --config skyMap.pixelScale=0.26 makeCoaddTempExp.py (now slurm-ready for the HiTS2014 dataset assuming data in ccds 1, 3-60, and 62) assembleCoadd.py (now slurm-ready for the HiTS2014 dataset assuming a 0.26"/pixel scale) Then the ap_pipe rerun can proceed as usual with ap_pipe.py via slurm. See, e.g., DM-22258 and DM-22729 . I am adding these already-existing slurm scripts as well, which also assume data in ccds 1, 3-60, and 62.
            Hide
            ebellm Eric Bellm added a comment -

            Looks good.

            Show
            ebellm Eric Bellm added a comment - Looks good.
            Hide
            mrawls Meredith Rawls added a comment -

            Kind of you to say, but unfortunately I found a problem when running ap_pipe for DM-22729 with these templates. First, visit 289818 is utter astrometric garbage (all ccds) and must be excluded from the final assembly step. Second, and more critically, many (maybe 1/3?) of the ccds in HiTS2015 fail to successfully make it through difference imaging due to RuntimeError: Cannot find any objects suitable for KernelCandidacy. I pushed an update which addresses the first problem but am still investigating the second. 

            Show
            mrawls Meredith Rawls added a comment - Kind of you to say, but unfortunately I found a problem when running ap_pipe for DM-22729 with these templates. First, visit 289818 is utter astrometric garbage (all ccds) and must be excluded from the final assembly step. Second, and more critically, many (maybe 1/3?) of the ccds in HiTS2015 fail to successfully make it through difference imaging due to RuntimeError: Cannot find any objects suitable for KernelCandidacy . I pushed an update which addresses the first problem but am still investigating the second. 
            Hide
            mrawls Meredith Rawls added a comment -

            I redid the processing, and the output (i.e., results for DM-22729) looks much better now. The slurm scripts now create warps without much parallelization - just two slurm processes, one for the solo field and one for the two overlapping fields. This is necessary to accommodate the current limitations of the good seeing selector. I also excluded the erroneous patches originating from visit 289818 during coadd assembly. I won't detail an analysis of the output APDB and related data products here, but the slurm scripts are now working as intended, so I think this is done now.

            Show
            mrawls Meredith Rawls added a comment - I redid the processing, and the output (i.e., results for DM-22729 ) looks much better now. The slurm scripts now create warps without much parallelization - just two slurm processes, one for the solo field and one for the two overlapping fields. This is necessary to accommodate the current limitations of the good seeing selector. I also excluded the erroneous patches originating from visit 289818 during coadd assembly. I won't detail an analysis of the output APDB and related data products here, but the slurm scripts are now working as intended, so I think this is done now.

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                ebellm Eric Bellm
                Reviewers:
                Eric Bellm
                Watchers:
                Eric Bellm, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel