Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11021

Generalize prototype pipeline script

    Details

    • Story Points:
      6
    • Sprint:
      Alert Production F17 - 7, Alert Production F17 - 8
    • Team:
      Alert Production

      Description

      The present AP Prototype Pipeline script decam_process.py has a few hard-wired settings to define repository names etc., requires an additional step outside of the script to ingest defects, and only uses visits as templates. In principle it should be a command-line python script that can be used on any DECam images. This ticket includes

      • Changing hard-wired inputs in main() to be handled via argparse or similar
      • Adding the ingestion of DECam defect images to the script
      • Extending the script to handle either a visit or a coadd to be a template
      • Think about the best way to handle datasets with more than one filter (ok not to implement this yet, but don't hard-wire any filter business)
      • Updating the technote DMTN-039 and lsst-dm/decam_hits repo README to describe the new behavior

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment - - edited

            Changed the scope of this ticket somewhat. Primary goals are now:

            • Changing hard-wired inputs in main() to be handled via argparse or similar (unchanged)
            • Adding the ingestion of DECam defect images to the script (unchanged)
            • Extending the script to handle either a visit or a coadd to be a template (moved to DM-11422)
            • Think about the best way to handle datasets with more than one filter (this is an outstanding issue; for now, the prototype pipeline assumes that the reference catalogs are available for a given filter and that both the template visit and science visit are in the same filter)
            • Updating the technote DMTN-039 and lsst-dm/decam_hits repo README to describe the new behavior (I am postponing this until the UI is more finalized)
            • NEW: use the dataset initialized in DM-11117 as the input data, which includes hardwiring some assumptions about where things live inside this dataset repo
            Show
            mrawls Meredith Rawls added a comment - - edited Changed the scope of this ticket somewhat. Primary goals are now: Changing hard-wired inputs in main() to be handled via argparse or similar (unchanged) Adding the ingestion of DECam defect images to the script (unchanged) Extending the script to handle either a visit or a coadd to be a template (moved to DM-11422 ) Think about the best way to handle datasets with more than one filter (this is an outstanding issue; for now, the prototype pipeline assumes that the reference catalogs are available for a given filter and that both the template visit and science visit are in the same filter) Updating the technote DMTN-039 and lsst-dm/decam_hits repo README to describe the new behavior (I am postponing this until the UI is more finalized) NEW: use the dataset initialized in DM-11117 as the input data, which includes hardwiring some assumptions about where things live inside this dataset repo
            Hide
            mrawls Meredith Rawls added a comment - - edited

            Can you please review this David Reiss? I'm happy to answer any questions on Monday.

            If you would like to see where I've successfully run this version of the prototype pipeline, you are welcome to look on lsst-dev in /project/mrawls/prototype_pipeline/zomg. The command I ran from the prototype_pipeline directory was python decam_hits/decam_process.py --dataset verify_ap_hits2015/ --output zomg. Note that verify_ap_hits2015 is a symlink to the directory by the same name in /project/reiss.

            Show
            mrawls Meredith Rawls added a comment - - edited Can you please review this David Reiss ? I'm happy to answer any questions on Monday. If you would like to see where I've successfully run this version of the prototype pipeline, you are welcome to look on lsst-dev in /project/mrawls/prototype_pipeline/zomg . The command I ran from the prototype_pipeline directory was python decam_hits/decam_process.py --dataset verify_ap_hits2015/ --output zomg . Note that verify_ap_hits2015 is a symlink to the directory by the same name in /project/reiss .
            Hide
            reiss David Reiss added a comment -

            I have reviewed the PR #4 for this ticket and DM-10975. The modifications definitely look like a great step forward, and I left some (mostly stylistic) comments. Please see the comments in the PR. Since getting this working is of utmost urgency right now I am ok with postponing the stylistic issues for another ticket. If you choose to go that route, please make a new ticket and note the suggested therein.

            Show
            reiss David Reiss added a comment - I have reviewed the PR #4 for this ticket and DM-10975 . The modifications definitely look like a great step forward, and I left some (mostly stylistic) comments. Please see the comments in the PR. Since getting this working is of utmost urgency right now I am ok with postponing the stylistic issues for another ticket. If you choose to go that route, please make a new ticket and note the suggested therein.
            Hide
            mrawls Meredith Rawls added a comment -

            We ultimately decided to postpone using coadds as templates to DM-11422 and to update the technote documentation as part of completing DM-11422 and/or DM-11390. Tests will be written as part of DM-11390 once the pipeline is renamed to ap_pipe.

            In addition to the rest of the work on this ticket, each function in the pipeline now returns metadata that will be used by ap_verify for initial metrics. For the full discussion, see the PR.

            Show
            mrawls Meredith Rawls added a comment - We ultimately decided to postpone using coadds as templates to DM-11422 and to update the technote documentation as part of completing DM-11422 and/or DM-11390 . Tests will be written as part of DM-11390 once the pipeline is renamed to ap_pipe . In addition to the rest of the work on this ticket, each function in the pipeline now returns metadata that will be used by ap_verify for initial metrics. For the full discussion, see the PR.

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                mrawls Meredith Rawls
                Reviewers:
                David Reiss
                Watchers:
                David Reiss, Eric Bellm, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: