Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11390

Plug prototype pipeline script into verify_ap framework

    Details

    • Story Points:
      6
    • Sprint:
      Alert Production F17 - 8, Alert Production F17 - 9
    • Team:
      Alert Production

      Description

      Update the prototype pipeline (soon to be ap_pipe) so it works when a user runs the verify_ap script from the command line.

      Also write some tests for ap_pipe since none currently exist!

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            We have ingestion and calib ingestion working!

            Parts of it are a bit messy, but if one pulls the DM-11390 branch of ap_verify and ap_pipe, and does setup for both as well as for ap_verify_hits2015, it is possible to run this from the command line and have it work through calibration ingestion:

            python ap_verify/bin.src/ap_verify.py --dataset HiTS2015 --output your_desired_output_dir --dataIdString "visit=410985" --silent

            (Note that the visit number doesn't matter just yet because ingestion and calib ingestion do not use any dataId info and instead they just ingest everything in the dataset repo provided.)

            Show
            mrawls Meredith Rawls added a comment - We have ingestion and calib ingestion working! Parts of it are a bit messy, but if one pulls the DM-11390 branch of ap_verify and ap_pipe , and does setup for both as well as for ap_verify_hits2015 , it is possible to run this from the command line and have it work through calibration ingestion: python ap_verify/bin.src/ap_verify.py --dataset HiTS2015 --output your_desired_output_dir --dataIdString "visit=410985" --silent (Note that the visit number doesn't matter just yet because ingestion and calib ingestion do not use any dataId info and instead they just ingest everything in the dataset repo provided.)
            Hide
            mrawls Meredith Rawls added a comment -

            Thanks to help from Colin Slater, both ap_pipe and ap_verify may now successfully be run from the command line, for all steps from ingestion through difference imaging. For now, the template visit is hardwired to 410929 and the template ccdnum is hardwired to 25. This will be fixed when coadds are accepted as templates in DM-11422.

            To try it out:

            • setup obs_decam, ap_pipe, ap_verify_hits2015, and ap_verify (as well as a non-buggy daf_persistence)
            • cd to wherever you cloned ap_pipe and ap_verify
            • for ap_pipe alone: python ap_pipe/bin.src/ap_pipe.py -d ap_verify_hits2015/ -o zomg5 -i "visit=410985 ccdnum=25" (you should be able to choose a different visit from the Blind15A_40 field if you prefer)
            • for ap_verify: python ap_verify/bin.src/ap_verify.py --dataset HiTS2015 --output zomg6 --dataIdString "visit=410985 ccdnum=25" --silent (again, you may choose a different Blind15A_40 visit if desired)

            Both of these have successfully produced output on lsst-dev in /project/mrawls/prototype_ap/ in the zomg5 and zomg6 directories, respectively. However, after ap_verify finishes running all the parts of ap_pipe, it does throw the following exception

            ap.verify.pipeline_driver.run_ap_pipe INFO: Pipeline complete
            Traceback (most recent call last):
              File "ap_verify/bin.src/ap_verify.py", line 27, in <module>
                run_ap_verify()
              File "/project/mrawls/prototype_ap/ap_verify/python/lsst/ap/verify/ap_verify.py", line 162, in run_ap_verify
                _measure_final_properties(metadata, job)
              File "/project/mrawls/prototype_ap/ap_verify/python/lsst/ap/verify/ap_verify.py", line 140, in _measure_final_properties
                measurements = measure_from_metadata(metadata)
              File "/project/mrawls/prototype_ap/ap_verify/python/lsst/ap/verify/measurements/compute_metrics.py", line 60, in measure_from_metadata
                for metric, task in timing_map:
              File "/home/mrawls/daf_persistence/python/lsst/daf/persistence/policy.py", line 168, in __getitem__
                for key in name.split('.'):
            AttributeError: 'int' object has no attribute 'split'
            

            Once this is resolved (Krzysztof Findeisen?), and I write a few tests for ap_pipe as promised, this ticket will be ready for review. I plan to upate the DMTN-039 technote for ap_pipe as part of DM-11422 when the coadd template functionality is implemented.

            Show
            mrawls Meredith Rawls added a comment - Thanks to help from Colin Slater , both ap_pipe and ap_verify may now successfully be run from the command line, for all steps from ingestion through difference imaging. For now, the template visit is hardwired to 410929 and the template ccdnum is hardwired to 25. This will be fixed when coadds are accepted as templates in DM-11422 . To try it out: setup obs_decam , ap_pipe , ap_verify_hits2015 , and ap_verify (as well as a non-buggy daf_persistence ) cd to wherever you cloned ap_pipe and ap_verify for ap_pipe alone: python ap_pipe/bin.src/ap_pipe.py -d ap_verify_hits2015/ -o zomg5 -i "visit=410985 ccdnum=25" (you should be able to choose a different visit from the Blind15A_40 field if you prefer) for ap_verify : python ap_verify/bin.src/ap_verify.py --dataset HiTS2015 --output zomg6 --dataIdString "visit=410985 ccdnum=25" --silent (again, you may choose a different Blind15A_40 visit if desired) Both of these have successfully produced output on lsst-dev in /project/mrawls/prototype_ap/ in the zomg5 and zomg6 directories, respectively. However, after ap_verify finishes running all the parts of ap_pipe , it does throw the following exception ap.verify.pipeline_driver.run_ap_pipe INFO: Pipeline complete Traceback (most recent call last): File "ap_verify/bin.src/ap_verify.py", line 27, in <module> run_ap_verify() File "/project/mrawls/prototype_ap/ap_verify/python/lsst/ap/verify/ap_verify.py", line 162, in run_ap_verify _measure_final_properties(metadata, job) File "/project/mrawls/prototype_ap/ap_verify/python/lsst/ap/verify/ap_verify.py", line 140, in _measure_final_properties measurements = measure_from_metadata(metadata) File "/project/mrawls/prototype_ap/ap_verify/python/lsst/ap/verify/measurements/compute_metrics.py", line 60, in measure_from_metadata for metric, task in timing_map: File "/home/mrawls/daf_persistence/python/lsst/daf/persistence/policy.py", line 168, in __getitem__ for key in name.split('.'): AttributeError: 'int' object has no attribute 'split' Once this is resolved ( Krzysztof Findeisen ?), and I write a few tests for ap_pipe as promised, this ticket will be ready for review. I plan to upate the DMTN-039 technote for ap_pipe as part of DM-11422 when the coadd template functionality is implemented.
            Hide
            mrawls Meredith Rawls added a comment -

            Krzysztof Findeisen, please let me know if you would rather not do this review.

            Show
            mrawls Meredith Rawls added a comment - Krzysztof Findeisen , please let me know if you would rather not do this review.
            Hide
            krzys Krzysztof Findeisen added a comment -

            I've left comments on the various branches. I would still like to see ap_verify not need to know about ap_pipe's output organization, but hiding that inside ap_pipe is probably best left to a separate ticket.

            Show
            krzys Krzysztof Findeisen added a comment - I've left comments on the various branches. I would still like to see ap_verify not need to know about ap_pipe 's output organization, but hiding that inside ap_pipe is probably best left to a separate ticket.

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                mrawls Meredith Rawls
                Reviewers:
                Krzysztof Findeisen
                Watchers:
                Eric Bellm, Krzysztof Findeisen, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel