Thanks to help from Colin Slater, both ap_pipe and ap_verify may now successfully be run from the command line, for all steps from ingestion through difference imaging. For now, the template visit is hardwired to 410929 and the template ccdnum is hardwired to 25. This will be fixed when coadds are accepted as templates in DM-11422.
To try it out:
- setup obs_decam, ap_pipe, ap_verify_hits2015, and ap_verify (as well as a non-buggy daf_persistence)
- cd to wherever you cloned ap_pipe and ap_verify
- for ap_pipe alone: python ap_pipe/bin.src/ap_pipe.py -d ap_verify_hits2015/ -o zomg5 -i "visit=410985 ccdnum=25" (you should be able to choose a different visit from the Blind15A_40 field if you prefer)
- for ap_verify: python ap_verify/bin.src/ap_verify.py --dataset HiTS2015 --output zomg6 --dataIdString "visit=410985 ccdnum=25" --silent (again, you may choose a different Blind15A_40 visit if desired)
Both of these have successfully produced output on lsst-dev in /project/mrawls/prototype_ap/ in the zomg5 and zomg6 directories, respectively. However, after ap_verify finishes running all the parts of ap_pipe, it does throw the following exception
ap.verify.pipeline_driver.run_ap_pipe INFO: Pipeline complete
|
Traceback (most recent call last):
|
File "ap_verify/bin.src/ap_verify.py", line 27, in <module>
|
run_ap_verify()
|
File "/project/mrawls/prototype_ap/ap_verify/python/lsst/ap/verify/ap_verify.py", line 162, in run_ap_verify
|
_measure_final_properties(metadata, job)
|
File "/project/mrawls/prototype_ap/ap_verify/python/lsst/ap/verify/ap_verify.py", line 140, in _measure_final_properties
|
measurements = measure_from_metadata(metadata)
|
File "/project/mrawls/prototype_ap/ap_verify/python/lsst/ap/verify/measurements/compute_metrics.py", line 60, in measure_from_metadata
|
for metric, task in timing_map:
|
File "/home/mrawls/daf_persistence/python/lsst/daf/persistence/policy.py", line 168, in __getitem__
|
for key in name.split('.'):
|
AttributeError: 'int' object has no attribute 'split'
|
Once this is resolved (Krzysztof Findeisen?), and I write a few tests for ap_pipe as promised, this ticket will be ready for review. I plan to upate the DMTN-039 technote for ap_pipe as part of DM-11422 when the coadd template functionality is implemented.
We have ingestion and calib ingestion working!
Parts of it are a bit messy, but if one pulls the
DM-11390branch of ap_verify and ap_pipe, and does setup for both as well as for ap_verify_hits2015, it is possible to run this from the command line and have it work through calibration ingestion:python ap_verify/bin.src/ap_verify.py --dataset HiTS2015 --output your_desired_output_dir --dataIdString "visit=410985" --silent
(Note that the visit number doesn't matter just yet because ingestion and calib ingestion do not use any dataId info and instead they just ingest everything in the dataset repo provided.)