# Store begin/end times of ap_pipe in ap_verify

XMLWordPrintable

## Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
2
• Sprint:
AP F19-6 (November)
• Team:

## Description

Chris Morrison asked to get time stamps for the start and end of an ap_verify job (excluding metric calculation, if possible). Find a way to do so; probably the StartUtc and EndUtc values logged by pipe.base.timeMethod.

## Activity

Hide
Krzysztof Findeisen added a comment -

While the StartUtc and EndUtc metadata do the job (format looks like 2019-11-13T19:24:16.151550), they can't be turned into metrics because metrics need to be Astropy quantities, and therefore floating-point scalars.

One option is to add the end time as metadata to the measurements themselves. This is flexible, but will read a bit awkwardly in inspect_job.py:

  ap_pipe.ApPipeTime = 69.0439 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:16.151550'})  pipe_tasks.ProcessCcdTime = 18.4718 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:10.052496'}) 

Depending on what you need the time stamps for, it might be easier to read them directly from the Butler (i.e., get("apPipe_metadata", dataId)). Does this option work better for you, or does the info need to be inside the .json files?

Show
Krzysztof Findeisen added a comment - While the StartUtc and EndUtc metadata do the job (format looks like 2019-11-13T19:24:16.151550 ), they can't be turned into metrics because metrics need to be Astropy quantities, and therefore floating-point scalars. One option is to add the end time as metadata to the measurements themselves. This is flexible, but will read a bit awkwardly in inspect_job.py : ap_pipe.ApPipeTime = 69.0439 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:16.151550'}) pipe_tasks.ProcessCcdTime = 18.4718 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:10.052496'}) Depending on what you need the time stamps for, it might be easier to read them directly from the Butler (i.e., get("apPipe_metadata", dataId) ). Does this option work better for you, or does the info need to be inside the .json files?
Hide
Chris Morrison added a comment -

I don't necessarily need them to come through the json files. One of the reasons I can see in wanting to do that is that I'm reading a lot of the other quantities "nNewDiaobjects", "fracUpdated", etc. from the json files. If the butler access is the best way to take care of this I'm happy to do so.

Right now I'm reading the data directly from the json files with my own script instead of parsing the output from inspect_job.py.

Show
Chris Morrison added a comment - I don't necessarily need them to come through the json files. One of the reasons I can see in wanting to do that is that I'm reading a lot of the other quantities "nNewDiaobjects", "fracUpdated", etc. from the json files. If the butler access is the best way to take care of this I'm happy to do so. Right now I'm reading the data directly from the json files with my own script instead of parsing the output from inspect_job.py.
Hide
Krzysztof Findeisen added a comment -

Hi Chris Morrison, can you review this? I've included just the end time (no start time) as metadata, to keep the information in the json file without adding too much clutter.

Show
Krzysztof Findeisen added a comment - Hi Chris Morrison , can you review this? I've included just the end time (no start time) as metadata, to keep the information in the json file without adding too much clutter.

## People

• Assignee:
Krzysztof Findeisen
Reporter:
Krzysztof Findeisen
Reviewers:
Chris Morrison
Watchers:
Chris Morrison, Krzysztof Findeisen