Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22093

Store begin/end times of ap_pipe in ap_verify

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_pipe, verify
    • Labels:
      None
    • Story Points:
      2
    • Sprint:
      AP F19-6 (November)
    • Team:
      Alert Production

      Description

      Chris Morrison asked to get time stamps for the start and end of an ap_verify job (excluding metric calculation, if possible). Find a way to do so; probably the StartUtc and EndUtc values logged by pipe.base.timeMethod.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            While the StartUtc and EndUtc metadata do the job (format looks like 2019-11-13T19:24:16.151550), they can't be turned into metrics because metrics need to be Astropy quantities, and therefore floating-point scalars.

            One option is to add the end time as metadata to the measurements themselves. This is flexible, but will read a bit awkwardly in inspect_job.py:

                                  ap_pipe.ApPipeTime =  69.0439 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:16.151550'})
                           pipe_tasks.ProcessCcdTime =  18.4718 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:10.052496'})
            

            Depending on what you need the time stamps for, it might be easier to read them directly from the Butler (i.e., get("apPipe_metadata", dataId)). Does this option work better for you, or does the info need to be inside the .json files?

            Show
            krzys Krzysztof Findeisen added a comment - While the StartUtc and EndUtc metadata do the job (format looks like 2019-11-13T19:24:16.151550 ), they can't be turned into metrics because metrics need to be Astropy quantities, and therefore floating-point scalars. One option is to add the end time as metadata to the measurements themselves. This is flexible, but will read a bit awkwardly in inspect_job.py : ap_pipe.ApPipeTime = 69.0439 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:16.151550'}) pipe_tasks.ProcessCcdTime = 18.4718 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:10.052496'}) Depending on what you need the time stamps for, it might be easier to read them directly from the Butler (i.e., get("apPipe_metadata", dataId) ). Does this option work better for you, or does the info need to be inside the .json files?
            Hide
            cmorrison Chris Morrison added a comment -

            I don't necessarily need them to come through the json files. One of the reasons I can see in wanting to do that is that I'm reading a lot of the other quantities "nNewDiaobjects", "fracUpdated", etc. from the json files. If the butler access is the best way to take care of this I'm happy to do so.

            Right now I'm reading the data directly from the json files with my own script instead of parsing the output from inspect_job.py.

            Show
            cmorrison Chris Morrison added a comment - I don't necessarily need them to come through the json files. One of the reasons I can see in wanting to do that is that I'm reading a lot of the other quantities "nNewDiaobjects", "fracUpdated", etc. from the json files. If the butler access is the best way to take care of this I'm happy to do so. Right now I'm reading the data directly from the json files with my own script instead of parsing the output from inspect_job.py.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Hi Chris Morrison, can you review this? I've included just the end time (no start time) as metadata, to keep the information in the json file without adding too much clutter.

            Show
            krzys Krzysztof Findeisen added a comment - Hi Chris Morrison , can you review this? I've included just the end time (no start time) as metadata, to keep the information in the json file without adding too much clutter.

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                Chris Morrison
                Watchers:
                Chris Morrison, Krzysztof Findeisen
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel