Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22093

Store begin/end times of ap_pipe in ap_verify

    XMLWordPrintable

Details

    • Improvement
    • Status: Done
    • Resolution: Done
    • None
    • ap_pipe, verify
    • None
    • 2
    • AP F19-6 (November)
    • Alert Production

    Description

      cmorrison asked to get time stamps for the start and end of an ap_verify job (excluding metric calculation, if possible). Find a way to do so; probably the StartUtc and EndUtc values logged by pipe.base.timeMethod.

      Attachments

        Issue Links

          Activity

            While the StartUtc and EndUtc metadata do the job (format looks like 2019-11-13T19:24:16.151550), they can't be turned into metrics because metrics need to be Astropy quantities, and therefore floating-point scalars.

            One option is to add the end time as metadata to the measurements themselves. This is flexible, but will read a bit awkwardly in inspect_job.py:

                                  ap_pipe.ApPipeTime =  69.0439 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:16.151550'})
                           pipe_tasks.ProcessCcdTime =  18.4718 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:10.052496'})
            

            Depending on what you need the time stamps for, it might be easier to read them directly from the Butler (i.e., get("apPipe_metadata", dataId)). Does this option work better for you, or does the info need to be inside the .json files?

            krzys Krzysztof Findeisen added a comment - While the StartUtc and EndUtc metadata do the job (format looks like 2019-11-13T19:24:16.151550 ), they can't be turned into metrics because metrics need to be Astropy quantities, and therefore floating-point scalars. One option is to add the end time as metadata to the measurements themselves. This is flexible, but will read a bit awkwardly in inspect_job.py : ap_pipe.ApPipeTime = 69.0439 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:16.151550'}) pipe_tasks.ProcessCcdTime = 18.4718 s ({'estimator': 'pipe.base.timeMethod', 'end': '2019-11-13T19:24:10.052496'}) Depending on what you need the time stamps for, it might be easier to read them directly from the Butler (i.e., get("apPipe_metadata", dataId) ). Does this option work better for you, or does the info need to be inside the .json files?

            I don't necessarily need them to come through the json files. One of the reasons I can see in wanting to do that is that I'm reading a lot of the other quantities "nNewDiaobjects", "fracUpdated", etc. from the json files. If the butler access is the best way to take care of this I'm happy to do so.

            Right now I'm reading the data directly from the json files with my own script instead of parsing the output from inspect_job.py.

            cmorrison Chris Morrison [X] (Inactive) added a comment - I don't necessarily need them to come through the json files. One of the reasons I can see in wanting to do that is that I'm reading a lot of the other quantities "nNewDiaobjects", "fracUpdated", etc. from the json files. If the butler access is the best way to take care of this I'm happy to do so. Right now I'm reading the data directly from the json files with my own script instead of parsing the output from inspect_job.py.

            Hi cmorrison, can you review this? I've included just the end time (no start time) as metadata, to keep the information in the json file without adding too much clutter.

            krzys Krzysztof Findeisen added a comment - Hi cmorrison , can you review this? I've included just the end time (no start time) as metadata, to keep the information in the json file without adding too much clutter.

            People

              krzys Krzysztof Findeisen
              krzys Krzysztof Findeisen
              Chris Morrison [X] (Inactive)
              Chris Morrison [X] (Inactive), Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.