Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21916

SQuaSH upload of Gen 3 Measurements

    XMLWordPrintable

    Details

      Description

      The existing upload script (dispatch_verify.py) assumes that metric values have been persisted in Job objects. Once we can store metric values in Gen 3 repositories, we need to be able to read those as part of the upload process.

      Story point value assumes that using Gen 3 Butler inside something that's not a PipelineTask is difficult.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Per discussion on DM-21885, the new code will need to be able to get what's currently Job metadata out of the registry. Presumably Measurement metadata will be unaffected.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Per discussion on DM-21885 , the new code will need to be able to get what's currently Job metadata out of the registry. Presumably Measurement metadata will be unaffected.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            See verify/tests/test_measurements.py and verify/tests/butler_utils.py for some examples of using the Butler in arbitrary scripts. I don't try to read the registry, though.

            Show
            krzys Krzysztof Findeisen added a comment - - edited See verify/tests/test_measurements.py and verify/tests/butler_utils.py for some examples of using the Butler in arbitrary scripts. I don't try to read the registry, though.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Jonathan Sick asked on #dm-square about whether the SQuaSH API needs to be updated to handle uploads of up to one metric value (measurement) per metric per data ID, rather than just per metric.

            Show
            krzys Krzysztof Findeisen added a comment - Jonathan Sick asked on #dm-square about whether the SQuaSH API needs to be updated to handle uploads of up to one metric value (measurement) per metric per data ID, rather than just per metric.
            Hide
            afausti Angelo Fausti added a comment -

            Krzysztof Findeisen this relates to the work Simon Krughoff is doing with the V&V team PR #13 has code that collects metric values from a Gen3 Butler repository, adds job metadata and create a verification job that can be upload to SQuaSH.

            Show
            afausti Angelo Fausti added a comment - Krzysztof Findeisen this relates to the work Simon Krughoff is doing with the V&V team PR #13 has code that collects metric values from a Gen3 Butler repository, adds job metadata and create a verification job that can be upload to SQuaSH.
            Hide
            krzys Krzysztof Findeisen added a comment -

            I assumed the script in metric-pipeline-tasks was a workaround for this issue. Or do you plan for the Job class to always be required as a sort of intermediate step?

            Show
            krzys Krzysztof Findeisen added a comment - I assumed the script in metric-pipeline-tasks was a workaround for this issue. Or do you plan for the Job class to always be required as a sort of intermediate step?
            Hide
            afausti Angelo Fausti added a comment -

            I think reconstructing the Job as an intermediate step is a possible approach to use dispatch_verify.py to submit to SQuaSH. This code could live in the lsst.verify package and be used by dispatch_verify.py if pointed to a Butler Gen 3 registry. Haven't thought much about this yet but it seems promising.

            Show
            afausti Angelo Fausti added a comment - I think reconstructing the Job as an intermediate step is a possible approach to use dispatch_verify.py to submit to SQuaSH. This code could live in the lsst.verify package and be used by dispatch_verify.py if pointed to a Butler Gen 3 registry. Haven't thought much about this yet but it seems promising.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            I've been working on updating our SQuaSH stuff for DM-24262, and I noticed we don't seem to have a tag(?) that distinguishes between Gen 2 and Gen 3 runs. This would be very useful once we have more data from both.

            Show
            krzys Krzysztof Findeisen added a comment - - edited I've been working on updating our SQuaSH stuff for DM-24262 , and I noticed we don't seem to have a tag(?) that distinguishes between Gen 2 and Gen 3 runs. This would be very useful once we have more data from both.
            Hide
            krughoff Simon Krughoff added a comment -

            That is a good point. I hope to have validate_drp running on HSC in nightlies using both gen2 and gen3 in the next few weeks. It would be very good to be able to differentiate between the two using tags.

            I was initially thinking about starting another database, but I think separating by tags is probably better.

            Show
            krughoff Simon Krughoff added a comment - That is a good point. I hope to have validate_drp running on HSC in nightlies using both gen2 and gen3 in the next few weeks. It would be very good to be able to differentiate between the two using tags. I was initially thinking about starting another database, but I think separating by tags is probably better.
            Show
            ebellm Eric Bellm added a comment - Here is a Slack discussion of the faro implementation ( https://github.com/lsst-dmsst/metric-pipeline-tasks/blob/master/python/metric_pipeline_scripts/jobReporter.py )
            Hide
            krzys Krzysztof Findeisen added a comment -

            Simon Krughoff, Angelo Fausti, it seems I misunderstood the constraints of the existing verify framework. Would it be possible/practical to move the faro Gen 3 -> Job translator into verify, so that it's available to any package that uses verify?

            Show
            krzys Krzysztof Findeisen added a comment - Simon Krughoff , Angelo Fausti , it seems I misunderstood the constraints of the existing verify framework. Would it be possible/practical to move the faro Gen 3 -> Job translator into verify , so that it's available to any package that uses verify ?
            Hide
            krzys Krzysztof Findeisen added a comment -

            Discussed with Simon Krughoff on Slack, and he agreed to using jobReporter in lsst.verify.

            Show
            krzys Krzysztof Findeisen added a comment - Discussed with Simon Krughoff  on Slack, and he agreed to using jobReporter in lsst.verify .
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Simon Krughoff, I apparently don't have permissions to push ticket branches to lsst-dmsst/metric-pipeline-tasks. What's the SST procedure for dealing with this? Should I fork?

            For reference, I'm trying to follow https://developer.lsst.io/stack/transferring-code.html.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Simon Krughoff , I apparently don't have permissions to push ticket branches to lsst-dmsst/metric-pipeline-tasks . What's the SST procedure for dealing with this? Should I fork? For reference, I'm trying to follow https://developer.lsst.io/stack/transferring-code.html .
            Hide
            krughoff Simon Krughoff added a comment -

            Sure, forking seems reasonable. FWIW, it's going to move to the lsst org in the next week or so, if you'd prefer to just wait.

            Show
            krughoff Simon Krughoff added a comment - Sure, forking seems reasonable. FWIW, it's going to move to the lsst org in the next week or so, if you'd prefer to just wait.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Thanks for the tip, but given that rebasing the double-branches is excruciatingly painful, I'd like to minimize the number of changes I need to account for when I do so. I'll open PRs as soon as Jenkins finishes.

            Show
            krzys Krzysztof Findeisen added a comment - Thanks for the tip, but given that rebasing the double-branches is excruciatingly painful, I'd like to minimize the number of changes I need to account for when I do so. I'll open PRs as soon as Jenkins finishes.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited
            Show
            krzys Krzysztof Findeisen added a comment - - edited PRs are verify#77 and metric-pipeline-tasks#66 .
            Hide
            krughoff Simon Krughoff added a comment -

            Looks great. I have not tested it explicitly, but since the interfaces are all preserved in the mertric_pipeline_tasks repository, I don't see any reason why these changes would cause problems

            Show
            krughoff Simon Krughoff added a comment - Looks great. I have not tested it explicitly, but since the interfaces are all preserved in the mertric_pipeline_tasks repository, I don't see any reason why these changes would cause problems

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Simon Krughoff
              Watchers:
              Angelo Fausti, Eric Bellm, Ian Sullivan, Jim Bosch, Krzysztof Findeisen, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: