Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17489

Initial prototype of Gen3 interfaces for human-curated master calibrations

    Details

      Description

      DM-16467 added some Python and bash scripts that add brighter-fatter kernel and defect datasets to a Gen3 repo.  These use direct SQL calls instead of the Registry API, and have been broken by changes on DM-16227.  This ticket will move that work into the daf.butler.Instrument class and its obs_* package specializations, allowing it to be done automatically by the gen2convert tool (in addition to fixing the breakage, of course).

      Coming up with a new, consistent-across-obs-packages way to manage human-curated source for these datasets is definitely out of scope right now, but this might provide an example of how the Butler side of that might look to those who do design that.

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          Simon Krughoff, assigning this to you in the hopes it might be a good way to spread some knowledge in a good direction.  PRs are:

          Nate Lust, if it'd be helpful for your ongoing work to get this on master sooner, feel free to steal the review; I'll be around off-and-on over the weekend to follow up if that's the case.

          Show
          jbosch Jim Bosch added a comment - Simon Krughoff , assigning this to you in the hopes it might be a good way to spread some knowledge in a good direction.  PRs are: daf_butler obs_subaru ci_hsc Nate Lust , if it'd be helpful for your ongoing work to get this on master sooner, feel free to steal the review; I'll be around off-and-on over the weekend to follow up if that's the case.
          Hide
          krughoff Simon Krughoff added a comment -

          Looks generally very good. My only concern is that the ticket talks about guiding functional code into daf_butler, but that doesn't appear to have happened. The code seems to have been migrated into ci_hsc instead. I'm sure there's a reason, but it was not obvious to me from the review why that discrepancy exists.

          Show
          krughoff Simon Krughoff added a comment - Looks generally very good. My only concern is that the ticket talks about guiding functional code into daf_butler , but that doesn't appear to have happened. The code seems to have been migrated into ci_hsc instead. I'm sure there's a reason, but it was not obvious to me from the review why that discrepancy exists.
          Hide
          jbosch Jim Bosch added a comment -

          My only concern is that the ticket talks about guiding functional code into daf_butler, but that doesn't appear to have happened. The code seems to have been migrated into ci_hsc instead

          When writing up the ticket, I had forgotten that the gen2convert code in daf_butler doesn't really have a good way to obtain the right Instrument class.  We'll probably want to address that eventually if we can cook up a good way to run gen2convert generically from the command-line, but at present it's really more of a set of high-level tools that can be used to put together a convert for a particular repo (which is what we've done in ci_hsc).

          Show
          jbosch Jim Bosch added a comment - My only concern is that the ticket talks about guiding functional code into daf_butler , but that doesn't appear to have happened. The code seems to have been migrated into ci_hsc instead When writing up the ticket, I had forgotten that the gen2convert code in daf_butler doesn't really have a good way to obtain the right Instrument class.  We'll probably want to address that eventually if we can cook up a good way to run gen2convert generically from the command-line, but at present it's really more of a set of high-level tools that can be used to put together a convert for a particular repo (which is what we've done in ci_hsc).
          Hide
          krughoff Simon Krughoff added a comment -

          OK. Sounds fine to me.

          Show
          krughoff Simon Krughoff added a comment - OK. Sounds fine to me.

            People

            • Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Simon Krughoff
              Watchers:
              Christopher Waters, Jim Bosch, Nate Lust, Simon Krughoff
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel