Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17489

Initial prototype of Gen3 interfaces for human-curated master calibrations

    XMLWordPrintable

Details

    Description

      DM-16467 added some Python and bash scripts that add brighter-fatter kernel and defect datasets to a Gen3 repo.  These use direct SQL calls instead of the Registry API, and have been broken by changes on DM-16227.  This ticket will move that work into the daf.butler.Instrument class and its obs_* package specializations, allowing it to be done automatically by the gen2convert tool (in addition to fixing the breakage, of course).

      Coming up with a new, consistent-across-obs-packages way to manage human-curated source for these datasets is definitely out of scope right now, but this might provide an example of how the Butler side of that might look to those who do design that.

      Attachments

        Activity

          jbosch Jim Bosch added a comment -

          krughoff, assigning this to you in the hopes it might be a good way to spread some knowledge in a good direction.  PRs are:

          nlust, if it'd be helpful for your ongoing work to get this on master sooner, feel free to steal the review; I'll be around off-and-on over the weekend to follow up if that's the case.

          jbosch Jim Bosch added a comment - krughoff , assigning this to you in the hopes it might be a good way to spread some knowledge in a good direction.  PRs are: daf_butler obs_subaru ci_hsc nlust , if it'd be helpful for your ongoing work to get this on master sooner, feel free to steal the review; I'll be around off-and-on over the weekend to follow up if that's the case.

          Looks generally very good. My only concern is that the ticket talks about guiding functional code into daf_butler, but that doesn't appear to have happened. The code seems to have been migrated into ci_hsc instead. I'm sure there's a reason, but it was not obvious to me from the review why that discrepancy exists.

          krughoff Simon Krughoff (Inactive) added a comment - Looks generally very good. My only concern is that the ticket talks about guiding functional code into daf_butler , but that doesn't appear to have happened. The code seems to have been migrated into ci_hsc instead. I'm sure there's a reason, but it was not obvious to me from the review why that discrepancy exists.
          jbosch Jim Bosch added a comment -

          My only concern is that the ticket talks about guiding functional code into daf_butler, but that doesn't appear to have happened. The code seems to have been migrated into ci_hsc instead

          When writing up the ticket, I had forgotten that the gen2convert code in daf_butler doesn't really have a good way to obtain the right Instrument class.  We'll probably want to address that eventually if we can cook up a good way to run gen2convert generically from the command-line, but at present it's really more of a set of high-level tools that can be used to put together a convert for a particular repo (which is what we've done in ci_hsc).

          jbosch Jim Bosch added a comment - My only concern is that the ticket talks about guiding functional code into daf_butler , but that doesn't appear to have happened. The code seems to have been migrated into ci_hsc instead When writing up the ticket, I had forgotten that the gen2convert code in daf_butler doesn't really have a good way to obtain the right Instrument class.  We'll probably want to address that eventually if we can cook up a good way to run gen2convert generically from the command-line, but at present it's really more of a set of high-level tools that can be used to put together a convert for a particular repo (which is what we've done in ci_hsc).

          OK. Sounds fine to me.

          krughoff Simon Krughoff (Inactive) added a comment - OK. Sounds fine to me.

          People

            jbosch Jim Bosch
            jbosch Jim Bosch
            Simon Krughoff (Inactive)
            Christopher Waters, Jim Bosch, Nate Lust, Simon Krughoff (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.