Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31924

Design and implement configurable data ID packing system

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_base, skymap
    • Labels:
      None

      Description

      Implements RFC-785; see that proposal for design constraints and goals.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            John Parejko, can I ask you for a fairly big review? It's on topics you've worked on before, and you're one of the people I'd like to spread some of the knowledge about this system to.

            This implements RFC-785 (with some "FYI" notes on how that's played out on RFC-917), and it touches a lot of packages:

            • pex_config has a tiny change to add a RegistryField feature I want to use later.
            • daf_butler has some deprecations and changes to tests to avoid using deprecated things.
            • pipe_base has the default implementation of the logic that packs visit+detector and exposure+detector data IDs (moved from daf_butler, but heavily modified), along with a system to let the Instrument class set the packer without having to do obs_* package overrides for every single relevant PipelineTask. This is the first half of the real guts of the ticket.
            • skymap has some changes to an existing data ID packer to make it configurable.
            • obs_base just has a deprecation.
            • meas_base has the new high-level helper code that almost all tasks will use to configure and get ID generators. This is the other half of the guts.
            • ip_diffim, pipe_tasks, ap_association, and ap_verify have mostly-mechanical updates to reflect all of those upstream changes, complicated in some cases by it being unclear which Task methods other than "run" are public and hence need deprecation periods for changes (I've made some reasonable guesses, as you'll see).

            An earlier version of this has been through Jenkins, so while I'm just starting another run right now, I don't expect any big surprises. I'll also do a manual ap_verify run, since this is exactly the kind of ticket that otherwise results in a nightly failure of that.
            I'm hoping to merge this some time early neext
            I'd ideally like to merge this early week, if all goes well; if you're unavailable for a review of this size on that timescale, please let me know and I'll find another victim.

            Show
            jbosch Jim Bosch added a comment - John Parejko , can I ask you for a fairly big review? It's on topics you've worked on before, and you're one of the people I'd like to spread some of the knowledge about this system to. This implements RFC-785 (with some "FYI" notes on how that's played out on RFC-917 ), and it touches a lot of packages: pex_config has a tiny change to add a RegistryField feature I want to use later. daf_butler has some deprecations and changes to tests to avoid using deprecated things. pipe_base has the default implementation of the logic that packs visit+detector and exposure+detector data IDs (moved from daf_butler, but heavily modified), along with a system to let the Instrument class set the packer without having to do obs_* package overrides for every single relevant PipelineTask. This is the first half of the real guts of the ticket. skymap has some changes to an existing data ID packer to make it configurable. obs_base just has a deprecation. meas_base has the new high-level helper code that almost all tasks will use to configure and get ID generators. This is the other half of the guts. ip_diffim , pipe_tasks , ap_association , and ap_verify have mostly-mechanical updates to reflect all of those upstream changes, complicated in some cases by it being unclear which Task methods other than "run" are public and hence need deprecation periods for changes (I've made some reasonable guesses, as you'll see). An earlier version of this has been through Jenkins, so while I'm just starting another run right now, I don't expect any big surprises. I'll also do a manual ap_verify run, since this is exactly the kind of ticket that otherwise results in a nightly failure of that. I'm hoping to merge this some time early neext I'd ideally like to merge this early week, if all goes well; if you're unavailable for a review of this size on that timescale, please let me know and I'll find another victim.
            Hide
            jbosch Jim Bosch added a comment -

            Local ap_verify runs successful.

            Show
            jbosch Jim Bosch added a comment - Local ap_verify runs successful.
            Hide
            jbosch Jim Bosch added a comment -

            Ok, I think I've responded to all review comments and made all resulting changes I'd intended, so I'm ready for another round whenever you are.

            Show
            jbosch Jim Bosch added a comment - Ok, I think I've responded to all review comments and made all resulting changes I'd intended, so I'm ready for another round whenever you are.
            Hide
            Parejkoj John Parejko added a comment -

            I've finished my second pass of comments: I think everything remaining should be trivial to sort out. Thanks for the docs updates: this is a complicated enough system that good descriptions and examples are very important.

            Show
            Parejkoj John Parejko added a comment - I've finished my second pass of comments: I think everything remaining should be trivial to sort out. Thanks for the docs updates: this is a complicated enough system that good descriptions and examples are very important.
            Hide
            jbosch Jim Bosch added a comment -

            Thanks! Since it sounds like you'd done looking at PRs, I'll go start some rebasing and Jenkinsing.

            Show
            jbosch Jim Bosch added a comment - Thanks! Since it sounds like you'd done looking at PRs, I'll go start some rebasing and Jenkinsing.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              John Parejko
              Watchers:
              Jim Bosch, John Parejko
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.