Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21898

Create command-line tools for Gen3 repo administration

    XMLWordPrintable

    Details

    • Story Points:
      0
    • Sprint:
      DB_F20_09, DB_S21_12, DB_F21_06, DB_S22_12
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      This is an umbrella ticket for creating (or polishing) command-line repo administration tools for operations that don't involve running PipelineTasks.  These may include:

      • repo creation (makeButlerRepo.py exists, but may need polish/updates)
      • Gen2 repo conversion
      • skymap registration (makeSkyMapGen3.py exists, but may need polish/updates)
      • visit definition (will be split off from ingest in the future)
      • raw ingest
      • repo subset
      • common Registry queries
      • remove collections
      • manage master calibration collections
      • remove dataset types that have been registered (likely used when a mistake has happened)

      Actual work should go on new tickets.

        Attachments

          Issue Links

            Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue is blocked by DM-15257 [ DM-15257 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-21915 [ DM-21915 ]
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            tjenness Tim Jenness made changes -
            Watchers Andy Salnikov, Jim Bosch, Michelle Gower, Nate Lust [ Andy Salnikov, Jim Bosch, Michelle Gower, Nate Lust ] Andy Salnikov, Jim Bosch, Michelle Gower, Nate Lust, Nate Pease [ Andy Salnikov, Jim Bosch, Michelle Gower, Nate Lust, Nate Pease ]
            Hide
            tjenness Tim Jenness added a comment -

            Nate Pease [X] this is the umbrella ticket for all the command line tool work. If you could link to this ticket with a RelatesTo link that would be good. I don't think this ticket needs to be assigned to you since it's highly likely no work will be done on this directly.

            Show
            tjenness Tim Jenness added a comment - Nate Pease [X] this is the umbrella ticket for all the command line tool work. If you could link to this ticket with a RelatesTo link that would be good. I don't think this ticket needs to be assigned to you since it's highly likely no work will be done on this directly.
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            ok

            Show
            npease Nate Pease [X] (Inactive) added a comment - ok
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-24245 [ DM-24245 ]
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-24329 [ DM-24329 ]
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-24330 [ DM-24330 ]
            Hide
            tjenness Tim Jenness added a comment -

            I'm going to set the story points on this ticket to 0 since the work is happening elsewhere.

            Show
            tjenness Tim Jenness added a comment - I'm going to set the story points on this ticket to 0 since the work is happening elsewhere.
            tjenness Tim Jenness made changes -
            Story Points 20 0
            gruendl Robert Gruendl [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 25783 ]
            Hide
            tjenness Tim Jenness added a comment -

            We probably should shut this ticket down and replace it with explicit tickets for all the items on the list above that have not yet been done – one ticket per subcommand. Nate Pease [X] can you do that and then close this?

            Show
            tjenness Tim Jenness added a comment - We probably should shut this ticket down and replace it with explicit tickets for all the items on the list above that have not yet been done – one ticket per subcommand. Nate Pease [X] can you do that and then close this?
            npease Nate Pease [X] (Inactive) made changes -
            Assignee Nate Pease [ npease ]
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-25244 [ 435560 ]
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            Yes, will do. I'm changing the team & epic on this ticket to reflect that it's assigned to me to do that (and so it shows up in my sprint view)

            Show
            npease Nate Pease [X] (Inactive) added a comment - Yes, will do. I'm changing the team & epic on this ticket to reflect that it's assigned to me to do that (and so it shows up in my sprint view)
            npease Nate Pease [X] (Inactive) made changes -
            Sprint DB_F20_09 [ 1048 ]
            Team Data Release Production [ 10301 ] Data Access and Database [ 10204 ]
            Urgent? off
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-26684 [ DM-26684 ]
            Hide
            jbosch Jim Bosch added a comment -

            I'm creating a few tickets related to this ticket right now, in order to capture blockers for the "feature parity" milestone.  I will be linking them to this one as "relates to" and at least mostly assigning them to Nate Pease [X] and DAX to start with, as I figure it's a good bet he'll be the one working on them.

            Show
            jbosch Jim Bosch added a comment - I'm creating a few tickets related to this ticket right now, in order to capture blockers for the "feature parity" milestone.  I will be linking them to this one as "relates to" and at least mostly assigning them to Nate Pease [X] and DAX to start with, as I figure it's a good bet he'll be the one working on them.
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-26685 [ DM-26685 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-26686 [ DM-26686 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-26688 [ DM-26688 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-26689 [ DM-26689 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-26690 [ DM-26690 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-26691 [ DM-26691 ]
            nlust Nate Lust made changes -
            Description This is an umbrella ticket for creating (or polishing) command-line repo administration tools for operations that don't involve running PipelineTasks.  These may include:
             * repo creation (makeButlerRepo.py exists, but may need polish/updates)
             * Gen2 repo conversion
             * skymap registration (makeSkyMapGen3.py exists, but may need polish/updates)
             * visit definition (will be split off from ingest in the future)
             * raw ingest
             * repo subset
             * common Registry queries
             * remove collections
             * manage master calibration collections{color:#000000}
            {color}

            Actual work should go on new tickets.
            This is an umbrella ticket for creating (or polishing) command-line repo administration tools for operations that don't involve running PipelineTasks.  These may include:
             * repo creation (makeButlerRepo.py exists, but may need polish/updates)
             * Gen2 repo conversion
             * skymap registration (makeSkyMapGen3.py exists, but may need polish/updates)
             * visit definition (will be split off from ingest in the future)
             * raw ingest
             * repo subset
             * common Registry queries
             * remove collections
             * manage master calibration collections{color:#000000}
            * remove dataset types that have been registered (likely used when a mistake has happened)
            {color}

            Actual work should go on new tickets.
            nlust Nate Lust made changes -
            Description This is an umbrella ticket for creating (or polishing) command-line repo administration tools for operations that don't involve running PipelineTasks.  These may include:
             * repo creation (makeButlerRepo.py exists, but may need polish/updates)
             * Gen2 repo conversion
             * skymap registration (makeSkyMapGen3.py exists, but may need polish/updates)
             * visit definition (will be split off from ingest in the future)
             * raw ingest
             * repo subset
             * common Registry queries
             * remove collections
             * manage master calibration collections{color:#000000}
            * remove dataset types that have been registered (likely used when a mistake has happened)
            {color}

            Actual work should go on new tickets.
            This is an umbrella ticket for creating (or polishing) command-line repo administration tools for operations that don't involve running PipelineTasks.  These may include:
             * repo creation (makeButlerRepo.py exists, but may need polish/updates)
             * Gen2 repo conversion
             * skymap registration (makeSkyMapGen3.py exists, but may need polish/updates)
             * visit definition (will be split off from ingest in the future)
             * raw ingest
             * repo subset
             * common Registry queries
             * remove collections
             * manage master calibration collections{color:#000000}
             * remove dataset types that have been registered (likely used when a mistake has happened)
            {color}

            Actual work should go on new tickets.
            nlust Nate Lust made changes -
            Description This is an umbrella ticket for creating (or polishing) command-line repo administration tools for operations that don't involve running PipelineTasks.  These may include:
             * repo creation (makeButlerRepo.py exists, but may need polish/updates)
             * Gen2 repo conversion
             * skymap registration (makeSkyMapGen3.py exists, but may need polish/updates)
             * visit definition (will be split off from ingest in the future)
             * raw ingest
             * repo subset
             * common Registry queries
             * remove collections
             * manage master calibration collections{color:#000000}
             * remove dataset types that have been registered (likely used when a mistake has happened)
            {color}

            Actual work should go on new tickets.
            This is an umbrella ticket for creating (or polishing) command-line repo administration tools for operations that don't involve running PipelineTasks.  These may include:
             * repo creation (makeButlerRepo.py exists, but may need polish/updates)
             * Gen2 repo conversion
             * skymap registration (makeSkyMapGen3.py exists, but may need polish/updates)
             * visit definition (will be split off from ingest in the future)
             * raw ingest
             * repo subset
             * common Registry queries
             * remove collections
             * manage master calibration collections
             * remove dataset types that have been registered (likely used when a mistake has happened)
            {color}

            Actual work should go on new tickets.
            nlust Nate Lust made changes -
            Link This issue relates to DM-26600 [ DM-26600 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 25783 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-26856 [ DM-26856 ]
            fritzm Fritz Mueller made changes -
            Epic Link DM-25244 [ 435560 ] DM-27791 [ 442307 ]
            fritzm Fritz Mueller made changes -
            Sprint DB_F20_09 [ 1048 ] DB_F20_09, DB_S21_12 [ 1048, 1065 ]
            fritzm Fritz Mueller made changes -
            Epic Link DM-27791 [ 442307 ] DM-30609 [ 511856 ]
            fritzm Fritz Mueller made changes -
            Sprint DB_F20_09, DB_S21_12 [ 1048, 1065 ] DB_F20_09, DB_S21_12, DB_F21_06 [ 1048, 1065, 1103 ]
            fritzm Fritz Mueller made changes -
            Epic Link DM-30609 [ 511856 ] DM-30628 [ 513191 ]
            fritzm Fritz Mueller made changes -
            Sprint DB_F20_09, DB_S21_12, DB_F21_06 [ 1048, 1065, 1103 ] DB_F20_09, DB_S21_12, DB_F21_06, DB_S22_12 [ 1048, 1065, 1103, 1136 ]
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            I'm working on closing this ticket; I think most of the items listed above have butler CLI commands now. 

            • repo subset: I don't think we have command line tool for this, is one still needed?
            • manage master calibration collections: we have certify-calibrations, is that one part of this? And we have a ticket to add a CLI cmd for Registry.decertify (DM-26686), is that the other part of what's needed for this? 

            These are the rest of the tools listed above, and the butler CLI commands that I think implement them, let me know if I'm wrong about anything.

            • repo creation: butler create
            • Gen2 repo conversion: butler convert
            • skymap registration: butler register-skymap
            • visit definition: butler define-visits
            • raw ingest: butler ingest-raws
            • common Registry queries: query-collections, query-data-ids, query-dataset-types, query-datasets, query-dimension-records
            • remove collections butler remove-collections
            • remove dataset types that have been registered remove-dataset-type

             

            Show
            npease Nate Pease [X] (Inactive) added a comment - I'm working on closing this ticket; I think most of the items listed above have butler CLI commands now.  repo subset: I don't think we have command line tool for this, is one still needed? manage master calibration collections: we have certify-calibrations , is that one part of this? And we have a ticket to add a CLI cmd for Registry.decertify  ( DM-26686 ), is that the other part of what's needed for this?  These are the rest of the tools listed above, and the butler CLI commands that I think implement them, let me know if I'm wrong about anything. repo creation:  butler create Gen2 repo conversion: butler convert skymap registration: butler register-skymap visit definition: butler define-visits raw ingest: butler ingest-raws common Registry queries: query-collections , query-data-ids , query-dataset-types , query-datasets , query-dimension-records remove collections butler remove-collections remove dataset types that have been registered remove-dataset-type  
            Hide
            tjenness Tim Jenness added a comment -

            I'm happy for this umbrella ticket to close.

            I think the repo subset one can probably be folded into making transfer-datasets also transfer the dimension it needs. That is effectively moving some code out of execution butler creation into more reusable form. I think that would go a long way to allowing a subset of data to be directly transferred from one repo to another. A completely generic butler exporter command-line is much harder.

            We do indeed need a decertify command, we also need a way of listing a calibration collection so that the validation dates can be inspected. I don't think we have a ticket for that but we should create one and block it on whatever ticket Jim thinks is stopping that from working. I think decertify is related in that, like remove-runs, you'd probably want it to report the calibrations that are going to be decertified and possibly tell you the validation ranges for anything that is left (are we going to close up holes in validity? We have to at least report timespans where no calibration is valid).

            Show
            tjenness Tim Jenness added a comment - I'm happy for this umbrella ticket to close. I think the repo subset one can probably be folded into making transfer-datasets also transfer the dimension it needs. That is effectively moving some code out of execution butler creation into more reusable form. I think that would go a long way to allowing a subset of data to be directly transferred from one repo to another. A completely generic butler exporter command-line is much harder. We do indeed need a decertify command, we also need a way of listing a calibration collection so that the validation dates can be inspected. I don't think we have a ticket for that but we should create one and block it on whatever ticket Jim thinks is stopping that from working. I think decertify is related in that, like remove-runs, you'd probably want it to report the calibrations that are going to be decertified and possibly tell you the validation ranges for anything that is left (are we going to close up holes in validity? We have to at least report timespans where no calibration is valid).
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-34887 [ DM-34887 ]
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue relates to DM-34888 [ DM-34888 ]
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            Of the commands listed in the description, they are either already implemented or child tickets are created and linked to this issue.

            Show
            npease Nate Pease [X] (Inactive) added a comment - Of the commands listed in the description, they are either already implemented or child tickets are created and linked to this issue.
            npease Nate Pease [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Won't Fix [ 10405 ]

              People

              Assignee:
              npease Nate Pease [X] (Inactive)
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Andy Salnikov, Christopher Waters, Jim Bosch, Michelle Gower, Nate Lust, Nate Pease [X] (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.