Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15257

Refactor argument parsing to work with drivers that aren't CmdLineTask

    Details

    • Templates:
    • Story Points:
      12
    • Team:
      Data Access and Database

      Description

      I'd like to start looking at refactoring the argument parsing infrastructure in pipe_base to be more broadly useful as we transition away from CmdLineTask.  That includes (but is not limited to) making it useful for PipelineTask execution - there will be some command-line driver scripts in Gen3 (such as those for the ingest tasks I'm working on now on DM-15189) that will want a Gen3 Butler and/or to parse config overrides without actually being actual PipelineTasks.

      The specific goal I have is to have a library of argument parser components that allow users to do any combination of the following without assuming that any of them necessarily go together:

      • Construct a Gen2 Butler.
      • Construct a Gen3 Butler.
      • Accept a Gen2 Data ID expression.
      • Accept a Gen3 Data ID expression.
      • Accept pex_config overrides for both Pipelines and Tasks.
      • Set up and control loggers and diagnostic displays/outputs.

      Not all of these components need to go in pipe_base (the Gen3 Data ID parser, at least, absolutely should not), but I'd like them to be usable (and mix-and-match-able) in roughly the same way.  It's also worth pointing out that Andy Salnikov already has some command-line parsing code for configs in pipe_supertask right now that is at least heavily derived from what's in pipe_base, and that it may not be possible for that to use the exact same command-line syntax we use for CmdLineTasks today (because there is no single top-level Task/Config in the PipelineTask world).  I'd like for this ticket to include refactoring that code as well (while consulting with Andy Salnikov to make sure it still meets his needs and does not disrupt his other work, of course).

      A major part of this ticket is extracting information on the kinds of things that need to be command-line configurable from each of the above.  Please ask on #gen3-middleware on Slack if you're not sure who the right person to extract any of that information from.

        Attachments

          Container Issues

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Andy Salnikov, Christopher Waters, Jim Bosch, John Parejko, Krzysztof Findeisen, Paul Price, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel