Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler
-
Labels:
-
Story Points:4
-
Epic Link:
-
Sprint:DB_S22_12
-
Team:Data Access and Database
-
Urgent?:No
Description
MWCommand captures the arguments in the order they were passed in at the command line, for use in processing pipelines, which are sensitive to this order.
However, when an equal sign is used to separate the flag & value on the command line, e.g. --foo=bar, the two are captured as one list item, and this breaks the pipeline processing code.
The algorithm to split these is not simple, at least because of:
- "flag" options (which do not take a value on the command line).
- short unix-style options, which should include everything after the short option flag, including an equal sign if present (e.g. -c=myfile.py should parse to become ["-c", "=myfile.py"].
I think the best solution is to use the click parser to parse the args. It returns a dict of option name-value pairs (if the option accepts multiple values then the value will be a tuple of values), and a list of Parameter objects that describe the parameter (which is an Option or Argument) that is mostly in the same order as the command line.
From these two things, an approximate reconstruction of the command line call can be created. The only considerable differences are
- if an option has multiple option flags, it is not easy to determine which one was used. (However, all downstream code that uses the flags should be able to use any of the ones used by the function, so it should be fine to just pick one.)
- in the list of Parameter objects, Options come first and Arguments are last. The order of Options is preserved, as is the order of Arguments.
Attachments
Activity
I recently wrote an explanation of this in the PR that you've seen now, but I'll add a summary of that answer here too, for the record:
Click generally does the right thing, as you describe
taking -c=foo.py and sending =foo.py as the value
The reason we have this other behavior to cache the CLI args is that these captured args are used by pipetask to turn CLI options into pipeline actions. There are 5 different options that result in a pipeline action (--task, --delete, --config, --config-file, --instrument), and the order these options are used on the command line specifies the order of the actions to be run. Click doesn't include any facility get the order of different options, it's a non-standard use case I think.
Thanks for explaining. I understand why we are special now.
My mental model of the problem being solved was not at all the same as what was actually being solved. I think one problem I have is that the ticket does not demonstrate the problem with any code failure but goes straight into the possible fix.
What I thought was happening was that click was doing the correct thing of taking -c=foo.py and sending =foo.py as the value for the config file. This should have triggered a downstream failure trying to find the file =foo.py. It sounds like that does not happen. What does click actually do then? Does it treat it as lots of separate single character options? (but then surely click would complain about the "-." option? I'm confused by so much code having to be written to work around click's interpretation without really being clear as to what click thinks the right interpreation is.