I ran into a problem chaining the run subcommand after qgraph because it wants to use butler options passed into the qgraph command to create a new butler. I didn't want to make the user enter butler options & values twice (once for qgraph, again with same values for run). I could have made qgraph create the butler and add it to the objects passed to run as a chained command, but there was one option that wasn't used by qgraph (it may have been output, or output-run? I forget, but can look it up tomorrow), and I didn't really want to pass an option to qgraph for a future subcommand that qgraph wasn't using itself. It occurred to me that it might be reasonable for subcommands to "forward" their option values to next subcommands in the same command execution (that is, chained subcommands). I added the "option forwarding" feature so we can evaluate it and change or remove it as needed. There's documentation about it in the code and in the qgraph and run subcommand help menus. Options that are forwarded by a subcommand are annotated with an "(f)" in that option's metavar in the subcommand's help output.

Hey Andy, here's the review I threatened promised. I got the email to review your code today, I'll start looking in the morning.

There's one unit test that passes on my machine but is failing CI, it's because it can't find the generated test file in CI, I'm going to have to spend a little time figuring that out, but I figured you can start looking at the rest of the code whenever you're ready.

Tim Jenness please take a look at the "forwarded options" feature described above and in code & help menus, and let me know what you think. I'd be happy to discuss it more too. Thanks!

Looks good, though I cannot say I understand much of it. Looks like you need to know deep details of click to figure out what is going on and click was not exactly designed for what we are trying to do. The amount of boilerplate and repetition bothers me too, may be hard to manage all of that over years.

Removed the boilerplate, related to options and arguments, with a better mechanism for sharing options. 

Changes are merged. Will close this ticket after I write the Community post.

Community post is here https://community.lsst.org/t/requesting-feedback-re-changes-to-the-pipetask-command-line-interface/4336