Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ctrl_mpexec
-
Labels:
-
Story Points:2
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
pipetask creates output collections too early - there are many ways that things can fail in QG generation (at least empty QG and missing input dataset types), and we should not modify the repository at all until QG generation has succeeded. This may be a bit of a pain, because I know the logic for setting up collections according to the given argument is pretty twisty (sorry!).
In addition, when the QG is empty, no attempt should be made to register dataset types, as this causes a new exception that obscures the message about the empty QG.
Finally, when pipetask run is not given any input or output collections, the error message is confusing (I don't recall exactly what, but we should try leaving off various combinations of "required" options anyway to see what happens).
Attachments
Activity
Note to self: this seems mostly done (branches for pipe_base and ctrl_mpexec), but it depends on DM-27153's branches for daf_butler and ctrl_mpexec. I'll put off final testing until that has merged.
Nate Pease [X], mind taking this small review?
Changes in pipe_base and ctrl_mpexec.
looks good.
The first problem here looks easier than I first thought: it seems the code in cmdLineFwk.py already tries to avoid doing any writes during QG gen, so this is just the second problem again - when the QG is empty, various methods just return None instead of raising, so it isn't until after we've done the pre-exec-init write write operations that we actually notice that the QG is empty. Looks like it'd be an easy fix to just make an empty QG raise earlier, assuming that doesn't cause problems elsewhere.