# Unclear error messages in pipetask run

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
2
• Team:
Data Access and Database
• Urgent?:
No

#### Description

While trying to figure out pipetask run and a new pipeline simultaneously, I've encountered the following error messages that I believe could be improved:

• If I omit the --butler-config argument, I get sqlite3.OperationalError: no such table: collection. It would be more helpful to warn that no Butler repository has been found/selected.
• If I have a mismatch between the dimensions of a dataset as listed in PipelineTask connections and in the registry, I get ValueError: Supplied dataset type [dataset] inconsistent with registry definition [dataset]. It would help to also mention which task is the problem; the stack trace is all from general-purpose code.
• If I can't account for a particular dataset, I get RuntimeError: Expected exactly one instance of input [dataset]. As above, it would help for debugging to know which task is being processed.
• If I omit the --register-dataset-types argument, I get KeyError: "DatasetType [dataset] could not be found.", suggesting that there is a matching problem with the pipeline. Given that this is an extremely common user error, it would be helpful to suggest that it might be a registration problem instead.
• If I forget to provide an --input when it's mandatory, I get an Expected exactly one instance of input error with a seemingly arbitrary data ID. This message misleadingly suggests the problem is with the dataset, not the collection.
• An invalid storage class raises a bare KeyError. It would be more helpful to report which connection and/or which dataset type caused the error.
• If I pass a --data-query argument that does not match any IDs, I get UnboundLocalError: local variable 'n' referenced before assignment. An empty query should be handled explicitly by the code.
• If I'm running on a repository with an out-of-date schema, I get sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database. It would be more helpful to say that the schema is out of date.
• Debug log messages need a bit of work: ctrl.mpexec.mpGraphExecutor DEBUG: Executing QuantumIterData(index=16, quantum=<lsst.daf.butler.core.quantum.Quantum object at 0x7faeb68908c0>, taskDef=<lsst.pipe.base.pipeline.TaskDef object at 0x7faeb2e05090>, dependencies=frozenset()) to make them usable.
• If extend-run is given with an output chain that does not exist, we get an IndexError that has no explanation. Instead we should report that extend-run should not be used first time (or use this as an opportunity to turn off extend run and ignore the user).

#### Activity

Hide
Andy Salnikov added a comment -

Well, we do not propagate exception after handling it so this should not be an issue. Right now I print an exception and tell click to exit (this is done via special click exception type), I'm proposing to replace printing with logging it to a logger, but then exit. Logging takes care of printing it to stderr (once if it is configured correctly), but nothing else can see this exception.

Show
Andy Salnikov added a comment - Well, we do not propagate exception after handling it so this should not be an issue. Right now I print an exception and tell click to exit (this is done via special click exception type), I'm proposing to replace printing with logging it to a logger, but then exit. Logging takes care of printing it to stderr (once if it is configured correctly), but nothing else can see this exception.
Hide
Tim Jenness added a comment -

Yes, log it to a logger instead of printing it and everyone is happy.

Show
Tim Jenness added a comment - Yes, log it to a logger instead of printing it and everyone is happy.
Hide
Andy Salnikov added a comment -

OK, this is what was implemented. Tim Jenness, could you look once again at daf/butler/cli/utils.py after all my modifications, especially at the logging message, is that clear enough or do I need add more info? The output will look like:

 $pipetask qgraph -b TEST_REPO lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback: Traceback (most recent call last):  File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph  script.qgraph(pipelineObj=pipeline, **kwargs)  File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 148, in qgraph  qgraph = f.makeGraph(pipelineObj, args)  File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 544, in makeGraph  registry, collections, run = _ButlerFactory.makeRegistryAndCollections(args)  File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 340, in makeRegistryAndCollections  butler, inputs, self = cls._makeReadParts(args)  File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 278, in _makeReadParts  butler = Butler(args.butler_config, writeable=False)  File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butler.py", line 250, in __init__  self._config = ButlerConfig(config, searchPaths=searchPaths)  File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butlerConfig.py", line 112, in __init__  butlerConfig = Config(other)  File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 226, in __init__  self.__initFromUri(other)  File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 332, in __initFromUri  raise FileNotFoundError(f"Config location {uri} does not exist.") FileNotFoundError: Config location /home/salnikov/gen3-middleware/daf_butler/TEST_REPO does not exist.  Show Andy Salnikov added a comment - OK, this is what was implemented. Tim Jenness , could you look once again at daf/butler/cli/utils.py after all my modifications, especially at the logging message, is that clear enough or do I need add more info? The output will look like:$ pipetask qgraph -b TEST_REPO lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback: Traceback (most recent call last): File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph script.qgraph(pipelineObj=pipeline, **kwargs) File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 148, in qgraph qgraph = f.makeGraph(pipelineObj, args) File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 544, in makeGraph registry, collections, run = _ButlerFactory.makeRegistryAndCollections(args) File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 340, in makeRegistryAndCollections butler, inputs, self = cls._makeReadParts(args) File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 278, in _makeReadParts butler = Butler(args.butler_config, writeable=False) File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butler.py", line 250, in __init__ self._config = ButlerConfig(config, searchPaths=searchPaths) File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butlerConfig.py", line 112, in __init__ butlerConfig = Config(other) File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 226, in __init__ self.__initFromUri(other) File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 332, in __initFromUri raise FileNotFoundError(f"Config location {uri} does not exist.") FileNotFoundError: Config location /home/salnikov/gen3-middleware/daf_butler/TEST_REPO does not exist.
Hide
Tim Jenness added a comment -

Thanks. This does work and is much shorter. I'm not sure what we do about the case where someone enables --no-log-tty because you get this:

 $butler --no-log-tty create COPYRIGHT $ echo $? 1  I suppose you only have yourself to blame because you are effectively saying you don't want any output to the screen... Show Tim Jenness added a comment - Thanks. This does work and is much shorter. I'm not sure what we do about the case where someone enables --no-log-tty because you get this:$ butler --no-log-tty create COPYRIGHT $echo$? 1 I suppose you only have yourself to blame because you are effectively saying you don't want any output to the screen...
Hide
Andy Salnikov added a comment -

Thanks for review and all suggestions, merged all three packages. I don't think this is the end of unclear messages from pipetask/butler, I guess new ticket(s) need to be open for new ones.

Show
Andy Salnikov added a comment - Thanks for review and all suggestions, merged all three packages. I don't think this is the end of unclear messages from pipetask/butler, I guess new ticket(s) need to be open for new ones.

#### People

Assignee:
Andy Salnikov
Reporter:
Krzysztof Findeisen
Reviewers:
Tim Jenness
Watchers:
Andy Salnikov, Eli Rykoff, Jim Bosch, Kian-Tat Lim, Krzysztof Findeisen, Simon Krughoff, Tim Jenness