Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ctrl_mpexec
-
Story Points:2
-
Team:Data Access and Database
-
Urgent?:No
Description
While trying to figure out pipetask run and a new pipeline simultaneously, I've encountered the following error messages that I believe could be improved:
If I omit the --butler-config argument, I get sqlite3.OperationalError: no such table: collection. It would be more helpful to warn that no Butler repository has been found/selected.- If I have a mismatch between the dimensions of a dataset as listed in PipelineTask connections and in the registry, I get ValueError: Supplied dataset type [dataset] inconsistent with registry definition [dataset]. It would help to also mention which task is the problem; the stack trace is all from general-purpose code.
If I can't account for a particular dataset, I get RuntimeError: Expected exactly one instance of input [dataset]. As above, it would help for debugging to know which task is being processed.- If I omit the --register-dataset-types argument, I get KeyError: "DatasetType [dataset] could not be found.", suggesting that there is a matching problem with the pipeline. Given that this is an extremely common user error, it would be helpful to suggest that it might be a registration problem instead.
- If I forget to provide an --input when it's mandatory, I get an Expected exactly one instance of input error with a seemingly arbitrary data ID. This message misleadingly suggests the problem is with the dataset, not the collection.
- An invalid storage class raises a bare KeyError. It would be more helpful to report which connection and/or which dataset type caused the error.
- If I pass a --data-query argument that does not match any IDs, I get UnboundLocalError: local variable 'n' referenced before assignment. An empty query should be handled explicitly by the code.
- If I'm running on a repository with an out-of-date schema, I get sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database. It would be more helpful to say that the schema is out of date.
- Debug log messages need a bit of work: ctrl.mpexec.mpGraphExecutor DEBUG: Executing QuantumIterData(index=16, quantum=<lsst.daf.butler.core.quantum.Quantum object at 0x7faeb68908c0>, taskDef=<lsst.pipe.base.pipeline.TaskDef object at 0x7faeb2e05090>, dependencies=frozenset()) to make them usable.
- If extend-run is given with an output chain that does not exist, we get an IndexError that has no explanation. Instead we should report that extend-run should not be used first time (or use this as an opportunity to turn off extend run and ignore the user).
Attachments
Issue Links
Activity
Yes, log it to a logger instead of printing it and everyone is happy.
OK, this is what was implemented. Tim Jenness, could you look once again at daf/butler/cli/utils.py after all my modifications, especially at the logging message, is that clear enough or do I need add more info? The output will look like:
$ pipetask qgraph -b TEST_REPO
|
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
|
Traceback (most recent call last):
|
File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph
|
script.qgraph(pipelineObj=pipeline, **kwargs)
|
File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 148, in qgraph
|
qgraph = f.makeGraph(pipelineObj, args)
|
File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 544, in makeGraph
|
registry, collections, run = _ButlerFactory.makeRegistryAndCollections(args)
|
File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 340, in makeRegistryAndCollections
|
butler, inputs, self = cls._makeReadParts(args)
|
File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 278, in _makeReadParts
|
butler = Butler(args.butler_config, writeable=False)
|
File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butler.py", line 250, in __init__
|
self._config = ButlerConfig(config, searchPaths=searchPaths)
|
File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butlerConfig.py", line 112, in __init__
|
butlerConfig = Config(other)
|
File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 226, in __init__
|
self.__initFromUri(other)
|
File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 332, in __initFromUri
|
raise FileNotFoundError(f"Config location {uri} does not exist.")
|
FileNotFoundError: Config location /home/salnikov/gen3-middleware/daf_butler/TEST_REPO does not exist.
|
Thanks. This does work and is much shorter. I'm not sure what we do about the case where someone enables --no-log-tty because you get this:
$ butler --no-log-tty create COPYRIGHT
|
$ echo $?
|
1
|
I suppose you only have yourself to blame because you are effectively saying you don't want any output to the screen...
Thanks for review and all suggestions, merged all three packages. I don't think this is the end of unclear messages from pipetask/butler, I guess new ticket(s) need to be open for new ones.
Well, we do not propagate exception after handling it so this should not be an issue. Right now I print an exception and tell click to exit (this is done via special click exception type), I'm proposing to replace printing with logging it to a logger, but then exit. Logging takes care of printing it to stderr (once if it is configured correctly), but nothing else can see this exception.