Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25013

Unclear error messages in pipetask run

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      While trying to figure out pipetask run and a new pipeline simultaneously, I've encountered the following error messages that I believe could be improved:

      • If I omit the --butler-config argument, I get sqlite3.OperationalError: no such table: collection. It would be more helpful to warn that no Butler repository has been found/selected.
      • If I have a mismatch between the dimensions of a dataset as listed in PipelineTask connections and in the registry, I get ValueError: Supplied dataset type [dataset] inconsistent with registry definition [dataset]. It would help to also mention which task is the problem; the stack trace is all from general-purpose code.
      • If I can't account for a particular dataset, I get RuntimeError: Expected exactly one instance of input [dataset]. As above, it would help for debugging to know which task is being processed.
      • If I omit the --register-dataset-types argument, I get KeyError: "DatasetType [dataset] could not be found.", suggesting that there is a matching problem with the pipeline. Given that this is an extremely common user error, it would be helpful to suggest that it might be a registration problem instead.
      • If I forget to provide an --input when it's mandatory, I get an Expected exactly one instance of input error with a seemingly arbitrary data ID. This message misleadingly suggests the problem is with the dataset, not the collection.
      • An invalid storage class raises a bare KeyError. It would be more helpful to report which connection and/or which dataset type caused the error.
      • If I pass a --data-query argument that does not match any IDs, I get UnboundLocalError: local variable 'n' referenced before assignment. An empty query should be handled explicitly by the code.
      • If I'm running on a repository with an out-of-date schema, I get sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database. It would be more helpful to say that the schema is out of date.
      • Debug log messages need a bit of work: ctrl.mpexec.mpGraphExecutor DEBUG: Executing QuantumIterData(index=16, quantum=<lsst.daf.butler.core.quantum.Quantum object at 0x7faeb68908c0>, taskDef=<lsst.pipe.base.pipeline.TaskDef object at 0x7faeb2e05090>, dependencies=frozenset()) to make them usable.
      • If extend-run is given with an output chain that does not exist, we get an IndexError that has no explanation. Instead we should report that extend-run should not be used first time (or use this as an opportunity to turn off extend run and ignore the user).

        Attachments

          Issue Links

            Activity

            Hide
            salnikov Andy Salnikov added a comment -

            Well, we do not propagate exception after handling it so this should not be an issue. Right now I print an exception and tell click to exit (this is done via special click exception type), I'm proposing to replace printing with logging it to a logger, but then exit. Logging takes care of printing it to stderr (once if it is configured correctly), but nothing else can see this exception.

            Show
            salnikov Andy Salnikov added a comment - Well, we do not propagate exception after handling it so this should not be an issue. Right now I print an exception and tell click to exit (this is done via special click exception type), I'm proposing to replace printing with logging it to a logger, but then exit. Logging takes care of printing it to stderr (once if it is configured correctly), but nothing else can see this exception.
            Hide
            tjenness Tim Jenness added a comment -

            Yes, log it to a logger instead of printing it and everyone is happy.

            Show
            tjenness Tim Jenness added a comment - Yes, log it to a logger instead of printing it and everyone is happy.
            Hide
            salnikov Andy Salnikov added a comment -

            OK, this is what was implemented. Tim Jenness, could you look once again at daf/butler/cli/utils.py after all my modifications, especially at the logging message, is that clear enough or do I need add more info? The output will look like:

            $ pipetask qgraph -b TEST_REPO
            lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
            Traceback (most recent call last):
              File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph
                script.qgraph(pipelineObj=pipeline, **kwargs)
              File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 148, in qgraph
                qgraph = f.makeGraph(pipelineObj, args)
              File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 544, in makeGraph
                registry, collections, run = _ButlerFactory.makeRegistryAndCollections(args)
              File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 340, in makeRegistryAndCollections
                butler, inputs, self = cls._makeReadParts(args)
              File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 278, in _makeReadParts
                butler = Butler(args.butler_config, writeable=False)
              File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butler.py", line 250, in __init__
                self._config = ButlerConfig(config, searchPaths=searchPaths)
              File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butlerConfig.py", line 112, in __init__
                butlerConfig = Config(other)
              File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 226, in __init__
                self.__initFromUri(other)
              File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 332, in __initFromUri
                raise FileNotFoundError(f"Config location {uri} does not exist.")
            FileNotFoundError: Config location /home/salnikov/gen3-middleware/daf_butler/TEST_REPO does not exist.
            

            Show
            salnikov Andy Salnikov added a comment - OK, this is what was implemented. Tim Jenness , could you look once again at daf/butler/cli/utils.py after all my modifications, especially at the logging message, is that clear enough or do I need add more info? The output will look like: $ pipetask qgraph -b TEST_REPO lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback: Traceback (most recent call last): File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 96, in qgraph script.qgraph(pipelineObj=pipeline, **kwargs) File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 148, in qgraph qgraph = f.makeGraph(pipelineObj, args) File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 544, in makeGraph registry, collections, run = _ButlerFactory.makeRegistryAndCollections(args) File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 340, in makeRegistryAndCollections butler, inputs, self = cls._makeReadParts(args) File "/home/salnikov/gen3-middleware/ctrl_mpexec/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 278, in _makeReadParts butler = Butler(args.butler_config, writeable=False) File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butler.py", line 250, in __init__ self._config = ButlerConfig(config, searchPaths=searchPaths) File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/_butlerConfig.py", line 112, in __init__ butlerConfig = Config(other) File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 226, in __init__ self.__initFromUri(other) File "/home/salnikov/gen3-middleware/daf_butler/python/lsst/daf/butler/core/config.py", line 332, in __initFromUri raise FileNotFoundError(f"Config location {uri} does not exist.") FileNotFoundError: Config location /home/salnikov/gen3-middleware/daf_butler/TEST_REPO does not exist.
            Hide
            tjenness Tim Jenness added a comment -

            Thanks. This does work and is much shorter. I'm not sure what we do about the case where someone enables --no-log-tty because you get this:

            $ butler --no-log-tty create COPYRIGHT
            $ echo $?
            1
            

            I suppose you only have yourself to blame because you are effectively saying you don't want any output to the screen...

            Show
            tjenness Tim Jenness added a comment - Thanks. This does work and is much shorter. I'm not sure what we do about the case where someone enables --no-log-tty because you get this: $ butler --no-log-tty create COPYRIGHT $ echo $? 1 I suppose you only have yourself to blame because you are effectively saying you don't want any output to the screen...
            Hide
            salnikov Andy Salnikov added a comment -

            Thanks for review and all suggestions, merged all three packages. I don't think this is the end of unclear messages from pipetask/butler, I guess new ticket(s) need to be open for new ones.

            Show
            salnikov Andy Salnikov added a comment - Thanks for review and all suggestions, merged all three packages. I don't think this is the end of unclear messages from pipetask/butler, I guess new ticket(s) need to be open for new ones.

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Tim Jenness
              Watchers:
              Andy Salnikov, Eli Rykoff, Jim Bosch, Kian-Tat Lim, Krzysztof Findeisen, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.