Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33600

Inconsistencies in queryDimensionRecords

    XMLWordPrintable

Details

    • 5
    • Data Access and Database
    • No

    Description

      rowen points out some inconsistencies in the use of queryDimensionRecords.

      TLDR:

      • Inconsistency between using kwargs for missing instruments vs where clause with missing instrument.
      • Bind parameter not working for governor dimensions.

      Using the pipelines_check repo since I have it lying around and it has one instrument (HSC):

      from lsst.daf.butler import Butler
       
      b = Butler("DATA_REPO")
      registry = b.registry
      where = ""
      bind = {}
      for instrument in ("HSC", "LATISS"):
       
          record_iter = b.registry.queryDimensionRecords(
              "exposure",
              instrument=instrument,
              bind=bind,
              where=where,
          )
          print(f"Got {record_iter.count()} results")
      

      This fails on the second loop with:

      Got 1 results
      Traceback (most recent call last):
        File "x.py", line 9, in <module>
          record_iter = b.registry.queryDimensionRecords(
        File "/Users/timj/work/lsstsw3/build/daf_butler/python/lsst/daf/butler/registries/sql.py", line 1102, in queryDimensionRecords
          dataIds = self.queryDataIds(
        File "/Users/timj/work/lsstsw3/build/daf_butler/python/lsst/daf/butler/registries/sql.py", line 1026, in queryDataIds
          standardizedDataId = self.expandDataId(dataId, **kwargs)
        File "/Users/timj/work/lsstsw3/build/daf_butler/python/lsst/daf/butler/registries/sql.py", line 718, in expandDataId
          raise LookupError(
      LookupError: Could not fetch record for required dimension instrument via keys {'instrument': 'LATISS'}.
      

      if though instead of using kwargs I specify the instrument in the where clause it all works fine:

      from lsst.daf.butler import Butler
       
      b = Butler("DATA_REPO")
      registry = b.registry
      where = ""
      bind = {}
      for instrument in ("HSC", "LATISS"):
       
          record_iter = b.registry.queryDimensionRecords(
              "exposure",
              # instrument=instrument,
              bind=bind,
              where=f"instrument = '{instrument}'",
          )
          print(f"Got {record_iter.count()} results")
      

      Result:

      Got 1 results
      Got 0 results
      

      This inconsistency is not great.

      Furthermore, if we then say that we will commit to the WHERE clause and use bind I can't get it to work at all:

      from lsst.daf.butler import Butler
       
      b = Butler("DATA_REPO")
      registry = b.registry
      for instrument in ("HSC", "LATISS"):
       
          record_iter = b.registry.queryDimensionRecords(
              "exposure",
              # instrument=instrument,
              bind={"inst": instrument},
              where="instrument = inst",
          )
          print(f"Got {record_iter.count()} results")
      

      which gives:

      Traceback (most recent call last):
        File "x.py", line 15, in <module>
          print(f"Got {record_iter.count()} results")
        File "/Users/timj/work/lsstsw3/build/daf_butler/python/lsst/daf/butler/registry/queries/_results.py", line 1102, in count
          return self._dataIds.count(exact=exact)
        File "/Users/timj/work/lsstsw3/build/daf_butler/python/lsst/daf/butler/registry/queries/_results.py", line 450, in count
          return self._query.count(self._db, exact=exact)
        File "/Users/timj/work/lsstsw3/build/daf_butler/python/lsst/daf/butler/registry/queries/_results.py", line 201, in _query
          self._cached_query = self._query_factory(self._order_by, self._limit)
        File "/Users/timj/work/lsstsw3/build/daf_butler/python/lsst/daf/butler/registries/sql.py", line 1056, in query_factory
          summary = queries.QuerySummary(
        File "/Users/timj/work/lsstsw3/build/daf_butler/python/lsst/daf/butler/registry/queries/_structs.py", line 395, in __init__
          self.where = expression.attach(
        File "/Users/timj/work/lsstsw3/build/daf_butler/python/lsst/daf/butler/registry/queries/_structs.py", line 161, in attach
          raise RuntimeError(msg) from None
      RuntimeError: Error in query expression "instrument = inst": No value(s) for governor dimensions {instrument} in expression that references dependent dimensions. 'Governor' dimensions must always be specified completely in either the query expression (via simple 'name=<value>' terms, not 'IN' terms) or in a data ID passed to the query method.
      

      implying that governor dimensions are checked before the bind parameters are handled? Other bind parameters seem to work fine.

      Attachments

        Issue Links

          Activity

            rowen Russell Owen added a comment - - edited

            I will be content if and only if the exception raised for an unknown governer value both predicable and different than the exception raised for invalid queries – invalid in the database sense. My service does not want to know or care about your special "governer" dimensions. It wants to treat registry as much like a database server as it possibly can. Catching a special "this is a valid query and if this was a database this call would be returning no records" exception is certainly tolerable.

            It almost sounds like Registry is the wrong API to be using, but I don't know of any alternative.

            rowen Russell Owen added a comment - - edited I will be content if and only if the exception raised for an unknown governer value both predicable and different than the exception raised for invalid queries – invalid in the database sense. My service does not want to know or care about your special "governer" dimensions. It wants to treat registry as much like a database server as it possibly can. Catching a special "this is a valid query and if this was a database this call would be returning no records" exception is certainly tolerable. It almost sounds like Registry is the wrong API to be using, but I don't know of any alternative.

            I can agree that the current set of exceptions that we raise in those query methods (and many other methods) is hard to use to identify specific issues. What we have today is a mix of LookupError, KeyError, RuntimeError, ValueError and its subclasses; to make exceptions more useful we probably need more specific and more structured class hierarchy for exceptions. It's a non-trivial problem, of course, to try to update existing code to generate different exceptions, but I can try something reasonably simple on this ticket.

            From the client perspective, I think it's probably worth to try to distinguish following cases for these three query methods:

            • Unknown keyword argument name (e.g. querySomething(..., not_a_thing="thing")), today this raises KeyError, maybe worth defining something DimensionNameError? (Same for the key in dataId argument).
            • Unexpected value for a dimension passed in dataId or keyword arguments, LookupError is raised today, maybe raise new exception InvalidDataIdError?
            • For inconsistent/conflicting coordinate values we already raise InconsistentDataIdError, which inherits ValueError, maybe make a new base class for it common with InvalidDataIdError?
            • Bad syntax in user expression today raises RuntimeError, replace that with more specific UserExpressionSyntaxError?
            • Valid user expression syntax, but containing unknown governor dimension values (e.g. "instrument = 'LSTS'"), with the changes on this ticket raises LookupError, could replace it with the UserExpressionInvalidDataIdError (maybe a subclass of InvalidDataIdError)?
            • In other cases methods can return empty result, and explain_no_results() can be used to try to identify the issue

            This may not cover all possible errors though, other types of exceptions may leak from the code we have no control over, e.g. database connection may break at any moment and produce some SQLAlchemy errors. We could catch them all at registry boundary and re-throw as a generic exception type (RegistryError), or we can leave that to the caller. I believe general good practice in the cases like this is to catch and wrap/re-throw all exceptions at the module/library boundary and not let undocumented exceptions leak.

            salnikov Andy Salnikov added a comment - I can agree that the current set of exceptions that we raise in those query methods (and many other methods) is hard to use to identify specific issues. What we have today is a mix of LookupError, KeyError, RuntimeError, ValueError and its subclasses; to make exceptions more useful we probably need more specific and more structured class hierarchy for exceptions. It's a non-trivial problem, of course, to try to update existing code to generate different exceptions, but I can try something reasonably simple on this ticket. From the client perspective, I think it's probably worth to try to distinguish following cases for these three query methods: Unknown keyword argument name (e.g. querySomething(..., not_a_thing="thing") ), today this raises KeyError, maybe worth defining something DimensionNameError? (Same for the key in dataId argument). Unexpected value for a dimension passed in dataId or keyword arguments, LookupError is raised today, maybe raise new exception InvalidDataIdError? For inconsistent/conflicting coordinate values we already raise InconsistentDataIdError, which inherits ValueError, maybe make a new base class for it common with InvalidDataIdError? Bad syntax in user expression today raises RuntimeError, replace that with more specific UserExpressionSyntaxError? Valid user expression syntax, but containing unknown governor dimension values (e.g. "instrument = 'LSTS'"), with the changes on this ticket raises LookupError, could replace it with the UserExpressionInvalidDataIdError (maybe a subclass of InvalidDataIdError)? In other cases methods can return empty result, and explain_no_results() can be used to try to identify the issue This may not cover all possible errors though, other types of exceptions may leak from the code we have no control over, e.g. database connection may break at any moment and produce some SQLAlchemy errors. We could catch them all at registry boundary and re-throw as a generic exception type (RegistryError), or we can leave that to the caller. I believe general good practice in the cases like this is to catch and wrap/re-throw all exceptions at the module/library boundary and not let undocumented exceptions leak.

            jbosch, I think this is ready for review. We discussed exceptions yesterday, I pushed everything that I have so far in the last commit on this branch. I'm not very content with that myself, mostly because it's hard to test that there are no other exceptions leaking, and of course unit tests don't cover all exception-raising cases. As I said, I'm open to dropping that last commit if you think this is not a big improvement.

            rowen, could you look at the new docstrings for Registry methods to check that it works for you.

            I have not added news snippet yet, if new exceptions survive the review, I'll add something.

            daf_butler PR: https://github.com/lsst/daf_butler/pull/651
            pipe_base PR: https://github.com/lsst/pipe_base/pull/236
            obs_base PR: https://github.com/lsst/obs_base/pull/409

            salnikov Andy Salnikov added a comment - jbosch , I think this is ready for review. We discussed exceptions yesterday, I pushed everything that I have so far in the last commit on this branch. I'm not very content with that myself, mostly because it's hard to test that there are no other exceptions leaking, and of course unit tests don't cover all exception-raising cases. As I said, I'm open to dropping that last commit if you think this is not a big improvement. rowen , could you look at the new docstrings for Registry methods to check that it works for you. I have not added news snippet yet, if new exceptions survive the review, I'll add something. daf_butler PR: https://github.com/lsst/daf_butler/pull/651 pipe_base PR: https://github.com/lsst/pipe_base/pull/236 obs_base PR: https://github.com/lsst/obs_base/pull/409
            jbosch Jim Bosch added a comment -

            Looks good to me; just one comment on a single new exception type and its usage on the daf_butler PR.

            jbosch Jim Bosch added a comment - Looks good to me; just one comment on a single new exception type and its usage on the daf_butler PR.

            Thanks for review and fixing docs! Merged.

            salnikov Andy Salnikov added a comment - Thanks for review and fixing docs! Merged.

            People

              salnikov Andy Salnikov
              tjenness Tim Jenness
              Jim Bosch
              Andy Salnikov, Jim Bosch, Russell Owen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.