Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10966

Errors with incomplete data ids in Butler Gen2

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: daf_persistence
    • Labels:
      None

      Description

      Figuring out what I'm doing wrong when I give an incomplete data id on the command line is currently tricky. I typically get an error like the following:

        File "/global/common/cori/contrib/lsst/lsstDM/v13_0/Linux64/daf_persistence/13.0/python/lsst/daf/persistence/registries.py", line 316, in lookup
          cmd += " FROM " + " NATURAL JOIN ".join(reference)
      TypeError: sequence item 0: expected string, NoneType found
      

      This was encountered using the following command line:

      reportPatches.py . --id --config raDecRange="96.05665652768202, 96.37725617219749, -29.545497947342373, -29.268971984383857"
      

      I ran into the issue with a v13 stack, but Russell Owen has confirmed that it is still a problem on master.

      The above error can be rectified by adding filter='r' to the --id argument, but that's very difficult to figure out from the error. I don't know what the right answer is, but it would be nice to error as early as possible and with a message that clearly states the data id is the problem (if it's not possible do deduce what part of the id is causing the issue).

        Attachments

          Issue Links

            Activity

            Hide
            rhl Robert Lupton added a comment -

            I totally agree that the error message needs to be better when there's an incomplete dataId

            In cases where the user know what he/she needs, a method on DataId such as DataId.get("filter") which does a queryMetadata as necessary (with cacheing?) is a good solution (it may already support get with slightly different semantics). However, this is a slightly different case. I'm not sure whose job it is to know that we use the same definition of patch in all filters; if this isn't "known", then this specification is incomplete; otherwise the code should (Internally?) attempt to look up all needed keys.

            My DataId.get("filter") is actually a little more difficult as it depends on the type of data (e.g. raw-like and patch-like); I think that this is specified in the mapper file for each data type, but the DataId doesn't know; so we might need DataId.get("filter", dataClass="raw")

            Show
            rhl Robert Lupton added a comment - I totally agree that the error message needs to be better when there's an incomplete dataId In cases where the user know what he/she needs, a method on DataId such as DataId.get("filter") which does a queryMetadata as necessary (with cacheing?) is a good solution (it may already support get with slightly different semantics). However, this is a slightly different case. I'm not sure whose job it is to know that we use the same definition of patch in all filters; if this isn't "known", then this specification is incomplete; otherwise the code should (Internally?) attempt to look up all needed keys. My DataId.get("filter") is actually a little more difficult as it depends on the type of data (e.g. raw-like and patch-like); I think that this is specified in the mapper file for each data type, but the DataId doesn't know; so we might need DataId.get("filter", dataClass="raw")

              People

              • Assignee:
                Unassigned
                Reporter:
                krughoff Simon Krughoff
                Watchers:
                Fritz Mueller, John Parejko, Robert Lupton, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel