Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23538

Bug of Data Butler to Get the Raw Data

    Details

    • Urgent?:
      No

      Description

      It looks like there is the bug for butler to get the raw exposure data in lsst_distrib: w_2020_06. This bug did not happen in w_2020_04. The following is the test data: input.zip

      I use the followings to try to get the raw data:

       
      from lsst.daf.persistence import Butler
       butler = Butler(inputs="input/")
       dataId = dict(expId=20, raftName="R00", detectorName="S22")
       raw = butler.get("{color:#ff2600}raw{color}", dataId=dataId)
      

      The following is the error message (You can also see details in errorMsg.txt):

       {color:#0747a6}NoResults: No locations for get: datasetType:raw_amp dataId:DataId(initialdata=\{'expId': 20, 'raftName': 'R00', 'detectorName': 'S22', 'run': '20', 'detector': 197, 'channel': 1}, tag=set()){color}
      

      It looks like the dataset type becomes "raw_amp" even though I use "raw" in the get().
      The followins is the data ID of "raw" if I use butler.getKeys("raw"):

      {'run': str,
        'raftName': str,
        'expId': int,
        'detectorName': str,
        'detector': int}
      

      There should be no 'channel' because it is not "raw_amp".

        Attachments

          Activity

          Hide
          krughoff Simon Krughoff added a comment - - edited

          It sounds like you need to use visit instead of expId based on this post.

          Will you try substituting visit=20 for expId=20 and posting results?

          Show
          krughoff Simon Krughoff added a comment - - edited It sounds like you need to use visit instead of expId based on this post . Will you try substituting visit=20 for expId=20 and posting results?
          Hide
          ttsai Te-Wei Tsai added a comment -

          I tried the `visit` first (in `w_2020_04` and before) and realized no such key supported in `w_2020_06`. This is why I used `butler.getKeys("raw")` to have the idea of keys. You can see no such key in the raw data set type.

          Show
          ttsai Te-Wei Tsai added a comment - I tried the `visit` first (in `w_2020_04` and before) and realized no such key supported in `w_2020_06`. This is why I used `butler.getKeys("raw")` to have the idea of keys. You can see no such key in the raw data set type.
          Hide
          krughoff Simon Krughoff added a comment -

          Did you reingest the data with the _06 stack?

          Show
          krughoff Simon Krughoff added a comment - Did you reingest the data with the _06 stack?
          Hide
          krughoff Simon Krughoff added a comment -

          Actually, I've been informed that _07 is out and contains fixes in ISR. It would be good if you tried that one.

          Show
          krughoff Simon Krughoff added a comment - Actually, I've been informed that _07 is out and contains fixes in ISR. It would be good if you tried that one.
          Hide
          ttsai Te-Wei Tsai added a comment -

          I re-ingested the data with `w_2020_06` version. I could try it in `w_2020_07`. Thanks!

          Show
          ttsai Te-Wei Tsai added a comment - I re-ingested the data with `w_2020_06` version. I could try it in `w_2020_07`. Thanks!
          Hide
          ttsai Te-Wei Tsai added a comment -

          The error is still there even though I use the version of w_2020_07. You can see the test result here.

          ---------------------------------------------------------------------

          E lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:raw_amp dataId:DataId(initialdata={'expId': 20, 'raftName': 'R00', 'detectorName': 'S22', 'run': '20', 'detector': 197, 'channel': 1}, tag=set())

          Show
          ttsai Te-Wei Tsai added a comment - The error is still there even though I use the version of w_2020_07. You can see the test result here. --------------------------------------------------------------------- E lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:raw_amp dataId:DataId(initialdata={'expId': 20, 'raftName': 'R00', 'detectorName': 'S22', 'run': '20', 'detector': 197, 'channel': 1}, tag=set())
          Hide
          krughoff Simon Krughoff added a comment -

          FYI, the link in the previous comment doesn't work for me, for some reason.

          Can you provide the full list of commands you run to get the above exception?

          Show
          krughoff Simon Krughoff added a comment - FYI, the link in the previous comment doesn't work for me, for some reason. Can you provide the full list of commands you run to get the above exception?
          Hide
          ttsai Te-Wei Tsai added a comment -

          It is in the description. Let me list it in the following. Thanks!

          from lsst.daf.persistence import Butler
          butler = Butler(inputs="input/")
          dataId = dict(expId=20, raftName="R00", detectorName="S22")
          raw = butler.get("raw", dataId=dataId)

          Show
          ttsai Te-Wei Tsai added a comment - It is in the description. Let me list it in the following. Thanks! from lsst.daf.persistence import Butler butler = Butler(inputs="input/") dataId = dict(expId=20, raftName="R00", detectorName="S22") raw = butler.get(" raw ", dataId=dataId)
          Hide
          krughoff Simon Krughoff added a comment -

          OK the problem is that an image with expId=20 doesn't exist. I think you simply need to use an expId that exists.

          You can see which expId values are available with a query like:

          butler.queryMetadata('raw', format=['expId', 'raftName', 'detectorName'])
          

          Show
          krughoff Simon Krughoff added a comment - OK the problem is that an image with expId=20 doesn't exist. I think you simply need to use an expId that exists. You can see which expId values are available with a query like: butler.queryMetadata('raw', format=['expId', 'raftName', 'detectorName'])
          Hide
          ttsai Te-Wei Tsai added a comment -

          The problem is that I used the w_2020_06 to ingest the raw data. After using the w_2020_07, the problem is solved. Thanks for the help!

          Show
          ttsai Te-Wei Tsai added a comment - The problem is that I used the w_2020_06 to ingest the raw data. After using the w_2020_07, the problem is solved. Thanks for the help!
          Hide
          tjenness Tim Jenness added a comment -

          Given the most recent comment I'm closing this ticket as invalid.

          Show
          tjenness Tim Jenness added a comment - Given the most recent comment I'm closing this ticket as invalid.

            People

            • Assignee:
              Unassigned
              Reporter:
              ttsai Te-Wei Tsai
              Watchers:
              Kian-Tat Lim, Simon Krughoff, Te-Wei Tsai, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel