Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23538

Bug of Data Butler to Get the Raw Data

    Details

    • Urgent?:
      No

      Description

      It looks like there is the bug for butler to get the raw exposure data in lsst_distrib: w_2020_06. This bug did not happen in w_2020_04. The following is the test data: input.zip

      I use the followings to try to get the raw data:

       
      from lsst.daf.persistence import Butler
       butler = Butler(inputs="input/")
       dataId = dict(expId=20, raftName="R00", detectorName="S22")
       raw = butler.get("{color:#ff2600}raw{color}", dataId=dataId)
      

      The following is the error message (You can also see details in errorMsg.txt):

       {color:#0747a6}NoResults: No locations for get: datasetType:raw_amp dataId:DataId(initialdata=\{'expId': 20, 'raftName': 'R00', 'detectorName': 'S22', 'run': '20', 'detector': 197, 'channel': 1}, tag=set()){color}
      

      It looks like the dataset type becomes "raw_amp" even though I use "raw" in the get().
      The followins is the data ID of "raw" if I use butler.getKeys("raw"):

      {'run': str,
        'raftName': str,
        'expId': int,
        'detectorName': str,
        'detector': int}
      

      There should be no 'channel' because it is not "raw_amp".

        Attachments

          Activity

          Hide
          krughoff Simon Krughoff added a comment -

          FYI, the link in the previous comment doesn't work for me, for some reason.

          Can you provide the full list of commands you run to get the above exception?

          Show
          krughoff Simon Krughoff added a comment - FYI, the link in the previous comment doesn't work for me, for some reason. Can you provide the full list of commands you run to get the above exception?
          Hide
          ttsai Te-Wei Tsai added a comment -

          It is in the description. Let me list it in the following. Thanks!

          from lsst.daf.persistence import Butler
          butler = Butler(inputs="input/")
          dataId = dict(expId=20, raftName="R00", detectorName="S22")
          raw = butler.get("raw", dataId=dataId)

          Show
          ttsai Te-Wei Tsai added a comment - It is in the description. Let me list it in the following. Thanks! from lsst.daf.persistence import Butler butler = Butler(inputs="input/") dataId = dict(expId=20, raftName="R00", detectorName="S22") raw = butler.get(" raw ", dataId=dataId)
          Hide
          krughoff Simon Krughoff added a comment -

          OK the problem is that an image with expId=20 doesn't exist. I think you simply need to use an expId that exists.

          You can see which expId values are available with a query like:

          butler.queryMetadata('raw', format=['expId', 'raftName', 'detectorName'])
          

          Show
          krughoff Simon Krughoff added a comment - OK the problem is that an image with expId=20 doesn't exist. I think you simply need to use an expId that exists. You can see which expId values are available with a query like: butler.queryMetadata('raw', format=['expId', 'raftName', 'detectorName'])
          Hide
          ttsai Te-Wei Tsai added a comment -

          The problem is that I used the w_2020_06 to ingest the raw data. After using the w_2020_07, the problem is solved. Thanks for the help!

          Show
          ttsai Te-Wei Tsai added a comment - The problem is that I used the w_2020_06 to ingest the raw data. After using the w_2020_07, the problem is solved. Thanks for the help!
          Hide
          tjenness Tim Jenness added a comment -

          Given the most recent comment I'm closing this ticket as invalid.

          Show
          tjenness Tim Jenness added a comment - Given the most recent comment I'm closing this ticket as invalid.

            People

            • Assignee:
              Unassigned
              Reporter:
              ttsai Te-Wei Tsai
              Watchers:
              Kian-Tat Lim, Simon Krughoff, Te-Wei Tsai, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel