# Bug of Data Butler to Get the Raw Data

XMLWordPrintable

## Details

• Type: Bug
• Status: Invalid
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Urgent?:
No

## Description

It looks like there is the bug for butler to get the raw exposure data in lsst_distrib: w_2020_06. This bug did not happen in w_2020_04. The following is the test data: input.zip

I use the followings to try to get the raw data:

   from lsst.daf.persistence import Butler  butler = Butler(inputs="input/")  dataId = dict(expId=20, raftName="R00", detectorName="S22")  raw = butler.get("{color:#ff2600}raw{color}", dataId=dataId) 

The following is the error message (You can also see details in errorMsg.txt):

  {color:#0747a6}NoResults: No locations for get: datasetType:raw_amp dataId:DataId(initialdata=\{'expId': 20, 'raftName': 'R00', 'detectorName': 'S22', 'run': '20', 'detector': 197, 'channel': 1}, tag=set()){color} 

It looks like the dataset type becomes "raw_amp" even though I use "raw" in the get().
The followins is the data ID of "raw" if I use butler.getKeys("raw"):

 {'run': str,   'raftName': str,   'expId': int,   'detectorName': str,   'detector': int} 

There should be no 'channel' because it is not "raw_amp".

## Attachments

1. errorMsg.txt
3 kB
2. input.zip
14.90 MB

## Activity

Te-Wei Tsai created issue -
Field Original Value New Value
Assignee Simon Krughoff [ krughoff ]
Hide
Simon Krughoff added a comment - - edited

It sounds like you need to use visit instead of expId based on this post.

Will you try substituting visit=20 for expId=20 and posting results?

Show
Simon Krughoff added a comment - - edited It sounds like you need to use visit instead of expId based on this post . Will you try substituting visit=20 for expId=20 and posting results?
Hide
Te-Wei Tsai added a comment -

I tried the visit first (in w_2020_04 and before) and realized no such key supported in w_2020_06. This is why I used butler.getKeys("raw") to have the idea of keys. You can see no such key in the raw data set type.

Show
Te-Wei Tsai added a comment - I tried the visit first (in w_2020_04 and before) and realized no such key supported in w_2020_06. This is why I used butler.getKeys("raw") to have the idea of keys. You can see no such key in the raw data set type.
Hide
Simon Krughoff added a comment -

Did you reingest the data with the _06 stack?

Show
Simon Krughoff added a comment - Did you reingest the data with the _06 stack?
Hide
Simon Krughoff added a comment -

Actually, I've been informed that _07 is out and contains fixes in ISR. It would be good if you tried that one.

Show
Simon Krughoff added a comment - Actually, I've been informed that _07 is out and contains fixes in ISR. It would be good if you tried that one.
Hide
Te-Wei Tsai added a comment -

I re-ingested the data with w_2020_06 version. I could try it in w_2020_07. Thanks!

Show
Te-Wei Tsai added a comment - I re-ingested the data with w_2020_06 version. I could try it in w_2020_07. Thanks!
Hide
Te-Wei Tsai added a comment -

The error is still there even though I use the version of w_2020_07. You can see the test result here.

---------------------------------------------------------------------

E lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:raw_amp dataId:DataId(initialdata={'expId': 20, 'raftName': 'R00', 'detectorName': 'S22', 'run': '20', 'detector': 197, 'channel': 1}, tag=set())

Show
Te-Wei Tsai added a comment - The error is still there even though I use the version of w_2020_07. You can see the test result here. --------------------------------------------------------------------- E lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:raw_amp dataId:DataId(initialdata={'expId': 20, 'raftName': 'R00', 'detectorName': 'S22', 'run': '20', 'detector': 197, 'channel': 1}, tag=set())
 Component/s obs_lsst [ 16504 ] Component/s Science Pipelines [ 10706 ]
 Labels SciencePipelines
 Watchers Simon Krughoff, Te-Wei Tsai [ Simon Krughoff, Te-Wei Tsai ] Kian-Tat Lim, Simon Krughoff, Te-Wei Tsai [ Kian-Tat Lim, Simon Krughoff, Te-Wei Tsai ]
Hide
Simon Krughoff added a comment -

FYI, the link in the previous comment doesn't work for me, for some reason.

Can you provide the full list of commands you run to get the above exception?

Show
Simon Krughoff added a comment - FYI, the link in the previous comment doesn't work for me, for some reason. Can you provide the full list of commands you run to get the above exception?
Hide
Te-Wei Tsai added a comment -

It is in the description. Let me list it in the following. Thanks!

from lsst.daf.persistence import Butler
butler = Butler(inputs="input/")
dataId = dict(expId=20, raftName="R00", detectorName="S22")
raw = butler.get("raw", dataId=dataId)

Show
Te-Wei Tsai added a comment - It is in the description. Let me list it in the following. Thanks! from lsst.daf.persistence import Butler butler = Butler(inputs="input/") dataId = dict(expId=20, raftName="R00", detectorName="S22") raw = butler.get(" raw ", dataId=dataId)
Hide
Simon Krughoff added a comment -

OK the problem is that an image with expId=20 doesn't exist. I think you simply need to use an expId that exists.

You can see which expId values are available with a query like:

 butler.queryMetadata('raw', format=['expId', 'raftName', 'detectorName']) 

Show
Simon Krughoff added a comment - OK the problem is that an image with expId=20 doesn't exist. I think you simply need to use an expId that exists. You can see which expId values are available with a query like: butler.queryMetadata('raw', format=['expId', 'raftName', 'detectorName'])
Hide
Te-Wei Tsai added a comment -

The problem is that I used the w_2020_06 to ingest the raw data. After using the w_2020_07, the problem is solved. Thanks for the help!

Show
Te-Wei Tsai added a comment - The problem is that I used the w_2020_06 to ingest the raw data. After using the w_2020_07, the problem is solved. Thanks for the help!
 Resolution Done [ 10000 ] Status To Do [ 10001 ] Invalid [ 11005 ]
Hide
Tim Jenness added a comment -

Given the most recent comment I'm closing this ticket as invalid.

Show
Tim Jenness added a comment - Given the most recent comment I'm closing this ticket as invalid.
 Description It looks like there is the bug for butler to get the raw exposure data in lsst_distrib: w_2020_06. This bug did not happen in w_2020_04. The following is the test data: [^input.zip] I use the followings to try to get the raw data:  -----------------------------------------------------------------  from lsst.daf.persistence import Butler  butler = Butler(inputs="input/")  dataId = dict(expId=20, raftName="R00", detectorName="S22")  raw = butler.get("{color:#ff2600}raw{color}", dataId=dataId)  ----------------------------------------------------------------- The following is the error message (You can also see details in [^errorMsg.txt]): ------------------------------  {color:#0747a6}NoResults: No locations for get: datasetType:raw_amp dataId:DataId(initialdata=\{'expId': 20, 'raftName': 'R00', 'detectorName': 'S22', 'run': '20', 'detector': 197, 'channel': 1}, tag=set()){color}  ------------------------------ It looks like the dataset type becomes "raw_amp" even though I use "raw" in the get().  The followins is the data ID of "raw" if I use butler.getKeys("raw"): \{'run': str,   'raftName': str,   'expId': int,   'detectorName': str,   'detector': int} There should be no 'channel' because it is not "raw_amp". It looks like there is the bug for butler to get the raw exposure data in lsst_distrib: w_2020_06. This bug did not happen in w_2020_04. The following is the test data: [^input.zip] I use the followings to try to get the raw data: {code} from lsst.daf.persistence import Butler  butler = Butler(inputs="input/")  dataId = dict(expId=20, raftName="R00", detectorName="S22")  raw = butler.get("{color:#ff2600}raw{color}", dataId=dataId) {code} The following is the error message (You can also see details in [^errorMsg.txt]): {code}  {color:#0747a6}NoResults: No locations for get: datasetType:raw_amp dataId:DataId(initialdata=\{'expId': 20, 'raftName': 'R00', 'detectorName': 'S22', 'run': '20', 'detector': 197, 'channel': 1}, tag=set()){color} {code} It looks like the dataset type becomes "raw_amp" even though I use "raw" in the get().  The followins is the data ID of "raw" if I use butler.getKeys("raw"): {code} {'run': str,   'raftName': str,   'expId': int,   'detectorName': str,   'detector': int} {code} There should be no 'channel' because it is not "raw_amp".

## People

• Assignee:
Unassigned
Reporter:
Te-Wei Tsai
Watchers:
Kian-Tat Lim, Simon Krughoff, Te-Wei Tsai, Tim Jenness