Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:4
-
Team:Data Access and Database
Description
In the snippet:
import os |
|
|
dataPath = os.path.join("/datasets", "hsc", "repo", "rerun", |
"private", "hchiang2", "RC", "DM-10084-mosaic") |
|
|
butler = dafPersist.Butler(dataPath) |
|
|
dataId = dict(tract=8766, patch='2,1', filter='HSC-I') |
|
|
coadd = butler.get("deepCoadd_calexp", dataId) |
|
|
ci = coadd.getInfo().getCoaddInputs() |
|
|
for ccd in ci.ccds: |
fn = butler.get("calexp_filename", dataId, visit=ccd["visit"], ccd=ccd["ccd"])[0] |
print ccd["visit"], ccd["ccd"], fn, os.path.exists(fn), butler.get("calexp", dataId, visit=ccd["visit"], ccd=ccd["ccd"]).getDimensions() |
break |
Returns:
7304 8 /datasets/hsc/repo/rerun/private/hchiang2/RC/DM-10084-mosaic/00995/HSC-I/corr/CORR-0007304-008.fits False (2048, 4176)
|
In the case of a get (where write=False) the Butler should not be returning a path to a file that does not exist, it should look in parent repositories until an existing file is found, or return something meaning "file not found".
This seems to be an issue related to how bypass functions work. I think it’s because there’s no consideration for write=True or False in the filename bypass function (as there normally is for `map`ping), and so the map_xyz_filename function returns the first location its able to map (which would be appropriate for a `put` where write=True but is not appropriate for a get where write=False).
Attachments
Activity
A-ha, thanks. Should I add a test there as well?
It looks ok to me. FWIW I think it's worth being explicit somewhere/somehow that you're not trying to support composite datasets here (otherwise I think folks may come asking about that).
I missed your reply asking about adding a test. If you want to support composites I think that will be more work. I think it's a conversation you should have with stakeholders (which is not me at this point) about if the effort is worthwhile now, or maybe wait & see.
Implemented getUri(), which solves the problem that led to this ticket without changing the current functionality of _filename.
FWIW composites are tested in obs_base, see https://github.com/lsst/obs_base/blob/master/tests/test_composite.py