Details
-
Type:
Story
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler, daf_persistence
-
Labels:None
-
Team:Architecture
-
Urgent?:No
Description
When ingestImages.py is run, a series of filesystem links are created pointing from the data repo to raw image files. If the raw image files are subsequently compressed (e.g., gzipped), then the links no longer point to the data. This is despite the fact that the Butler would perfectly happy to read the gzipped files.
This ticket is to extend the Butler search routine so that if a link pointing to foo.fits can not be resolved to a file, then foo.fits.gz is tried as a fall back. Presumably other common compression extensions could also be checked (foo.fits.bz2, foo.fz) (though my use case at the moment is just .gz).
Also, if someone with more experience in this area wants to come and clean up my very non-expert description in this ticket, please do!
Attachments
Issue Links
- mentioned in
-
Page Loading...
Activity
Changing the link names when the data is compressed sounds very reasonable.
I'm going to close this on the basis that I don't think gen3 should be doing this either.
At first glance, I don't think it is appropriate for the Butler to determine that it is looking at a broken symlink and check for alternate filenames.
Other possibilities would be to not have ingestImages.py create links (if the Butler owns the repo, it ought to own all aspects of the storage) or, less severely, to change the link names when the data is compressed.