Details
-
Type:
Improvement
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler
-
Labels:
-
Story Points:0.25
-
Team:Architecture
Description
Many Butler operations (ingest, import, export, etc.) have an option to handle files through links. However, applying this option to datasets that are themselves links leads to chains of links, which rely on every intermediate copy of the data to continue existing. It would be helpful to have an option that resolves links before processing them, so that any links in the output point directly to the original file.
Jim Bosch and Nate Lust suggest much of the functionality for this could be implemented in lsst.daf.butler.datastores.posixDatastore.PosixDatastore.
Attachments
Issue Links
- relates to
-
DM-21915 Support Gen 3 ingestion of ap_verify datasets
- Done
Activity
The tests are always much more work than the fixes. I'll see if the simple ingest test in daf_butler is amenable to a symlink to a symlink.
Krzysztof Findeisen I've added a test if you want to take a look at it. Test fails with old code.
I found the test a bit hard to read, but it looks like it tests the key issue. 
Krzysztof Findeisen
added a comment - I found the test a bit hard to read, but it looks like it tests the key issue. Yes, I have to create a file and then I have to get a second temporary file path so that I can symlink to that one and then I need to do the ingest and check that that ingested version is a symlink that points to the first file and not the second one.
It would be nice to create a unit test, but I assume that's not possible? Thanks for the quick response!