Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24314

Support following links in Butler operations

    Details

    • Story Points:
      0.25
    • Team:
      Architecture

      Description

      Many Butler operations (ingest, import, export, etc.) have an option to handle files through links. However, applying this option to datasets that are themselves links leads to chains of links, which rely on every intermediate copy of the data to continue existing. It would be helpful to have an option that resolves links before processing them, so that any links in the output point directly to the original file.

      Jim Bosch and Nate Lust suggest much of the functionality for this could be implemented in lsst.daf.butler.datastores.posixDatastore.PosixDatastore.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            It would be nice to create a unit test, but I assume that's not possible? Thanks for the quick response!

            Show
            krzys Krzysztof Findeisen added a comment - - edited It would be nice to create a unit test, but I assume that's not possible? Thanks for the quick response!
            Hide
            tjenness Tim Jenness added a comment -

            The tests are always much more work than the fixes. I'll see if the simple ingest test in daf_butler is amenable to a symlink to a symlink.

            Show
            tjenness Tim Jenness added a comment - The tests are always much more work than the fixes. I'll see if the simple ingest test in daf_butler is amenable to a symlink to a symlink.
            Hide
            tjenness Tim Jenness added a comment -

            Krzysztof Findeisen I've added a test if you want to take a look at it. Test fails with old code.

            Show
            tjenness Tim Jenness added a comment - Krzysztof Findeisen I've added a test if you want to take a look at it. Test fails with old code.
            Hide
            krzys Krzysztof Findeisen added a comment -

            I found the test a bit hard to read, but it looks like it tests the key issue.

            Show
            krzys Krzysztof Findeisen added a comment - I found the test a bit hard to read, but it looks like it tests the key issue.
            Hide
            tjenness Tim Jenness added a comment -

            Yes, I have to create a file and then I have to get a second temporary file path so that I can symlink to that one and then I need to do the ingest and check that that ingested version is a symlink that points to the first file and not the second one.

            Show
            tjenness Tim Jenness added a comment - Yes, I have to create a file and then I have to get a second temporary file path so that I can symlink to that one and then I need to do the ingest and check that that ingested version is a symlink that points to the first file and not the second one.

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                Krzysztof Findeisen
                Watchers:
                Jim Bosch, Krzysztof Findeisen, Nate Lust, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: