Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29106

Allow butler to "forget" datasets without deleting them

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      We can now ingest datasets with "direct" transfer, adding them to a data repository while leaving them in an external, usually read-only location. We now need the inverse operation: a way to remove datasets from the data repository's records without actually deleting the ingested files.

      I have started this on the DM-28636 branch (where I need it to clean up a incomplete gen2to3 conversion), and will put it on the branch for this ticket for separate review and merge.

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          Good point. If you prune a dataset the datastore should automatically spot that it doesn't own the file. I completely forgot about that. This seems to be orthogonal to an explicit forget() method though.

          Show
          tjenness Tim Jenness added a comment - Good point. If you prune a dataset the datastore should automatically spot that it doesn't own the file. I completely forgot about that. This seems to be orthogonal to an explicit forget() method though.
          Hide
          jbosch Jim Bosch added a comment -

          Yeah, I think there's a need for both guessing whether to forget or delete for a dataset (based on whether it's in the tree) and specifying that explicitly. Guessing seemed harder, and I didn't need it, so I just decided to quickly do the explicit route here.

          Show
          jbosch Jim Bosch added a comment - Yeah, I think there's a need for both guessing whether to forget or delete for a dataset (based on whether it's in the tree) and specifying that explicitly. Guessing seemed harder, and I didn't need it, so I just decided to quickly do the explicit route here.
          Hide
          tjenness Tim Jenness added a comment -

          Guessing is easy because the entry in the datastore will be a fully-specified URI rather than a relative path.

          Show
          tjenness Tim Jenness added a comment - Guessing is easy because the entry in the datastore will be a fully-specified URI rather than a relative path.
          Hide
          jbosch Jim Bosch added a comment -

          This is ready for review but doesn't need to be done in a hurry; I want to give people a chance to digest and comment on the related CLO post (see PR) before starting us down that path.

          PR is https://github.com/lsst/daf_butler/pull/486.

          Show
          jbosch Jim Bosch added a comment - This is ready for review but doesn't need to be done in a hurry; I want to give people a chance to digest and comment on the related CLO post (see PR) before starting us down that path. PR is https://github.com/lsst/daf_butler/pull/486 .
          Hide
          salnikov Andy Salnikov added a comment -

          Looks good, couple of comments on PR.

          Show
          salnikov Andy Salnikov added a comment - Looks good, couple of comments on PR.

            People

            Assignee:
            jbosch Jim Bosch
            Reporter:
            jbosch Jim Bosch
            Reviewers:
            Andy Salnikov
            Watchers:
            Andy Salnikov, Jim Bosch, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: