Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18577

Fix PosixDataStore naming

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: daf_butler
    • Labels:
      None

      Description

      Currently the name associated with a Posix data store includes the path to the directory root. This is usually a full path and the name is used to indicate to the butler registry that the named datastore has a particular dataset.

      If a datastore is moved to another location the name will change and the config file must be edited to reflect the new root location. This will not update the name associated with the datasets already known to the registry. Root can be a relative path but it is relative to the person using the butler and not relative to the butler directory.

      Things sort of work by luck at the moment since butler never calls getDatasetLocations.

      The posix datastore config also specifies a name for the records.table which must be unique. It is not related to the name at all and I wonder if it would be easier if it was.

      For this ticket I will:

      • Investigate specifying a name field in posix datastore config and using that instead of the root.
      • Consider allowing root to be specified relative to butler root to allow the simple case of relocating a posix datastore with butler sqlite registry.
      • Consider deriving the name of table.records from datastore name without having to specify a unique string in two places.

      Opinions welcomed.

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          Jim Bosch on this ticket I've made a number of changes relating to repository relocation:

          • You can now use <root> in the path to the posix datastore or registry. This will then be replaced by the root coming from the butler (either from the butler yaml location or from the name of the root defined in that yaml).
          • The name of the posix datastore can now be specified in the datastore yaml, overriding the value derived from the root.
          Show
          tjenness Tim Jenness added a comment - Jim Bosch on this ticket I've made a number of changes relating to repository relocation: You can now use <root> in the path to the posix datastore or registry. This will then be replaced by the root coming from the butler (either from the butler yaml location or from the name of the root defined in that yaml). The name of the posix datastore can now be specified in the datastore yaml, overriding the value derived from the root.
          Hide
          tjenness Tim Jenness added a comment -

          Jim Bosch I needed to fix one line in obs_subaru ingest tests since the test was assuming files are written to the same directory as the butler.yaml.

          Show
          tjenness Tim Jenness added a comment - Jim Bosch I needed to fix one line in obs_subaru ingest tests since the test was assuming files are written to the same directory as the butler.yaml.
          Hide
          tjenness Tim Jenness added a comment -

          Jim Bosch thanks for your review. I've had to make some changes to ci_hsc to get it to respect the dynamic root. I have removed the "/datastore" from the makeRepo yaml output file and instead added datastore/ to the default file template in ci_hsc. I needed to make some related changes in daf_butler to support explicit search paths in the ButlerConfig constructor. Please can you take a look at the ci_hsc changes (and the related daf_butler changes if you are interested).

          Show
          tjenness Tim Jenness added a comment - Jim Bosch thanks for your review. I've had to make some changes to ci_hsc to get it to respect the dynamic root. I have removed the "/datastore" from the makeRepo yaml output file and instead added datastore/ to the default file template in ci_hsc . I needed to make some related changes in daf_butler to support explicit search paths in the ButlerConfig constructor. Please can you take a look at the ci_hsc changes (and the related daf_butler changes if you are interested).
          Hide
          tjenness Tim Jenness added a comment -

          Merged.

          Show
          tjenness Tim Jenness added a comment - Merged.

            People

            • Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Michelle Gower, Nate Lust, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel