Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4683

Implementation of multiple repositories v2

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None
    • Story Points:
      15
    • Sprint:
      DB_W16_02, DB_W16_03
    • Team:
      Data Access and Database

      Attachments

        Issue Links

          Activity

          Hide
          ktl Kian-Tat Lim added a comment -

          On the whole, this looks reasonable in terms of the Access/Storage/Repository design. Some concerns:

          • Storage and PosixStorage should probably be in their own file.
          • Possible confusion between your Python Storage and its subclasses and the C++ Storage classes.
          • The Repository constructor calls setCfg() to write the repository configuration out. This has potential problems if the file already exists and is different or if many nodes are writing to the repository at the same time.
          • Static method Repository.Repository() might be renamed makeRepository() or createRepository().
          • Because this is in transition, the compatibility between old repositories using _parent links and the new configurations is a little messy. I believe that _parent will be handled before anything in the _parents list, correct?
          • setCfg() might be named better as saveCfg(); that's the usual antonym for load.
          • We should be more clear about what various Mapper methods return. For example, map() could always return a list of ButlerLocations or it could return either a ButlerLocation or a list of ButlerLocations, but it shouldn't be able to return anything else.
          • I think Gregory came up with a use case where we need to be able to label parent repositories, not just handle them as an ordered list, but that can wait for a future ticket.
          • Squashing and reorganizing the commits to distinguish better between moving existing code and creating new code would be appreciated.
          Show
          ktl Kian-Tat Lim added a comment - On the whole, this looks reasonable in terms of the Access/Storage/Repository design. Some concerns: Storage and PosixStorage should probably be in their own file. Possible confusion between your Python Storage and its subclasses and the C++ Storage classes. The Repository constructor calls setCfg() to write the repository configuration out. This has potential problems if the file already exists and is different or if many nodes are writing to the repository at the same time. Static method Repository.Repository() might be renamed makeRepository() or createRepository() . Because this is in transition, the compatibility between old repositories using _parent links and the new configurations is a little messy. I believe that _parent will be handled before anything in the _parents list, correct? setCfg() might be named better as saveCfg() ; that's the usual antonym for load . We should be more clear about what various Mapper methods return. For example, map() could always return a list of ButlerLocations or it could return either a ButlerLocation or a list of ButlerLocations , but it shouldn't be able to return anything else. I think Gregory came up with a use case where we need to be able to label parent repositories, not just handle them as an ordered list, but that can wait for a future ticket. Squashing and reorganizing the commits to distinguish better between moving existing code and creating new code would be appreciated.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          What did this ticket do? The description is blank.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - What did this ticket do? The description is blank.
          Hide
          npease Nate Pease added a comment - - edited

          at this point the easiest thing to do is read about repositories in documentation (that is WIP on its own ticket branch) (LDM-463. In short, repo used to be specified by passing root="c:/path/to/repository" to butler init. Now: there is what's described in the LDM.

          Show
          npease Nate Pease added a comment - - edited at this point the easiest thing to do is read about repositories in documentation (that is WIP on its own ticket branch) ( LDM-463 . In short, repo used to be specified by passing root="c:/path/to/repository" to butler init. Now: there is what's described in the LDM.
          Hide
          tjenness Tim Jenness added a comment -

          In future please ensure a description is included in the ticket even if that is just a link to some other documentation.

          Show
          tjenness Tim Jenness added a comment - In future please ensure a description is included in the ticket even if that is just a link to some other documentation.
          Hide
          tjenness Tim Jenness added a comment -

          Nate Pease I'm trying to write the weekly summary and want to include this work in it but you aren't making it easy. I don't think in the comment above that you really want to point me to LDM-135. I assume I am meant to be looking at the document mentioned in the community post at http://ldm-463.lsst.io/en/tickets-dm-5091/ but that is initially confusing because it cites DM-5091 which is a ticket describing the general act of documenting the butler not specifically documenting this item. DM-5091 does not have any description and does not have any linked tickets. Do I assume that DM-4683 was a blocker for DM-5091? I'm also going to make a quick comment on the community post to clarify things for me.

          Show
          tjenness Tim Jenness added a comment - Nate Pease I'm trying to write the weekly summary and want to include this work in it but you aren't making it easy. I don't think in the comment above that you really want to point me to LDM-135. I assume I am meant to be looking at the document mentioned in the community post at http://ldm-463.lsst.io/en/tickets-dm-5091/ but that is initially confusing because it cites DM-5091 which is a ticket describing the general act of documenting the butler not specifically documenting this item. DM-5091 does not have any description and does not have any linked tickets. Do I assume that DM-4683 was a blocker for DM-5091 ? I'm also going to make a quick comment on the community post to clarify things for me.
          Hide
          npease Nate Pease added a comment - - edited

          I corrected the LDM links. DM-4683 and DM-5091 were developed in parallel. One did not block the other, but since I had to write about butler in the LDM I figured I'd write about the soon-to-be current (now current) implementation.

          Sorry for no description in this ticket.

          Tim Jenness Also I'm sorry for not making it easy for you. Figuring out how to use all the correct channels and checkboxes to receive a feature request, gather consensus, design, implement, and submit code in LSST is also not easy. I'm doing the best I can figure out. Thanks for bearing with me.

          Show
          npease Nate Pease added a comment - - edited I corrected the LDM links. DM-4683 and DM-5091 were developed in parallel. One did not block the other, but since I had to write about butler in the LDM I figured I'd write about the soon-to-be current (now current) implementation. Sorry for no description in this ticket. Tim Jenness Also I'm sorry for not making it easy for you. Figuring out how to use all the correct channels and checkboxes to receive a feature request, gather consensus, design, implement, and submit code in LSST is also not easy . I'm doing the best I can figure out. Thanks for bearing with me.

            People

            Assignee:
            npease Nate Pease
            Reporter:
            npease Nate Pease
            Reviewers:
            Kian-Tat Lim
            Watchers:
            Jacek Becla, Kian-Tat Lim, Michael Wood-Vasey, Nate Pease, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: