Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13626

Butler inconsistently resolves duplicate repositories

    XMLWordPrintable

    Details

    • Story Points:
      2

      Description

      I've run into a situation where I wish to create a Butler that may have one or more input repositories. In other words, I'm calling (slightly simplified):

      Butler(inputs=[{'root': repo1}, {'root': repo2}], outputs={'root': outputRepo, 'mode': 'rw'})
      

      where the URIs repo1 and repo2 may represent the same location. When I first do this, the Butler detects whether the two repositories are the same, and if so removes the duplicate parent from outputRepo. So far, so good.

      However, if I rerun the program with the same repositories, I get the following error:

      ParentMismatch exception: The beginning of the passed-in parents list:
      [RepositoryCfg(root=<repodir>,
                     ...,
                     parents=[],
                     policy=None),
       RepositoryCfg(root=<repodir>,
                     ...,
                     parents=[],
                     policy=None)] does not match the existing parents list in this RepositoryCfg:
      [RepositoryCfg(root=<repodir>,
                     ...,
                     parents=[],
                     policy=None)]
      

      Aside from relative vs. absolute paths, the only difference between the two lists is that the former has a duplicate entry while the latter does not. Adding an explicit os.path.samefile(repo1, repo2) test to the calling code causes the problem to go away, so I assume it's the not-yet-pruned duplicates that are the problem. However, I'd prefer to not have to assume that the repository URIs are filesystem paths.

      Would it be possible to fix the test so that it's the "reduced" parent lists that get compared?

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            Nate Pease [X] is this something you have time to look into?

            Show
            rowen Russell Owen added a comment - Nate Pease [X] is this something you have time to look into?
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            We do have some time set aside for butler work, Fritz Mueller is the gatekeeper and can add it to the epic. Is the priority of the ticket set correctly? (It's 'minor'). 

             

            Show
            npease Nate Pease [X] (Inactive) added a comment - We do have some time set aside for butler work, Fritz Mueller is the gatekeeper and can add it to the epic. Is the priority of the ticket set correctly? (It's 'minor').   
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            (I'm estimating 2 points for this)

            Show
            npease Nate Pease [X] (Inactive) added a comment - (I'm estimating 2 points for this)
            Hide
            krzys Krzysztof Findeisen added a comment -

            Yes, minor. As noted in the issue text, there's a straightforward workaround.

            Show
            krzys Krzysztof Findeisen added a comment - Yes, minor. As noted in the issue text, there's a straightforward workaround.
            Hide
            tjenness Tim Jenness added a comment -

            Closing since we are not going to do any more work on daf_persistence.

            Show
            tjenness Tim Jenness added a comment - Closing since we are not going to do any more work on daf_persistence.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              krzys Krzysztof Findeisen
              Watchers:
              Krzysztof Findeisen, Nate Pease [X] (Inactive), Russell Owen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.