Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5268

Allow pluggable db access in daf_persistence

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler, daf_persistence
    • Labels:
      None
    • Team:
      Data Access and Database

      Description

      It was noticed by Robert Lupton on HipChat that daf_persistence depends on mariadbclient for persisting database tables (he was having trouble building mariadbclient on OS X because of the cmake dependency). There was a question as to whether database persistence should be implemented as a plug in system or use a more generic and portable DB abstraction layer. This is partly motivated by AFW depending on daf_persistence thereby causing any dependencies here to have a broad impact.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            When daf_butler eventually replaces daf_persistence, it should have no dependency on a database except SQLite. Until then, I didn't think it was much of a problem to leave it in. It would take a little effort to remove it. It looks like that would require changes in afw, cat, ctrl_orca/provenance, and mops_*.

            Low-level database access is being provided through SqlAlchemy, but this is not desirable for pipelines, which should use Butlerized access to databases from DM-4169.

            Show
            ktl Kian-Tat Lim added a comment - When daf_butler eventually replaces daf_persistence , it should have no dependency on a database except SQLite. Until then, I didn't think it was much of a problem to leave it in. It would take a little effort to remove it. It looks like that would require changes in afw , cat , ctrl_orca/provenance , and mops_* . Low-level database access is being provided through SqlAlchemy, but this is not desirable for pipelines, which should use Butlerized access to databases from DM-4169.
            Hide
            rhl Robert Lupton added a comment -

            That makes sense. Using daf_butler to fix this is fine with me (but I hope it isn't too far away)

            Show
            rhl Robert Lupton added a comment - That makes sense. Using daf_butler to fix this is fine with me (but I hope it isn't too far away)
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment - - edited

            I agree very much with the request in this ticket that access should be pluggable.

            I'm also quite happy to wait for pluggability to be part of the new daf_butler. Is daf_butler replacing daf_persistence with such a pluggable functionality something that will be completed in the next short 3-month cycle, or will it be in the next cycle?

            (Even though this particular one didn't bite me, I sadly realize I have lost a month of my life over the years to mysql client installation frustrations.)

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - - edited I agree very much with the request in this ticket that access should be pluggable. I'm also quite happy to wait for pluggability to be part of the new daf_butler . Is daf_butler replacing daf_persistence with such a pluggable functionality something that will be completed in the next short 3-month cycle, or will it be in the next cycle? (Even though this particular one didn't bite me, I sadly realize I have lost a month of my life over the years to mysql client installation frustrations.)
            Hide
            npease Nate Pease added a comment - - edited

            I don't know yet when the switch from daf_persistence to daf_butler will happen, until now there hasn't been a compelling need. (does this constitute a compelling need? Can an architect weigh in?)

            I agree that any storage access (which would include choice of database package) should be pluggable. It raises an interesting question: the daf_butler will contain some default implementations for pluggable components. If the butler is going to have a default database component, we'll have to decide how to integrate a database without adding a dependency to everyone who uses the butler. I'm sure there's several ways to overcome the issue, but we will have to figure out how to do The Right Thing).

            Show
            npease Nate Pease added a comment - - edited I don't know yet when the switch from daf_persistence to daf_butler will happen, until now there hasn't been a compelling need. (does this constitute a compelling need? Can an architect weigh in?) I agree that any storage access (which would include choice of database package) should be pluggable. It raises an interesting question: the daf_butler will contain some default implementations for pluggable components. If the butler is going to have a default database component, we'll have to decide how to integrate a database without adding a dependency to everyone who uses the butler. I'm sure there's several ways to overcome the issue, but we will have to figure out how to do The Right Thing).
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            sqlite3 has been a very workable default that is very widely installed.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - sqlite3 has been a very workable default that is very widely installed.
            Hide
            ktl Kian-Tat Lim added a comment -

            Removal of the mariadbclient dependency was performed in DM-13788 in March 2018.

            Show
            ktl Kian-Tat Lim added a comment - Removal of the mariadbclient dependency was performed in DM-13788 in March 2018.
            Hide
            ktl Kian-Tat Lim added a comment -

            And daf_butler depends only on sqlalchemy, which is pluggable, with sqlite3 used for testing.

            Show
            ktl Kian-Tat Lim added a comment - And daf_butler depends only on sqlalchemy , which is pluggable, with sqlite3 used for testing.

              People

              • Assignee:
                ktl Kian-Tat Lim
                Reporter:
                tjenness Tim Jenness
                Watchers:
                Fritz Mueller, Kian-Tat Lim, Michael Wood-Vasey, Nate Pease, Robert Lupton, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel