Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23214

Migrate Cassandra development branch to APDB

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • dax_ppdb
    • 2
    • DB_S20_01
    • Data Access and Database

    Description

      Cassandra development branch (u/andy-slac/cassandra) for dax_ppdb and l1dbproto needs an update after dax_ppdb was renamed to dax_apdb.

      Attachments

        Issue Links

          Activity

            Doing usual rebase is complicated as the files have been renamed/modified on both branches. I think it's probably easier to just create another branch on top of master and try to move stuff that works from old branch without keeping whole history.

            salnikov Andy Salnikov added a comment - Doing usual rebase is complicated as the files have been renamed/modified on both branches. I think it's probably easier to just create another branch on top of master and try to move stuff that works from old branch without keeping whole history.
            salnikov Andy Salnikov added a comment - - edited

            Here is  the summary of changes that were applied to u/andy-slac/cassandra branch since commit c7c8782:

            1. (406935c) renamed ppdbSchema.py to ppdbSqlSchema.py (no changes to the code)
            2. (01e5696) factrored out PpdbBaseSchemaConfig and PpdbBaseSchema from the above, these are base classes with a functionality common to all implementations.
              • new class PpdbSqlSchemaConfig inheriting from PpdbBaseSchemaConfig
              • PpdbConfig now inherits from PpdbSqlSchemaConfig
              • PPdbSchema renamed to PpdbSqlSchema inheriting PpdbBaseSchema
              • small changes to Ppdb class
              • test_ppdbSchema.py updated
            3. (ced16f0) implemented PpdbCassandraSchemaConfig(PpdbBaseSchemaConfig), PpdbCassandraSchema(PpdbBaseSchema), PpdbCassandraConfig(PpdbCassandraSchemaConfig) and no-op PpdbCassandra class
            4. (ea1dd4d) added implementation for few PpdbCassandra methods
            5. (1538a8e) added couple of more store methods
            6. (4e65d1d) added getDiaObjects implementation
            7. (b2349b9) implemented getDiaSources
            8. (53145bd4) improved query generation
            9. (a014b64c) month-based partitioning for sources
            10. (c656916) change hard-coded timeout value

            The non-trivial part here is refactoring of sql-based implementation (first two commits), I'm going to repeat that process to make sure that nothing breaks and then just copy Cassandra implemention in its current state from old branch (and rename ppdb -> apdb)

            salnikov Andy Salnikov added a comment - - edited Here is  the summary of changes that were applied to u/andy-slac/cassandra branch since commit c7c8782: (406935c) renamed ppdbSchema.py to ppdbSqlSchema.py (no changes to the code) (01e5696) factrored out PpdbBaseSchemaConfig and PpdbBaseSchema from the above, these are base classes with a functionality common to all implementations. new class PpdbSqlSchemaConfig inheriting from PpdbBaseSchemaConfig PpdbConfig now inherits from PpdbSqlSchemaConfig PPdbSchema renamed to PpdbSqlSchema inheriting PpdbBaseSchema small changes to Ppdb class test_ppdbSchema.py updated (ced16f0) implemented PpdbCassandraSchemaConfig(PpdbBaseSchemaConfig), PpdbCassandraSchema(PpdbBaseSchema), PpdbCassandraConfig(PpdbCassandraSchemaConfig) and no-op PpdbCassandra class (ea1dd4d) added implementation for few PpdbCassandra methods (1538a8e) added couple of more store methods (4e65d1d) added getDiaObjects implementation (b2349b9) implemented getDiaSources (53145bd4) improved query generation (a014b64c) month-based partitioning for sources (c656916) change hard-coded timeout value The non-trivial part here is refactoring of sql-based implementation (first two commits), I'm going to repeat that process to make sure that nothing breaks and then just copy Cassandra implemention in its current state from old branch (and rename ppdb -> apdb)
            salnikov Andy Salnikov added a comment - - edited

            Changes on master branch since commit c7c8782:

            1. Chris have moved the code from ap_verify_queries.py into Ppdb class
            2. lots of changes for remaining PPDB into APDB, but no change in the logic

            I think it should be OK to copy ppdbBaseSchema and ppdbSqlSchema from old branch instead of trying to reproduce that refactoring process, but it need care with renaming part.

            salnikov Andy Salnikov added a comment - - edited Changes on master branch since commit c7c8782: Chris have moved the code from ap_verify_queries.py into Ppdb class lots of changes for remaining PPDB into APDB, but no change in the logic I think it should be OK to copy ppdbBaseSchema and ppdbSqlSchema from old branch instead of trying to reproduce that refactoring process, but it need care with renaming part.

            Similarly I kept old branch u/andy-slac/cassandra in l1dbproto and replayed it on top of the new master making new branch u/andy-slac/cassandra-2.

            salnikov Andy Salnikov added a comment - Similarly I kept old branch u/andy-slac/cassandra in l1dbproto and replayed it on top of the new master making new branch u/andy-slac/cassandra-2 .
            salnikov Andy Salnikov added a comment - - edited

            Merged all changes, u/andy-slac/cassandra-2 is a new development branch for both packages. Tested that it works OK on my dev VM.

            salnikov Andy Salnikov added a comment - - edited Merged all changes, u/andy-slac/cassandra-2 is a new development branch for both packages. Tested that it works OK on my dev VM.

            People

              salnikov Andy Salnikov
              salnikov Andy Salnikov
              Andy Salnikov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.