Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-520

Remove old partitioner/ loader and duplicator

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Once Fabrice has migrated the integrated tests towards using the new partitioner and duplicator, we should delete the old partitioner/duplicator (in client/examples).

        Attachments

          Issue Links

            Activity

            Hide
            jbecla Jacek Becla added a comment -

            Once this is done, we can update https://confluence.lsstcorp.org/display/DM/DM+Third+Party+Software and remove dependency between Qserv and numpy.

            Show
            jbecla Jacek Becla added a comment - Once this is done, we can update https://confluence.lsstcorp.org/display/DM/DM+Third+Party+Software and remove dependency between Qserv and numpy.
            Hide
            jammes Fabrice Jammes added a comment -

            Jacek Becla,

            Ok, above documentation updated.

            Show
            jammes Fabrice Jammes added a comment - Jacek Becla , Ok, above documentation updated.
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Daniel Wang [X],

            I removed some code from core/ in this ticket so I would like you to review it.
            It's very small, it only removes some files.

            Thanks,

            Fabrice

            Show
            jammes Fabrice Jammes added a comment - Hi Daniel Wang [X] , I removed some code from core/ in this ticket so I would like you to review it. It's very small, it only removes some files. Thanks, Fabrice
            Hide
            danielw Daniel Wang [X] (Inactive) added a comment -

            This looks mostly fine, but... we are removing functionality.

            plotChunksCopies.tcl, while dependent on output from obsolete code, is still the only way we have of visualizing the chunk layout. I recommend we keep it and plan on some small work to port it for the current infrastructure.

            rebalance.py provides a chunk-to-node mapping. It should work fine on the new partitioner's output. It depends on loader.py. Do we have a replacement for it? We might borrow code from this and the old loader in building the multi-node loader, so I think we should wait until the multi-node loader is done before removing this (and loader.py, even though we won't be using loader.py to load anymore).

            Show
            danielw Daniel Wang [X] (Inactive) added a comment - This looks mostly fine, but... we are removing functionality. plotChunksCopies.tcl , while dependent on output from obsolete code, is still the only way we have of visualizing the chunk layout. I recommend we keep it and plan on some small work to port it for the current infrastructure. rebalance.py provides a chunk-to-node mapping. It should work fine on the new partitioner's output. It depends on loader.py. Do we have a replacement for it? We might borrow code from this and the old loader in building the multi-node loader, so I think we should wait until the multi-node loader is done before removing this (and loader.py, even though we won't be using loader.py to load anymore).
            Hide
            danielw Daniel Wang [X] (Inactive) added a comment -

            So, I suggest keeping plotChunksCopies, rebalance, and loader. Maybe put a note in loader.py to say that it is only being kept in support of rebalance, and both should go away when a multi-node loader lands.

            Show
            danielw Daniel Wang [X] (Inactive) added a comment - So, I suggest keeping plotChunksCopies, rebalance, and loader. Maybe put a note in loader.py to say that it is only being kept in support of rebalance, and both should go away when a multi-node loader lands.
            Hide
            jammes Fabrice Jammes added a comment -

            I Daniel,

            I put the tree files in core/examples/obsolete, added core/examples/obsolete/README.txt and created a new ticket DM-1902 for plotChunkCopies.tcl port.

            Thanks.

            Show
            jammes Fabrice Jammes added a comment - I Daniel, I put the tree files in core/examples/obsolete , added core/examples/obsolete/README.txt and created a new ticket DM-1902 for plotChunkCopies.tcl port. Thanks.

              People

              Assignee:
              jammes Fabrice Jammes
              Reporter:
              smonkewitz Serge Monkewitz
              Reviewers:
              Daniel Wang [X] (Inactive)
              Watchers:
              Daniel Wang [X] (Inactive), Fabrice Jammes, Jacek Becla, Serge Monkewitz
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.