Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-211

Revise design for Qserv front-end rearchitect

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • Qserv
    • None

    Description

      Revisit the architecture. This includes proxy, down to XRootD client (mysqld, python, zookeeper). Capture all findings in new stories, add these stories to DM-1707

      Attachments

        Issue Links

          Activity

            See the comments in czar: appInterface.py and indexing.py (related to obsolete/unused code), and remove the obsolete code.

            jbecla Jacek Becla (Inactive) added a comment - See the comments in czar: appInterface.py and indexing.py (related to obsolete/unused code), and remove the obsolete code.

            Also, from a past email thread:

            • consider renaming to qcontrol and qmerger
            • re-design cluttered thread.cc and dispatcher.cc
            danielw Daniel Wang [X] (Inactive) added a comment - Also, from a past email thread: consider renaming to qcontrol and qmerger re-design cluttered thread.cc and dispatcher.cc

            Here are the things I noticed.

            • QuerySession->QueryPipeline. The "session" abstraction has moved to a
              better place. The iterator portion should be shifted into its own
              separate class, though perhaps still associated with
              QueryPipeline. The iterator portion's new home should be amenable to
              eventually moving the actual query materialization to the worker,
              though we shouldn't introduce new abstractions until we are actually
              ready to move the substitution/materialization to the worker.
            • SessionManagerAsync can go.
            • DynamicWorkQueue is nice thread-pooling work queue implementation
              (thanks, Serge!) that we don't need right now because XrdSsi has
              taken over its original use case, but it's likely we'll find it
              useful in other parts (replica management?).
            • QueryContext needs to be split into incoming external QueryContext
              and a sort of QueryClipboard for passing information between
              analysis/manipulation plugins. Eventually, I imagine a chain/tree of
              them attached to the select statements themselves in order to
              represent subquery scope nesting (which is complicated to represent
              and to reason about--nesting and the resulting namespace resolution
              is tricky), but I don't think we should try doing the chaining in
              the first phase.
            • thread and dispatcher are already taken care of by Fritz in another
              ticket, I think. They can be put in the attic.
            • TaskMsgFactory2 -->TaskMsgFactory. The original python
              TaskMsgFactory seems a distant memory now, so there shouldn't be
              anymore confusion.
            • The code that BillC put in that introduces random errors for fault-recovery testing was never ported to XrdSsi. I wonder if there's a way to interpose a similar fault-generating mechanism that can be toggled on/off for testing fault-recovery and robustness.
            danielw Daniel Wang [X] (Inactive) added a comment - Here are the things I noticed. QuerySession->QueryPipeline. The "session" abstraction has moved to a better place. The iterator portion should be shifted into its own separate class, though perhaps still associated with QueryPipeline. The iterator portion's new home should be amenable to eventually moving the actual query materialization to the worker, though we shouldn't introduce new abstractions until we are actually ready to move the substitution/materialization to the worker. SessionManagerAsync can go. DynamicWorkQueue is nice thread-pooling work queue implementation (thanks, Serge!) that we don't need right now because XrdSsi has taken over its original use case, but it's likely we'll find it useful in other parts (replica management?). QueryContext needs to be split into incoming external QueryContext and a sort of QueryClipboard for passing information between analysis/manipulation plugins. Eventually, I imagine a chain/tree of them attached to the select statements themselves in order to represent subquery scope nesting (which is complicated to represent and to reason about--nesting and the resulting namespace resolution is tricky), but I don't think we should try doing the chaining in the first phase. thread and dispatcher are already taken care of by Fritz in another ticket, I think. They can be put in the attic. TaskMsgFactory2 -->TaskMsgFactory. The original python TaskMsgFactory seems a distant memory now, so there shouldn't be anymore confusion. The code that BillC put in that introduces random errors for fault-recovery testing was never ported to XrdSsi. I wonder if there's a way to interpose a similar fault-generating mechanism that can be toggled on/off for testing fault-recovery and robustness.

            No new code in this ticket. Does anyone want to have a look before we close it? I haven't made the new tickets yet... would like to get back to geom integration now.

            danielw Daniel Wang [X] (Inactive) added a comment - No new code in this ticket. Does anyone want to have a look before we close it? I haven't made the new tickets yet... would like to get back to geom integration now.

            I need to add tickets for these. Will do that now/this week.

            danielw Daniel Wang [X] (Inactive) added a comment - I need to add tickets for these. Will do that now/this week.

            Removal of SessionManagerAsync will get rolled in on some tickets I already have in progress I think, so no new ticket needed there.

            fritzm Fritz Mueller added a comment - Removal of SessionManagerAsync will get rolled in on some tickets I already have in progress I think, so no new ticket needed there.

            Tickets created: DM-2059 DM-2060 DM-2061
            Closing.

            danielw Daniel Wang [X] (Inactive) added a comment - Tickets created: DM-2059 DM-2060 DM-2061 Closing.

            People

              fritzm Fritz Mueller
              fritzm Fritz Mueller
              Jacek Becla (Inactive)
              Daniel Wang [X] (Inactive), Fritz Mueller, Jacek Becla (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.