Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-211

Revise design for Qserv front-end rearchitect

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Revisit the architecture. This includes proxy, down to XRootD client (mysqld, python, zookeeper). Capture all findings in new stories, add these stories to DM-1707

        Attachments

          Issue Links

            Activity

            Hide
            jbecla Jacek Becla added a comment -

            See the comments in czar: appInterface.py and indexing.py (related to obsolete/unused code), and remove the obsolete code.

            Show
            jbecla Jacek Becla added a comment - See the comments in czar: appInterface.py and indexing.py (related to obsolete/unused code), and remove the obsolete code.
            Hide
            danielw Daniel Wang [X] (Inactive) added a comment -

            Also, from a past email thread:

            • consider renaming to qcontrol and qmerger
            • re-design cluttered thread.cc and dispatcher.cc
            Show
            danielw Daniel Wang [X] (Inactive) added a comment - Also, from a past email thread: consider renaming to qcontrol and qmerger re-design cluttered thread.cc and dispatcher.cc
            Hide
            danielw Daniel Wang [X] (Inactive) added a comment -

            Here are the things I noticed.

            • QuerySession->QueryPipeline. The "session" abstraction has moved to a
              better place. The iterator portion should be shifted into its own
              separate class, though perhaps still associated with
              QueryPipeline. The iterator portion's new home should be amenable to
              eventually moving the actual query materialization to the worker,
              though we shouldn't introduce new abstractions until we are actually
              ready to move the substitution/materialization to the worker.
            • SessionManagerAsync can go.
            • DynamicWorkQueue is nice thread-pooling work queue implementation
              (thanks, Serge!) that we don't need right now because XrdSsi has
              taken over its original use case, but it's likely we'll find it
              useful in other parts (replica management?).
            • QueryContext needs to be split into incoming external QueryContext
              and a sort of QueryClipboard for passing information between
              analysis/manipulation plugins. Eventually, I imagine a chain/tree of
              them attached to the select statements themselves in order to
              represent subquery scope nesting (which is complicated to represent
              and to reason about--nesting and the resulting namespace resolution
              is tricky), but I don't think we should try doing the chaining in
              the first phase.
            • thread and dispatcher are already taken care of by Fritz in another
              ticket, I think. They can be put in the attic.
            • TaskMsgFactory2 -->TaskMsgFactory. The original python
              TaskMsgFactory seems a distant memory now, so there shouldn't be
              anymore confusion.
            • The code that BillC put in that introduces random errors for fault-recovery testing was never ported to XrdSsi. I wonder if there's a way to interpose a similar fault-generating mechanism that can be toggled on/off for testing fault-recovery and robustness.
            Show
            danielw Daniel Wang [X] (Inactive) added a comment - Here are the things I noticed. QuerySession->QueryPipeline. The "session" abstraction has moved to a better place. The iterator portion should be shifted into its own separate class, though perhaps still associated with QueryPipeline. The iterator portion's new home should be amenable to eventually moving the actual query materialization to the worker, though we shouldn't introduce new abstractions until we are actually ready to move the substitution/materialization to the worker. SessionManagerAsync can go. DynamicWorkQueue is nice thread-pooling work queue implementation (thanks, Serge!) that we don't need right now because XrdSsi has taken over its original use case, but it's likely we'll find it useful in other parts (replica management?). QueryContext needs to be split into incoming external QueryContext and a sort of QueryClipboard for passing information between analysis/manipulation plugins. Eventually, I imagine a chain/tree of them attached to the select statements themselves in order to represent subquery scope nesting (which is complicated to represent and to reason about--nesting and the resulting namespace resolution is tricky), but I don't think we should try doing the chaining in the first phase. thread and dispatcher are already taken care of by Fritz in another ticket, I think. They can be put in the attic. TaskMsgFactory2 -->TaskMsgFactory. The original python TaskMsgFactory seems a distant memory now, so there shouldn't be anymore confusion. The code that BillC put in that introduces random errors for fault-recovery testing was never ported to XrdSsi. I wonder if there's a way to interpose a similar fault-generating mechanism that can be toggled on/off for testing fault-recovery and robustness.
            Hide
            danielw Daniel Wang [X] (Inactive) added a comment -

            No new code in this ticket. Does anyone want to have a look before we close it? I haven't made the new tickets yet... would like to get back to geom integration now.

            Show
            danielw Daniel Wang [X] (Inactive) added a comment - No new code in this ticket. Does anyone want to have a look before we close it? I haven't made the new tickets yet... would like to get back to geom integration now.
            Hide
            danielw Daniel Wang [X] (Inactive) added a comment -

            I need to add tickets for these. Will do that now/this week.

            Show
            danielw Daniel Wang [X] (Inactive) added a comment - I need to add tickets for these. Will do that now/this week.
            Hide
            fritzm Fritz Mueller added a comment -

            Removal of SessionManagerAsync will get rolled in on some tickets I already have in progress I think, so no new ticket needed there.

            Show
            fritzm Fritz Mueller added a comment - Removal of SessionManagerAsync will get rolled in on some tickets I already have in progress I think, so no new ticket needed there.
            Hide
            danielw Daniel Wang [X] (Inactive) added a comment -

            Tickets created: DM-2059 DM-2060 DM-2061
            Closing.

            Show
            danielw Daniel Wang [X] (Inactive) added a comment - Tickets created: DM-2059 DM-2060 DM-2061 Closing.

              People

              Assignee:
              fritzm Fritz Mueller
              Reporter:
              fritzm Fritz Mueller
              Reviewers:
              Jacek Becla
              Watchers:
              Daniel Wang [X] (Inactive), Fritz Mueller, Jacek Becla
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.