Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-864

XXX Replace mysql-proxy with czar internal handling of mysql protocol

    Details

    • Type: Epic
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
    • Epic Name:
      mysql-proxy replacement
    • Team:
      Data Access and Database

      Description

      Currently mysql connection from clients is handled by mysql-proxy which passes queries to czar and returns query results back to user. This proxy-czar combination has some significant issues which limit what we can do with it:

      • very limited possibility to generate data on proxy side (it has to result from SQL query of some sort)
      • proxy has very little information about result data and cannot do transformations on that

      It looks like we can achieve better result if we can implement our own proxy which can work at the mysql wire-level protocol and integrate that proxy directly into czar. This would eliminate one server process from our current setup and should help both performance and stability.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            Please do not implement the MySQL protocol or even try to feed it directly. Look for other proxies; there are a number out there, including some that may be more customizable than mysql-proxy.

            Show
            ktl Kian-Tat Lim added a comment - Please do not implement the MySQL protocol or even try to feed it directly. Look for other proxies; there are a number out there, including some that may be more customizable than mysql-proxy.
            Hide
            salnikov Andy Salnikov added a comment -

            Hi K-T,

            you are absolutely right that there is more than one mysql proxy out there (I personally wrote more than one). My impression is that most are focused on high performance and availability and are not very big on customization. I did some research and mysql-proxy actually looks like the most customizable one and even with that we are struggling to do what we would like to do. I'm adding Jacek and Daniel, maybe they can add more historical insight on mysql-proxy .

            Anyways, we have something that is kind of working now, there is a wish to improve that but it's not clear if we have time/resources to do it. I opened this epic as a reminder in case we actually find it useful to start working on that.

            Show
            salnikov Andy Salnikov added a comment - Hi K-T, you are absolutely right that there is more than one mysql proxy out there (I personally wrote more than one). My impression is that most are focused on high performance and availability and are not very big on customization. I did some research and mysql-proxy actually looks like the most customizable one and even with that we are struggling to do what we would like to do. I'm adding Jacek and Daniel, maybe they can add more historical insight on mysql-proxy . Anyways, we have something that is kind of working now, there is a wish to improve that but it's not clear if we have time/resources to do it. I opened this epic as a reminder in case we actually find it useful to start working on that.
            Hide
            danielw Daniel Wang [X] (Inactive) added a comment -

            There didn't used to be a number of alternative proxy implementations. But it seems that there are two that we might look at: MaxScale and ProxySQL. Both seem lower in maturity than mysqlproxy. In both, the focus is on scalability and load-balancing, and customizability is focused on rewriting queries (I think one of them has a parser). I guess if one of them is C++, we can find a good place to cut-in. Another thing we could do is to let ourselves hack on mysqlproxy at the C level--I don't think there will be any upstream changes to worry about.

            Show
            danielw Daniel Wang [X] (Inactive) added a comment - There didn't used to be a number of alternative proxy implementations. But it seems that there are two that we might look at: MaxScale and ProxySQL. Both seem lower in maturity than mysqlproxy. In both, the focus is on scalability and load-balancing, and customizability is focused on rewriting queries (I think one of them has a parser). I guess if one of them is C++, we can find a good place to cut-in. Another thing we could do is to let ourselves hack on mysqlproxy at the C level--I don't think there will be any upstream changes to worry about.
            Hide
            jbecla Jacek Becla added a comment -

            Related: we talked about replacing xml-rpc proxy-czar protocol with RESTful interface. See https://confluence.lsstcorp.org/display/DM/Qserv+Hangout+2015-02-04

            Show
            jbecla Jacek Becla added a comment - Related: we talked about replacing xml-rpc proxy-czar protocol with RESTful interface. See https://confluence.lsstcorp.org/display/DM/Qserv+Hangout+2015-02-04
            Hide
            jbecla Jacek Becla added a comment -

            Comments related to MaxScale - see DM-2057

            Show
            jbecla Jacek Becla added a comment - Comments related to MaxScale - see DM-2057
            Hide
            swinbank John Swinbank added a comment -

            Fritz Mueller — the DMCCB wonders whether this ticket is still relevant.

            Show
            swinbank John Swinbank added a comment - Fritz Mueller — the DMCCB wonders whether this ticket is still relevant.
            Hide
            salnikov Andy Salnikov added a comment -

            I guess after 6 years we are not going to do anything like it. I think current idea is that if we are ever to replace mysql-proxy we'll get rid of mysql protocol altogether.

            Show
            salnikov Andy Salnikov added a comment - I guess after 6 years we are not going to do anything like it. I think current idea is that if we are ever to replace mysql-proxy we'll get rid of mysql protocol altogether.

              People

              • Assignee:
                salnikov Andy Salnikov
                Reporter:
                salnikov Andy Salnikov
                Watchers:
                Andy Salnikov, Daniel Wang [X] (Inactive), Jacek Becla, John Swinbank, Kian-Tat Lim
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel