Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4348

Replace XML-RPC with in-process communication

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:

      Description

      With all recent development in CSS sector we should be able now to get rid of Python in czar entirely. This is also a good opportunity to move from XML-RPC between proxy and czar with direct in-process "communication". Daniel said it's a good idea

        Attachments

          Issue Links

            Activity

            Hide
            jammes Fabrice Jammes added a comment -

            Hi Andy Salnikov, I'm quite impressed by this ticket. Will you wrap czar inside a lua API? If yes, I've understood mysql-proxy was mono-threaded, so how would you workaround this? The lua script would exit, but a C++/czar thread would wait for the result and unlock the result table?

            Show
            jammes Fabrice Jammes added a comment - Hi Andy Salnikov , I'm quite impressed by this ticket. Will you wrap czar inside a lua API? If yes, I've understood mysql-proxy was mono-threaded, so how would you workaround this? The lua script would exit, but a C++/czar thread would wait for the result and unlock the result table?
            Hide
            salnikov Andy Salnikov added a comment -

            Hi Fabrice Jammes, all multi-threading will happen on C++ side, not in lua. Lua will work just as before, only instead of using XML-RPC to submit the query it will call czar directly (via Lua/C in-process wrapper). Czar will return the same info to Lua ~immediately and will spawn a separate thread to process the query.

            Show
            salnikov Andy Salnikov added a comment - Hi Fabrice Jammes , all multi-threading will happen on C++ side, not in lua. Lua will work just as before, only instead of using XML-RPC to submit the query it will call czar directly (via Lua/C in-process wrapper). Czar will return the same info to Lua ~immediately and will spawn a separate thread to process the query.
            Hide
            salnikov Andy Salnikov added a comment - - edited

            OK, this is rather big bunch of commits so I think I need couple of reviewers (at least). I propose to split this approximately:

            • Fabrice: all admin/ stuff and qserv_testdata
            • Fritz: everything else

            I think it's not too terrible, there should more stuff deleted (all czar python stuff) then added. Still there were a lot of changes everywhere so there are likely some new bugs (or unexpected new features) in there, hope you'll discover some of them

            Show
            salnikov Andy Salnikov added a comment - - edited OK, this is rather big bunch of commits so I think I need couple of reviewers (at least). I propose to split this approximately: Fabrice: all admin/ stuff and qserv_testdata Fritz: everything else I think it's not too terrible, there should more stuff deleted (all czar python stuff) then added. Still there were a lot of changes everywhere so there are likely some new bugs (or unexpected new features) in there, hope you'll discover some of them
            Hide
            fritzm Fritz Mueller added a comment -

            Okay – reviewed – I think this is great, thanks! John Gates expressed an interest in reviewing as well, so adding him here. John, since Fabrice and I have already taken a look, would you drag to "reviewed" after you have had a look?

            Show
            fritzm Fritz Mueller added a comment - Okay – reviewed – I think this is great, thanks! John Gates expressed an interest in reviewing as well, so adding him here. John, since Fabrice and I have already taken a look, would you drag to "reviewed" after you have had a look?
            Hide
            jammes Fabrice Jammes added a comment -

            Review ok for me

            Show
            jammes Fabrice Jammes added a comment - Review ok for me
            Hide
            fritzm Fritz Mueller added a comment -

            Andy Salnikov – took a look at the recent changes, and they look good to me – thanks!

            Show
            fritzm Fritz Mueller added a comment - Andy Salnikov – took a look at the recent changes, and they look good to me – thanks!
            Hide
            jgates John Gates added a comment -

            Looks fine to me. Good to see this change being made.

            Show
            jgates John Gates added a comment - Looks fine to me. Good to see this change being made.
            Hide
            salnikov Andy Salnikov added a comment -

            Thanks all for review. Fixed all issues, squashed and merged. Even ran through Jenkins Done.

            Show
            salnikov Andy Salnikov added a comment - Thanks all for review. Fixed all issues, squashed and merged. Even ran through Jenkins Done.

              People

              • Assignee:
                salnikov Andy Salnikov
                Reporter:
                salnikov Andy Salnikov
                Reviewers:
                Fritz Mueller, John Gates
                Watchers:
                Andy Salnikov, Brian Van Klaveren, Fabrice Jammes, Fritz Mueller, John Gates
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: