Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2176

Worker management service - impl

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      We need to replace direct worker-mysql communication and other administrative channels with a special service which will control all worker communication. Some light-weight service running alongside other worker servers, probably HTTP-based. Data loading, start/stop should be handled by this service.

        Attachments

          Issue Links

            Activity

            Hide
            jbecla Jacek Becla added a comment -

            I think having a quick readme, or some example would be useful.

            Show
            jbecla Jacek Becla added a comment - I think having a quick readme, or some example would be useful.
            Hide
            jbecla Jacek Becla added a comment - - edited

            Looks good, some minor comments in PR and in comments here. Thanks

            Show
            jbecla Jacek Becla added a comment - - edited Looks good, some minor comments in PR and in comments here. Thanks
            Hide
            salnikov Andy Salnikov added a comment -

            Thanks Jacek, few answers:

            Re _specialDbs:

            • yes, my assumption is that everything in worker database "belongs" to worker in a sense that we can do whatever we want with the contents that we are allowed to see/modify. I was not aware there is an attempt to separate different workers using the same mysql server. This could potentially be done of course if needed either with prefix or simply using separate accounts with corresponding permissions.

            Re createDb vs registerDb:

            • agree, I'll update names to make them clearer. My current naming reflects REST resources and operations that we do on them, but I can change it.

            Re deleteDb vs dbDelete:

            • this is again influenced mostly by REST resource+operation scheme but it does not have to be that way. I can change that as well.

            I'll add README.md. Full documentation is in a separate ticket.

            Show
            salnikov Andy Salnikov added a comment - Thanks Jacek, few answers: Re _specialDbs: yes, my assumption is that everything in worker database "belongs" to worker in a sense that we can do whatever we want with the contents that we are allowed to see/modify. I was not aware there is an attempt to separate different workers using the same mysql server. This could potentially be done of course if needed either with prefix or simply using separate accounts with corresponding permissions. Re createDb vs registerDb: agree, I'll update names to make them clearer. My current naming reflects REST resources and operations that we do on them, but I can change it. Re deleteDb vs dbDelete: this is again influenced mostly by REST resource+operation scheme but it does not have to be that way. I can change that as well. I'll add README.md. Full documentation is in a separate ticket.
            Hide
            salnikov Andy Salnikov added a comment - - edited

            you are using pretty wide screen, some like eg in dbMgr.py are quite long. I'd suggest to keep things shorter

            Everything is below 110 columns, which is the upper limit that our coding standard sets. Typically long lines are those which contain message text (like in logging or exceptions). I find them harder to read if they are split into multiple short lines. But if standard would say that we should try hard to keep everything below 80 characters for readability reasons I'd be happy to oblige

            Show
            salnikov Andy Salnikov added a comment - - edited you are using pretty wide screen, some like eg in dbMgr.py are quite long. I'd suggest to keep things shorter Everything is below 110 columns, which is the upper limit that our coding standard sets. Typically long lines are those which contain message text (like in logging or exceptions). I find them harder to read if they are split into multiple short lines. But if standard would say that we should try hard to keep everything below 80 characters for readability reasons I'd be happy to oblige
            Hide
            salnikov Andy Salnikov added a comment -

            I added a bunch of small fixes to resolve Fabrice's and Jacek's comments. Merged and pushed. New service is started now if you run qserv-start.sh (even in mono mode), it's not used yet, but you can play with it if you like.

            Show
            salnikov Andy Salnikov added a comment - I added a bunch of small fixes to resolve Fabrice's and Jacek's comments. Merged and pushed. New service is started now if you run qserv-start.sh (even in mono mode), it's not used yet, but you can play with it if you like.

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              salnikov Andy Salnikov
              Reviewers:
              Jacek Becla
              Watchers:
              Andy Salnikov, Fabrice Jammes, Jacek Becla
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.