Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2419

Implement authentication mechanism for worker management service

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      We need some reasonable security for access to new worker management service. It should be lightweight and not depend on complex things that require infrastructure. Something based on a shared secret should be adequate for our immediate needs and likely for the long term.

        Attachments

          Issue Links

            Activity

            Hide
            salnikov Andy Salnikov added a comment -

            I think we have couple of options for secure authentication (based on shared secret):

            • HTTPS with Basic auth
            • HTTP with Digest auth
            • can also do HTTPS+Digest but his is an overkill, if we have HTTPS then Basic is enough and it is simpler than Digest

            Flask has good-enough built-in support for Basic auth: http://flask.pocoo.org/snippets/8/ (just looking at the request.authorization data which is pre-filled by Flask). Digest auth is a bit more complex and there is no simple built-in implementation in Flask (it only provides access to auth headers). We could use one of the ready extensions though: https://github.com/miguelgrinberg/Flask-HTTPAuth or http://flask.pocoo.org/snippets/31/. This needs an update of the Flask or werkzeug installation (or maybe just implementing something similar in our new lsst/webcommon repo.

            Show
            salnikov Andy Salnikov added a comment - I think we have couple of options for secure authentication (based on shared secret): HTTPS with Basic auth HTTP with Digest auth can also do HTTPS+Digest but his is an overkill, if we have HTTPS then Basic is enough and it is simpler than Digest Flask has good-enough built-in support for Basic auth: http://flask.pocoo.org/snippets/8/ (just looking at the request.authorization data which is pre-filled by Flask). Digest auth is a bit more complex and there is no simple built-in implementation in Flask (it only provides access to auth headers). We could use one of the ready extensions though: https://github.com/miguelgrinberg/Flask-HTTPAuth or http://flask.pocoo.org/snippets/31/ . This needs an update of the Flask or werkzeug installation (or maybe just implementing something similar in our new lsst/webcommon repo.
            Hide
            salnikov Andy Salnikov added a comment -

            Safe storage for secret

            If we are going to authenticate using shared secret then this secret needs to be stored somewhere and be accessible to both client and server. Given the nature of our installation, the only reasonable thing that I can imagine now is the file on a file system in a pre-defined location. Location should be specific to the service (release), which means that it's probably in the qserv run directory, etc/ is probably most reasonable location. the file of course has to be protected from anyone except the account which runs worker manager service and clients. If they are one and the same account then we don't need any fancy ACLs support for that. The secret will be stored unencrypted (but it could be somewhat obfuscated); on server side secret could be encrypted though, but there is no point in doing that as long as we store password for client, so to simplify things we should store just one copy of a secret and it needs to be unecrypted.

            The password needs to be known to client and server only, client means also non-humans, like scripts and services. It does not need to be well-known and it can be different on every installation. We can generate it as a part of installation (qserv-configure).

            Show
            salnikov Andy Salnikov added a comment - Safe storage for secret If we are going to authenticate using shared secret then this secret needs to be stored somewhere and be accessible to both client and server. Given the nature of our installation, the only reasonable thing that I can imagine now is the file on a file system in a pre-defined location. Location should be specific to the service (release), which means that it's probably in the qserv run directory, etc/ is probably most reasonable location. the file of course has to be protected from anyone except the account which runs worker manager service and clients. If they are one and the same account then we don't need any fancy ACLs support for that. The secret will be stored unencrypted (but it could be somewhat obfuscated); on server side secret could be encrypted though, but there is no point in doing that as long as we store password for client, so to simplify things we should store just one copy of a secret and it needs to be unecrypted. The password needs to be known to client and server only, client means also non-humans, like scripts and services. It does not need to be well-known and it can be different on every installation. We can generate it as a part of installation (qserv-configure).
            Hide
            bvan Brian Van Klaveren added a comment -

            From a security and a developer standpoint, I think HTTPS+Basic Auth or HTTPS+Application Tokens (or cookies for users logged in via a browser) is easiest, but that of course requires the unofficial route where either all clients are updated to trust the webserver or we go the official route and get an SSL Certificate, which does limit you on some flexibility. For other projects I'm working on, we are planning on using nginx as a load balancer and SSL/TLS endpoint and going the official route.

            If we don't want to bother with SSL, I've implemented HMAC-based auth in clients before. It's very straightforward. I think I should be able to do it with Flask just fine as well (using that snippet code). It's similar to Digest Auth.

            The basic implementation requires a shared key, a key id (which can be either a UUID or a username/user id). The user performs a digest of request attributes using the secret key, adds that digest and the key id to the request, and the server looks up the private key using the key id (this can be memoized/cached for a while). Every request is sent with a date, and the server always drops requests older than a few seconds. In one implementation I've written, I've allowed multiple UUID/key pairs to be assigned to a user, with each key having it's own expiration date. This slightly improves security, but you still need a way to communicate the keys securely (usually an application which uses HTTPS).

            See Amazon's page on HMAC based auth.
            See datacat documentation on authentication.

            Finally, you can always just use iptables on the server if you trust the host, or have an equivalent application-level version of iptables as well. This is definitely the easy way forward for good-enough security if we want to defer making a decision for a bit, assuming the machines accessing the worker management service aren't open to just anyone.

            For password storage, storing the secret somewhere under /etc, with proper permissions set, is how we usually do it as well.

            Show
            bvan Brian Van Klaveren added a comment - From a security and a developer standpoint, I think HTTPS+Basic Auth or HTTPS+Application Tokens (or cookies for users logged in via a browser) is easiest, but that of course requires the unofficial route where either all clients are updated to trust the webserver or we go the official route and get an SSL Certificate, which does limit you on some flexibility. For other projects I'm working on, we are planning on using nginx as a load balancer and SSL/TLS endpoint and going the official route. If we don't want to bother with SSL, I've implemented HMAC-based auth in clients before. It's very straightforward. I think I should be able to do it with Flask just fine as well (using that snippet code). It's similar to Digest Auth. The basic implementation requires a shared key, a key id (which can be either a UUID or a username/user id). The user performs a digest of request attributes using the secret key, adds that digest and the key id to the request, and the server looks up the private key using the key id (this can be memoized/cached for a while). Every request is sent with a date, and the server always drops requests older than a few seconds. In one implementation I've written, I've allowed multiple UUID/key pairs to be assigned to a user, with each key having it's own expiration date. This slightly improves security, but you still need a way to communicate the keys securely (usually an application which uses HTTPS). See Amazon's page on HMAC based auth . See datacat documentation on authentication . Finally, you can always just use iptables on the server if you trust the host, or have an equivalent application-level version of iptables as well. This is definitely the easy way forward for good-enough security if we want to defer making a decision for a bit, assuming the machines accessing the worker management service aren't open to just anyone. For password storage, storing the secret somewhere under /etc, with proper permissions set, is how we usually do it as well.
            Hide
            salnikov Andy Salnikov added a comment -

            Brian, thanks for comment.

            Regarding SSL - I believe that we are required to keep everything private and that I think implies encryption for all traffic (question for K-T - do we also have to encrypt xrootd traffic and client-qserv data exchange via mysql protocol? If we don't then I do not really see a point why do we have to encrypt worker manager traffic). So in the end we are supposed to have SSL, and this of course needs some decision regarding use of either self-signed certificates or official ones. But this is discussion for separate ticket.

            Regarding use of different auth schemes - I'm not an expert in this, could you explain what is the benefit of HMAC-based auth vs. standard Digest auth. I'm a bit worried about timestamp-based request invalidation, in the not-too-well-controlled environment this could result in things suddenly falling apart.

            Show
            salnikov Andy Salnikov added a comment - Brian, thanks for comment. Regarding SSL - I believe that we are required to keep everything private and that I think implies encryption for all traffic (question for K-T - do we also have to encrypt xrootd traffic and client-qserv data exchange via mysql protocol? If we don't then I do not really see a point why do we have to encrypt worker manager traffic). So in the end we are supposed to have SSL, and this of course needs some decision regarding use of either self-signed certificates or official ones. But this is discussion for separate ticket. Regarding use of different auth schemes - I'm not an expert in this, could you explain what is the benefit of HMAC-based auth vs. standard Digest auth. I'm a bit worried about timestamp-based request invalidation, in the not-too-well-controlled environment this could result in things suddenly falling apart.
            Hide
            salnikov Andy Salnikov added a comment -

            We still have some unanswered questions (see me previous comment) but can discuss that later and figure out if we need to do anything differently. Meanwhile I have implemented authentication mechanism which supports both basic and digest authentication (selectable via configuration). Brian, could you review PR when you have time for it?

            Show
            salnikov Andy Salnikov added a comment - We still have some unanswered questions (see me previous comment) but can discuss that later and figure out if we need to do anything differently. Meanwhile I have implemented authentication mechanism which supports both basic and digest authentication (selectable via configuration). Brian, could you review PR when you have time for it?
            Hide
            salnikov Andy Salnikov added a comment -

            Brian, I have updated the code which reads secret to do some validation and squashed/rebased it to current master.
            If you finished reviewing you can click Workflow -> Review Complete

            Show
            salnikov Andy Salnikov added a comment - Brian, I have updated the code which reads secret to do some validation and squashed/rebased it to current master. If you finished reviewing you can click Workflow -> Review Complete
            Hide
            salnikov Andy Salnikov added a comment -

            I'm marking this as review complete.

            Show
            salnikov Andy Salnikov added a comment - I'm marking this as review complete.

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              salnikov Andy Salnikov
              Reviewers:
              Brian Van Klaveren
              Watchers:
              Andy Salnikov, Brian Van Klaveren, Kian-Tat Lim
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.