Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-723

Changes to LDF dev resources

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: LSST
    • Labels:
      None

      Description

      In the interest of getting the word out to everyone, a RFC was suggested to help with that.  
       

      SSH Access Migrating To New Login Nodes

      Starting Oct 1, all external SSH access into LDF services will require going through the new lsst-login nodes. These nodes are available for use now. This change centralizes NCSA DUO two-factor authentication (2FA) and minimizes the number of jump hosts necessary for accessing the LDF resources. These new login nodes replace the need to use NCSA cerberus or lsst-bastion jump hosts to access LDF environments.

      New devl Nodes

      The login nodes are lightweight servers intended primarily for access to other systems and to centralize the DUO 2FA. Other work, especially intensive work requiring high CPU/memory usage, long running jobs, storage IO, etc. should be performed on new devl nodes. These new devl nodes essentially replace the current dev nodes – they can be used for software development, new batch job submission, and longer running interactive work. The following new devl nodes will be available as follows:

      • lsst-devl01.ncsa.illinois.edu (Dell Intel - available now)
      • lsst-devl02.ncsa.illinois.edu (Dell Intel - available by Sep 14)
      • lsst-devl03.ncsa.illinois.edu (Dell AMD Rome - available by Sep 14)

      Batch Submission Changes

      • Users are encouraged to submit batch jobs for work that requires significant resources (e.g. some historical data processing that happened on lsst-dev).
      • Documentation for the updated batch resources at NCSA is available here: https://developer.lsst.io/services/batch.html
      • HTCondor: The HTCondor DAC pool (available now) is available for general use. Users working on "formal data products" can also use the HTCondor Prod pool.
      • Slurm: A new Slurm cluster has been built (also available now) and compute nodes will be slowly migrated to it throughout August and September. Jobs can be submitted to the new Slurm cluster from any of the new devl or condor submit nodes.

      Services Shutting Down On Oct 1

      • lsst-dev nodes will be unavailable after Oct 1 according to the schedule below. Developers should instead use login or devl nodes. (dev03 will be turned off Sept 21, dev02 turned off Sept 28, dev01 turned off Oct 1.)
      • lsst-dev-db MySQL/MariaDB server will be turned off on Oct 1.
      • lsst-bastion01 server will be turned off on Oct 1. Use the lsst-login nodes instead.
      • Legacy Slurm “verification cluster” will be turned off on Oct 1. It is being refreshed as the new general-use Slurm Cluster.

      Issues

      To report any issues, log into LSST JIRA and file a JIRA ticket in the IT Helpdesk Support (IHS) project tagging NCSA as the responsible organization.|

        Attachments

          Issue Links

            Activity

            Hide
            rhl Robert Lupton added a comment -

            Oh, and "Your control master connection will persist in the background after your initial client connection terminates, according to the value of ControlPersist". If that means that shutting down a laptop loses the connection that's not good enough – it means that the lifetime is less than the minimal acceptable one week.

            Show
            rhl Robert Lupton added a comment - Oh, and "Your control master connection will persist in the background after your initial client connection terminates, according to the value of ControlPersist". If that means that shutting down a laptop loses the connection that's not good enough – it means that the lifetime is less than the minimal acceptable one week.
            Hide
            cclausen Christopher Clausen [X] (Inactive) added a comment -

            I just checked and more than half of the files in user .ssh directories named id_rsa (default name for an RSA private key) are not encrypted so this is not just a theory.  And this isn't even looking at private keys kept locally which I suspect have an even lower chance of being encrypted.  The reality is that private keys are not stored properly by all users and attackers are able to use this fact to steal credentials.  (I am actually surprised by the number of keys that do seem to be encrypted.  I would guess this rate to be higher than for other NCSA resources, although many resources are also moving to requiring 2FA so use of keys may be more limited in the future.)

             

            Shutting down a laptop means there is no state on the server side to track any longer and a new authentication is needed to reestablish identity.  One would need to have something stored on the laptop which is what we are attempting to prevent by asking for a password and maintaining the "something you know" over reverting to "something you have."

             

            I am curious - how does your setup work with SSH key passphrases?  Doesn't one need to re-type the passphrase at least once each time one logs in locally?  I am assuming there is nothing in place to cache SSH key passphrases across system shutdowns but perhaps I am wrong about that.

             

            If one really wanted to work around the spirit of the 2FA requirement, one could simply create a Kerberos keytab file stored locally on their system and kinit using that file and then use GSSAPI credential forwarding without needing to type a password.  There isn't a way to block or differentiate this on the SSH server side nor on the Kerberos KDC. See the ktutil man page for more info on creating keytabs.

            Show
            cclausen Christopher Clausen [X] (Inactive) added a comment - I just checked and more than half of the files in user .ssh directories named id_rsa (default name for an RSA private key) are not encrypted so this is not just a theory.  And this isn't even looking at private keys kept locally which I suspect have an even lower chance of being encrypted.  The reality is that private keys are not stored properly by all users and attackers are able to use this fact to steal credentials.  (I am actually surprised by the number of keys that do seem to be encrypted.  I would guess this rate to be higher than for other NCSA resources, although many resources are also moving to requiring 2FA so use of keys may be more limited in the future.)   Shutting down a laptop means there is no state on the server side to track any longer and a new authentication is needed to reestablish identity.  One would need to have something stored on the laptop which is what we are attempting to prevent by asking for a password and maintaining the "something you know" over reverting to "something you have."   I am curious - how does your setup work with SSH key passphrases?  Doesn't one need to re-type the passphrase at least once each time one logs in locally?  I am assuming there is nothing in place to cache SSH key passphrases across system shutdowns but perhaps I am wrong about that.   If one really wanted to work around the spirit of the 2FA requirement, one could simply create a Kerberos keytab file stored locally on their system and kinit using that file and then use GSSAPI credential forwarding without needing to type a password.  There isn't a way to block or differentiate this on the SSH server side nor on the Kerberos KDC. See the ktutil man page for more info on creating keytabs.
            Hide
            rra Russ Allbery added a comment -

            SSH public key plus Duo (with a very similar albeit somewhat simpler PAM hack) is the same approach I've used elsewhere for access to internal systems.

            I agree with the theoretical concern that this isn't a true second factor since the SSH key and the Duo authenticator are both things you have, but I would not weigh the classes of factor analysis too heavily. It's intended as shorthand to try to guide a security policy towards defending against more threats. In practice, Duo push plus an SSH public key requires that an attacker control two separate devices to directly bypass the authentication step, one of which (the phone for Duo push) probably requires physical theft, which is a high bar. There is some theoretical benefit of protection against stolen devices in scenarios where one believes a laptop and an unlocked phone may be stolen together, but in terms of risk modeling, I think it's around the edges.

            Under either setup, the most likely attack is attacker compromise of the laptop followed by piggybacking on any open connection or available credentials, which works nearly as well with both password and SSH public key. ControlMaster makes this attack easier to perform and removes any marginal benefit of password over SSH public key against that attacker, so if you're willing to accept the risk of ControlMaster, I'm dubious accepting SSH public keys instead of passwords adds much additional risk.

            Password plus Duo is slightly more secure (in that it protects against a few more attack paths) than SSH public key plus Duo, but I think it's a fairly minor benefit for the usability loss. The good news is that we have the luxury of knowing that the only attractive attack target for most attackers is abuse of resources; the data isn't very sensitive or of much attacker interest, so we're unlikely to attract a sophisticated attacker who would be willing to do things like compromise phones or use targeted malware.

            Show
            rra Russ Allbery added a comment - SSH public key plus Duo (with a very similar albeit somewhat simpler PAM hack) is the same approach I've used elsewhere for access to internal systems. I agree with the theoretical concern that this isn't a true second factor since the SSH key and the Duo authenticator are both things you have, but I would not weigh the classes of factor analysis too heavily. It's intended as shorthand to try to guide a security policy towards defending against more threats. In practice, Duo push plus an SSH public key requires that an attacker control two separate devices to directly bypass the authentication step, one of which (the phone for Duo push) probably requires physical theft, which is a high bar. There is some theoretical benefit of protection against stolen devices in scenarios where one believes a laptop and an unlocked phone may be stolen together, but in terms of risk modeling, I think it's around the edges. Under either setup, the most likely attack is attacker compromise of the laptop followed by piggybacking on any open connection or available credentials, which works nearly as well with both password and SSH public key. ControlMaster makes this attack easier to perform and removes any marginal benefit of password over SSH public key against that attacker, so if you're willing to accept the risk of ControlMaster , I'm dubious accepting SSH public keys instead of passwords adds much additional risk. Password plus Duo is slightly more secure (in that it protects against a few more attack paths) than SSH public key plus Duo, but I think it's a fairly minor benefit for the usability loss. The good news is that we have the luxury of knowing that the only attractive attack target for most attackers is abuse of resources; the data isn't very sensitive or of much attacker interest, so we're unlikely to attract a sophisticated attacker who would be willing to do things like compromise phones or use targeted malware.
            Hide
            rhl Robert Lupton added a comment -

            With ssh you authenticate once after you reboot your laptop, which is every month or two. I'm talking about shutting the lid (or going off network for a few hours on a flight down to Chile in the old days).

            So ssh + duo would be OK, providing it remains authorised for long enough. I suppose we could live with that once per day, but I would very much prefer to leave things as they are. We're trying to get work done, and ssh with a passphrase IMHO provides a suitable ratio of risk/security.

            If NCSA wants to support the same level of functionality in some other way, via e.g. kerberos, such that users, including users who only use these resources relatively seldom, that's OK of course.

            Show
            rhl Robert Lupton added a comment - With ssh you authenticate once after you reboot your laptop, which is every month or two. I'm talking about shutting the lid (or going off network for a few hours on a flight down to Chile in the old days). So ssh + duo would be OK, providing it remains authorised for long enough. I suppose we could live with that once per day, but I would very much prefer to leave things as they are. We're trying to get work done, and ssh with a passphrase IMHO provides a suitable ratio of risk/security. If NCSA wants to support the same level of functionality in some other way, via e.g. kerberos, such that users, including users who only use these resources relatively seldom, that's OK of course.
            Hide
            mbutler Michelle Butler [X] (Inactive) added a comment -

            posted and finished.  

            Show
            mbutler Michelle Butler [X] (Inactive) added a comment - posted and finished.  

              People

              Assignee:
              mbutler Michelle Butler [X] (Inactive)
              Reporter:
              mbutler Michelle Butler [X] (Inactive)
              Watchers:
              Christopher Clausen [X] (Inactive), Colin Slater, Ian Sullivan, Jacob Rundall, John Swinbank, Michelle Butler [X] (Inactive), Paul Price, Robert Lupton, Russ Allbery, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.