I've been asked to mention here that connections to Princeton's clusters uses Duo 2FA first, and then allows me to login with my ssh keys. I can find out the details if you like. Example:

price@price-laptop:~ $ slogin tiger

Duo two-factor login for pprice

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-9024

 2. Phone call to XXX-XXX-9024

 3. SMS passcodes to XXX-XXX-9024 (next code starts with: 4)

Passcode or option (1-3): 1

Last login: Mon Aug 24 15:52:54 2020 from 10.36.16.17

pprice@tiger2-sumire:~ $ 

Kerberos is annoying because not all ssh installations support it: even having the options in your config file cause ssh to crash. That means I can't have a single ssh config file for all my logins.

Strong +1 to what Paul said. Duo is a manageable inconvenience, but it's extra annoying to have to dig my NCSA password out of a password manager and manually copy-paste it into SSH every time.

I asked the Princeton CSES team about setup for Duo+ssh, and Josko Plazonic replied:

I'll start with RHEL7:

Only sshd_config changes with explanations:

# required for duo prompts, and we need PAM both for duo and other pam modules

PasswordAuthentication no

ChallengeResponseAuthentication yes

UsePAM yes

# This is the essential setting that forces keyboard-interactive authentication, i.e. PAM, even if public key succeeded

AuthenticationMethods publickey,keyboard-interactive keyboard-interactive

# Fills SSH_USER_AUTH env variable with info on what authentication succeeded

ExposeAuthenticationMethods pam-only

AuthenticationMethods setting forces that at least one of the space separate settings have to succeed - so either publickey AND keyboard-interactive or only keyboard-interactive. This is what makes ssh go through PAM auth even if ssh key auth worked.

So now PAM auth is forced but by then SSH_USER_AUTH variable is present (because of ExposeAuth... and because publickey succeeded) which we can then use in PAM stack to detect it and act accordingly - details below.

This is auth section from /etc/pam.d/sshd (the rest is the same as usual but adds noise so will drop it here). I am also putting line numbers for easier reference in explanation:

01 auth required pam_sepermit.so

02 auth required pam_env.so

03 auth requisite pam_succeed_if.so shell != /sbin/nologin quiet_success

04 auth [success=2 ignore=ignore default=ignore] pam_succeed_if.so uid >= 1000 quiet_success

05 auth sufficient pam_ssh_user_auth.so

06 auth [success=done default=die] pam_unix.so nullok try_first_pass

07 auth [success=2 new_authtok_reqd=ok ignore=ignore default=die] pam_ssh_user_auth.so

08 auth [success=1 new_authtok_reqd=ok default=ignore] pam_unix.so nullok try_first_pass

09 auth requisite pam_ldap.so try_first_pass

10 auth sufficient pam_openmfa_access.so

11 auth sufficient pam_duo.so conf=/etc/duo/oit.conf

12 auth required pam_deny.so

13 auth include postlogin

14 auth optional pam_reauthorize.so prepare

This is not too easy to read due to ridiculous if/then/goto setup pam modules have but let's start with line 04 - that's there to allow root and service accounts (all uid number < 1000) to login without dual factor being involved. So if uid>=1000 then skip next 2 lines and jump to line 7.

Regular users will proceed from line 7 - pam_ssh_user_auth is a pam module that just checks for existence of SSH_USER_AUTH in pam environment. If it is present (i.e. ssh keys used successfully) it skips to line 10. pam_openmfa_access module allows us to have custom rules according to where the user is logging from. E.g. we use it to skip duo if one is coming from a trusted machine or from VPN. If it succeeds result is sufficient and user is authenticated. It if does not it proceeds to duo on line 11.

If user did not use public keys then line 7 fails and it proceeds to line 8/9/10/....

Regarding supporting modules - pam_ssh_user_auth we got from https://github.com/CERN-CERT/pam_2fa and pam_openmfa comes from TACC https://github.com/TACC/OpenMFA

Openmfa module config file might be interesting as well, these are some of the lines from /etc/security/openmfa_access.conf, e.g. trust anyone coming from VPN to skip duo:

# VPN

+ : ALL : ALL : 128.112.64.96/255.255.224.0

+ : ALL : ALL : 128.112.64.224/255.255.224.0

+ : ALL : ALL : 128.112.65.0/255.255.255.128

+ : ALL : ALL : 128.112.67.0/255.255.255.0

+ : ALL : ALL : 128.112.68.0/255.255.255.0

+ : ALL : ALL : 128.112.69.0/255.255.255.0

+ : ALL : ALL : 128.112.70.0/255.255.255.0

+ : ALL : ALL : 128.112.71.0/255.255.255.0

We have rpms for this and puppet rules though those are probably hard to apply to a different setup.

I can do RHEL8 setup later but it is essentially the same idea with differences in ssh part and therefore PAM.

More from Josko, on RHEL 8:

I'll be briefer now for it is the same story, different actors:

# sshd_config relevant bits

PasswordAuthentication no

ChallengeResponseAuthentication yes

UsePAM yes

AuthenticationMethods publickey,keyboard-interactive keyboard-interactive

Then auth excerpt from /etc/pam.d/sshd, this time with comments already embedded (as I was going nuts figuring this awkward if/then business):

auth required pam_env.so

auth required pam_faildelay.so delay=2000000

# LDAP case

####################################################################################################

# check if user is not local and has uid < 1000, if so ABORT

auth [default=1 ignore=ignore success=ok] pam_succeed_if.so uid < 1000 quiet

auth requisite pam_localuser.so

# only check unix and ssh, no duo, for local accounts with uid < 1000

auth [default=3 ignore=ignore success=ok] pam_localuser.so

auth [default=2 ignore=ignore success=ok] pam_succeed_if.so uid < 1000 quiet

auth sufficient pam_ssh_user_auth.so

auth [success=done new_authtok_reqd=done default=die] pam_unix.so nulloktry_first_pass

# otherwise insist on mfa or duo

auth [success=3 ignore=ignore] pam_ssh_user_auth.so

auth [success=2 new_authtok_reqd=ok default=ignore] pam_unix.so nulloktry_first_pass

auth requisite pam_succeed_if.so uid >= 1000 quiet_success

auth requisite pam_ldap.so forward_pass

auth sufficient pam_openmfa_access.so

auth sufficient pam_duo.so conf=/etc/duo/oit.conf

# END LDAP case

################################################################################################

auth required pam_deny.so

auth include postlogin

Note that we still have here pam_openmfa_access.so, same source but the
piece checking if ssh keys were used is pam_ssh_user_auth.so which still
comes from https://cern-cert.github.io/pam_2fa/ but it's a newer release
or dev or something - I don't remember the details.

If you do not wish to enter a password as often, please setup your SSH client to reuse connections such as through controlmaster and controlpersist (as is mentioned at https://developer.lsst.io/services/lsst-login.html#lsst-login-connect ) or kinit locally first and forward credentials using GSSAPI.  I enter my NCSA password once per week and my Kerberos client (Heimdal and NIM on Windows) renews this ticket for the whole week until the maximum renewable lifetime (7 days.)

If the comments on PAM and sshd_config are trying to prove that it is technically possible to allow publickey AND Duo auth via SSH, we already believe this is true (although this has not actually been tested.)

sshd on lsst-login nodes are currently using:

PubkeyAuthentication no

AuthenticationMethods gssapi-with-mic,keyboard-interactive:pam password,keyboard-interactive:pam 

and changing this to:

PubkeyAuthentication yes

AuthenticationMethods gssapi-with-mic,keyboard-interactive:pam password,keyboard-interactive:pam publickey,keyboard-interactive:pam

should also allow first for a publickey and then go through the PAM stack which is set to prompt for Duo (based on guidance from https://help.duo.com/s/article/5085 as well as configs that the admins of https://www.sherlock.stanford.edu/docs/advanced-topics/connection/ sent us after I found this page and contacted them.)

However, my general concern with allowing publickey authentication is more about two factor authentication and actually keeping it as two different factors.  A password is "something you know" and Duo is "something you have" and in my experience most people store private keys unencrypted (likely for ease of use) and thus that reduces keys to also being "something you have,"  Relying twice on "something you have" is not as strong of an assurance level as "something you know" and then "something you have."  For the same reason, using two different passwords is generally not considered two factor as it is the same factor, twice.

In addition, we don't have a way to know on the server side if a user is properly encrypting their private key on the client side (and thus being asked for passphrase to unlock it when it is used.)  And we also do not know if unencrypted private keys exist on multiple hosts or are in use by multiple users and thus could be more likely to be compromised.  Users could also have permissions set too open on their home directory and .ssh contents might be readable by others on the system and may have a copy of the private key stored unencrypted.  Key management adds another level of complexity to the existing identity management.

(And yes, I realize people can also share or store passwords in plaintext as well, but in my experience it is much more common to see unencrypted or shared SSH private keys.)

Also, some functions (such as HTCondor) require Kerberos tickets to be present either on the login nodes or on other hosts behind them. Requiring either a password (which can be used to obtain a Kerberos ticket by the ssh daemon) or forwarded Kerberos ticket for initial login generally ensures that Kerberos tickets are present after login and available for use for other services that need them.  Users logging in via SSH key would need to kinit to obtain a Kerberos ticket and thus type in their password anyway.

If the section about not requiring Duo for the VPN range is a request for such an allowance on lsst-login nodes, that is a reasonable request and something that we can look into.  It does indeed seem redundant to require Duo to connect to the VPN and then require Duo for SSH as well.

The request is for users to be able to ssh to NCSA machines without typing a password. I think that this is entirely reasonable – if you force people to use passwords + duo they will end up using weak passwords. I understand the concern about ssh keys without a passphrase, but it's only a hypothetical problem and if you want to make your users certify that they have one set that should be sufficient.

Transparent ssh access to the login nodes is used for things like remote editing, so it is important. Once a week is marginally acceptable if you make it sufficiently easy for users. The cost of making it hard for some of the Rubin team to login is significant. Just allowing duo from within the VPN isn't good enough, as the VPN is already a significant obstacle, especially when working remotely.

Oh, and "Your control master connection will persist in the background after your initial client connection terminates, according to the value of ControlPersist". If that means that shutting down a laptop loses the connection that's not good enough – it means that the lifetime is less than the minimal acceptable one week.

I just checked and more than half of the files in user .ssh directories named id_rsa (default name for an RSA private key) are not encrypted so this is not just a theory.  And this isn't even looking at private keys kept locally which I suspect have an even lower chance of being encrypted.  The reality is that private keys are not stored properly by all users and attackers are able to use this fact to steal credentials.  (I am actually surprised by the number of keys that do seem to be encrypted.  I would guess this rate to be higher than for other NCSA resources, although many resources are also moving to requiring 2FA so use of keys may be more limited in the future.)

Shutting down a laptop means there is no state on the server side to track any longer and a new authentication is needed to reestablish identity.  One would need to have something stored on the laptop which is what we are attempting to prevent by asking for a password and maintaining the "something you know" over reverting to "something you have."

I am curious - how does your setup work with SSH key passphrases?  Doesn't one need to re-type the passphrase at least once each time one logs in locally?  I am assuming there is nothing in place to cache SSH key passphrases across system shutdowns but perhaps I am wrong about that.

If one really wanted to work around the spirit of the 2FA requirement, one could simply create a Kerberos keytab file stored locally on their system and kinit using that file and then use GSSAPI credential forwarding without needing to type a password.  There isn't a way to block or differentiate this on the SSH server side nor on the Kerberos KDC. See the ktutil man page for more info on creating keytabs.

SSH public key plus Duo (with a very similar albeit somewhat simpler PAM hack) is the same approach I've used elsewhere for access to internal systems.

I agree with the theoretical concern that this isn't a true second factor since the SSH key and the Duo authenticator are both things you have, but I would not weigh the classes of factor analysis too heavily. It's intended as shorthand to try to guide a security policy towards defending against more threats. In practice, Duo push plus an SSH public key requires that an attacker control two separate devices to directly bypass the authentication step, one of which (the phone for Duo push) probably requires physical theft, which is a high bar. There is some theoretical benefit of protection against stolen devices in scenarios where one believes a laptop and an unlocked phone may be stolen together, but in terms of risk modeling, I think it's around the edges.

Under either setup, the most likely attack is attacker compromise of the laptop followed by piggybacking on any open connection or available credentials, which works nearly as well with both password and SSH public key. ControlMaster makes this attack easier to perform and removes any marginal benefit of password over SSH public key against that attacker, so if you're willing to accept the risk of ControlMaster, I'm dubious accepting SSH public keys instead of passwords adds much additional risk.

Password plus Duo is slightly more secure (in that it protects against a few more attack paths) than SSH public key plus Duo, but I think it's a fairly minor benefit for the usability loss. The good news is that we have the luxury of knowing that the only attractive attack target for most attackers is abuse of resources; the data isn't very sensitive or of much attacker interest, so we're unlikely to attract a sophisticated attacker who would be willing to do things like compromise phones or use targeted malware.

With ssh you authenticate once after you reboot your laptop, which is every month or two. I'm talking about shutting the lid (or going off network for a few hours on a flight down to Chile in the old days).

So ssh + duo would be OK, providing it remains authorised for long enough. I suppose we could live with that once per day, but I would very much prefer to leave things as they are. We're trying to get work done, and ssh with a passphrase IMHO provides a suitable ratio of risk/security.

If NCSA wants to support the same level of functionality in some other way, via e.g. kerberos, such that users, including users who only use these resources relatively seldom, that's OK of course.

posted and finished.