ssh-agent and explicit key authentication puzzles

Note: The SSH pubkey and key file names, hosts, and users you see below have been changed to protect the innocent.

I noticed something odd last night. I was testing out a newish SSH key I’d created for some infrastructure deployment within my personal network. I was attempting to test the key against a host to make sure it worked correctly.

I’d already deployed the public key to the host and, in fact, had run a few successful deployments using the key. I was trying to use that same key from my MacBook Pro to login to the machine now. However, I found that despite specifying the key explicitly with the -i switch, it failed:

~ ❯❯❯ ssh -i ~/.ssh/id_ed25519_test user@host.org
Received disconnect from 1.2.3.4 port 22:2: Too many authentication failures
Disconnected from 1.2.3.4 port 22

Down the rabbit hole…

That’s weird - I know the key works. I double checked the pubkey in the ~/.ssh directory with the pubkey I had for the user on the host (I logged in via different administrator user). Just to be extra sure, I also extracted the pubkey for the private key I had and double checked that as well:

~ ❯❯❯ ssh-keygen -y -f ~/.ssh/id_ed25519_test
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMO2a0E4awq8Flr33I+epxrwqzWSmVhXOJf/JcZPi+6u test
~ ❯❯❯ cat ~/.ssh/id_ed25519_test.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMO2a0E4awq8Flr33I+epxrwqzWSmVhXOJf/JcZPi+6u test

Yep - it definitely matches.

Guess we’d better run the ssh command again but with verbose output. I only show output from ssh with the -v switch. Using -vvvv was noisier but wasn’t any more enlightening:

~ ❯❯❯ ssh -v -i ~/.ssh/id_ed25519_test -p 22 user@host.org
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/zbrown/.ssh/config
debug1: /Users/zbrown/.ssh/config line 34: Applying options for host.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to host.org port 22.
debug1: Connection established.
debug1: identity file /Users/zbrown/.ssh/id_ed25519_test type 3
debug1: identity file /Users/zbrown/.ssh/id_ed25519_test-cert type -1
<TRUNCATED>
debug1: Will attempt key: zac@host.org home ED25519 SHA256:<REDACTED> agent
debug1: Will attempt key: /Users/zbrown/.ssh/id_rsa RSA SHA256:<REDACTED> agent
debug1: Will attempt key: zac@other ED25519 SHA256:<REDACTED> agent
debug1: Will attempt key: /Users/zbrown/.ssh/id_ed25519_test ED25519 SHA256:lHvQ9ojyEUDawn7g/S/EApZdJt/rQYzKtLG/Yv6PNsY explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: zac@host.org home ED25519 SHA256:<REDACTED> agent
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key: /Users/zbrown/.ssh/id_rsa RSA SHA256:<REDACTED> agent
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Offering public key: zac@other ED25519 SHA256:<REDACTED> agent
Received disconnect from 1.2.3.4 port 22:2: Too many authentication failures
Disconnected from 1.2.3.4 port 22

Huh. That’s weird. In the middle of that debug output, there’s a line that says:

debug1: Will attempt key: zac@other ED25519 SHA256:<REDACTED> agent
debug1: Will attempt key: /Users/zbrown/.ssh/id_ed25519_test ED25519 SHA256:lHvQ9ojyEUDawn7g/S/EApZdJt/rQYzKtLG/Yv6PNsY explicit

So that’s the key I was trying to test. But then at the end, we got:

debug1: Offering public key: zac@other ED25519 SHA256:<REDACTED> agent
Received disconnect from 1.2.3.4 port 22:2: Too many authentication failures
Disconnected from 1.2.3.4 port 22

So it says it tried the zac@other key which is resident in ssh-agent and then it received a “Too many authentication failures” error. It never even tried the id_ed25519_test key. And if you read that debug output carefully from the first blob - there’s four total keys it’s going to try, in order. Three of them are resident in ssh-agent and the last is our explicitly specified key.

Oh. Right. This server faces the internet and I’ve setup a particularly paranoid SSH config on the machine. I bet we’re only allowed three authentication attempts before it kills the TCP connection.

Let’s login to the server and see what MaxAuthTries is set to in /etc/ssh/sshd_config:

root@host ~# cat /etc/ssh/sshd_config | grep MaxAuthTries
MaxAuthTries 3
root@host ~#

Yup - that’ll do it. The ssh command tries all three keys I had loaded in ssh-agent first and then tries the explicit key I provided via the -i switch.

What did we learn?

Well, there’s two things to take away from this adventure:

In the end, the failure to login was the result of a couple of configurations colliding in a way I hadn’t previously predicted. It was fun to debug though - I did not realize that for every key you want to authenticate with, it counts as a distinct “authentication”. In retrospect this is obvious, it just never occurred to me.

Addendum

One side story I didn’t include, for brevity, was the side quest where I tried the same key from a different client machine without issue. I tried variations of configurations with and without ssh-agent involved. This puzzled me initially, but in retrospect, the reason it worked was because I didn’t try to load three keys into ssh-agent.

Posted on 2021-04-06