Locked out of Drobo SSH

Drobo - End of Sale Drobo FS
1

I’m pretty sure DRI’s nasd black box has bitten me hard. A while ago I replaced dropbear with openssh (as provided by Ricardo’s DroboPort), and all has been well for many weeks across plenty of use, reboots, etc. I also previously set up the authorized keys so I could log in without passwords, etc. All was well.

I went to log in today and all I get is “ssh_exchange_identification: Connection closed by remote host”. I created a new share yesterday, so I assume nasd blithely went along and trashed a bunch of files that it had no business messing with. I despise the worthless lackey who wrote that POS.

What’s the most reliable way to regain shell access to the Drobo? I have no idea if it nuked just /etc/passwd, or if it’s done a wholesale job on the root home folder. I can’t get in, and the folders involved are far below the typical share points. For now, I’m trying mucking with sshd_config to see if I can’t cajole a way in via different server configs. Of course, I have to reboot the Drobo every time for new options to take effect…

In the future, what can be done to avoid this and provide an “emergency parachute” of sorts?

2

Have you tried a PHP Terminal? Something like http://sourceforge.net/projects/phpterm/ for example.

My version of lighttpd runs as root (I know, I know, it shouldn’t :P), so you probably will be able to do anything you want. I know I managed to pull myself out of a situation like this in this way.

While we’re at it: the last time this happened to me, it was because /var/run/wtmp was somehow corrupted.

3

Okay, I managed to log in by disabling key-based authentication in DroboApps/openssh/etc/sshd_config, then used DroboAdmin to disable/enable openssh. This at least got me in via password-based authentication. I later discovered a way to force password-auth even when public key is preferred:

ssh -o PreferredAuthentications=password root@drobo

Everything looks okay for public key authentication. Permissions are properly set on all folders/files. Confirmed that the key in authorized_keys on the DroboFS is an exact match for id_rsa.pub on my laptop. Even this didn’t fix it:

scp ~/.ssh/id_rsa.pub root@drobo:~/.ssh/authorized_keys

Oh, and there’s nothing being logged to sshd.log - nada. May be a bug to look into in the sshd package, although I certainly don’t see anything amiss. Does make troubleshooting a bit more involved though.

So what did nasd do that just killed my SSH access?
[hr]
Interesting - now OpenSSH is just closing the connection entirely. Disabling public key authentication still gets me in via password, but I’m really scratching my head as to what’s wrong. Here’s the verbose output:

jochs@lightning ~ $ ssh -vvv root@drobo OpenSSH_5.2p1, OpenSSL 0.9.8l 5 Nov 2009 debug1: Reading configuration data /Users/jochs/.ssh/config debug1: Reading configuration data /etc/ssh_config debug2: ssh_connect: needpriv 0 debug1: Connecting to drobo [10.0.1.107] port 22. debug1: Connection established. debug1: identity file /Users/jochs/.ssh/identity type -1 debug3: Not a RSA1 key file /Users/jochs/.ssh/id_rsa. debug2: key_type_from_name: unknown key type '-----BEGIN' debug3: key_read: missing keytype debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug2: key_type_from_name: unknown key type '-----END' debug3: key_read: missing keytype debug1: identity file /Users/jochs/.ssh/id_rsa type 1 debug1: identity file /Users/jochs/.ssh/id_dsa type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.8 debug1: match: OpenSSH_5.8 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.2 debug2: fd 3 setting O_NONBLOCK debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256 debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_setup: found hmac-md5 debug1: kex: server->client aes128-ctr hmac-md5 none debug2: mac_setup: found hmac-md5 debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug2: dh_gen_key: priv key bits set: 124/256 debug2: bits set: 503/1024 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug3: check_host_in_hostfile: filename /Users/jochs/.ssh/known_hosts debug3: check_host_in_hostfile: match line 3 debug3: check_host_in_hostfile: filename /Users/jochs/.ssh/known_hosts debug3: check_host_in_hostfile: match line 3 debug1: Host 'drobo' is known and matches the RSA host key. debug1: Found key in /Users/jochs/.ssh/known_hosts:3 debug2: bits set: 507/1024 debug1: ssh_rsa_verify: signature correct debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: /Users/jochs/.ssh/identity (0x0) debug2: key: /Users/jochs/.ssh/id_rsa (0x1001190e0) debug2: key: /Users/jochs/.ssh/id_dsa (0x0) debug1: Authentications that can continue: publickey,password,keyboard-interactive debug3: start over, passed a different list publickey,password,keyboard-interactive debug3: preferred publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Trying private key: /Users/jochs/.ssh/identity debug3: no such identity: /Users/jochs/.ssh/identity debug1: Offering public key: /Users/jochs/.ssh/id_rsa debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply Connection closed by 10.0.1.107[hr]

I have no such file. Just a folder for the avahi-daemon and an sshd.pid.

4

Are you sure that authorized_keys has the proper permission on the drobo? It is very picky about it:

chmod 600 ~/.ssh/authorized_keys

I vaguely remember you saying that you wanted to move the root home folder, and maybe the permissions got messed up during the copy (did you use ‘cp -a’?).

Hmm, that is weird. It should be able to log stuff (service.sh does set the permissions correctly), although mine is also empty… there may be something wrong with the configuration. I’ll check it again.

I don’t know, but it seems your keyfile may be corrupted on your Mac.

5

I finally finished a bunch of backups so I could take the Drobo offline and give it a reboot. On reboot, it worked normally - which certainly confirms that all the keys are fine (although I knew they were, since I could SSH into any other host with no problem).

So… what did nasd do to break it (when I created a new share), and what did a reboot fix? And why does nasd feel the need to keep meddling with things that it has no business meddling with?[hr]

I never got around to this (thankfully). If I had, there would have been far too many variables in play to make sense of anything. For now, that little project is on indefinite hold.[hr]

I thought the exact same when I saw that, but that is apparently completely normal for the RSA key, and still occurs even though my public-key login is working again. Seems to me that OpenSSH could improve that little bit of debugging/error code, as it’s very misleading.