From kvhwyucjkp at snkmail.com Sun May 1 08:59:59 2011 From: kvhwyucjkp at snkmail.com (Ryan) Date: Sat, 30 Apr 2011 15:59:59 -0700 Subject: enhanced permitopen matching (patch in bugzilla) Message-ID: <1973-1304204406-857797@sneakemail.com> I'd really like to get some feedback on the patch I posted to bugzilla about a month ago. https://bugzilla.mindrot.org/show_bug.cgi?id=1513#c3 This patch adds support for CIDR ranges, wildcard matches, negated matches, port ranges and makes matches for ip addresses and hostnames work interchangeably. This was done primarily by using already existing functions for from= in ssh keys. I'm not very experienced with sockets or dynamic memory management so some more sets of eyes would be great. -Ryan From medsalim.bouhlel at enis.rnu.tn Sun May 1 14:00:43 2011 From: medsalim.bouhlel at enis.rnu.tn (Med Salim BOUHLEL) Date: Sun, 1 May 2011 06:00:43 +0200 Subject: Deadline submission for the SETIT 2011-TUNISIA- Message-ID: <1164650861441468121254@admin> Appologies if you received multiple copies. Please forward this message to any potential colleagues in the areas of interest Dear, Just a small reminder that the deadline submission for the sixth International Conference Sciences of Electronics, Technologies of Information and Telecommunications SETIT 2011 is coming up. If you are interested to submit a regular paper, tutorial or a proposal for a special session you are pleased to do it before June 1st 2011. This conference will be held in Tunisia, 26-29 October 2011. It is technically sponsored by IEEE and the SAI. Until now, more than 750 papers have been proposed among which 350 have been retained for publication. You can find more details in: http://www.setit.rnu.tn/. All papers in conference proceedings will be included in the IEEE Xplore. The selected best papers will be published in one of our partener journals http://www.setit.rnu.tn/list_of_journals.html . See you in Sousse, Mohamed Salim BOUHLEL General Co-Chair, SETIT 2011 Head of Research Unit: Sciences & Technologies of Image and Telecommunications (Sfax University) GSM +216 20 200005 Find us on Facbook Find us on Twitter ============================================================================= This email is sent out to all those on the SETIT database. If you want to be removed from this database, please send an email to unsubscribe.setit at gmail.com with subject Unsubscribe ============================================================================= From djm at cvs.openbsd.org Tue May 3 10:48:57 2011 From: djm at cvs.openbsd.org (Damien Miller) Date: Mon, 2 May 2011 18:48:57 -0600 (MDT) Subject: Revised: Portable OpenSSH security advisory: portable-keysign-rand-helper.adv Message-ID: <201105030048.p430mvxS001384@cvs.openbsd.org> OpenSSH Security Advisory: portable-keysign-rand-helper.adv This document may be found at: http://www.openssh.com/txt/portable-keysign-rand-helper.adv 1. Vulnerability Portable OpenSSH's ssh-keysign utility may allow unauthorised local access to host keys on platforms if ssh-rand-helper is used. 2. Affected configurations Portable OpenSSH prior to version 5.8p2 only on platforms that are configured to use ssh-rand-helper for entropy collection. ssh-rand-helper is enabled at configure time when it is detected that OpenSSL does not have a built-in source of randomness, and only used at runtime if this condition remains. Platforms that support /dev/random or otherwise configure OpenSSL with a random number provider are not vulnerable. In particular, *BSD, OS X, Cygwin and Linux are not affected. 3. Mitigation If host-based authentication is not in use (enabled using HostBasedAuthentication or RhostsRSAAuthentication in sshd_config), then remove the setuid bit from ssh-keysign. 4. Details ssh-keysign is a setuid helper program that is used to mediate access to the host's private host keys during host-based authentication. It would use its elevated privilege to open the keys and then immediately drop privileges to complete its cryptographic signing operations. After privilege was dropped, ssh-keysign would ensure that the OpenSSL random number generator that it depends upon was adequately prepared. On configurations that lacked a built-in source of entropy in OpenSSL, ssh-keysign would execute the ssh-rand-helper program to attempt to retrieve some from the system environment. However, the file descriptors to the host private key files were not closed prior to executing ssh-rand-helper. Since this process was "born unprivileged" and inherited the sensitive file descriptors, there was no protection against an attacker using ptrace(2) to attach to it and instructing it to read out the private keys. 5. Credit This issue was privately reported by Tomas Mraz on April 26, 2011. 6. Fix OpenSSH 5.8p2 contains a fix for this vulnerability. Future releases of portable OpenSSH will remove support for ssh-rand-helper - in 2011, there is no excuse for not providing a /dev/random-like interface as part of the OS. Users stuck on one of these platforms may use PRNGd (http://prngd.sf.net) to provide a host-wide random pool. Users of older versions that do not wish to upgrade immediately may apply this patch: Index: ssh-keysign.c =================================================================== RCS file: /var/cvs/openssh/ssh-keysign.c,v retrieving revision 1.43 diff -u -p -r1.43 ssh-keysign.c --- ssh-keysign.c 10 Sep 2010 01:12:09 -0000 1.43 +++ ssh-keysign.c 29 Apr 2011 01:25:55 -0000 @@ -167,6 +167,9 @@ main(int argc, char **argv) key_fd[0] = open(_PATH_HOST_RSA_KEY_FILE, O_RDONLY); key_fd[1] = open(_PATH_HOST_DSA_KEY_FILE, O_RDONLY); + if (fcntl(key_fd[0], F_SETFD, FD_CLOEXEC) != 0 || + fcntl(key_fd[1], F_SETFD, FD_CLOEXEC) != 0) + fatal("fcntl failed"); original_real_uid = getuid(); /* XXX readconf.c needs this */ if ((pw = getpwuid(original_real_uid)) == NULL) From djm at cvs.openbsd.org Tue May 3 10:50:41 2011 From: djm at cvs.openbsd.org (Damien Miller) Date: Mon, 2 May 2011 18:50:41 -0600 (MDT) Subject: Announce: Portable OpenSSH 5.8p2 released Message-ID: <201105030050.p430ofQu024783@cvs.openbsd.org> Portable OpenSSH 5.8p2 has just been released. It will be available from the mirrors listed at http://www.openssh.com/ shortly. OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0 implementation and includes sftp client and server support. Once again, we would like to thank the OpenSSH community for their continued support of the project, especially those who contributed code or patches, reported bugs, tested snapshots or donated to the project. More information on donations may be found at: http://www.openssh.com/donations.html Changes since OpenSSH 5.8p1 =========================== Security: * Fix local private host key compromise on platforms without host- level randomness support (e.g. /dev/random) reported by Tomas Mraz On hosts that did not have a randomness source configured in OpenSSL and were not configured to use EGD/PRNGd (using the --with-prngd-socket configure option), the ssh-rand-helper command was being implicitly executed by ssh-keysign with open file descriptors to the host private keys. An attacker could use ptrace(2) to attach to ssh-rand-helper and exfiltrate the keys. Most modern operating systems are not vulnerable. In particular, *BSD, Linux, OS X and Cygwin do not use ssh-rand-helper. A full advisory for this issue is available at: http://www.openssh.com/txt/portable-keysign-rand-helper.adv Portable OpenSSH Bugfixes: * Fix compilation failure when enabling SELinux support. * Revised Cygwin ssh-{host,user}-config that include ECDSA key support. * Revised Cygwin ssh-host-config to be more thorough in error checking and reporting. Checksums: ========== - SHA1 (openssh-5.8p2.tar.gz) = e610270e0c5484fb291cd81bbcbefbeb5e391a62 Reporting Bugs: =============== - Please read http://www.openssh.com/report.html Security bugs should be reported directly to openssh at openssh.com OpenSSH is brought to you by Markus Friedl, Niels Provos, Theo de Raadt, Kevin Steves, Damien Miller, Darren Tucker, Jason McIntyre, Tim Rice and Ben Lindstrom. From bob at proulx.com Tue May 3 12:53:22 2011 From: bob at proulx.com (Bob Proulx) Date: Mon, 2 May 2011 20:53:22 -0600 Subject: advisory file locks in linux - do they work? In-Reply-To: <4DA5E8C5.7000303@spscommerce.com> References: <4DA478B8.7050409@spscommerce.com> <20110413164536.GA18908@hysteria.proulx.com> <4DA5E8C5.7000303@spscommerce.com> Message-ID: <20110503025322.GA14543@hysteria.proulx.com> Jeff Gibson wrote: > Bob Proulx wrote: > > You seem to think that all programs do file locking? Or perhaps you > > are aware that they do not but you think they /should/ use file > > locking and this is your way of lobbying to add it? > > Whoa - Please don't start attacking. I had just read on some > websites that SFTP supported it (perhaps this was not OpenSSH), and > I *thought* I had it working on Solaris at one point. Guess not. Sorry we got off on the wrong foot. > Anyway it's good to know this so I don't spend any more time trying > to get it to work! :) > Perhaps I wasn't very clear - we have clients connecting to us via > SFTP - Pushing files to you? Or pulling files from you. Obviously if you want implement locking then it needs to be on the write side. > We cannot reasonably expect them to know enough about UNIX/SFTP to > do the above, and many of the 3rd party programs they use are closed > source with limited feature sets. Right. > Instead I was hoping that it was possible for the internal-sftp > process to automatically lock incoming files. As I have now > learned, OpenSSH/SFTP does not support this natively. However, we > have had an internal developer add the functionality and so far it > seems to work. Personally I have always been an advocate of writing the file off to the side in a temporary file and then renaming it into place. That doesn't need a lockfile. If you are hacking on the code to add cooperative file semaphores then you might consider writing to a temporary file and renaming it instead. Since rename(2) is atomic the file will either be the previous file or the next file. (Although recently with the Linux filesystems this semantic has been changed.) > OK - so is there a way to do this with the internal-sftp command > while using a chroot directory? I guess I'm not grokking how to do > this on the server-side without adding code or copying binaries to > the user's directory. As far as I know the only way to do this would be to make code changes. I don't know if this applies to you but a common file upload strategy is to have people drop files into a directory with unique names and then have a process notice these files as they appear and then to move them to wherever they need to go. Bob From william at 25thandClement.com Tue May 3 13:26:14 2011 From: william at 25thandClement.com (William Ahern) Date: Mon, 2 May 2011 20:26:14 -0700 Subject: advisory file locks in linux - do they work? In-Reply-To: <20110503025322.GA14543@hysteria.proulx.com> References: <4DA478B8.7050409@spscommerce.com> <20110413164536.GA18908@hysteria.proulx.com> <4DA5E8C5.7000303@spscommerce.com> <20110503025322.GA14543@hysteria.proulx.com> Message-ID: <20110503032614.GA564@wilbur.25thandClement.com> On Mon, May 02, 2011 at 08:53:22PM -0600, Bob Proulx wrote: > Jeff Gibson wrote: > > Instead I was hoping that it was possible for the internal-sftp > > process to automatically lock incoming files. As I have now > > learned, OpenSSH/SFTP does not support this natively. However, we > > have had an internal developer add the functionality and so far it > > seems to work. > > Personally I have always been an advocate of writing the file off to > the side in a temporary file and then renaming it into place. That > doesn't need a lockfile. If you are hacking on the code to add > cooperative file semaphores then you might consider writing to a > temporary file and renaming it instead. Since rename(2) is atomic the > file will either be the previous file or the next file. (Although > recently with the Linux filesystems this semantic has been changed.) rename was always atomic. You must be referring to the brief episode where ext4 wouldn't guarantee the ordering of writes viz-a-viz rename in the transaction log. That is, the rename might be committed before any prior writes, rather than rename acting as an implicit write barrier. You still ended up with an atomic rename--the old file intact or gone--but the new file may not be whole after a crash even if the rename succeeded. The defaults were quickly reverted to the expected behavior after raucous protest. But to be completely safe according to a strict interpretation of POSIX, fdatasync() or, if unavailable, fsync() should be issued _before_ the rename. That defeats the use of rename as a cheap transaction primitive, and that's why the behavior was reverted. From djm at mindrot.org Tue May 3 14:45:10 2011 From: djm at mindrot.org (Damien Miller) Date: Tue, 3 May 2011 14:45:10 +1000 (EST) Subject: heads-up: snapshots now being built using autoconf-2.68 Message-ID: Hi, I have just switched snapshot building over to autoconf 2.68 (from 2.61). Please report any problems you may encounter to the list. If all goes well the 5.9 release will use autoconf 2.68. -d From des at des.no Wed May 4 06:50:29 2011 From: des at des.no (=?utf-8?Q?Dag-Erling_Sm=C3=B8rgrav?=) Date: Tue, 03 May 2011 22:50:29 +0200 Subject: IPTOS constants Message-ID: <86mxj3tsoa.fsf@ds4.des.no> defines.h defines a bunch of IPTOS constants if they're not already available: #ifndef IPTOS_LOWDELAY # define IPTOS_LOWDELAY 0x10 # define IPTOS_THROUGHPUT 0x08 # define IPTOS_RELIABILITY 0x04 # define IPTOS_LOWCOST 0x02 # define IPTOS_MINCOST IPTOS_LOWCOST #endif /* IPTOS_LOWDELAY */ A few lines further down, it includes , which is where those constants are defined (at least in FreeBSD). --- /home/des/freebsd/base/vendor-crypto/openssh/dist/defines.h 2011-01-17 11:15:31.000000000 +0100 +++ defines.h 2011-05-03 22:34:47.724047823 +0200 @@ -45,6 +45,7 @@ /* * Definitions for IP type of service (ip_tos) */ +#include #ifndef IPTOS_LOWDELAY # define IPTOS_LOWDELAY 0x10 # define IPTOS_THROUGHPUT 0x08 @@ -57,7 +58,6 @@ * Definitions for DiffServ Codepoints as per RFC2474 */ #include -#include #ifndef IPTOS_DSCP_AF11 # define IPTOS_DSCP_AF11 0x28 # define IPTOS_DSCP_AF12 0x30 These include directives should really be moved to includes.h - in fact, is already there. DES -- Dag-Erling Sm?rgrav - des at des.no From oren at held.org.il Thu May 5 05:23:18 2011 From: oren at held.org.il (Oren Held) Date: Wed, 4 May 2011 22:23:18 +0300 Subject: ssh 'connection reset by peer' problem: updates Message-ID: <20110504192317.GA2129@orenhe-laptop> Hi, The new interesting piece of information regarding the problem I (and others) reported here on February, is that it's reproducible on some Cisco-firewalled networks. I concluded that because that's what my workplace is using, and also seen this report on https://www.nowhere.dk/articles/natty-narwhal-problems-connecting-to-servers-behind-cisco-firewalls-using-ssh I've also summarized all information known to me so far, from multiple sources: http://www.held.org.il/blog/2011/05/the-myterious-case-of-broken-ssh-client-connection-reset-by-peer/ Note that now that Ubuntu 11.04 is out, more people are upgrading to 5.8p1 and are about to be exposed to this problem. Maybe it's just a Cisco bug and there's nothing to change in OpenSSH, and still we should remember that the problem is triggered only from 5.7p1. Best Regards, Oren From charles at infoplatter.com Thu May 5 08:43:18 2011 From: charles at infoplatter.com (Charles Johnston) Date: Wed, 04 May 2011 16:43:18 -0600 Subject: PATCH: update version to p2 in version.h Message-ID: <4DC1D686.5080302@infoplatter.com> --- a/version.h +++ a/version.h @@ -2,5 +2,5 @@ #define SSH_VERSION "OpenSSH_5.8" -#define SSH_PORTABLE "p1" +#define SSH_PORTABLE "p2" #define SSH_RELEASE SSH_VERSION SSH_PORTABLE From djm at mindrot.org Thu May 5 11:50:59 2011 From: djm at mindrot.org (Damien Miller) Date: Thu, 5 May 2011 11:50:59 +1000 (EST) Subject: IPTOS constants In-Reply-To: <86mxj3tsoa.fsf@ds4.des.no> References: <86mxj3tsoa.fsf@ds4.des.no> Message-ID: On Tue, 3 May 2011, Dag-Erling Sm?rgrav wrote: > defines.h defines a bunch of IPTOS constants if they're not already > available: > > #ifndef IPTOS_LOWDELAY > # define IPTOS_LOWDELAY 0x10 > # define IPTOS_THROUGHPUT 0x08 > # define IPTOS_RELIABILITY 0x04 > # define IPTOS_LOWCOST 0x02 > # define IPTOS_MINCOST IPTOS_LOWCOST > #endif /* IPTOS_LOWDELAY */ > > A few lines further down, it includes , which is where > those constants are defined (at least in FreeBSD). > > --- /home/des/freebsd/base/vendor-crypto/openssh/dist/defines.h > 2011-01-17 11:15:31.000000000 +0100 > +++ defines.h 2011-05-03 22:34:47.724047823 +0200 applied - thanks. > These include directives should really be moved to includes.h - in fact, > is already there. I'm a bit ambivalent about this - includes.h is a bit of a dump that I'd like to do away with (but probably won't). defines.h actually exists for a reason other than laziness :) -d From djm at mindrot.org Thu May 5 11:55:21 2011 From: djm at mindrot.org (Damien Miller) Date: Thu, 5 May 2011 11:55:21 +1000 (EST) Subject: ssh 'connection reset by peer' problem: updates In-Reply-To: <20110504192317.GA2129@orenhe-laptop> References: <20110504192317.GA2129@orenhe-laptop> Message-ID: On Wed, 4 May 2011, Oren Held wrote: > Hi, > > The new interesting piece of information regarding the problem I > (and others) reported here on February, is that it's reproducible > on some Cisco-firewalled networks. I concluded that because > that's what my workplace is using, and also seen this report on > https://www.nowhere.dk/articles/natty-narwhal-problems-connecting-to-s > ervers-behind-cisco-firewalls-using-ssh > > I've also summarized all information known > to me so far, from multiple sources: > http://www.held.org.il/blog/2011/05/the-myterious-case-of-broken-ssh-c > lient-connection-reset-by-peer/ > > Note that now that Ubuntu 11.04 is out, more people are upgrading to > 5.8p1 and are about to be exposed to this problem. > > Maybe it's just a Cisco bug and there's nothing to change in OpenSSH, > and still we should remember that the problem is triggered only from > 5.7p1. If you are able to reproduce it in your environment, e.g. with the Ciphers comma hack then you can exhonerate (or otherwise) OpenSSH by tcpdumping the session at both the client and the server and seeing who actually closes the connection. If a middlebox is screwing up DPI, then you would expect to see a RST appear at the client that was not sent by the server. -d From djm at mindrot.org Thu May 5 12:03:11 2011 From: djm at mindrot.org (Damien Miller) Date: Thu, 5 May 2011 12:03:11 +1000 (EST) Subject: Incorrect version identifier in 5.8p2 release Message-ID: Hi, The OpenSSH 5.8p2 release shipped with the wrong version identified embedded in version.h (it still identified itself as "5.8p1"). I have cut and uploaded a new release with this mistake corrected. The checksum for the revised tarball is: SHA1 (openssh-5.8p2.tar.gz) = 64798328d310e4f06c9f01228107520adbc8b3e5 As usual, this file is signed with gzsig and PGP. Apologies for any inconvenience and thanks to Charles Johnston for bringing the problem to my attention. -d From djm at mindrot.org Thu May 5 13:58:37 2011 From: djm at mindrot.org (Damien Miller) Date: Thu, 5 May 2011 13:58:37 +1000 (EST) Subject: heads-up: removal of ssh-rand-helper In-Reply-To: <201105030048.p430mvxS001384@cvs.openbsd.org> References: <201105030048.p430mvxS001384@cvs.openbsd.org> Message-ID: On Mon, 2 May 2011, Damien Miller wrote: > OpenSSH Security Advisory: portable-keysign-rand-helper.adv ... > Future releases of portable OpenSSH will remove support for > ssh-rand-helper - in 2011, there is no excuse for not > providing a /dev/random-like interface as part of the OS. > Users stuck on one of these platforms may use PRNGd > (http://prngd.sf.net) to provide a host-wide random pool. I have committed the diff to remove ssh-rand-helper from -current. Systems that were using ssh-rand-helper should install PRNGd from http://prngd.sf.net/ instead - it is less complex and a better source of random numbers. How less complex? > 22 files changed, 230 insertions(+), 1550 deletions(-) Previously, portable OpenSSH's support for PRNGd depended on ssh-rand-helper to make the connections. The removal diff moved this support into libssh.a, so all programs that need random numbers will now connect to PRNGd directly. Support for this will be in tomorrow's snapshot, so if you use a system without /dev/random or a similar source of random numbers in OpenSSL then please test one. -d From markus.r.friedl at arcor.de Sat May 7 00:42:15 2011 From: markus.r.friedl at arcor.de (Markus Friedl) Date: Fri, 6 May 2011 16:42:15 +0200 Subject: new option ssh-add -v to verify if key is loaded into the agent In-Reply-To: <4D9D63C8.6060909@open.ch> References: <4D9D63C8.6060909@open.ch> Message-ID: <20110506144215.GA11339@folly> On Thu, Apr 07, 2011 at 09:12:08AM +0200, Konrad Bucheli wrote: > Dear openssh developers > > In a shell script I need to verify if a key belonging to a given public > key file is already loaded into the agent. To achieve this, I added a > new option -v to ssh-add which does this verification. > > The patch bases on openssh v5.8p1. The regression test agent.sh was > extended to test this new feature. > > Is there any chance for inclusion of attached patch? fwiw, some of my ssh trees had something like this (restricted to ssh v2 keys). Index: ssh-add.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/ssh-add.c,v retrieving revision 1.100 diff -u -p -u -r1.100 ssh-add.c --- ssh-add.c 31 Aug 2010 12:33:38 -0000 1.100 +++ ssh-add.c 6 May 2011 14:34:44 -0000 @@ -261,6 +261,32 @@ update_card(AuthenticationConnection *ac } static int +test_key(AuthenticationConnection *ac, const char *filename) +{ + Key *key = NULL; + u_char *sig = NULL; + u_int slen = 0; + int ret = -1; + char data[1024]; + + if ((key = key_load_public(filename, NULL)) == NULL) { + error("Loading key from '%s' failed", filename); + goto done; + } + arc4random_buf(data, sizeof(data)); + if (ssh_agent_sign(ac, key, &sig, &slen, data, sizeof(data)) == -1) + goto done; + if (key_verify(key, sig, slen, data, sizeof(data)) == 1) + ret = 0; + done: + if (sig) + xfree(sig); + if (key) + key_free(key); + return (ret); +} + +static int list_identities(AuthenticationConnection *ac, int do_fp) { Key *key; @@ -351,6 +377,7 @@ usage(void) fprintf(stderr, " -c Require confirmation to sign using identities\n"); fprintf(stderr, " -s pkcs11 Add keys from PKCS#11 provider.\n"); fprintf(stderr, " -e pkcs11 Remove keys provided by PKCS#11 provider.\n"); + fprintf(stderr, " -T pubkey Test if ssh-agent can access matching private key.\n"); } int @@ -360,6 +387,7 @@ main(int argc, char **argv) extern int optind; AuthenticationConnection *ac = NULL; char *pkcs11provider = NULL; + char *testing = NULL; int i, ch, deleting = 0, ret = 0; /* Ensure that fds 0, 1 and 2 are open or directed to /dev/null */ @@ -374,7 +402,7 @@ main(int argc, char **argv) "Could not open a connection to your authentication agent.\n"); exit(2); } - while ((ch = getopt(argc, argv, "lLcdDxXe:s:t:")) != -1) { + while ((ch = getopt(argc, argv, "lLcdDxXe:s:t:T:")) != -1) { switch (ch) { case 'l': case 'L': @@ -410,6 +438,9 @@ main(int argc, char **argv) goto done; } break; + case 'T': + testing = optarg; + break; default: usage(); ret = 1; @@ -418,6 +449,10 @@ main(int argc, char **argv) } argc -= optind; argv += optind; + if (testing != NULL) { + ret = (test_key(ac, testing) == 0) ? 0 : 1; + goto done; + } if (pkcs11provider != NULL) { if (update_card(ac, !deleting, pkcs11provider) == -1) ret = 1; From bert.wesarg at googlemail.com Sat May 7 05:14:26 2011 From: bert.wesarg at googlemail.com (Bert Wesarg) Date: Fri, 6 May 2011 21:14:26 +0200 Subject: [PATCH/trivial] Fix numbering in PROTOCOL.mux Message-ID: diff --git i/PROTOCOL.mux w/PROTOCOL.mux index 05bb146..b53af19 100644 PROTOCOL.mux --- i/PROTOCOL.mux +++ w/PROTOCOL.mux @@ -180,7 +180,7 @@ The MUX_S_PERMISSION_DENIED and MUX_S_FAILURE include a reason: uint32 client request id string reason -9. Protocol numbers +10. Protocol numbers #define MUX_MSG_HELLO 0x00000001 #define MUX_C_NEW_SESSION 0x10000002 From lip at lip.net.ua Sat May 7 05:59:08 2011 From: lip at lip.net.ua (F 10) Date: Fri, 6 May 2011 22:59:08 +0300 Subject: hacking attempt Message-ID: Hello, today I find in my logs May 6 01:36:14 xxx sshd[27880]: Address x.x.x.x maps to xxx.com, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! May 6 01:36:15 xxx sshd[27880]: *Accepted publickey* for root from x.x.x.x port 55707 ssh2 May 6 01:36:15 xxx sshd[27880]: pam_unix(sshd:session): session opened for user root by (uid=0) May 6 01:36:15 xxx sshd[27880]: subsystem request for sftp In the sshd_config was always PermitRootLogin no /root/.ssh always was empty md5sum /usr/sbin/sshd f8c11462e8f2a7bf80e212e06041492b /usr/sbin/sshd md5sum sshd #binary from .deb f8c11462e8f2a7bf80e212e06041492b sshd OS Debian GNU/Linux 6.0 SSH-2.0-OpenSSH_5.5p1 Debian-6 How it's possible? From keisial at gmail.com Sat May 7 07:09:10 2011 From: keisial at gmail.com (=?ISO-8859-1?Q?=C1ngel_Gonz=E1lez?=) Date: Fri, 06 May 2011 23:09:10 +0200 Subject: hacking attempt In-Reply-To: References: Message-ID: <4DC46376.6000907@gmail.com> F 10 wrote: > Hello, > today I find in my logs > > May 6 01:36:14 xxx sshd[27880]: Address x.x.x.x maps to xxx.com, but this > does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! > May 6 01:36:15 xxx sshd[27880]: *Accepted publickey* for root from x.x.x.x > port 55707 ssh2 > May 6 01:36:15 xxx sshd[27880]: pam_unix(sshd:session): session opened for > user root by (uid=0) > May 6 01:36:15 xxx sshd[27880]: subsystem request for sftp > > In the sshd_config was always PermitRootLogin no > > /root/.ssh always was empty > > md5sum /usr/sbin/sshd > f8c11462e8f2a7bf80e212e06041492b /usr/sbin/sshd > > md5sum sshd #binary from .deb > f8c11462e8f2a7bf80e212e06041492b sshd > > OS Debian GNU/Linux 6.0 > SSH-2.0-OpenSSH_5.5p1 Debian-6 > > How it's possible? Perhaps 27880 wasn't the normal sshd instance, but run with different config/a trojaned one? (that bears the question on how they could launch such hypothetical 'evil sshd' before, though) From dgtlshdw at gmail.com Sat May 7 07:48:58 2011 From: dgtlshdw at gmail.com (Kevin Boers) Date: Fri, 6 May 2011 16:48:58 -0500 Subject: hacking attempt In-Reply-To: <4DC46376.6000907@gmail.com> References: <4DC46376.6000907@gmail.com> Message-ID: That's the exact scenario I've seen before: someone got in with another exploit and replaced my sshd with a hacked one. I'd search to see if there's more than one sshd, and if any of the settings have been replaced. I've since put monitoring around the files to make sure they don't change. K 2011/5/6 ?ngel Gonz?lez : > F 10 wrote: >> Hello, >> today I find in my logs >> >> May ?6 01:36:14 xxx sshd[27880]: Address x.x.x.x maps to xxx.com, but this >> does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! >> May ?6 01:36:15 xxx sshd[27880]: *Accepted publickey* for root from x.x.x.x >> port 55707 ssh2 >> May ?6 01:36:15 xxx sshd[27880]: pam_unix(sshd:session): session opened for >> user root by (uid=0) >> May ?6 01:36:15 xxx sshd[27880]: subsystem request for sftp >> >> In the sshd_config was always PermitRootLogin no >> >> /root/.ssh always was empty >> >> md5sum /usr/sbin/sshd >> f8c11462e8f2a7bf80e212e06041492b ?/usr/sbin/sshd >> >> md5sum sshd #binary from .deb >> f8c11462e8f2a7bf80e212e06041492b ?sshd >> >> OS Debian GNU/Linux 6.0 >> SSH-2.0-OpenSSH_5.5p1 Debian-6 >> >> How it's possible? > Perhaps 27880 wasn't the normal sshd instance, but run with different > config/a trojaned one? > (that bears the question on how they could launch such hypothetical > 'evil sshd' before, though) > > > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev > From scott_n at xypro.com Sat May 7 07:55:53 2011 From: scott_n at xypro.com (Scott Neugroschl) Date: Fri, 6 May 2011 14:55:53 -0700 Subject: hacking attempt In-Reply-To: <4DC46376.6000907@gmail.com> References: <4DC46376.6000907@gmail.com> Message-ID: <78DD71C304F38B41885A242996B96F7302AD1E9C@xyservd.XYPRO-23.LOCAL> Do you normally run your sshd on 55707? > -----Original Message----- > From: openssh-unix-dev-bounces+scott_n=xypro.com at mindrot.org > [mailto:openssh-unix-dev-bounces+scott_n=xypro.com at mindrot.org] On > Behalf Of ?ngel Gonz?lez > Sent: Friday, May 06, 2011 2:09 PM > To: F 10; openssh-unix-dev at mindrot.org > Subject: Re: hacking attempt > > F 10 wrote: > > Hello, > > today I find in my logs > > > > May 6 01:36:14 xxx sshd[27880]: Address x.x.x.x maps to xxx.com, but > this > > does not map back to the address - POSSIBLE BREAK-IN ATTEMPT! > > May 6 01:36:15 xxx sshd[27880]: *Accepted publickey* for root from > x.x.x.x > > port 55707 ssh2 > > May 6 01:36:15 xxx sshd[27880]: pam_unix(sshd:session): session > opened for > > user root by (uid=0) > > May 6 01:36:15 xxx sshd[27880]: subsystem request for sftp > > > > In the sshd_config was always PermitRootLogin no > > > > /root/.ssh always was empty > > > > md5sum /usr/sbin/sshd > > f8c11462e8f2a7bf80e212e06041492b /usr/sbin/sshd > > > > md5sum sshd #binary from .deb > > f8c11462e8f2a7bf80e212e06041492b sshd > > > > OS Debian GNU/Linux 6.0 > > SSH-2.0-OpenSSH_5.5p1 Debian-6 > > > > How it's possible? > Perhaps 27880 wasn't the normal sshd instance, but run with different > config/a trojaned one? > (that bears the question on how they could launch such hypothetical > 'evil sshd' before, though) > > > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev From dtucker at zip.com.au Sat May 7 08:18:49 2011 From: dtucker at zip.com.au (Darren Tucker) Date: Sat, 7 May 2011 08:18:49 +1000 Subject: hacking attempt In-Reply-To: <78DD71C304F38B41885A242996B96F7302AD1E9C@xyservd.XYPRO-23.LOCAL> References: <4DC46376.6000907@gmail.com> <78DD71C304F38B41885A242996B96F7302AD1E9C@xyservd.XYPRO-23.LOCAL> Message-ID: On Sat, May 7, 2011 at 7:55 AM, Scott Neugroschl wrote: > Do you normally run your sshd on 55707? That log message is of the source port of the connection not the listening port. To the OP: I agree with the previous posters that there's probably another sshd running on your machine. You can look for listening ports (eg "lsof | grep LISTEN") but it certainly looks like the machine is compromised so the only safe action is to determine how then reinstall the system from trusted media (fixing the problem, obviously). -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4? 37C9 C982 80C7 8FF4 FA69 ? ? Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement. From djm at mindrot.org Sat May 7 09:44:06 2011 From: djm at mindrot.org (Damien Miller) Date: Sat, 7 May 2011 09:44:06 +1000 (EST) Subject: [PATCH/trivial] Fix numbering in PROTOCOL.mux In-Reply-To: References: Message-ID: applied, thanks On Fri, 6 May 2011, Bert Wesarg wrote: > diff --git i/PROTOCOL.mux w/PROTOCOL.mux > index 05bb146..b53af19 100644 PROTOCOL.mux > --- i/PROTOCOL.mux > +++ w/PROTOCOL.mux > @@ -180,7 +180,7 @@ The MUX_S_PERMISSION_DENIED and MUX_S_FAILURE > include a reason: > uint32 client request id > string reason > > -9. Protocol numbers > +10. Protocol numbers > > #define MUX_MSG_HELLO 0x00000001 > #define MUX_C_NEW_SESSION 0x10000002 > From phil at hands.com Sat May 7 19:03:29 2011 From: phil at hands.com (Philip Hands) Date: Sat, 07 May 2011 10:03:29 +0100 Subject: hacking attempt In-Reply-To: References: <4DC46376.6000907@gmail.com> <78DD71C304F38B41885A242996B96F7302AD1E9C@xyservd.XYPRO-23.LOCAL> Message-ID: <87tyd6c272.fsf@poker.hands.com> On Sat, 7 May 2011 08:18:49 +1000, Darren Tucker wrote: > On Sat, May 7, 2011 at 7:55 AM, Scott Neugroschl wrote: > > Do you normally run your sshd on 55707? > > That log message is of the source port of the connection not the listening port. > > To the OP: I agree with the previous posters that there's probably > another sshd running on your machine. You can look for listening > ports (eg "lsof | grep LISTEN") but it certainly looks like the > machine is compromised so the only safe action is to determine how > then reinstall the system from trusted media (fixing the problem, > obviously). If you want to get a decent idea of what happened, you could try booting From a matching version of Debian Live CD, and then mounting the disk(s), read-only, and doing recursive diffs to see what's different. debsums can be helpful, as can cruft You'll probably find that there's a /var/tmp/.foo directory, or some such that has your rogue ssh in it. The reason to boot from a known good CD is that if they've done a proper job (which I doubt, given that they didn't persuade their sshd to not log the fact that they were hacking in -- Doh!) then they'll have installed new versions of all the utilities that you'd use to find that directory or check the contents of the files. As mentioned, lsof (if it's the real one and the kernel hasn't been tampered with) will show you the extra ssh, including the file it was loaded from, which may make things a bit clearer. Have a look for anything odd in all .bash_history files (or similar for whatever shells you use). Script kiddies tend to forget about this. find can be handy too: find / -mtime -10 will show you all the things touched in the last 10 days, for example, assuming they've not put effort into hiding their tracks. Cheers, Phil. -- |)| Philip Hands [+44 (0)20 8530 9560] http://www.hands.com/ |-| HANDS.COM Ltd. http://www.uk.debian.org/ |(| 10 Onslow Gardens, South Woodford, London E18 1NE ENGLAND -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From lip at lip.net.ua Sat May 7 21:08:29 2011 From: lip at lip.net.ua (F 10) Date: Sat, 7 May 2011 14:08:29 +0300 Subject: hacking attempt In-Reply-To: <87tyd6c272.fsf@poker.hands.com> References: <4DC46376.6000907@gmail.com> <78DD71C304F38B41885A242996B96F7302AD1E9C@xyservd.XYPRO-23.LOCAL> <87tyd6c272.fsf@poker.hands.com> Message-ID: Thanks for your help, I think he got the root, before he used the SSH. But, as he has access to the root I can not explain. From list2rado at gmx.de Tue May 10 00:21:03 2011 From: list2rado at gmx.de (Rado S) Date: Mon, 9 May 2011 16:21:03 +0200 Subject: backdoor by authorized_keys2 leftovers Message-ID: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> Hi devs, recently I had to replace authorized_keys on several systems to enforce an access policy change. I was badly surprised that authorized_keys2(!) was still processed, which allowed some old keys to enter the systems again, because I wasn't aware of the file's existance on the server and use by sshd, since this "backward compatibility" isn't documented, not even a historical reference about "obsolete" or "deprecated". Maybe it's time to drop the old stuff not to get haunted by such leftovers again. Thanks, regards, Rado -- ? Rado S. -- You must provide YOUR effort for your goal! EVERY effort counts: at least to show your attitude. You're responsible for ALL you do: you get what you give. From dkg at fifthhorseman.net Tue May 10 02:06:27 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Mon, 09 May 2011 12:06:27 -0400 Subject: ssh-agent subprocess parentage Message-ID: <4DC81103.40609@fifthhorseman.net> I regularly use ssh-agent with a subcommand; my X11 session is spawned through ssh-agent, and sometimes i'll run a special agent for a certain subset of commands, like this: ssh-agent bash ... and then do work within that shell. From the man page: > If a commandline is given, this is executed as a subprocess of the agent. > When the command dies, so does the agent. But looking at the process table (and at ssh-agent.c) this isn't what actually happens. It looks like the agent lives on as the subprocess, and the subcommand is execed from the parent process. This has troublesome implications for the agent being able to detect when the command dies, since it can't rely on SIGCHLD handlers. Indeed, check_parent_exists() relies on simply storing the process ID and checking to see whether such a process is signalable. Given that process IDs tend to be relatively small numbers (15 bits on Linux), and are reused frequently, it's not hard to imagine this test giving a false positive. It's also troublesome for process supervision -- if i invoke ssh-agent and i want to ensure that i know when the agent dies (via SIGCHLD), invoking it with a subcommand will obscure the ssh-agent's dying sigchld from my supervising process. That is, the subcommand can continue living (as the supervising process' immediate child) even if the agent segfaults or terminates due to ssh-agent -k or any other reason. Is there a reason for doing things in this order? Is there interest in a patch to make the code behave as documented in the man page? --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From keisial at gmail.com Tue May 10 03:29:10 2011 From: keisial at gmail.com (=?ISO-8859-1?Q?=C1ngel_Gonz=E1lez?=) Date: Mon, 09 May 2011 19:29:10 +0200 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC81103.40609@fifthhorseman.net> References: <4DC81103.40609@fifthhorseman.net> Message-ID: <4DC82466.7060502@gmail.com> Daniel Kahn Gillmor wrote: > It looks like the agent lives on as the subprocess, and the subcommand > is execed from the parent process. > > This has troublesome implications for the agent being able to detect > when the command dies, since it can't rely on SIGCHLD handlers. Indeed, > check_parent_exists() relies on simply storing the process ID and > checking to see whether such a process is signalable. Given that > process IDs tend to be relatively small numbers (15 bits on Linux), and > are reused frequently, it's not hard to imagine this test giving a false > positive. Such premature reuse looks like an OS bug, although it could theoretically happen. Making the original ssh-agent a process group can probably avoid it, though: > > > 4.12 Process ID Reuse > > A process group ID shall not be reused by the system until the process > group lifetime ends. > > A process ID shall not be reused by the system until the process > lifetime ends. In addition, if there exists a process group whose > process group ID is equal to that process ID, the process ID shall not > be reused by the system until the process group lifetime ends. A > process that is not a system process shall not have a process ID of 1. > http://pubs.opengroup.org/onlinepubs/009604599/basedefs/xbd_chap04.html Another option to ensure it is our real parent would be to use getpid() and look if we have been reparented, although it would need to be careful with getpid() wrappers. > It's also troublesome for process supervision -- if i invoke ssh-agent > and i want to ensure that i know when the agent dies (via SIGCHLD), > invoking it with a subcommand will obscure the ssh-agent's dying sigchld > from my supervising process. That is, the subcommand can continue > living (as the supervising process' immediate child) even if the agent > segfaults or terminates due to ssh-agent -k or any other reason. > > Is there a reason for doing things in this order? Is there interest in > a patch to make the code behave as documented in the man page? I guess it's so you can supervise the "right process", as the important one would be the parameter, so you could replace transparently the called command with "ssh-agent oldcommand". It's also cleaner having the agent showing as child of bash. It's part of your bash session (although 'adopted'). From abduljabbar.vja at gmail.com Tue May 10 06:18:09 2011 From: abduljabbar.vja at gmail.com (abdul jabbar) Date: Tue, 10 May 2011 01:48:09 +0530 Subject: configure: error: *** Can't find recent OpenSSL libcrypto (see config.log for details) *** Message-ID: HI, Getting below error while trying to compile openssh-5.8p2 on *Centos 5.6_X86-64* *configure: error: *** Can't find recent OpenSSL libcrypto (see config.log for details) **** I recently compiled latest open ssl version *OpenSSL 1.0.0d 8 Feb 2011 *please help me to solve this issue. ** Thanks&Regards, Abdul Jabbar * * From dkg at fifthhorseman.net Tue May 10 06:39:27 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Mon, 09 May 2011 16:39:27 -0400 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC82466.7060502@gmail.com> References: <4DC81103.40609@fifthhorseman.net> <4DC82466.7060502@gmail.com> Message-ID: <4DC850FF.2080506@fifthhorseman.net> On 05/09/2011 01:29 PM, ?ngel Gonz?lez wrote: > Making the original ssh-agent a process group can probably avoid it, though: which process (the parent or the child) would need to be its own process group for this to work? Would there be other side effects of doing this? > I guess it's so you can supervise the "right process", as the important one > would be the parameter, so you could replace transparently the called > command with > "ssh-agent oldcommand". but in the example i gave, i actually want to supervise the agent, not the "oldcommand". and indeed, the man page suggests that i should be able to do that :/ I see what you're saying about how this could be useful in some cases, but it is explicitly not the documented behavior. > It's also cleaner having the agent showing as child of bash. It's part > of your bash session > (although 'adopted'). Why is this cleaner than the other way around? --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From phil.pennock at globnix.org Tue May 10 06:54:45 2011 From: phil.pennock at globnix.org (Phil Pennock) Date: Mon, 9 May 2011 16:54:45 -0400 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC81103.40609@fifthhorseman.net> References: <4DC81103.40609@fifthhorseman.net> Message-ID: <20110509205444.GA564@redoubt.spodhuis.org> On 2011-05-09 at 12:06 -0400, Daniel Kahn Gillmor wrote: > I regularly use ssh-agent with a subcommand; my X11 session is spawned > through ssh-agent, and sometimes i'll run a special agent for a certain > subset of commands, like this: > > ssh-agent bash > > ... and then do work within that shell. From the man page: > > > If a commandline is given, this is executed as a subprocess of the agent. > > When the command dies, so does the agent. > > But looking at the process table (and at ssh-agent.c) this isn't what > actually happens. > > It looks like the agent lives on as the subprocess, and the subcommand > is execed from the parent process. No, that's not how Unix works. A child process can't indirectly cause its parent to run something without pre-arranged RPCs to communicate application-layer instructions to do so, which I doubt your shell has. There is a common idiom to use: eval `ssh-agent -s` to set up ssh which results in the process layout you describe. This is probably happening somewhere that you just haven't tracked down yet. If you really can't track it down, mv ssh-agent to ssh-agent.real and write a wrapper-script which calls pstree, writing the output to a log-file, before exec'ing ssh-agent.real and wait for that to track it down. From keisial at gmail.com Tue May 10 07:16:51 2011 From: keisial at gmail.com (=?UTF-8?B?w4FuZ2VsIEdvbnrDoWxleg==?=) Date: Mon, 09 May 2011 23:16:51 +0200 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC850FF.2080506@fifthhorseman.net> References: <4DC81103.40609@fifthhorseman.net> <4DC82466.7060502@gmail.com> <4DC850FF.2080506@fifthhorseman.net> Message-ID: <4DC859C3.20501@gmail.com> Daniel Kahn Gillmor wrote: > On 05/09/2011 01:29 PM, ?ngel Gonz?lez wrote: >> Making the original ssh-agent a process group can probably avoid it, though: > which process (the parent or the child) would need to be its own process > group for this to work? Would there be other side effects of doing this? The parent. The parent pid would then also be a process group id, and while there is a process belonging to it (in this case the child agent), it can't be reused. I think shelles usually do that already, but there are cases with side effects, although I doubt they are common. Note that my knowledge of process groups is very limited, though. >> I guess it's so you can supervise the "right process", as the important one >> would be the parameter, so you could replace transparently the called >> command with >> "ssh-agent oldcommand". > but in the example i gave, i actually want to supervise the agent, not > the "oldcommand". and indeed, the man page suggests that i should be > able to do that :/ > > I see what you're saying about how this could be useful in some cases, > but it is explicitly not the documented behavior. Yes. Either the code or the documentation should be changed. Note that you could supervise it by running the commands in the shell. So instead of $ supervise ssh-agent bash you can do $ bash bash$ eval $(supervise ssh-agent) Or even bash -c 'eval $(supervise ssh-agent); exec bash' There's probably a way to avoid that second bash, though. >> It's also cleaner having the agent showing as child of bash. It's part >> of your bash session >> (although 'adopted'). > Why is this cleaner than the other way around? When you login into a system, the shell is the ancestor of everything. Thus it looks more natural that it is the same for ssh-agent, too. But your view may be completely different :) From dkg at fifthhorseman.net Tue May 10 07:13:33 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Mon, 09 May 2011 17:13:33 -0400 Subject: ssh-agent subprocess parentage In-Reply-To: <20110509205444.GA564@redoubt.spodhuis.org> References: <4DC81103.40609@fifthhorseman.net> <20110509205444.GA564@redoubt.spodhuis.org> Message-ID: <4DC858FD.4090703@fifthhorseman.net> On 05/09/2011 04:54 PM, Phil Pennock wrote: > No, that's not how Unix works. A child process can't indirectly cause > its parent to run something without pre-arranged RPCs to communicate > application-layer instructions to do so, which I doubt your shell has. I'm pretty sure this is actually how it works. ssh-agent forks, and is two processes concurrently. Each process (the parent and the child) gets to decide what to do next. In ssh-agent, the parent process exec()'s the subcommand, while the child process lives on as an agent. Search for "fork" in ssh-agent.c to see what i'm talking about. > There is a common idiom to use: > eval `ssh-agent -s` > to set up ssh which results in the process layout you describe. This is > probably happening somewhere that you just haven't tracked down yet. This does *not* result in the process layout i describe -- this results in a daemonized process via a forking and dying parent (that is, only the child process remains, and its parent becomes PID 1). This arrangement is even harder to supervise properly. --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From dkg at fifthhorseman.net Tue May 10 07:22:33 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Mon, 09 May 2011 17:22:33 -0400 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC859C3.20501@gmail.com> References: <4DC81103.40609@fifthhorseman.net> <4DC82466.7060502@gmail.com> <4DC850FF.2080506@fifthhorseman.net> <4DC859C3.20501@gmail.com> Message-ID: <4DC85B19.7070108@fifthhorseman.net> On 05/09/2011 05:16 PM, ?ngel Gonz?lez wrote: > Note that you could supervise it by running the commands in the shell. > So instead of > $ supervise ssh-agent bash > you can do > $ bash > bash$ eval $(supervise ssh-agent) > > Or even > bash -c 'eval $(supervise ssh-agent); exec bash' so yes -- i can certainly supervise it if i don't supply a subcommand at all; but in this case, i want it to run a subcommand. For concreteness' sake, imagine a daemon that stays in the foreground, but needs to talk to an ssh-agent which handles its keys. If either the agent or the daemon dies, you'd like the supervising process to restart the whole chain. With the documented behavior, this would work fine. With the existing actual behavior, if the agent dies (for whatever reason), the supervising process has no way of knowing. > When you login into a system, the shell is the ancestor of everything. > Thus it looks > more natural that it is the same for ssh-agent, too. But your view may > be completely > different :) i'd expect that the first process invoked is the ancestor of everything, which in this case would be ssh-agent :) --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From keisial at gmail.com Tue May 10 07:58:27 2011 From: keisial at gmail.com (=?ISO-8859-1?Q?=C1ngel_Gonz=E1lez?=) Date: Mon, 09 May 2011 23:58:27 +0200 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC85B19.7070108@fifthhorseman.net> References: <4DC81103.40609@fifthhorseman.net> <4DC82466.7060502@gmail.com> <4DC850FF.2080506@fifthhorseman.net> <4DC859C3.20501@gmail.com> <4DC85B19.7070108@fifthhorseman.net> Message-ID: <4DC86383.5010503@gmail.com> Daniel Kahn Gillmor wrote: > On 05/09/2011 05:16 PM, ?ngel Gonz?lez wrote: >> Note that you could supervise it by running the commands in the shell. >> So instead of >> $ supervise ssh-agent bash >> you can do >> $ bash >> bash$ eval $(supervise ssh-agent) >> >> Or even >> bash -c 'eval $(supervise ssh-agent); exec bash' > so yes -- i can certainly supervise it if i don't supply a subcommand at > all; but in this case, i want it to run a subcommand. > > For concreteness' sake, imagine a daemon that stays in the foreground, > but needs to talk to an ssh-agent which handles its keys. If either the > agent or the daemon dies, you'd like the supervising process to restart > the whole chain. > > With the documented behavior, this would work fine. > > With the existing actual behavior, if the agent dies (for whatever > reason), the supervising process has no way of knowing. What about having the supervisor as parent of both the agent and the daemon ? From keisial at gmail.com Tue May 10 08:14:23 2011 From: keisial at gmail.com (=?UTF-8?B?w4FuZ2VsIEdvbnrDoWxleg==?=) Date: Tue, 10 May 2011 00:14:23 +0200 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC859C3.20501@gmail.com> References: <4DC81103.40609@fifthhorseman.net> <4DC82466.7060502@gmail.com> <4DC850FF.2080506@fifthhorseman.net> <4DC859C3.20501@gmail.com> Message-ID: <4DC8673F.5080500@gmail.com> ?ngel Gonz?lez wrote: > Daniel Kahn Gillmor wrote: >> On 05/09/2011 01:29 PM, ?ngel Gonz?lez wrote: >>> Making the original ssh-agent a process group can probably avoid it, though: >> which process (the parent or the child) would need to be its own process >> group for this to work? Would there be other side effects of doing this? > The parent. The parent pid would then also be a process group id, and while > there is a process belonging to it (in this case the child agent), it > can't be reused. > I think shelles usually do that already, but there are cases with side > effects, although > I doubt they are common. > Note that my knowledge of process groups is very limited, though. Actually, there is a non-portable way in Linux to get a when your parent dies. I had been looking for it in clone(2) from your previous mail, but it actually lives at prctl(2) prctl(PR_SET_PDEATHSIG,) I could look into using it under Linux if there's interest on it. From dkg at fifthhorseman.net Tue May 10 08:21:42 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Mon, 09 May 2011 18:21:42 -0400 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC86383.5010503@gmail.com> References: <4DC81103.40609@fifthhorseman.net> <4DC82466.7060502@gmail.com> <4DC850FF.2080506@fifthhorseman.net> <4DC859C3.20501@gmail.com> <4DC85B19.7070108@fifthhorseman.net> <4DC86383.5010503@gmail.com> Message-ID: <4DC868F6.5030305@fifthhorseman.net> On 05/09/2011 05:58 PM, ?ngel Gonz?lez wrote: > What about having the supervisor as parent of both the agent and the > daemon ? Because that would mean writing and maintaining my own supervision suite :) I'm using runit at the moment, and runsv supervises a single process quite well without any tweaking on my behalf, and it works as documented. --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From dkg at fifthhorseman.net Tue May 10 08:23:27 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Mon, 09 May 2011 18:23:27 -0400 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC8673F.5080500@gmail.com> References: <4DC81103.40609@fifthhorseman.net> <4DC82466.7060502@gmail.com> <4DC850FF.2080506@fifthhorseman.net> <4DC859C3.20501@gmail.com> <4DC8673F.5080500@gmail.com> Message-ID: <4DC8695F.50502@fifthhorseman.net> On 05/09/2011 06:14 PM, ?ngel Gonz?lez wrote: > Actually, there is a non-portable way in Linux to get a when your parent > dies. > I had been looking for it in clone(2) from your previous mail, but it > actually > lives at prctl(2) > prctl(PR_SET_PDEATHSIG,) > > I could look into using it under Linux if there's interest on it. Ah, nice find! That would certainly help the concern that an ssh-agent process would linger on Linux (maybe similar approaches are available for other kernels?), but it doesn't resolve the process supervision concern, does it? --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From keisial at gmail.com Tue May 10 08:51:28 2011 From: keisial at gmail.com (=?ISO-8859-1?Q?=C1ngel_Gonz=E1lez?=) Date: Tue, 10 May 2011 00:51:28 +0200 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC8695F.50502@fifthhorseman.net> References: <4DC81103.40609@fifthhorseman.net> <4DC82466.7060502@gmail.com> <4DC850FF.2080506@fifthhorseman.net> <4DC859C3.20501@gmail.com> <4DC8673F.5080500@gmail.com> <4DC8695F.50502@fifthhorseman.net> Message-ID: <4DC86FF0.8020302@gmail.com> Daniel Kahn Gillmor wrote: > On 05/09/2011 06:14 PM, ?ngel Gonz?lez wrote: >> Actually, there is a non-portable way in Linux to get a when your parent >> dies. >> I had been looking for it in clone(2) from your previous mail, but it >> actually >> lives at prctl(2) >> prctl(PR_SET_PDEATHSIG,) >> >> I could look into using it under Linux if there's interest on it. > Ah, nice find! That would certainly help the concern that an ssh-agent > process would linger on Linux (maybe similar approaches are available > for other kernels?), but it doesn't resolve the process supervision > concern, does it? > > --dkg No. It only applies to the "lingering" case, where the parent id is reused before the agent notices with a long-lived process owned by the same user. From dtucker at zip.com.au Tue May 10 09:26:10 2011 From: dtucker at zip.com.au (Darren Tucker) Date: Tue, 10 May 2011 09:26:10 +1000 Subject: configure: error: *** Can't find recent OpenSSL libcrypto (see config.log for details) *** In-Reply-To: References: Message-ID: On Tue, May 10, 2011 at 6:18 AM, abdul jabbar wrote: [...] > ?*configure: error: *** Can't find recent OpenSSL libcrypto (see config.log > for details) **** What did you find when you looking in config.log for details? -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4? 37C9 C982 80C7 8FF4 FA69 ? ? Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement. From kb at open.ch Tue May 10 17:58:52 2011 From: kb at open.ch (Konrad Bucheli) Date: Tue, 10 May 2011 09:58:52 +0200 Subject: new option ssh-add -v to verify if key is loaded into the agent In-Reply-To: <20110506144215.GA11339@folly> References: <4D9D63C8.6060909@open.ch> <20110506144215.GA11339@folly> Message-ID: <4DC8F03C.10005@open.ch> Hi Thanks for the input. I have now a few questions: > + char data[1024]; > + arc4random_buf(data, sizeof(data)); You use a long pseudo-random token for to verify the private key, while my patch uses a short (20 bytes) random token. What is recommended? > + fprintf(stderr, " -T pubkey Test if ssh-agent can access matching private key.\n"); I think this documentation goes more to the point. Still the verb to use is open: -T for test -C for check -v for verify -V for verify I guess verify is the word with the least general meaning. And if we put a capital V, it will not be mistaken as "version". Any opinions? Cheers Konrad -- konrad bucheli security engineer open systems ag raeffelstrasse 29 ch-8045 zurich t: +41 44 455 74 00 f: +41 44 455 74 01 kb at open.ch http://www.open.ch From phil.pennock at globnix.org Tue May 10 20:10:14 2011 From: phil.pennock at globnix.org (Phil Pennock) Date: Tue, 10 May 2011 06:10:14 -0400 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC858FD.4090703@fifthhorseman.net> References: <4DC81103.40609@fifthhorseman.net> <20110509205444.GA564@redoubt.spodhuis.org> <4DC858FD.4090703@fifthhorseman.net> Message-ID: <20110510101013.GA21754@redoubt.spodhuis.org> On 2011-05-09 at 17:13 -0400, Daniel Kahn Gillmor wrote: > I'm pretty sure this is actually how it works. ssh-agent forks, and is > two processes concurrently. Each process (the parent and the child) > gets to decide what to do next. In ssh-agent, the parent process > exec()'s the subcommand, while the child process lives on as an agent. I misinterpreted which process you meant when you said parent. I apologise. Curiously, [gpg-agent --daemon cmd] does the exact same thing. Safety measure so that if the agent dies you don't get left with a login program reaping its child and resetting the tty while the shell is still active? ssh to localhost, then run a second shell, in that kill -9 $PPID, see yourself disconnected while the shell you were typing into is still running. -Phil From lee.hambley at gmail.com Wed May 11 00:24:02 2011 From: lee.hambley at gmail.com (Lee Hambley) Date: Tue, 10 May 2011 16:24:02 +0200 Subject: Rationale for an absence of a native file-logging configuration option for OpenSSH-Server Message-ID: Good afternoon, I wanted to ask quickly about the rationale for not allowing explicit declaration of a log file to receive logging output. Whilst using syslogd is convenient for system processes, for short-lived applications, such as booting an SSH server (and wishing to monitor it) for testing purposes? having to also configure syslogd to receive output is inconvenient. I ask because it seems logical to allow either a native file, or a syslog type system to be used, and the absence of a native-log-file directive in the configuration is inconvenient for me. Sincerely yours, Lee Hambley From dkg at fifthhorseman.net Wed May 11 00:32:34 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Tue, 10 May 2011 10:32:34 -0400 Subject: Rationale for an absence of a native file-logging configuration option for OpenSSH-Server In-Reply-To: References: Message-ID: <4DC94C82.4060407@fifthhorseman.net> On 05/10/2011 10:24 AM, Lee Hambley wrote: > Good afternoon, I wanted to ask quickly about the rationale for not allowing > explicit declaration of a log file to receive logging output. Whilst using > syslogd is convenient for system processes, for short-lived applications, > such as booting an SSH server (and wishing to monitor it) for testing > purposes? having to also configure syslogd to receive output is > inconvenient. You can use sshd's -e option to send logs to stderr, while mapping the stderr file descriptor (fd 2) to whatever native file (or process) you like. Does this solve your problem? Note that if you plan to rotate your logs, you may need to restart sshd at the appropriate time. hth, --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From lee.hambley at gmail.com Wed May 11 00:40:44 2011 From: lee.hambley at gmail.com (Lee Hambley) Date: Tue, 10 May 2011 16:40:44 +0200 Subject: Rationale for an absence of a native file-logging configuration option for OpenSSH-Server In-Reply-To: <4DC94C82.4060407@fifthhorseman.net> References: <4DC94C82.4060407@fifthhorseman.net> Message-ID: > > You can use sshd's -e option to send logs to stderr, while mapping the > stderr file descriptor (fd 2) to whatever native file (or process) you > like. > I hadn't considered that, interesting point? > Does this solve your problem? > I expect so, I'm working on Ruby bindings for libssh, and the server is booted by a Ruby package "Daemon Controller", DaemonConrtoller requires only that there be a logfile for it to check for errors? > Note that if you plan to rotate your logs, you may need to restart sshd > at the appropriate time. > Not an issue here, but thanks for keeping it in mind! For what it's worth, and if anyone is interested in my project, contributing or following - here's the link: https://github.com/leehambley/libssh.rb and specifically my server wrapper implementation https://github.com/leehambley/libssh.rb/blob/master/test/support/ssh_daemon.rb From peter at stuge.se Wed May 11 00:44:24 2011 From: peter at stuge.se (Peter Stuge) Date: Tue, 10 May 2011 16:44:24 +0200 Subject: Rationale for an absence of a native file-logging configuration option for OpenSSH-Server In-Reply-To: References: <4DC94C82.4060407@fifthhorseman.net> Message-ID: <20110510144424.25763.qmail@stuge.se> Lee Hambley wrote: > I expect so, I'm working on Ruby bindings for libssh, and the server > is booted by a Ruby package "Daemon Controller", DaemonConrtoller > requires only that there be a logfile for it to check for errors? Hmm. Note that libssh is a separate project, and it inclues both client an server. How does your work relate to OpenSSH? //Peter From keisial at gmail.com Wed May 11 01:12:02 2011 From: keisial at gmail.com (=?ISO-8859-1?Q?=C1ngel_Gonz=E1lez?=) Date: Tue, 10 May 2011 17:12:02 +0200 Subject: Rationale for an absence of a native file-logging configuration option for OpenSSH-Server In-Reply-To: <4DC94C82.4060407@fifthhorseman.net> References: <4DC94C82.4060407@fifthhorseman.net> Message-ID: <4DC955C2.7080700@gmail.com> Daniel Kahn Gillmor wrote: > You can use sshd's -e option to send logs to stderr, while mapping the > stderr file descriptor (fd 2) to whatever native file (or process) you like. > > Does this solve your problem? > > Note that if you plan to rotate your logs, you may need to restart sshd > at the appropriate time. > > hth, > > --dkg I don't think that's enough. I was recently in such situation, and the logging stopped after a few lines. Seemed that after forking the child closed the descriptor. So the real problem, that happened later, was not being recorded. From dkg at fifthhorseman.net Wed May 11 01:13:59 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Tue, 10 May 2011 11:13:59 -0400 Subject: ssh-agent subprocess parentage In-Reply-To: <20110510101013.GA21754@redoubt.spodhuis.org> References: <4DC81103.40609@fifthhorseman.net> <20110509205444.GA564@redoubt.spodhuis.org> <4DC858FD.4090703@fifthhorseman.net> <20110510101013.GA21754@redoubt.spodhuis.org> Message-ID: <4DC95637.4090904@fifthhorseman.net> On 05/10/2011 06:10 AM, Phil Pennock wrote: > Safety measure so that if the agent dies you don't get left with a login > program reaping its child and resetting the tty while the shell is still > active? ssh to localhost, then run a second shell, in that kill -9 > $PPID, see yourself disconnected while the shell you were typing into is > still running. Yep, i can see how it's useful in this context. But i still think the documented behavior would be useful in other contexts (and it provides a standard, cross-platform way to actually be sure that the subprocess actually dies). It seems like one of three things should happen: 0) the documentation should be changed to describe the actual behavior 1) the behavior should change to match the documentation 2) both behaviors should be available (since there seem to be reasonable arguments for either use) -- i'm not sure how the user would choose between them, though. --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From keisial at gmail.com Wed May 11 01:43:49 2011 From: keisial at gmail.com (=?ISO-8859-1?Q?=C1ngel_Gonz=E1lez?=) Date: Tue, 10 May 2011 17:43:49 +0200 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC95637.4090904@fifthhorseman.net> References: <4DC81103.40609@fifthhorseman.net> <20110509205444.GA564@redoubt.spodhuis.org> <4DC858FD.4090703@fifthhorseman.net> <20110510101013.GA21754@redoubt.spodhuis.org> <4DC95637.4090904@fifthhorseman.net> Message-ID: <4DC95D35.1090102@gmail.com> Daniel Kahn Gillmor wrote: > 2) both behaviors should be available (since there seem to be > reasonable arguments for either use) -- i'm not sure how the user would > choose between them, though. A command line switch seems the obvious way. --execute-parent / --execute-child perhaps. From jmknoble at pobox.com Wed May 11 02:56:07 2011 From: jmknoble at pobox.com (Jim Knoble) Date: Tue, 10 May 2011 09:56:07 -0700 Subject: Rationale for an absence of a native file-logging configuration option for OpenSSH-Server In-Reply-To: <4DC955C2.7080700@gmail.com> References: <4DC94C82.4060407@fifthhorseman.net> <4DC955C2.7080700@gmail.com> Message-ID: <4DC96E27.5020701@pobox.com> You need sshd -D -e. For log rotation without restarting sshd, use something like http://cr.yp.to/daemontools/multilog.html : mkdir /var/log/sshd sshd -D -e 2>&1 |multilog t /var/log/sshd is a simple example to send sshd messages to /var/log/sshd/current. -- jim knoble | jmknoble at pobox.com | http://www.pobox.com/~jmknoble/ On 2011-05-10 08:12, ?ngel Gonz?lez wrote: > Daniel Kahn Gillmor wrote: >> You can use sshd's -e option to send logs to stderr, while mapping the >> stderr file descriptor (fd 2) to whatever native file (or process) you like. >> >> Does this solve your problem? >> >> Note that if you plan to rotate your logs, you may need to restart sshd >> at the appropriate time. >> >> hth, >> >> --dkg > I don't think that's enough. I was recently in such situation, and the > logging > stopped after a few lines. Seemed that after forking the child closed > the descriptor. > So the real problem, that happened later, was not being recorded. > > > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev > From djm at mindrot.org Wed May 11 14:47:46 2011 From: djm at mindrot.org (Damien Miller) Date: Wed, 11 May 2011 14:47:46 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> Message-ID: On Mon, 9 May 2011, Rado S wrote: > Hi devs, > > recently I had to replace authorized_keys on several systems to > enforce an access policy change. > I was badly surprised that authorized_keys2(!) was still processed, > which allowed some old keys to enter the systems again, because I > wasn't aware of the file's existance on the server and use by sshd, > since this "backward compatibility" isn't documented, not even a > historical reference about "obsolete" or "deprecated". > > Maybe it's time to drop the old stuff not to get haunted by such > leftovers again. Good point - I just committed a change to remove it for openssh-5.9 -d From dan at doxpara.com Wed May 11 16:01:14 2011 From: dan at doxpara.com (Dan Kaminsky) Date: Tue, 10 May 2011 23:01:14 -0700 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> Message-ID: <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> Sent from my iPhone On May 10, 2011, at 9:47 PM, Damien Miller wrote: > On Mon, 9 May 2011, Rado S wrote: > >> Hi devs, >> >> recently I had to replace authorized_keys on several systems to >> enforce an access policy change. >> I was badly surprised that authorized_keys2(!) was still processed, >> which allowed some old keys to enter the systems again, because I >> wasn't aware of the file's existance on the server and use by sshd, >> since this "backward compatibility" isn't documented, not even a >> historical reference about "obsolete" or "deprecated". >> >> Maybe it's time to drop the old stuff not to get haunted by such >> leftovers again. > > Good point - I just committed a change to remove it for openssh-5.9 > I'd document, rather than remove. I think all my systems use authorized_keys2. You will end up locking users and admins out. > -d > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev From jrollins at finestructure.net Wed May 11 16:23:30 2011 From: jrollins at finestructure.net (Jameson Graef Rollins) Date: Tue, 10 May 2011 23:23:30 -0700 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> Message-ID: <87wrhxiwm5.fsf@servo.factory.finestructure.net> On Tue, 10 May 2011 23:01:14 -0700, Dan Kaminsky wrote: > I'd document, rather than remove. I think all my systems use > authorized_keys2. You will end up locking users and admins out. I definitely agree with this sentiment. I also think that being able to specify multiple authorized_keys files is very useful, so I would prefer to just see this as a documented feature. jamie. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From efo at basefarm.no Wed May 11 16:42:34 2011 From: efo at basefarm.no (=?ISO-8859-1?Q?Espen_Fjellv=E6r_Olsen?=) Date: Wed, 11 May 2011 08:42:34 +0200 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <87wrhxiwm5.fsf@servo.factory.finestructure.net> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> Message-ID: <4DCA2FDA.8090806@basefarm.no> On 11. mai 2011 08:23, Jameson Graef Rollins wrote: > On Tue, 10 May 2011 23:01:14 -0700, Dan Kaminsky wrote: >> I'd document, rather than remove. I think all my systems use >> authorized_keys2. You will end up locking users and admins out. > I definitely agree with this sentiment. > > I also think that being able to specify multiple authorized_keys files > is very useful, so I would prefer to just see this as a documented > feature. > > jamie. I say either remove it, or make it a configuration option to disable it. Where authorized_keys are controlled by the AuthorizedKeysFile option, authorized_keys2 are not, which makes our distribution regimes a bit troublesome as we will have to make use of /etc/ssh/sshrc to delete/die/remove/something if %h/.ssh/authorized_keys2 is found. -- BR Espen Fjellv?r Olsen Basefarm AS From djm at mindrot.org Wed May 11 18:44:59 2011 From: djm at mindrot.org (Damien Miller) Date: Wed, 11 May 2011 18:44:59 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> Message-ID: On Tue, 10 May 2011, Dan Kaminsky wrote: > >> Maybe it's time to drop the old stuff not to get haunted by such > >> leftovers again. > > > > Good point - I just committed a change to remove it for openssh-5.9 > > > > I'd document, rather than remove. I think all my systems use > authorized_keys2. You will end up locking users and admins out. We'll document the removal :) Really, there is no reason to have two files that do exactly the same thing. -d From philipp.marek at linbit.com Wed May 11 18:52:36 2011 From: philipp.marek at linbit.com (Philipp Marek) Date: Wed, 11 May 2011 10:52:36 +0200 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> Message-ID: <201105111052.37253.philipp.marek@linbit.com> On Wednesday 11 May 2011, Damien Miller wrote: > On Tue, 10 May 2011, Dan Kaminsky wrote: > > >> Maybe it's time to drop the old stuff not to get haunted by such > > >> leftovers again. > > > > > > Good point - I just committed a change to remove it for openssh-5.9 > > > > I'd document, rather than remove. I think all my systems use > > authorized_keys2. You will end up locking users and admins out. > > We'll document the removal :) Really, there is no reason to have two > files that do exactly the same thing. Well, there is a very good reason - easier configurability. Having one file for the "static" admins, and one for the per-server (application) executives is nice, IMO. There are lots of places where instead of a file a directory is used - the famous /etc/rc*.d/, /etc/cron.d, etc. etc. Perhaps this should be an alternative - either have ~/.ssh/authorized_keys a file, or a directory, but not both? Regards, Phil From jrollins at finestructure.net Wed May 11 19:58:55 2011 From: jrollins at finestructure.net (Jameson Graef Rollins) Date: Wed, 11 May 2011 02:58:55 -0700 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> Message-ID: <87r585imn4.fsf@servo.factory.finestructure.net> On Wed, 11 May 2011 18:44:59 +1000 (EST), Damien Miller wrote: > > I'd document, rather than remove. I think all my systems use > > authorized_keys2. You will end up locking users and admins out. > > We'll document the removal :) Really, there is no reason to have two > files that do exactly the same thing. Actually, there are a lot of reasons to have multiple authorized_keys files. One user controlled and one admin controlled is just the first thing that pops in to my head. I'm sure we can think of lots of other reasons as well. jamie. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From djm at mindrot.org Wed May 11 21:48:38 2011 From: djm at mindrot.org (Damien Miller) Date: Wed, 11 May 2011 21:48:38 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <87wrhxiwm5.fsf@servo.factory.finestructure.net> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> Message-ID: On Tue, 10 May 2011, Jameson Graef Rollins wrote: > On Tue, 10 May 2011 23:01:14 -0700, Dan Kaminsky wrote: > > I'd document, rather than remove. I think all my systems use > > authorized_keys2. You will end up locking users and admins out. > > I definitely agree with this sentiment. > > I also think that being able to specify multiple authorized_keys files > is very useful, so I would prefer to just see this as a documented > feature. Perhaps we should make options.authorized_keys_file an array to let people who want to use multiple files do so. -d From dkg at fifthhorseman.net Thu May 12 00:44:17 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Wed, 11 May 2011 10:44:17 -0400 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> Message-ID: <4DCAA0C1.7060505@fifthhorseman.net> On 05/11/2011 07:48 AM, Damien Miller wrote: > Perhaps we should make options.authorized_keys_file an array to let > people who want to use multiple files do so. There are at least two bugzilla issues open about this: https://bugzilla.mindrot.org/show_bug.cgi?id=172 https://bugzilla.mindrot.org/show_bug.cgi?id=1684 Either way, the current arrangement is pretty counter-intuitive, so i'm glad something is being done to resolve it. In the current setup, IIRC, if AuthorizedKeysFile is not set, then both ~/.authorized_keys and ~/.authorized_keys2 are parsed. But if AuthorizedKeysFile *is* set, then that and only that is parsed (no ~/.authorized_keys2). This is confusing/surprising no matter which direction you're coming from. Thanks for addressing the issue, --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From imorgan at nas.nasa.gov Thu May 12 02:50:08 2011 From: imorgan at nas.nasa.gov (Iain Morgan) Date: Wed, 11 May 2011 09:50:08 -0700 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> Message-ID: <20110511165008.GB492@linux124.nas.nasa.gov> On Wed, May 11, 2011 at 06:48:38 -0500, Damien Miller wrote: > On Tue, 10 May 2011, Jameson Graef Rollins wrote: > > > On Tue, 10 May 2011 23:01:14 -0700, Dan Kaminsky wrote: > > > I'd document, rather than remove. I think all my systems use > > > authorized_keys2. You will end up locking users and admins out. > > > > I definitely agree with this sentiment. > > > > I also think that being able to specify multiple authorized_keys files > > is very useful, so I would prefer to just see this as a documented > > feature. > > Perhaps we should make options.authorized_keys_file an array to let > people who want to use multiple files do so. > > -d > I was going to suggest something similar, but you beat me to it. :-) One scenario that could potentially be useful in a cluster environment would be to allow per-host authorized_keys files. Support for the following syntax might be useful: AuthorizedKeysFile %h/.ssh/authorized_keys.%H,%h/.ssh/authorized_keys where '%H' would be expanded as the server's hostname. (I don't particulary like '%H', but '%h' is already used.) This would allow clusters which use a shared home filesystem to have authorized_keys files which are tailored for a specific host and the capability to fall back to a more generic file in the absence of a host-specific one. By the way, I applaud getting rid of the old cruft. -- Iain Morgan From keisial at gmail.com Thu May 12 03:24:16 2011 From: keisial at gmail.com (=?ISO-8859-1?Q?=C1ngel_Gonz=E1lez?=) Date: Wed, 11 May 2011 19:24:16 +0200 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <20110511165008.GB492@linux124.nas.nasa.gov> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <20110511165008.GB492@linux124.nas.nasa.gov> Message-ID: <4DCAC640.7020606@gmail.com> Iain Morgan wrote: > I was going to suggest something similar, but you beat me to it. :-) > > One scenario that could potentially be useful in a cluster environment > would be to allow per-host authorized_keys files. Support for the > following syntax might be useful: > > AuthorizedKeysFile %h/.ssh/authorized_keys.%H,%h/.ssh/authorized_keys > > where '%H' would be expanded as the server's hostname. (I don't > particulary like '%H', but '%h' is already used.) > > This would allow clusters which use a shared home filesystem to have > authorized_keys files which are tailored for a specific host and the > capability to fall back to a more generic file in the absence of a > host-specific one. > > By the way, I applaud getting rid of the old cruft. To fall back? As I understood it, they would be additive. From imorgan at nas.nasa.gov Thu May 12 04:20:30 2011 From: imorgan at nas.nasa.gov (Iain Morgan) Date: Wed, 11 May 2011 11:20:30 -0700 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <4DCAC640.7020606@gmail.com> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <20110511165008.GB492@linux124.nas.nasa.gov> <4DCAC640.7020606@gmail.com> Message-ID: <20110511182030.GC492@linux124.nas.nasa.gov> On Wed, May 11, 2011 at 12:24:16 -0500, ?ngel Gonz?lez wrote: > Iain Morgan wrote: > > I was going to suggest something similar, but you beat me to it. :-) > > > > One scenario that could potentially be useful in a cluster environment > > would be to allow per-host authorized_keys files. Support for the > > following syntax might be useful: > > > > AuthorizedKeysFile %h/.ssh/authorized_keys.%H,%h/.ssh/authorized_keys > > > > where '%H' would be expanded as the server's hostname. (I don't > > particulary like '%H', but '%h' is already used.) > > > > This would allow clusters which use a shared home filesystem to have > > authorized_keys files which are tailored for a specific host and the > > capability to fall back to a more generic file in the absence of a > > host-specific one. > > > > By the way, I applaud getting rid of the old cruft. > To fall back? As I understood it, they would be additive. > By "fall back," I didn't necessarily mean "stop at the first file," but there might be some scenarios where that behaviour is desirable. However, most of the discussion on this list has been with the expectation that all files would be examined. -- Iain Morgan From ward at gnu.org Thu May 12 04:29:40 2011 From: ward at gnu.org (Ward Vandewege) Date: Wed, 11 May 2011 14:29:40 -0400 Subject: more user friendly OpenSSH client error messages? Message-ID: <20110511182940.GA7068@countzero.vandewege.net> Hi all, RMS would like to see more user friendly OpenSSH client error messages. Specifically, he would like to see the "permission denied" error messages easier to understand. If one tries to log into a machine that is configured to only accept publickey authentication and the key doesn't match, the client output is $ ssh user at server Permission denied (publickey). If the server also supports password authentication, the output is: $ ssh user at server user at server's password: Permission denied, please try again. user at server's password: Permission denied, please try again. user at server's password: Permission denied (publickey,password). How about rewording the error message to something like this: Access denied: the server did not accept your [ssh key/password/ssh key or password] for user $username. Thanks, Ward. -- Ward Vandewege Free Software Foundation - Senior Systems Administrator From cyrille.lefevre-lists at laposte.net Thu May 12 08:19:03 2011 From: cyrille.lefevre-lists at laposte.net (Cyrille Lefevre) Date: Thu, 12 May 2011 00:19:03 +0200 Subject: ssh command line option processing behaviour Message-ID: <4DCB0B57.6040008@laposte.net> Hi, usualy, in the unix world, while processing command line options, unix programs keep the last option value, overriding config file options or previous command line ones, unfortunately, ssh only keep the first option processed. ex. : ssh -o BatchMode=no -o BatchMode=yes host cmd arg... the unix common expected behaviour is to have BatchMode = true, while ssh says BatchMode = false. how about to change the ssh command line processing bahaviour ? Regards, Cyrille Lefevre -- mailto:Cyrille.Lefevre-lists at laposte.net From djm at mindrot.org Thu May 12 11:19:13 2011 From: djm at mindrot.org (Damien Miller) Date: Thu, 12 May 2011 11:19:13 +1000 (EST) Subject: ssh command line option processing behaviour In-Reply-To: <4DCB0B57.6040008@laposte.net> References: <4DCB0B57.6040008@laposte.net> Message-ID: On Thu, 12 May 2011, Cyrille Lefevre wrote: > ssh -o BatchMode=no -o BatchMode=yes host cmd arg... > > the unix common expected behaviour is to have BatchMode = true, while > ssh says BatchMode = false. > > how about to change the ssh command line processing bahaviour ? No. The command-line behaviour is consistent with the configuration file parsing logic and is widely depended upon. The "common expected behaviour" is hardly universal; while many programs do it, others will detect the inconsistency fatally error and some will do the wrong thing. -d From mfriedl at gmail.com Fri May 13 04:49:07 2011 From: mfriedl at gmail.com (Markus Friedl) Date: Thu, 12 May 2011 20:49:07 +0200 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> Message-ID: looks like we've been waiting too long :) http://www.openssh.com/txt/release-3.0 2) The files /etc/ssh_known_hosts2 ~/.ssh/known_hosts2 ~/.ssh/authorized_keys2 are now obsolete, you can use /etc/ssh_known_hosts ~/.ssh/known_hosts ~/.ssh/authorized_keys For backward compatibility ~/.ssh/authorized_keys2 will still used for authentication and hostkeys are still read from the known_hosts2. However, those deprecated files are considered 'readonly'. Future releases are likely not to read these files. On Mittwoch, 11. Mai 2011 at 08:01, Dan Kaminsky wrote: > > > Sent from my iPhone > > On May 10, 2011, at 9:47 PM, Damien Miller wrote: > > > On Mon, 9 May 2011, Rado S wrote: > > > > > Hi devs, > > > > > > recently I had to replace authorized_keys on several systems to > > > enforce an access policy change. > > > I was badly surprised that authorized_keys2(!) was still processed, > > > which allowed some old keys to enter the systems again, because I > > > wasn't aware of the file's existance on the server and use by sshd, > > > since this "backward compatibility" isn't documented, not even a > > > historical reference about "obsolete" or "deprecated". > > > > > > Maybe it's time to drop the old stuff not to get haunted by such > > > leftovers again. > > > > Good point - I just committed a change to remove it for openssh-5.9 > > I'd document, rather than remove. I think all my systems use authorized_keys2. You will end up locking users and admins out. > > > -d > > _______________________________________________ > > openssh-unix-dev mailing list > > openssh-unix-dev at mindrot.org > > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev > _______________________________________________ > openssh-unix-dev mailing list > openssh-unix-dev at mindrot.org > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev > From dan at doxpara.com Fri May 13 05:14:02 2011 From: dan at doxpara.com (Dan Kaminsky) Date: Thu, 12 May 2011 12:14:02 -0700 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> Message-ID: On Thu, May 12, 2011 at 11:49 AM, Markus Friedl wrote: > looks like we've been waiting too long :) > > http://www.openssh.com/txt/release-3.0 > > 2) The files > /etc/ssh_known_hosts2 > ~/.ssh/known_hosts2 > ~/.ssh/authorized_keys2 > are now obsolete, you can use > /etc/ssh_known_hosts > ~/.ssh/known_hosts > ~/.ssh/authorized_keys > For backward compatibility ~/.ssh/authorized_keys2 will still used for > authentication and hostkeys are still read from the known_hosts2. > However, those deprecated files are considered 'readonly'. Future > releases are likely not to read these files. > In no uncertain terms, removal of authorized_keys2 support will cause outages, up to and including requiring physical access for administrators to resolve. Documentation is not an excuse to make this change. It's completely reasonable, desirable even, to allow a new configuration option to explicitly define the set of files that can contain authorized keys. It'd even be convenient to have an AuthorizationCommand option, that sent properly escaped strings to a command for external testing and validation. > > On Mittwoch, 11. Mai 2011 at 08:01, Dan Kaminsky wrote: > > > > > > Sent from my iPhone > > > > On May 10, 2011, at 9:47 PM, Damien Miller wrote: > > > > > On Mon, 9 May 2011, Rado S wrote: > > > > > > > Hi devs, > > > > > > > > recently I had to replace authorized_keys on several systems to > > > > enforce an access policy change. > > > > I was badly surprised that authorized_keys2(!) was still processed, > > > > which allowed some old keys to enter the systems again, because I > > > > wasn't aware of the file's existance on the server and use by sshd, > > > > since this "backward compatibility" isn't documented, not even a > > > > historical reference about "obsolete" or "deprecated". > > > > > > > > Maybe it's time to drop the old stuff not to get haunted by such > > > > leftovers again. > > > > > > Good point - I just committed a change to remove it for openssh-5.9 > > > > I'd document, rather than remove. I think all my systems use > authorized_keys2. You will end up locking users and admins out. > > > > > -d > > > _______________________________________________ > > > openssh-unix-dev mailing list > > > openssh-unix-dev at mindrot.org > > > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev > > _______________________________________________ > > openssh-unix-dev mailing list > > openssh-unix-dev at mindrot.org > > https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev > > > > From apb at cequrux.com Fri May 13 05:19:30 2011 From: apb at cequrux.com (Alan Barrett) Date: Thu, 12 May 2011 21:19:30 +0200 Subject: ssh-agent subprocess parentage In-Reply-To: <4DC81103.40609@fifthhorseman.net> References: <4DC81103.40609@fifthhorseman.net> Message-ID: <20110512191930.GA1625@apb-laptoy.apb.alt.za> On Mon, 09 May 2011, Daniel Kahn Gillmor wrote: >check_parent_exists() relies on simply storing the process ID and >checking to see whether such a process is signalable. I reported a bug around that code in April 2006. See for the problem and a patch that still works for me. --apb (Alan Barrett) From dkg at fifthhorseman.net Fri May 13 05:26:24 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Thu, 12 May 2011 15:26:24 -0400 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> Message-ID: <4DCC3460.5010800@fifthhorseman.net> On 05/12/2011 03:14 PM, Dan Kaminsky wrote: > It's completely reasonable, desirable even, to allow a new configuration > option to explicitly define the set of files that can contain authorized > keys. This doesn't need to be a "new configuration option", since it's what AuthorizedKeysFile does right now. I think if you define an AuthorizedKeysFile, then ~/.ssh/authorized_keys2 is no longer checked. The proposal to make AuthorizedKeysFile an array would allow for saying "the default is ~/.ssh/authorized_keys,~/.ssh/authorized_keys2" if we wanted to permanently enshrine the current behavior and keep a simple explanation. > It'd even be convenient to have an AuthorizationCommand option, that > sent properly escaped strings to a command for external testing and > validation. There is an outstanding proposal for AuthorizedKeysCommand which works slightly differently: https://bugzilla.mindrot.org/show_bug.cgi?id=1663 The command itself just produces the equivalent of an AuthorizedKeysFile on stdout, which is read by sshd in the same way that it currently reads the AuthorizedKeysFile. An alternate implementation (e.g. one that feeds the submitted key (properly escaped) to the command and checks the return value) would be a welcome contribution. Regards, --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From dkg at fifthhorseman.net Fri May 13 05:35:29 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Thu, 12 May 2011 15:35:29 -0400 Subject: ssh-agent subprocess parentage In-Reply-To: <20110512191930.GA1625@apb-laptoy.apb.alt.za> References: <4DC81103.40609@fifthhorseman.net> <20110512191930.GA1625@apb-laptoy.apb.alt.za> Message-ID: <4DCC3681.10100@fifthhorseman.net> On 05/12/2011 03:19 PM, Alan Barrett wrote: > On Mon, 09 May 2011, Daniel Kahn Gillmor wrote: >> check_parent_exists() relies on simply storing the process ID and >> checking to see whether such a process is signalable. > > I reported a bug around that code in April 2006. See > > > for the problem and a patch that still works for me. This makes sense to me, and looks like a clean way to deal with the process checking for a parent. Thanks! I've put it into bugzilla here: https://bugzilla.mindrot.org/show_bug.cgi?id=1905 This still doesn't deal with the issue about process supervision, though. --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From imorgan at nas.nasa.gov Fri May 13 08:47:19 2011 From: imorgan at nas.nasa.gov (Iain Morgan) Date: Thu, 12 May 2011 15:47:19 -0700 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> Message-ID: <20110512224719.GF492@linux124.nas.nasa.gov> On Thu, May 12, 2011 at 14:14:02 -0500, Dan Kaminsky wrote: > On Thu, May 12, 2011 at 11:49 AM, Markus Friedl wrote: > > > looks like we've been waiting too long :) > > > > http://www.openssh.com/txt/release-3.0 > > > > 2) The files > > /etc/ssh_known_hosts2 > > ~/.ssh/known_hosts2 > > ~/.ssh/authorized_keys2 > > are now obsolete, you can use > > /etc/ssh_known_hosts > > ~/.ssh/known_hosts > > ~/.ssh/authorized_keys > > For backward compatibility ~/.ssh/authorized_keys2 will still used for > > authentication and hostkeys are still read from the known_hosts2. > > However, those deprecated files are considered 'readonly'. Future > > releases are likely not to read these files. > > > > In no uncertain terms, removal of authorized_keys2 support will cause > outages, up to and including requiring physical access for administrators to > resolve. Documentation is not an excuse to make this change. > > It's completely reasonable, desirable even, to allow a new configuration > option to explicitly define the set of files that can contain authorized > keys. It'd even be convenient to have an AuthorizationCommand option, that > sent properly escaped strings to a command for external testing and > validation. > Provided that sysadmins are aware of the change ahead of time, proactive measures can easily be taken. On a per-system basis, it's obviously a simple matter to find and rename such deprecated files. However, we can expect that some will be caught off guard. In hindsight, what probably should have been done is have sshd return a warning to the client about the deprecated file after successful login. And, for good measure, an appropriate message should be logged on the server, if that's not already the case. Given that support for authorized_keys2 hasn't been documented for a number of years, I have to wonder just how widespread its use is. -- Iain morgan From dan at doxpara.com Fri May 13 11:49:13 2011 From: dan at doxpara.com (Dan Kaminsky) Date: Thu, 12 May 2011 21:49:13 -0400 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <20110512224719.GF492@linux124.nas.nasa.gov> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <20110512224719.GF492@linux124.nas.nasa.gov> Message-ID: Sent from my iPhone On May 12, 2011, at 6:47 PM, Iain Morgan wrote: > On Thu, May 12, 2011 at 14:14:02 -0500, Dan Kaminsky wrote: >> On Thu, May 12, 2011 at 11:49 AM, Markus Friedl wrote: >> >>> looks like we've been waiting too long :) >>> >>> http://www.openssh.com/txt/release-3.0 >>> >>> 2) The files >>> /etc/ssh_known_hosts2 >>> ~/.ssh/known_hosts2 >>> ~/.ssh/authorized_keys2 >>> are now obsolete, you can use >>> /etc/ssh_known_hosts >>> ~/.ssh/known_hosts >>> ~/.ssh/authorized_keys >>> For backward compatibility ~/.ssh/authorized_keys2 will still used for >>> authentication and hostkeys are still read from the known_hosts2. >>> However, those deprecated files are considered 'readonly'. Future >>> releases are likely not to read these files. >>> >> >> In no uncertain terms, removal of authorized_keys2 support will cause >> outages, up to and including requiring physical access for administrators to >> resolve. Documentation is not an excuse to make this change. >> >> It's completely reasonable, desirable even, to allow a new configuration >> option to explicitly define the set of files that can contain authorized >> keys. It'd even be convenient to have an AuthorizationCommand option, that >> sent properly escaped strings to a command for external testing and >> validation. >> > Provided that sysadmins are aware of the change ahead of time, proactive > measures can easily be taken. On a per-system basis, it's obviously a > simple matter to find and rename such deprecated files. However, we can > expect that some will be caught off guard. > > In hindsight, what probably should have been done is have sshd return a > warning to the client about the deprecated file after successful login. > And, for good measure, an appropriate message should be logged on the > server, if that's not already the case. > > Given that support for authorized_keys2 hasn't been documented for a > number of years, I have to wonder just how widespread its use is. > Doesn't matter. A failure here locks the admin out of the server, requiring up to and including physical recovery. A vaguely nice security change cannot justify extensive notification and lockout risk. And we're not really in the position to collect data saying the risk is small enough. > -- > Iain morgan From phil at hands.com Fri May 13 22:03:09 2011 From: phil at hands.com (Philip Hands) Date: Fri, 13 May 2011 13:03:09 +0100 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <20110512224719.GF492@linux124.nas.nasa.gov> Message-ID: <87sjsivmde.fsf@poker.hands.com> On Thu, 12 May 2011 21:49:13 -0400, Dan Kaminsky wrote: ... > > Provided that sysadmins are aware of the change ahead of time, proactive > > measures can easily be taken. On a per-system basis, it's obviously a > > simple matter to find and rename such deprecated files. However, we can > > expect that some will be caught off guard. > > > > In hindsight, what probably should have been done is have sshd return a > > warning to the client about the deprecated file after successful login. > > And, for good measure, an appropriate message should be logged on the > > server, if that's not already the case. > > > > Given that support for authorized_keys2 hasn't been documented for a > > number of years, I have to wonder just how widespread its use is. > > > > Doesn't matter. A failure here locks the admin out of the server, > requiring up to and including physical recovery. How about if for the first release: if AuthorizedKeysFile is set, then that's fine, we're not defaulting to using the authorized_keys2 file anyway if it is not set, and authorized_keys2 file(s) exist on the system, put up a warning if it's an interactive login, log a warning, and pause for 20 seconds before then allowing the login. That should get people's attention. To be certain that you're going to alert the sysadmins, you'd want to check for any authorized_keys2 anywhere, but that's a bit cumbersome. Perhaps it would be OK to only do it when one were using the keys out of an authorized_keys2, although there is some danger that only automated logins would be affected, and the delay and logs would go unnoticed. Perhaps one could set a flag when an authorized_keys2 file had been found at some point during the lifetime of the sshd, and change the behaviour for all subsequent logins. Leave that behaviour in place for a release or two, then just drop the default use of authorized_keys2. If anyone then gets caught out, they've really not been paying attention. If the above seems like too much effort (which it probably is), just put it in the release notes, and suggest that people who package ssh make sure that the user is alerted to the change (if AuthorizedKeysFile is unset). Anyone who upgrades ssh remotely without testing that they can still log in before they log out, is a dimwit. That said, I've been that dimwit ;-) Cheers, Phil. -- |)| Philip Hands [+44 (0)20 8530 9560] http://www.hands.com/ |-| HANDS.COM Ltd. http://www.uk.debian.org/ |(| 10 Onslow Gardens, South Woodford, London E18 1NE ENGLAND -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: From djm at mindrot.org Sat May 14 21:02:21 2011 From: djm at mindrot.org (Damien Miller) Date: Sat, 14 May 2011 21:02:21 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> Message-ID: On Wed, 11 May 2011, Damien Miller wrote: > > I also think that being able to specify multiple authorized_keys files > > is very useful, so I would prefer to just see this as a documented > > feature. > > Perhaps we should make options.authorized_keys_file an array to let > people who want to use multiple files do so. Here's a lightly-tested patch against -current (OpenBSD version, it will probably need some editing for portable) Index: auth.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth.c,v retrieving revision 1.91 diff -u -p -r1.91 auth.c --- auth.c 29 Nov 2010 23:45:51 -0000 1.91 +++ auth.c 10 May 2011 11:47:16 -0000 @@ -271,12 +271,6 @@ authorized_keys_file(struct passwd *pw) } char * -authorized_keys_file2(struct passwd *pw) -{ - return expand_authorized_keys(options.authorized_keys_file2, pw); -} - -char * authorized_principals_file(struct passwd *pw) { if (options.authorized_principals_file == NULL) Index: auth.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth.h,v retrieving revision 1.67 diff -u -p -r1.67 auth.h --- auth.h 10 Mar 2011 11:34:25 -0000 1.67 +++ auth.h 10 May 2011 11:47:16 -0000 @@ -146,7 +146,6 @@ char *get_challenge(Authctxt *); int verify_response(Authctxt *, const char *); char *authorized_keys_file(struct passwd *); -char *authorized_keys_file2(struct passwd *); char *authorized_principals_file(struct passwd *); FILE *auth_openkeyfile(const char *, struct passwd *, int); Index: auth2-pubkey.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth2-pubkey.c,v retrieving revision 1.27 diff -u -p -r1.27 auth2-pubkey.c --- auth2-pubkey.c 20 Nov 2010 05:12:38 -0000 1.27 +++ auth2-pubkey.c 10 May 2011 11:47:16 -0000 @@ -450,13 +450,7 @@ user_key_allowed(struct passwd *pw, Key file = authorized_keys_file(pw); success = user_key_allowed2(pw, key, file); xfree(file); - if (success) - return success; - /* try suffix "2" for backward compat, too */ - file = authorized_keys_file2(pw); - success = user_key_allowed2(pw, key, file); - xfree(file); return success; } Index: pathnames.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/pathnames.h,v retrieving revision 1.20 diff -u -p -r1.20 pathnames.h --- pathnames.h 31 Aug 2010 11:54:45 -0000 1.20 +++ pathnames.h 10 May 2011 11:47:17 -0000 @@ -88,9 +88,6 @@ */ #define _PATH_SSH_USER_PERMITTED_KEYS ".ssh/authorized_keys" -/* backward compat for protocol v2 */ -#define _PATH_SSH_USER_PERMITTED_KEYS2 ".ssh/authorized_keys2" - /* * Per-user and system-wide ssh "rc" files. These files are executed with * /bin/sh before starting the shell or command if they exist. They will be Index: servconf.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/servconf.c,v retrieving revision 1.214 diff -u -p -r1.214 servconf.c --- servconf.c 29 Mar 2011 18:54:17 -0000 1.214 +++ servconf.c 10 May 2011 11:47:17 -0000 @@ -120,7 +120,6 @@ initialize_server_options(ServerOptions options->client_alive_interval = -1; options->client_alive_count_max = -1; options->authorized_keys_file = NULL; - options->authorized_keys_file2 = NULL; options->num_accept_env = 0; options->permit_tun = -1; options->num_permitted_opens = -1; @@ -250,13 +249,6 @@ fill_default_server_options(ServerOption options->client_alive_interval = 0; if (options->client_alive_count_max == -1) options->client_alive_count_max = 3; - if (options->authorized_keys_file2 == NULL) { - /* authorized_keys_file2 falls back to authorized_keys_file */ - if (options->authorized_keys_file != NULL) - options->authorized_keys_file2 = xstrdup(options->authorized_keys_file); - else - options->authorized_keys_file2 = xstrdup(_PATH_SSH_USER_PERMITTED_KEYS2); - } if (options->authorized_keys_file == NULL) options->authorized_keys_file = xstrdup(_PATH_SSH_USER_PERMITTED_KEYS); if (options->permit_tun == -1) @@ -1207,9 +1199,6 @@ process_server_config_line(ServerOptions case sAuthorizedKeysFile: charptr = &options->authorized_keys_file; goto parse_tilde_filename; - case sAuthorizedKeysFile2: - charptr = &options->authorized_keys_file2; - goto parse_tilde_filename; case sAuthorizedPrincipalsFile: charptr = &options->authorized_principals_file; parse_tilde_filename: @@ -1474,7 +1463,6 @@ copy_set_server_options(ServerOptions *d M_CP_STROPT(trusted_user_ca_keys); M_CP_STROPT(revoked_keys_file); M_CP_STROPT(authorized_keys_file); - M_CP_STROPT(authorized_keys_file2); M_CP_STROPT(authorized_principals_file); } @@ -1687,7 +1675,6 @@ dump_config(ServerOptions *o) dump_cfg_string(sMacs, o->macs); dump_cfg_string(sBanner, o->banner); dump_cfg_string(sAuthorizedKeysFile, o->authorized_keys_file); - dump_cfg_string(sAuthorizedKeysFile2, o->authorized_keys_file2); dump_cfg_string(sForceCommand, o->adm_forced_command); dump_cfg_string(sChrootDirectory, o->chroot_directory); dump_cfg_string(sTrustedUserCAKeys, o->trusted_user_ca_keys); Index: servconf.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/servconf.h,v retrieving revision 1.95 diff -u -p -r1.95 servconf.h --- servconf.h 13 Nov 2010 23:27:50 -0000 1.95 +++ servconf.h 10 May 2011 11:47:17 -0000 @@ -146,7 +146,6 @@ typedef struct { */ char *authorized_keys_file; /* File containing public keys */ - char *authorized_keys_file2; char *adm_forced_command; From dkg at fifthhorseman.net Sun May 15 01:23:56 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Sat, 14 May 2011 11:23:56 -0400 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> Message-ID: <4DCE9E8C.4060101@fifthhorseman.net> On 05/14/2011 07:02 AM, Damien Miller wrote: > On Wed, 11 May 2011, Damien Miller wrote: > >>> I also think that being able to specify multiple authorized_keys files >>> is very useful, so I would prefer to just see this as a documented >>> feature. >> >> Perhaps we should make options.authorized_keys_file an array to let >> people who want to use multiple files do so. > > Here's a lightly-tested patch against -current (OpenBSD version, it > will probably need some editing for portable) This patch appears to removes authorized_keys2 support, but does not appear to make authorized_keys_file an array, as the comments above would suggest. --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From djm at mindrot.org Sun May 15 08:28:58 2011 From: djm at mindrot.org (Damien Miller) Date: Sun, 15 May 2011 08:28:58 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> Message-ID: On Sat, 14 May 2011, Damien Miller wrote: > On Wed, 11 May 2011, Damien Miller wrote: > > > > I also think that being able to specify multiple authorized_keys files > > > is very useful, so I would prefer to just see this as a documented > > > feature. > > > > Perhaps we should make options.authorized_keys_file an array to let > > people who want to use multiple files do so. > > Here's a lightly-tested patch against -current (OpenBSD version, it > will probably need some editing for portable) doh, wrong diff Index: auth-rsa.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth-rsa.c,v retrieving revision 1.79 diff -u -p -r1.79 auth-rsa.c --- auth-rsa.c 3 Dec 2010 23:55:27 -0000 1.79 +++ auth-rsa.c 13 May 2011 12:22:18 -0000 @@ -157,44 +157,27 @@ auth_rsa_challenge_dialog(Key *key) return (success); } -/* - * check if there's user key matching client_n, - * return key if login is allowed, NULL otherwise - */ - -int -auth_rsa_key_allowed(struct passwd *pw, BIGNUM *client_n, Key **rkey) +static int +rsa_key_allowed_in_file(struct passwd *pw, char *file, + const BIGNUM *client_n, Key **rkey) { - char line[SSH_MAX_PUBKEY_BYTES], *file; + char line[SSH_MAX_PUBKEY_BYTES]; int allowed = 0; u_int bits; FILE *f; u_long linenum = 0; Key *key; - /* Temporarily use the user's uid. */ - temporarily_use_uid(pw); - - /* The authorized keys. */ - file = authorized_keys_file(pw); debug("trying public RSA key file %s", file); - f = auth_openkeyfile(file, pw, options.strict_modes); - if (!f) { - xfree(file); - restore_uid(); - return (0); - } - - /* Flag indicating whether the key is allowed. */ - allowed = 0; - - key = key_new(KEY_RSA1); + if ((f = auth_openkeyfile(file, pw, options.strict_modes)) == NULL) + return 0; /* * Go though the accepted keys, looking for the current key. If * found, perform a challenge-response dialog to verify that the * user really has the corresponding private key. */ + key = key_new(KEY_RSA1); while (read_keyfile_line(f, file, line, sizeof(line), &linenum) != -1) { char *cp; char *key_options; @@ -232,7 +215,10 @@ auth_rsa_key_allowed(struct passwd *pw, } /* cp now points to the comment part. */ - /* Check if the we have found the desired key (identified by its modulus). */ + /* + * Check if the we have found the desired key (identified + * by its modulus). + */ if (BN_cmp(key->rsa->n, client_n) != 0) continue; @@ -261,11 +247,7 @@ auth_rsa_key_allowed(struct passwd *pw, break; } - /* Restore the privileged uid. */ - restore_uid(); - /* Close the file. */ - xfree(file); fclose(f); /* return key if allowed */ @@ -273,7 +255,33 @@ auth_rsa_key_allowed(struct passwd *pw, *rkey = key; else key_free(key); - return (allowed); + + return allowed; +} + +/* + * check if there's user key matching client_n, + * return key if login is allowed, NULL otherwise + */ + +int +auth_rsa_key_allowed(struct passwd *pw, BIGNUM *client_n, Key **rkey) +{ + char *file; + u_int i, allowed = 0; + + temporarily_use_uid(pw); + + for (i = 0; !allowed && i < options.num_authkeys_files; i++) { + file = expand_authorized_keys( + options.authorized_keys_files[i], pw); + allowed = rsa_key_allowed_in_file(pw, file, client_n, rkey); + xfree(file); + } + + restore_uid(); + + return allowed; } /* Index: auth.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth.c,v retrieving revision 1.92 diff -u -p -r1.92 auth.c --- auth.c 11 May 2011 04:47:06 -0000 1.92 +++ auth.c 13 May 2011 12:22:18 -0000 @@ -241,7 +241,7 @@ auth_root_allowed(char *method) * * This returns a buffer allocated by xmalloc. */ -static char * +char * expand_authorized_keys(const char *filename, struct passwd *pw) { char *file, ret[MAXPATHLEN]; @@ -262,12 +262,6 @@ expand_authorized_keys(const char *filen fatal("expand_authorized_keys: path too long"); xfree(file); return (xstrdup(ret)); -} - -char * -authorized_keys_file(struct passwd *pw) -{ - return expand_authorized_keys(options.authorized_keys_file, pw); } char * Index: auth.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth.h,v retrieving revision 1.68 diff -u -p -r1.68 auth.h --- auth.h 11 May 2011 04:47:06 -0000 1.68 +++ auth.h 13 May 2011 12:22:18 -0000 @@ -145,7 +145,7 @@ struct passwd * getpwnamallow(const char char *get_challenge(Authctxt *); int verify_response(Authctxt *, const char *); -char *authorized_keys_file(struct passwd *); +char *expand_authorized_keys(const char *, struct passwd *pw); char *authorized_principals_file(struct passwd *); FILE *auth_openkeyfile(const char *, struct passwd *, int); Index: auth2-pubkey.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth2-pubkey.c,v retrieving revision 1.28 diff -u -p -r1.28 auth2-pubkey.c --- auth2-pubkey.c 11 May 2011 04:47:06 -0000 1.28 +++ auth2-pubkey.c 13 May 2011 12:22:18 -0000 @@ -435,7 +435,7 @@ user_cert_trusted_ca(struct passwd *pw, int user_key_allowed(struct passwd *pw, Key *key) { - int success; + u_int success, i; char *file; if (auth_key_is_revoked(key)) @@ -447,9 +447,12 @@ user_key_allowed(struct passwd *pw, Key if (success) return success; - file = authorized_keys_file(pw); - success = user_key_allowed2(pw, key, file); - xfree(file); + for (i = 0; !success && i < options.num_authkeys_files; i++) { + file = expand_authorized_keys( + options.authorized_keys_files[i], pw); + success = user_key_allowed2(pw, key, file); + xfree(file); + } return success; } Index: pathnames.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/pathnames.h,v retrieving revision 1.21 diff -u -p -r1.21 pathnames.h --- pathnames.h 11 May 2011 04:47:06 -0000 1.21 +++ pathnames.h 13 May 2011 12:22:18 -0000 @@ -88,6 +88,9 @@ */ #define _PATH_SSH_USER_PERMITTED_KEYS ".ssh/authorized_keys" +/* backward compat for protocol v2 */ +#define _PATH_SSH_USER_PERMITTED_KEYS2 ".ssh/authorized_keys2" + /* * Per-user and system-wide ssh "rc" files. These files are executed with * /bin/sh before starting the shell or command if they exist. They will be Index: servconf.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/servconf.c,v retrieving revision 1.215 diff -u -p -r1.215 servconf.c --- servconf.c 11 May 2011 04:47:06 -0000 1.215 +++ servconf.c 13 May 2011 12:22:19 -0000 @@ -119,7 +119,7 @@ initialize_server_options(ServerOptions options->use_dns = -1; options->client_alive_interval = -1; options->client_alive_count_max = -1; - options->authorized_keys_file = NULL; + options->num_authkeys_files = 0; options->num_accept_env = 0; options->permit_tun = -1; options->num_permitted_opens = -1; @@ -249,8 +249,12 @@ fill_default_server_options(ServerOption options->client_alive_interval = 0; if (options->client_alive_count_max == -1) options->client_alive_count_max = 3; - if (options->authorized_keys_file == NULL) - options->authorized_keys_file = xstrdup(_PATH_SSH_USER_PERMITTED_KEYS); + if (options->num_authkeys_files == 0) { + options->authorized_keys_files[options->num_authkeys_files++] = + xstrdup(_PATH_SSH_USER_PERMITTED_KEYS); + options->authorized_keys_files[options->num_authkeys_files++] = + xstrdup(_PATH_SSH_USER_PERMITTED_KEYS2); + } if (options->permit_tun == -1) options->permit_tun = SSH_TUNMODE_NO; if (options->zero_knowledge_password_authentication == -1) @@ -286,7 +290,7 @@ typedef enum { sMaxStartups, sMaxAuthTries, sMaxSessions, sBanner, sUseDNS, sHostbasedAuthentication, sHostbasedUsesNameFromPacketOnly, sClientAliveInterval, - sClientAliveCountMax, sAuthorizedKeysFile, sAuthorizedKeysFile2, + sClientAliveCountMax, sAuthorizedKeysFile, sGssAuthentication, sGssCleanupCreds, sAcceptEnv, sPermitTunnel, sMatch, sPermitOpen, sForceCommand, sChrootDirectory, sUsePrivilegeSeparation, sAllowAgentForwarding, @@ -391,7 +395,6 @@ static struct { { "clientaliveinterval", sClientAliveInterval, SSHCFG_GLOBAL }, { "clientalivecountmax", sClientAliveCountMax, SSHCFG_GLOBAL }, { "authorizedkeysfile", sAuthorizedKeysFile, SSHCFG_ALL }, - { "authorizedkeysfile2", sAuthorizedKeysFile2, SSHCFG_ALL }, { "useprivilegeseparation", sUsePrivilegeSeparation, SSHCFG_GLOBAL}, { "acceptenv", sAcceptEnv, SSHCFG_GLOBAL }, { "permittunnel", sPermitTunnel, SSHCFG_ALL }, @@ -1197,11 +1200,19 @@ process_server_config_line(ServerOptions * AuthorizedKeysFile /etc/ssh_keys/%u */ case sAuthorizedKeysFile: - charptr = &options->authorized_keys_file; - goto parse_tilde_filename; + while ((arg = strdelim(&cp)) && *arg != '\0') { + if (options->num_authkeys_files >= MAX_AUTHKEYS_FILES) + fatal("%s line %d: " + "too many authorized keys files.", + filename, linenum); + options->authorized_keys_files[ + options->num_authkeys_files++] = + tilde_expand_filename(arg, getuid()); + } + break; + case sAuthorizedPrincipalsFile: charptr = &options->authorized_principals_file; - parse_tilde_filename: arg = strdelim(&cp); if (!arg || *arg == '\0') fatal("%s line %d: missing file name.", @@ -1420,6 +1431,10 @@ parse_server_match_config(ServerOptions dst->n = src->n; \ } \ } while(0) +#define M_CP_STRARRAYOPT(n, num_n) do {\ + for (dst->num_n = 0; dst->num_n < src->num_n; dst->num_n++) \ + dst->n[dst->num_n] = xstrdup(src->n[dst->num_n]); \ +} while(0) /* * Copy any supported values that are set. @@ -1462,8 +1477,8 @@ copy_set_server_options(ServerOptions *d M_CP_STROPT(chroot_directory); M_CP_STROPT(trusted_user_ca_keys); M_CP_STROPT(revoked_keys_file); - M_CP_STROPT(authorized_keys_file); M_CP_STROPT(authorized_principals_file); + M_CP_STRARRAYOPT(authorized_keys_files, num_authkeys_files); } #undef M_CP_INTOPT @@ -1674,7 +1689,6 @@ dump_config(ServerOptions *o) dump_cfg_string(sCiphers, o->ciphers); dump_cfg_string(sMacs, o->macs); dump_cfg_string(sBanner, o->banner); - dump_cfg_string(sAuthorizedKeysFile, o->authorized_keys_file); dump_cfg_string(sForceCommand, o->adm_forced_command); dump_cfg_string(sChrootDirectory, o->chroot_directory); dump_cfg_string(sTrustedUserCAKeys, o->trusted_user_ca_keys); @@ -1687,6 +1701,8 @@ dump_config(ServerOptions *o) dump_cfg_string(sLogFacility, log_facility_name(o->log_facility)); /* string array arguments */ + dump_cfg_strarray(sAuthorizedKeysFile, o->num_authkeys_files, + o->authorized_keys_files); dump_cfg_strarray(sHostKeyFile, o->num_host_key_files, o->host_key_files); dump_cfg_strarray(sHostKeyFile, o->num_host_cert_files, Index: servconf.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/servconf.h,v retrieving revision 1.96 diff -u -p -r1.96 servconf.h --- servconf.h 11 May 2011 04:47:06 -0000 1.96 +++ servconf.h 13 May 2011 12:22:19 -0000 @@ -27,6 +27,7 @@ #define MAX_HOSTCERTS 256 /* Max # host certificates. */ #define MAX_ACCEPT_ENV 256 /* Max # of env vars. */ #define MAX_MATCH_GROUPS 256 /* Max # of groups for Match. */ +#define MAX_AUTHKEYS_FILES 256 /* Max # of authorized_keys files. */ /* permit_root_login */ #define PERMIT_NOT_SET -1 @@ -145,7 +146,8 @@ typedef struct { * disconnect the session */ - char *authorized_keys_file; /* File containing public keys */ + u_int num_authkeys_files; /* Files containing public keys */ + char *authorized_keys_files[MAX_AUTHKEYS_FILES]; char *adm_forced_command; Index: sshd.8 =================================================================== RCS file: /cvs/src/usr.bin/ssh/sshd.8,v retrieving revision 1.260 diff -u -p -r1.260 sshd.8 --- sshd.8 28 Oct 2010 18:33:28 -0000 1.260 +++ sshd.8 13 May 2011 12:22:19 -0000 @@ -435,7 +435,7 @@ is run, and if that does not exist either, xauth is used to add the cookie. .Sh AUTHORIZED_KEYS FILE FORMAT .Cm AuthorizedKeysFile -specifies the file containing public keys for +specifies the file or files containing public keys for public key authentication; if none is specified, the default is .Pa ~/.ssh/authorized_keys . Index: sshd_config =================================================================== RCS file: /cvs/src/usr.bin/ssh/sshd_config,v retrieving revision 1.83 diff -u -p -r1.83 sshd_config --- sshd_config 6 May 2011 01:03:35 -0000 1.83 +++ sshd_config 13 May 2011 12:22:19 -0000 @@ -42,7 +42,7 @@ #RSAAuthentication yes #PubkeyAuthentication yes -#AuthorizedKeysFile .ssh/authorized_keys +#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #RhostsRSAAuthentication no Index: sshd_config.5 =================================================================== RCS file: /cvs/src/usr.bin/ssh/sshd_config.5,v retrieving revision 1.131 diff -u -p -r1.131 sshd_config.5 --- sshd_config.5 8 Dec 2010 04:02:47 -0000 1.131 +++ sshd_config.5 13 May 2011 12:22:19 -0000 @@ -170,6 +170,10 @@ is taken to be an absolute path or one r directory. The default is .Dq .ssh/authorized_keys . +Multiple files may be listed, either on a single line separated by +whitespace or on additional +.Cm AuthorizedKeysFile +lines. .It Cm AuthorizedPrincipalsFile Specifies a file that lists principal names that are accepted for certificate authentication. From djm at mindrot.org Sun May 15 08:52:46 2011 From: djm at mindrot.org (Damien Miller) Date: Sun, 15 May 2011 08:52:46 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> Message-ID: On Sun, 15 May 2011, Damien Miller wrote: > On Sat, 14 May 2011, Damien Miller wrote: > > > On Wed, 11 May 2011, Damien Miller wrote: > > > > > > I also think that being able to specify multiple authorized_keys files > > > > is very useful, so I would prefer to just see this as a documented > > > > feature. > > > > > > Perhaps we should make options.authorized_keys_file an array to let > > > people who want to use multiple files do so. > > > > Here's a lightly-tested patch against -current (OpenBSD version, it > > will probably need some editing for portable) > > doh, wrong diff This will apply to -current portable as checked out from cvs, hg or from the 20110516 snapshot tomorrow. -d From dtucker at zip.com.au Sun May 15 19:31:49 2011 From: dtucker at zip.com.au (Darren Tucker) Date: Sun, 15 May 2011 19:31:49 +1000 Subject: configure: error: *** Can't find recent OpenSSL libcrypto (see config.log for details) *** In-Reply-To: References: Message-ID: On Tue, May 10, 2011 at 9:26 AM, Darren Tucker wrote: > On Tue, May 10, 2011 at 6:18 AM, abdul jabbar wrote: > [...] >> ?*configure: error: *** Can't find recent OpenSSL libcrypto (see config.log >> for details) **** > > What did ?you find when you looking in config.log for details? (log supplied offline) The interesting bits are below, but it looks like you're trying to build a 64 bit binary against a 32 bit library (I don't know if centos does anything cute with 64 bit libraries like stick them in a lib64). The functions __umoddi3 and friends are usually provided by libgcc. My suggestions: - tell configure to build a 32 bit binary (./configure --with-cflags=-m32) - check that your compiler is installed properly (in particular, that libgcc.a exists and matches the version of gcc you're trying to use). - if you have a lib64, try telling configure to use it (./configure --with-ldflags=-L/path/to/lib64). configure:17548: gcc -o conftest -g -O2 -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wno-pointer-sign -fno-strict-aliasing -fno-builtin-memset -fstack-protector-all -std=gnu99 -I/usr/local/ssl/include -L/usr/local/ssl/lib -fstack-protector-all conftest.c /usr/lib/libcrypto.a -ldl -ldl -lutil -lz -lnsl >&5 /usr/bin/ld: warning: i386 architecture of input file `/usr/lib/libcrypto.a(rand_lib.o)' is incompatible with i386:x86-64 output [repeated many times] /usr/lib/libcrypto.a(bn_word.o): In function `BN_mod_word': (.text+0x86): undefined reference to `__umoddi3' /usr/lib/libcrypto.a(bn_asm.o): In function `bn_div_words': (.text+0x4e2): undefined reference to `__udivdi3' /usr/lib/libcrypto.a(bn_div.o): In function `BN_div': (.text+0x329): undefined reference to `__udivdi3' /usr/lib/libcrypto.a(bn_div.o): In function `BN_div': (.text+0x7a6): undefined reference to `__udivdi3' -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4? 37C9 C982 80C7 8FF4 FA69 ? ? Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement. From dkg at fifthhorseman.net Mon May 16 02:43:12 2011 From: dkg at fifthhorseman.net (Daniel Kahn Gillmor) Date: Sun, 15 May 2011 12:43:12 -0400 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> Message-ID: <4DD002A0.6040406@fifthhorseman.net> On 05/14/2011 06:28 PM, Damien Miller wrote: > Index: sshd_config.5 > =================================================================== > RCS file: /cvs/src/usr.bin/ssh/sshd_config.5,v > retrieving revision 1.131 > diff -u -p -r1.131 sshd_config.5 > --- sshd_config.5 8 Dec 2010 04:02:47 -0000 1.131 > +++ sshd_config.5 13 May 2011 12:22:19 -0000 > @@ -170,6 +170,10 @@ is taken to be an absolute path or one r > directory. > The default is > .Dq .ssh/authorized_keys . > +Multiple files may be listed, either on a single line separated by > +whitespace or on additional > +.Cm AuthorizedKeysFile > +lines. > .It Cm AuthorizedPrincipalsFile > Specifies a file that lists principal names that are accepted for > certificate authentication. It seems somewhat unclear how AuthorizedKeysFile interacts with a Match clause. If the following makes an array of two authorizedkeysfiles: AuthorizedKeysFile foo AuthorizedKeysFile bar then what does this mean for user X: AuthorizedKeysFile foo Match user x AuthorizedKeysFile bar Is it worth explicitly stating that, for a matching connection, setting an AuthorizedKeysFile within a Match block explicitly removes all other AuthorizedKeysFile settings *not* in that match block? --dkg -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1030 bytes Desc: OpenPGP digital signature URL: From djm at mindrot.org Mon May 16 12:51:14 2011 From: djm at mindrot.org (Damien Miller) Date: Mon, 16 May 2011 12:51:14 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <4DD002A0.6040406@fifthhorseman.net> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <4DD002A0.6040406@fifthhorseman.net> Message-ID: On Sun, 15 May 2011, Daniel Kahn Gillmor wrote: > It seems somewhat unclear how AuthorizedKeysFile interacts with a Match > clause. > > If the following makes an array of two authorizedkeysfiles: > > AuthorizedKeysFile foo > AuthorizedKeysFile bar So the question is whether to allow multiple directives that add to the list (as is the case in the slightly-broken patch I sent out yesterday) or to allow a single directive that specifies all the files on one line. The latter is more clear for Match, but long lines are more likely to wrap and are harder to read in sshd_config. That being said, there is plenty of room for the common cases that I can think of: AuthorizedKeysFile .ssh/authorized_keys AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 AuthorizedKeysFile /etc/ssh/authorized_keys/keys_%u .ssh/authorized_keys So maybe all-keys-on-one-line is better. -d From djm at mindrot.org Mon May 16 13:14:15 2011 From: djm at mindrot.org (Damien Miller) Date: Mon, 16 May 2011 13:14:15 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <4DD002A0.6040406@fifthhorseman.net> Message-ID: On Mon, 16 May 2011, Damien Miller wrote: > On Sun, 15 May 2011, Daniel Kahn Gillmor wrote: > > > It seems somewhat unclear how AuthorizedKeysFile interacts with a Match > > clause. > > > > If the following makes an array of two authorizedkeysfiles: > > > > AuthorizedKeysFile foo > > AuthorizedKeysFile bar > > So the question is whether to allow multiple directives that add to the > list (as is the case in the slightly-broken patch I sent out yesterday) > or to allow a single directive that specifies all the files on one line. > > The latter is more clear for Match, but long lines are more likely to wrap > and are harder to read in sshd_config. > > That being said, there is plenty of room for the common cases that I can > think of: > > AuthorizedKeysFile .ssh/authorized_keys > AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 > AuthorizedKeysFile /etc/ssh/authorized_keys/keys_%u .ssh/authorized_keys > > So maybe all-keys-on-one-line is better. Here's a diff that implements this: Index: auth-rsa.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth-rsa.c,v retrieving revision 1.79 diff -u -p -r1.79 auth-rsa.c --- auth-rsa.c 3 Dec 2010 23:55:27 -0000 1.79 +++ auth-rsa.c 16 May 2011 03:12:58 -0000 @@ -157,44 +157,27 @@ auth_rsa_challenge_dialog(Key *key) return (success); } -/* - * check if there's user key matching client_n, - * return key if login is allowed, NULL otherwise - */ - -int -auth_rsa_key_allowed(struct passwd *pw, BIGNUM *client_n, Key **rkey) +static int +rsa_key_allowed_in_file(struct passwd *pw, char *file, + const BIGNUM *client_n, Key **rkey) { - char line[SSH_MAX_PUBKEY_BYTES], *file; + char line[SSH_MAX_PUBKEY_BYTES]; int allowed = 0; u_int bits; FILE *f; u_long linenum = 0; Key *key; - /* Temporarily use the user's uid. */ - temporarily_use_uid(pw); - - /* The authorized keys. */ - file = authorized_keys_file(pw); debug("trying public RSA key file %s", file); - f = auth_openkeyfile(file, pw, options.strict_modes); - if (!f) { - xfree(file); - restore_uid(); - return (0); - } - - /* Flag indicating whether the key is allowed. */ - allowed = 0; - - key = key_new(KEY_RSA1); + if ((f = auth_openkeyfile(file, pw, options.strict_modes)) == NULL) + return 0; /* * Go though the accepted keys, looking for the current key. If * found, perform a challenge-response dialog to verify that the * user really has the corresponding private key. */ + key = key_new(KEY_RSA1); while (read_keyfile_line(f, file, line, sizeof(line), &linenum) != -1) { char *cp; char *key_options; @@ -232,7 +215,10 @@ auth_rsa_key_allowed(struct passwd *pw, } /* cp now points to the comment part. */ - /* Check if the we have found the desired key (identified by its modulus). */ + /* + * Check if the we have found the desired key (identified + * by its modulus). + */ if (BN_cmp(key->rsa->n, client_n) != 0) continue; @@ -261,11 +247,7 @@ auth_rsa_key_allowed(struct passwd *pw, break; } - /* Restore the privileged uid. */ - restore_uid(); - /* Close the file. */ - xfree(file); fclose(f); /* return key if allowed */ @@ -273,7 +255,33 @@ auth_rsa_key_allowed(struct passwd *pw, *rkey = key; else key_free(key); - return (allowed); + + return allowed; +} + +/* + * check if there's user key matching client_n, + * return key if login is allowed, NULL otherwise + */ + +int +auth_rsa_key_allowed(struct passwd *pw, BIGNUM *client_n, Key **rkey) +{ + char *file; + u_int i, allowed = 0; + + temporarily_use_uid(pw); + + for (i = 0; !allowed && i < options.num_authkeys_files; i++) { + file = expand_authorized_keys( + options.authorized_keys_files[i], pw); + allowed = rsa_key_allowed_in_file(pw, file, client_n, rkey); + xfree(file); + } + + restore_uid(); + + return allowed; } /* Index: auth.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth.c,v retrieving revision 1.92 diff -u -p -r1.92 auth.c --- auth.c 11 May 2011 04:47:06 -0000 1.92 +++ auth.c 16 May 2011 03:12:58 -0000 @@ -241,7 +241,7 @@ auth_root_allowed(char *method) * * This returns a buffer allocated by xmalloc. */ -static char * +char * expand_authorized_keys(const char *filename, struct passwd *pw) { char *file, ret[MAXPATHLEN]; @@ -262,12 +262,6 @@ expand_authorized_keys(const char *filen fatal("expand_authorized_keys: path too long"); xfree(file); return (xstrdup(ret)); -} - -char * -authorized_keys_file(struct passwd *pw) -{ - return expand_authorized_keys(options.authorized_keys_file, pw); } char * Index: auth.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth.h,v retrieving revision 1.68 diff -u -p -r1.68 auth.h --- auth.h 11 May 2011 04:47:06 -0000 1.68 +++ auth.h 16 May 2011 03:12:58 -0000 @@ -145,7 +145,7 @@ struct passwd * getpwnamallow(const char char *get_challenge(Authctxt *); int verify_response(Authctxt *, const char *); -char *authorized_keys_file(struct passwd *); +char *expand_authorized_keys(const char *, struct passwd *pw); char *authorized_principals_file(struct passwd *); FILE *auth_openkeyfile(const char *, struct passwd *, int); Index: auth2-pubkey.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth2-pubkey.c,v retrieving revision 1.28 diff -u -p -r1.28 auth2-pubkey.c --- auth2-pubkey.c 11 May 2011 04:47:06 -0000 1.28 +++ auth2-pubkey.c 16 May 2011 03:12:58 -0000 @@ -435,7 +435,7 @@ user_cert_trusted_ca(struct passwd *pw, int user_key_allowed(struct passwd *pw, Key *key) { - int success; + u_int success, i; char *file; if (auth_key_is_revoked(key)) @@ -447,9 +447,12 @@ user_key_allowed(struct passwd *pw, Key if (success) return success; - file = authorized_keys_file(pw); - success = user_key_allowed2(pw, key, file); - xfree(file); + for (i = 0; !success && i < options.num_authkeys_files; i++) { + file = expand_authorized_keys( + options.authorized_keys_files[i], pw); + success = user_key_allowed2(pw, key, file); + xfree(file); + } return success; } Index: pathnames.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/pathnames.h,v retrieving revision 1.21 diff -u -p -r1.21 pathnames.h --- pathnames.h 11 May 2011 04:47:06 -0000 1.21 +++ pathnames.h 16 May 2011 03:12:58 -0000 @@ -88,6 +88,9 @@ */ #define _PATH_SSH_USER_PERMITTED_KEYS ".ssh/authorized_keys" +/* backward compat for protocol v2 */ +#define _PATH_SSH_USER_PERMITTED_KEYS2 ".ssh/authorized_keys2" + /* * Per-user and system-wide ssh "rc" files. These files are executed with * /bin/sh before starting the shell or command if they exist. They will be Index: servconf.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/servconf.c,v retrieving revision 1.215 diff -u -p -r1.215 servconf.c --- servconf.c 11 May 2011 04:47:06 -0000 1.215 +++ servconf.c 16 May 2011 03:12:58 -0000 @@ -119,7 +119,7 @@ initialize_server_options(ServerOptions options->use_dns = -1; options->client_alive_interval = -1; options->client_alive_count_max = -1; - options->authorized_keys_file = NULL; + options->num_authkeys_files = 0; options->num_accept_env = 0; options->permit_tun = -1; options->num_permitted_opens = -1; @@ -249,8 +249,12 @@ fill_default_server_options(ServerOption options->client_alive_interval = 0; if (options->client_alive_count_max == -1) options->client_alive_count_max = 3; - if (options->authorized_keys_file == NULL) - options->authorized_keys_file = xstrdup(_PATH_SSH_USER_PERMITTED_KEYS); + if (options->num_authkeys_files == 0) { + options->authorized_keys_files[options->num_authkeys_files++] = + xstrdup(_PATH_SSH_USER_PERMITTED_KEYS); + options->authorized_keys_files[options->num_authkeys_files++] = + xstrdup(_PATH_SSH_USER_PERMITTED_KEYS2); + } if (options->permit_tun == -1) options->permit_tun = SSH_TUNMODE_NO; if (options->zero_knowledge_password_authentication == -1) @@ -286,7 +290,7 @@ typedef enum { sMaxStartups, sMaxAuthTries, sMaxSessions, sBanner, sUseDNS, sHostbasedAuthentication, sHostbasedUsesNameFromPacketOnly, sClientAliveInterval, - sClientAliveCountMax, sAuthorizedKeysFile, sAuthorizedKeysFile2, + sClientAliveCountMax, sAuthorizedKeysFile, sGssAuthentication, sGssCleanupCreds, sAcceptEnv, sPermitTunnel, sMatch, sPermitOpen, sForceCommand, sChrootDirectory, sUsePrivilegeSeparation, sAllowAgentForwarding, @@ -391,7 +395,6 @@ static struct { { "clientaliveinterval", sClientAliveInterval, SSHCFG_GLOBAL }, { "clientalivecountmax", sClientAliveCountMax, SSHCFG_GLOBAL }, { "authorizedkeysfile", sAuthorizedKeysFile, SSHCFG_ALL }, - { "authorizedkeysfile2", sAuthorizedKeysFile2, SSHCFG_ALL }, { "useprivilegeseparation", sUsePrivilegeSeparation, SSHCFG_GLOBAL}, { "acceptenv", sAcceptEnv, SSHCFG_GLOBAL }, { "permittunnel", sPermitTunnel, SSHCFG_ALL }, @@ -1197,11 +1200,22 @@ process_server_config_line(ServerOptions * AuthorizedKeysFile /etc/ssh_keys/%u */ case sAuthorizedKeysFile: - charptr = &options->authorized_keys_file; - goto parse_tilde_filename; + if (*activep && options->num_authkeys_files == 0) { + while ((arg = strdelim(&cp)) && *arg != '\0') { + if (options->num_authkeys_files >= + MAX_AUTHKEYS_FILES) + fatal("%s line %d: " + "too many authorized keys files.", + filename, linenum); + options->authorized_keys_files[ + options->num_authkeys_files++] = + tilde_expand_filename(arg, getuid()); + } + } + return 0; + case sAuthorizedPrincipalsFile: charptr = &options->authorized_principals_file; - parse_tilde_filename: arg = strdelim(&cp); if (!arg || *arg == '\0') fatal("%s line %d: missing file name.", @@ -1420,6 +1434,12 @@ parse_server_match_config(ServerOptions dst->n = src->n; \ } \ } while(0) +#define M_CP_STRARRAYOPT(n, num_n) do {\ + if (src->num_n != 0) { \ + for (dst->num_n = 0; dst->num_n < src->num_n; dst->num_n++) \ + dst->n[dst->num_n] = xstrdup(src->n[dst->num_n]); \ + } \ +} while(0) /* * Copy any supported values that are set. @@ -1455,19 +1475,23 @@ copy_set_server_options(ServerOptions *d M_CP_INTOPT(ip_qos_interactive); M_CP_INTOPT(ip_qos_bulk); + M_CP_STROPT(authorized_principals_file); M_CP_STROPT(banner); + M_CP_STROPT(revoked_keys_file); + M_CP_STROPT(trusted_user_ca_keys); + + M_CP_STRARRAYOPT(authorized_keys_files, num_authkeys_files); + if (preauth) return; + M_CP_STROPT(adm_forced_command); M_CP_STROPT(chroot_directory); - M_CP_STROPT(trusted_user_ca_keys); - M_CP_STROPT(revoked_keys_file); - M_CP_STROPT(authorized_keys_file); - M_CP_STROPT(authorized_principals_file); } #undef M_CP_INTOPT #undef M_CP_STROPT +#undef M_CP_STRARRAYOPT void parse_server_config(ServerOptions *options, const char *filename, Buffer *conf, @@ -1674,7 +1698,6 @@ dump_config(ServerOptions *o) dump_cfg_string(sCiphers, o->ciphers); dump_cfg_string(sMacs, o->macs); dump_cfg_string(sBanner, o->banner); - dump_cfg_string(sAuthorizedKeysFile, o->authorized_keys_file); dump_cfg_string(sForceCommand, o->adm_forced_command); dump_cfg_string(sChrootDirectory, o->chroot_directory); dump_cfg_string(sTrustedUserCAKeys, o->trusted_user_ca_keys); @@ -1687,6 +1710,8 @@ dump_config(ServerOptions *o) dump_cfg_string(sLogFacility, log_facility_name(o->log_facility)); /* string array arguments */ + dump_cfg_strarray(sAuthorizedKeysFile, o->num_authkeys_files, + o->authorized_keys_files); dump_cfg_strarray(sHostKeyFile, o->num_host_key_files, o->host_key_files); dump_cfg_strarray(sHostKeyFile, o->num_host_cert_files, Index: servconf.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/servconf.h,v retrieving revision 1.96 diff -u -p -r1.96 servconf.h --- servconf.h 11 May 2011 04:47:06 -0000 1.96 +++ servconf.h 16 May 2011 03:12:58 -0000 @@ -27,6 +27,7 @@ #define MAX_HOSTCERTS 256 /* Max # host certificates. */ #define MAX_ACCEPT_ENV 256 /* Max # of env vars. */ #define MAX_MATCH_GROUPS 256 /* Max # of groups for Match. */ +#define MAX_AUTHKEYS_FILES 256 /* Max # of authorized_keys files. */ /* permit_root_login */ #define PERMIT_NOT_SET -1 @@ -145,7 +146,8 @@ typedef struct { * disconnect the session */ - char *authorized_keys_file; /* File containing public keys */ + u_int num_authkeys_files; /* Files containing public keys */ + char *authorized_keys_files[MAX_AUTHKEYS_FILES]; char *adm_forced_command; Index: sshd.8 =================================================================== RCS file: /cvs/src/usr.bin/ssh/sshd.8,v retrieving revision 1.260 diff -u -p -r1.260 sshd.8 --- sshd.8 28 Oct 2010 18:33:28 -0000 1.260 +++ sshd.8 16 May 2011 03:12:59 -0000 @@ -435,10 +435,12 @@ is run, and if that does not exist either, xauth is used to add the cookie. .Sh AUTHORIZED_KEYS FILE FORMAT .Cm AuthorizedKeysFile -specifies the file containing public keys for +specifies the file or files containing public keys for public key authentication; -if none is specified, the default is -.Pa ~/.ssh/authorized_keys . +if none is specified, the default is both +.Pa ~/.ssh/authorized_keys +and +.Pa ~/.ssh/authorized_keys2 . Each line of the file contains one key (empty lines and lines starting with a .Ql # Index: sshd_config.5 =================================================================== RCS file: /cvs/src/usr.bin/ssh/sshd_config.5,v retrieving revision 1.131 diff -u -p -r1.131 sshd_config.5 --- sshd_config.5 8 Dec 2010 04:02:47 -0000 1.131 +++ sshd_config.5 16 May 2011 03:12:59 -0000 @@ -168,8 +168,14 @@ After expansion, .Cm AuthorizedKeysFile is taken to be an absolute path or one relative to the user's home directory. -The default is -.Dq .ssh/authorized_keys . +The default is both +.Dq .ssh/authorized_keys +and +.Dq .ssh/authorized_keys2 . +Multiple files may be listed, either on a single line separated by +whitespace or on additional +.Cm AuthorizedKeysFile +lines. .It Cm AuthorizedPrincipalsFile Specifies a file that lists principal names that are accepted for certificate authentication. From ricks at alldigital.com Wed May 18 12:19:40 2011 From: ricks at alldigital.com (Rick Stevens) Date: Tue, 17 May 2011 19:19:40 -0700 Subject: Bug found in openssh 5.8p1 source Message-ID: <4DD32CBC.5010802@alldigital.com> I was compiling the 5.8p1 source for CentOS 5.6 by installing the CentOS 5.6 source RPM for 4.3p2 and stealing the configure command they use to ensure I had all the same bits. I needed this to chroot some SFTP users and need the "Match" operator. The configure went well (once I had satisfied all the requirements). Upon compiling, however, I hit a syntax error in one of the files. In "openbsd-compat/port-linux.c", line 216 is missing an opening "{". WAS: if (path == NULL) setfscreatecon(NULL); return; } SHOULD BE: if (path == NULL) { setfscreatecon(NULL); return; } Just thought you should know. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks at alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - ...Had this been an actual emergency, we would have fled in terror - - and you'd be on your own, pal! - ---------------------------------------------------------------------- From dtucker at zip.com.au Wed May 18 12:58:06 2011 From: dtucker at zip.com.au (Darren Tucker) Date: Wed, 18 May 2011 12:58:06 +1000 Subject: Bug found in openssh 5.8p1 source In-Reply-To: <4DD32CBC.5010802@alldigital.com> References: <4DD32CBC.5010802@alldigital.com> Message-ID: On Wed, May 18, 2011 at 12:19 PM, Rick Stevens wrote: [...] > Upon compiling, however, I hit a syntax error in one of the files. > In "openbsd-compat/port-linux.c", line 216 is missing an opening "{". Thanks, this was fixed in the recently released 5.8p2. -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4? 37C9 C982 80C7 8FF4 FA69 ? ? Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement. From mattdm at mattdm.org Thu May 19 01:12:53 2011 From: mattdm at mattdm.org (Matthew Miller) Date: Wed, 18 May 2011 11:12:53 -0400 Subject: Might a patch to ssh-agent to allow relaxing of peer euid check be accepted? Message-ID: <20110518151253.GA3322@jadzia.bu.edu> Hi everyone. I have a system where I'd like to give certain users time-limited access to the use of certain SSH private keys without actually exposing the keys. I have the idea of using ssh-agent to do this. The agent would run as a "keyholder" user, and group permissions on the UNIX-domain socket would allow read-write by both that account and the actual ssh user. Right now, ssh-agent makes a check using getpeereid(), and declines access if it fails. This is very sensible in general, but breaks this particular case. Might a patch to allow an option to ssh-agent to relax the check be accepted? (Attached is a draft patch against 5.8p2.) -- Matthew Miller mattdm at mattdm.org -------------- next part -------------- diff -ur openssh-5.8p2.orig/ssh-agent.1 openssh-5.8p2.dontbestrict//ssh-agent.1 --- openssh-5.8p2.orig/ssh-agent.1 2010-11-30 19:50:35.000000000 -0500 +++ openssh-5.8p2.dontbestrict//ssh-agent.1 2011-05-18 11:10:00.000000000 -0400 @@ -46,6 +46,7 @@ .Op Fl d .Op Fl a Ar bind_address .Op Fl t Ar life +.Op Fl U .Op Ar command Op Ar arg ... .Nm ssh-agent .Op Fl c | s @@ -102,6 +103,13 @@ .Xr ssh-add 1 overrides this value. Without this option the default maximum lifetime is forever. +.It Fl U +Disables strict checking of the EUID of processes accessing the +.Ux Ns -domain +socket +to which the agent is bound. This allows the user accessing +the agent to be different from the account under which the +agent runs, protected only by file permissions. .El .Pp If a commandline is given, this is executed as a subprocess of the agent. diff -ur openssh-5.8p2.orig/ssh-agent.c openssh-5.8p2.dontbestrict//ssh-agent.c --- openssh-5.8p2.orig/ssh-agent.c 2010-11-30 19:50:35.000000000 -0500 +++ openssh-5.8p2.dontbestrict//ssh-agent.c 2011-05-18 10:25:33.000000000 -0400 @@ -137,6 +137,9 @@ /* Default lifetime (0 == forever) */ static int lifetime = 0; +/* Flag for allowing mismatched peer EUIDs */ +static int U_flag = 0; + static void close_socket(SocketEntry *e) { @@ -1023,11 +1026,18 @@ break; } if ((euid != 0) && (getuid() != euid)) { - error("uid mismatch: " - "peer euid %u != uid %u", - (u_int) euid, (u_int) getuid()); - close(sock); - break; + if (U_flag) { + verbose("uid mismatch (permitted by -U): " + "peer euid %u != uid %u", + (u_int) euid, (u_int) getuid()); + + } else { + error("uid mismatch: " + "peer euid %u != uid %u", + (u_int) euid, (u_int) getuid()); + close(sock); + break; + } } new_socket(AUTH_CONNECTION, sock); } @@ -1116,6 +1126,7 @@ fprintf(stderr, " -d Debug mode.\n"); fprintf(stderr, " -a socket Bind agent socket to given name.\n"); fprintf(stderr, " -t life Default identity lifetime (seconds).\n"); + fprintf(stderr, " -U Disable strict matching of peer EUID.\n"); exit(1); } @@ -1157,7 +1168,7 @@ init_rng(); seed_rng(); - while ((ch = getopt(ac, av, "cdksa:t:")) != -1) { + while ((ch = getopt(ac, av, "cdksa:t:U")) != -1) { switch (ch) { case 'c': if (s_flag) @@ -1186,6 +1197,9 @@ usage(); } break; + case 'U': + U_flag++; + break; default: usage(); } @@ -1193,7 +1207,7 @@ ac -= optind; av += optind; - if (ac > 0 && (c_flag || k_flag || s_flag || d_flag)) + if (ac > 0 && (c_flag || k_flag || s_flag || d_flag || U_flag)) usage(); if (ac == 0 && !c_flag && !s_flag) { From efo at basefarm.no Thu May 19 05:43:43 2011 From: efo at basefarm.no (=?ISO-8859-1?Q?Espen_Fjellv=E6r_Olsen?=) Date: Wed, 18 May 2011 21:43:43 +0200 Subject: Problem SSHing to HP ILO SSH-2.0-mpSSH_0.1.0 with 5.8p1 Message-ID: <4DD4216F.5070507@basefarm.no> Hi everyone, We are recently seeing a problem with OpenSSH 5.8p1 and SSH to ILO cards running SSH-2.0-mpSSH_0.1.0. This has previously worked with OpenSSH 5.5p1 (last known version for us to work). ssh ilohost -vvv gives the following on 5.8p1: debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Applying options for *.* debug2: ssh_connect: needpriv 0 debug1: Connecting to ilohost [10.20.12.30] port 22. debug1: Connection established. debug1: identity file /home/efo/.ssh/identity type -1 debug1: identity file /home/efo/.ssh/identity-cert type -1 debug3: Incorrect RSA1 identifier debug3: Could not load "/home/efo/.ssh/id_rsa" as a RSA1 public key debug2: key_type_from_name: unknown key type '-----BEGIN' debug3: key_read: missing keytype debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug2: key_type_from_name: unknown key type '-----END' debug3: key_read: missing keytype debug1: identity file /home/efo/.ssh/id_rsa type 1 debug1: identity file /home/efo/.ssh/id_rsa-cert type -1 debug1: identity file /home/efo/.ssh/id_dsa type -1 debug1: identity file /home/efo/.ssh/id_dsa-cert type -1 debug1: identity file /home/efo/.ssh/id_ecdsa type -1 debug1: identity file /home/efo/.ssh/id_ecdsa-cert type -1 debug1: Remote protocol version 2.0, remote software version mpSSH_0.1.0 debug1: no match: mpSSH_0.1.0 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.8p1 debug2: fd 3 setting O_NONBLOCK debug3: load_hostkeys: loading entries for host "ilohost" from file "/home/efo/.ssh/known_hosts" debug3: load_hostkeys: found key type RSA in file /home/efo/.ssh/known_hosts:185 debug3: load_hostkeys: loaded 1 keys debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01 at openssh.com,ssh-rsa-cert-v00 at openssh.com,ssh-rsa debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa-cert-v01 at openssh.com,ssh-rsa-cert-v00 at openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01 at openssh.com,ecdsa-sha2-nistp384-cert-v01 at openssh.com,ecdsa-sha2-nistp521-cert-v01 at openssh.com,ssh-dss-cert-v01 at openssh.com,ssh-dss-cert-v00 at openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc at lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc at lysator.liu.se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64 at openssh.com,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64 at openssh.com,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib debug2: kex_parse_kexinit: none,zlib at openssh.com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-cbc,3des-cbc debug2: kex_parse_kexinit: aes128-cbc,3des-cbc debug2: kex_parse_kexinit: hmac-md5,hmac-sha1 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1 debug2: kex_parse_kexinit: none debug2: kex_parse_kexinit: none debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_setup: found hmac-md5 debug1: kex: server->client aes128-cbc hmac-md5 none debug2: mac_setup: found hmac-md5 debug1: kex: client->server aes128-cbc hmac-md5 none debug2: dh_gen_key: priv key bits set: 134/256 debug2: bits set: 511/1024 debug1: sending SSH2_MSG_KEXDH_INIT debug1: expecting SSH2_MSG_KEXDH_REPLY Received disconnect from 10.20.12.30: 2: Client Disconnect The received disconnect is almost instant, so it is not a timeout or anything like that. I have also tried unsetting LANG, ssh -a, ssh -x etc. etc. -- Best regards Espen Fjellv?r Olsen efo at basefarm.no Basefarm AS From djm at mindrot.org Thu May 19 07:15:50 2011 From: djm at mindrot.org (Damien Miller) Date: Thu, 19 May 2011 07:15:50 +1000 (EST) Subject: Problem SSHing to HP ILO SSH-2.0-mpSSH_0.1.0 with 5.8p1 In-Reply-To: <4DD4216F.5070507@basefarm.no> References: <4DD4216F.5070507@basefarm.no> Message-ID: On Wed, 18 May 2011, Espen Fjellv?r Olsen wrote: > Hi everyone, > We are recently seeing a problem with OpenSSH 5.8p1 and SSH to ILO cards > running SSH-2.0-mpSSH_0.1.0. > This has previously worked with OpenSSH 5.5p1 (last known version for us to > work). > > ssh ilohost -vvv gives the following on 5.8p1: Could you try ssh -vvv -oKexAlgorithms=diffie-hellman-group1-sha1 ilohost? If that doesn't work, try adding "-oServerHostkeyAlgorithms=ssh-rsa" -d From efo at basefarm.no Thu May 19 07:19:04 2011 From: efo at basefarm.no (=?ISO-8859-1?Q?Espen_Fjellv=E6r_Olsen?=) Date: Wed, 18 May 2011 23:19:04 +0200 Subject: Problem SSHing to HP ILO SSH-2.0-mpSSH_0.1.0 with 5.8p1 In-Reply-To: References: <4DD4216F.5070507@basefarm.no> Message-ID: <4DD437C8.3070603@basefarm.no> On 18. mai 2011 23:15, Damien Miller wrote: > On Wed, 18 May 2011, Espen Fjellv?r Olsen wrote: > >> Hi everyone, >> We are recently seeing a problem with OpenSSH 5.8p1 and SSH to ILO cards >> running SSH-2.0-mpSSH_0.1.0. >> This has previously worked with OpenSSH 5.5p1 (last known version for us to >> work). >> >> ssh ilohost -vvv gives the following on 5.8p1: > Could you try > > ssh -vvv -oKexAlgorithms=diffie-hellman-group1-sha1 ilohost? > > If that doesn't work, try adding "-oServerHostkeyAlgorithms=ssh-rsa" > Aha, Heres something; -oKexAlgorithms=diffie-hellman-group1-sha1 did not work. -oServerHostkeyAlgorithms=ssh-rsa wasnt recognized as an option, but -oHostKeyAlgorithms=ssh-rsa on the other hand, did infact work! -- Espen FJellv?r Olsen Basefarm AS From djm at mindrot.org Thu May 19 08:24:01 2011 From: djm at mindrot.org (Damien Miller) Date: Thu, 19 May 2011 08:24:01 +1000 (EST) Subject: Problem SSHing to HP ILO SSH-2.0-mpSSH_0.1.0 with 5.8p1 In-Reply-To: <4DD437C8.3070603@basefarm.no> References: <4DD4216F.5070507@basefarm.no> <4DD437C8.3070603@basefarm.no> Message-ID: On Wed, 18 May 2011, Espen Fjellv?r Olsen wrote: > On 18. mai 2011 23:15, Damien Miller wrote: > > On Wed, 18 May 2011, Espen Fjellv?r Olsen wrote: > > > > > Hi everyone, > > > We are recently seeing a problem with OpenSSH 5.8p1 and SSH to ILO cards > > > running SSH-2.0-mpSSH_0.1.0. > > > This has previously worked with OpenSSH 5.5p1 (last known version for us > > > to > > > work). > > > > > > ssh ilohost -vvv gives the following on 5.8p1: > > Could you try > > > > ssh -vvv -oKexAlgorithms=diffie-hellman-group1-sha1 ilohost? > > > > If that doesn't work, try adding "-oServerHostkeyAlgorithms=ssh-rsa" > > > Aha, > Heres something; > -oKexAlgorithms=diffie-hellman-group1-sha1 did not work. > -oServerHostkeyAlgorithms=ssh-rsa wasnt recognized as an option, but > -oHostKeyAlgorithms=ssh-rsa on the other hand, did infact work! ok, so HP's ILO SSH implementation is junk. Harmlessly ignoring unsupported algorithms is the very point of the initial SSH negotiation, so that the HP code gets this really basic thing wrong is hugely worrying - if they can't get the simple stuff right, what else have they botched? Anyway, adding Host ilo1 ilo2 omfgilo ... KexAlgorithms diffie-hellman-group1-sha1 HostkeyAlgorithms ssh-rsa to your ~/.ssh/config (replaceing the host names) should let you connect. Could you please file a bug with HP? I'd love to hear what they say. -d From peter at stuge.se Thu May 19 10:30:09 2011 From: peter at stuge.se (Peter Stuge) Date: Thu, 19 May 2011 02:30:09 +0200 Subject: Might a patch to ssh-agent to allow relaxing of peer euid check be accepted? In-Reply-To: <20110518151253.GA3322@jadzia.bu.edu> References: <20110518151253.GA3322@jadzia.bu.edu> Message-ID: <20110519003009.9161.qmail@stuge.se> Matthew Miller wrote: > Right now, ssh-agent makes a check using getpeereid(), and declines access > if it fails. This is very sensible in general, but breaks this particular > case. Might a patch to allow an option to ssh-agent to relax the check be > accepted? I doubt it. I would suggest that you implement an ssh-agent proxy to sit in front of the actual agent, running as keyholder, where you implement policy. //Peter From Winston.Lee at commscope.com Thu May 19 11:18:38 2011 From: Winston.Lee at commscope.com (Lee, Winston) Date: Thu, 19 May 2011 09:18:38 +0800 Subject: Incorrect IP address in syslog when logging commands in Unix Message-ID: <242F1C57D225834BA3ED1B0BF8107120340171DDBD@SISPE7MB2.commscope.com> Hi All, I've turned on logging in the sshd_config file and it now logs all commands, which is great. The trouble is, the source IP address in the log is always the same, regardless of where I log in from. e.g. In syslog, "May 19 10:58:11 n1 audit: [ID 702911 audit.notice] execve(2) ok session 3800170877 by emob as root:root from 1.2.3.4 obj /usr/bin/less" IP address 1.2.3.4 doesn't change at all... This could jolly well be the IP address of the machine that was first used to connect using ssh. I'm not sure if this is an ssh problem but telnet seems to show the right address... Using OpenSSH 5.3 Cheers, Winston From mattdm at mattdm.org Thu May 19 23:25:30 2011 From: mattdm at mattdm.org (Matthew Miller) Date: Thu, 19 May 2011 09:25:30 -0400 Subject: Might a patch to ssh-agent to allow relaxing of peer euid check be accepted? In-Reply-To: <20110519003009.9161.qmail@stuge.se> References: <20110518151253.GA3322@jadzia.bu.edu> Message-ID: <20110519132530.GA26130@jadzia.bu.edu> Peter Stuge wrote: >> Right now, ssh-agent makes a check using getpeereid(), and declines >> access if it fails. This is very sensible in general, but breaks this >> particular case. Might a patch to allow an option to ssh-agent to relax >> the check be accepted? > I doubt it. I would suggest that you implement an ssh-agent proxy to sit > in front of the actual agent, running as keyholder, where you implement > policy. That's an interesting idea. However, for this case, that introduces complication without particular benefit, as we're not wanting to implement any particular policy but rather have ssh-agent _refrain_ from enforcing a hard-coded one. Without the check, simple policy can be implemented at the filesystem level (or through various security modules). It's worth noting that sshd itself doesn't implement the policy that ssh-agent does. If you forward your ssh-agent connection to a remote machine, there is no similar check. So, I do hope the simple patch can be considered. -- Matthew Miller mattdm at mattdm.org From mattdm at mattdm.org Thu May 19 23:32:42 2011 From: mattdm at mattdm.org (Matthew Miller) Date: Thu, 19 May 2011 09:32:42 -0400 Subject: Might a patch to ssh-agent to allow relaxing of peer euid check be accepted? In-Reply-To: <20110519132530.GA26130@jadzia.bu.edu> References: <20110518151253.GA3322@jadzia.bu.edu> <20110519132530.GA26130@jadzia.bu.edu> Message-ID: <20110519133242.GA27359@jadzia.bu.edu> On Thu, May 19, 2011 at 09:25:30AM -0400, Matthew Miller wrote: > So, I do hope the simple patch can be considered. Oh, actually, I see that it's already in the bugzilla, with a patch very similar to mine (it's simple and there's not too many ways to do it!) written by Damien Miller. See: https://bugzilla.mindrot.org/show_bug.cgi?id=1247 What would it take to move this from there into an accepted patch? -- Matthew Miller mattdm at mattdm.org From wmertens at cisco.com Thu May 19 23:51:46 2011 From: wmertens at cisco.com (Wout Mertens) Date: Thu, 19 May 2011 15:51:46 +0200 Subject: Might a patch to ssh-agent to allow relaxing of peer euid check be accepted? In-Reply-To: <20110519132530.GA26130@jadzia.bu.edu> References: <20110518151253.GA3322@jadzia.bu.edu> <20110519132530.GA26130@jadzia.bu.edu> Message-ID: On May 19, 2011, at 15:25 , Matthew Miller wrote: > Peter Stuge wrote: >>> Right now, ssh-agent makes a check using getpeereid(), and declines >>> access if it fails. This is very sensible in general, but breaks this >>> particular case. Might a patch to allow an option to ssh-agent to relax >>> the check be accepted? >> I doubt it. I would suggest that you implement an ssh-agent proxy to sit >> in front of the actual agent, running as keyholder, where you implement >> policy. > > That's an interesting idea. However, for this case, that introduces > complication without particular benefit, as we're not wanting to implement > any particular policy but rather have ssh-agent _refrain_ from enforcing a > hard-coded one. Without the check, simple policy can be implemented at the > filesystem level (or through various security modules). Why not simply give each user their own private key and add/remove it from the authorized_keys at the appropriate times? Wout. From mattdm at mattdm.org Fri May 20 01:29:50 2011 From: mattdm at mattdm.org (Matthew Miller) Date: Thu, 19 May 2011 11:29:50 -0400 Subject: Might a patch to ssh-agent to allow relaxing of peer euid check be accepted? In-Reply-To: References: <20110518151253.GA3322@jadzia.bu.edu> <20110519132530.GA26130@jadzia.bu.edu> Message-ID: <20110519152950.GA1032@jadzia.bu.edu> On Thu, May 19, 2011 at 03:51:46PM +0200, Wout Mertens wrote: > Why not simply give each user their own private key and add/remove it from > the authorized_keys at the appropriate times? With that model, there's a lot to keep track of. Individual users must keep track of their keys, and the various authorized_keys files must be managed carefully. There's no way to enforce "good" behavior with private key files: they might have no passphrase, they might get copied around or stolen -- potentially without the users' knowledge. And if there are a large number of accounts on different systems accessed in this way, one needs a system to manage those (and it's likely things will get overlooked). I'm open to entertaining more conversations of this nature, but I think it's really off-topic for this list and I don't want to trouble everyone with it -- I think it'd be better to send me such messages directly. Thanks. -- Matthew Miller mattdm at mattdm.org From dtucker at zip.com.au Fri May 20 10:24:53 2011 From: dtucker at zip.com.au (Darren Tucker) Date: Fri, 20 May 2011 10:24:53 +1000 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <4DD002A0.6040406@fifthhorseman.net> Message-ID: <4DD5B4D5.3010402@zip.com.au> On 16/05/11 12:51 PM, Damien Miller wrote: [...] > The latter is more clear for Match, but long lines are more likely to wrap > and are harder to read in sshd_config. > > That being said, there is plenty of room for the common cases that I can > think of: > > AuthorizedKeysFile .ssh/authorized_keys > AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 > AuthorizedKeysFile /etc/ssh/authorized_keys/keys_%u .ssh/authorized_keys > > So maybe all-keys-on-one-line is better. FWIW I agree with all-one-line as it's consistent with other (long) config directives like Ciphers. The only place that I can think of where we use multiple lines is Subsystem, and that's key-value pairs rather than a simple list. -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69 Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement. From dtucker at zip.com.au Fri May 20 11:05:05 2011 From: dtucker at zip.com.au (Darren Tucker) Date: Fri, 20 May 2011 11:05:05 +1000 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <4DD002A0.6040406@fifthhorseman.net> Message-ID: <4DD5BE41.600@zip.com.au> On 16/05/11 1:14 PM, Damien Miller wrote: > On Mon, 16 May 2011, Damien Miller wrote: [...] >> AuthorizedKeysFile .ssh/authorized_keys >> AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 >> AuthorizedKeysFile /etc/ssh/authorized_keys/keys_%u .ssh/authorized_keys >> >> So maybe all-keys-on-one-line is better. > > Here's a diff that implements this: Diff looks mostly OK, however I suggest the following: - all-one-line as mentioned earlier - continue to accept authorized_keys2 in sshd_config and stash in serveropts. - when the config file parsing is done, if it's set append the value of authorized_keys2 to the authorized_keys_files array. This should be equivalent to the current behaviour (maybe log a deprecation warning or something). - explicitly set AuthorizedKeysFile in the shipped sshd_config without authorized_keys2 (similar to the Protocol 1 deprecation). This should mean new installs won't look at authorized_keys2, but the behaviour of existing installs won't change during an upgrade. -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69 Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement. From djm at mindrot.org Fri May 20 11:19:29 2011 From: djm at mindrot.org (Damien Miller) Date: Fri, 20 May 2011 11:19:29 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <4DD5BE41.600@zip.com.au> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <4DD002A0.6040406@fifthhorseman.net> <4DD5BE41.600@zip.com.au> Message-ID: On Fri, 20 May 2011, Darren Tucker wrote: > On 16/05/11 1:14 PM, Damien Miller wrote: > > On Mon, 16 May 2011, Damien Miller wrote: > [...] > > > AuthorizedKeysFile .ssh/authorized_keys > > > AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 > > > AuthorizedKeysFile /etc/ssh/authorized_keys/keys_%u .ssh/authorized_keys > > > > > > So maybe all-keys-on-one-line is better. > > > > Here's a diff that implements this: > > Diff looks mostly OK, however I suggest the following: > - all-one-line as mentioned earlier done > - continue to accept authorized_keys2 in sshd_config and stash in serveropts. > - when the config file parsing is done, if it's set append the value of > authorized_keys2 to the authorized_keys_files array. This should be > equivalent to the current behaviour (maybe log a deprecation warning or > something). I disagree here: I accept the argument against changing the silently-working behaviour of looking for .ssh/authorized_keys2, but I think we should just deprecate the AuthorizedKeysFile2 option. It hasn't been documented for (10?) years and has been broken wrt Match blocks. By deprecating AuthorizedKeysFile2, server admins would get a clear warning on server start and restart and have a simple path to achieve the same result (multiple AuthorizedKeysFile). This wasn't the case with actually looking for authorized_keys2 files - there the faliure was completely silent. > - explicitly set AuthorizedKeysFile in the shipped sshd_config without > authorized_keys2 (similar to the Protocol 1 deprecation). Yes - new installations shouldn't look for authorized_keys2 files. We can remove looking for them by default completely some time in 2015 :) -d From jmknoble at pobox.com Fri May 20 12:26:46 2011 From: jmknoble at pobox.com (Jim Knoble) Date: Thu, 19 May 2011 19:26:46 -0700 Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <4DD5B4D5.3010402@zip.com.au> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <4DD002A0.6040406@fifthhorseman.net> <4DD5B4D5.3010402@zip.com.au> Message-ID: <8EA916E7-72E3-4DB9-9D88-2597CAFB2AE5@pobox.com> On May 19, 2011, at 17:24, Darren Tucker wrote: > On 16/05/11 12:51 PM, Damien Miller wrote: > [...] >> The latter is more clear for Match, but long lines are more likely >> to wrap >> and are harder to read in sshd_config. >> >> That being said, there is plenty of room for the common cases that >> I can >> think of: >> >> AuthorizedKeysFile .ssh/authorized_keys >> AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2 >> AuthorizedKeysFile /etc/ssh/authorized_keys/keys_%u .ssh/ >> authorized_keys >> >> So maybe all-keys-on-one-line is better. > > FWIW I agree with all-one-line as it's consistent with other (long) > config directives like Ciphers. The only place that I can think of > where we use multiple lines is Subsystem, and that's key-value pairs > rather than a simple list. While we're at it, Ciphers and MACs use commas for separators. Using spaces may prevent placing authorized_keys files in a directory whose name contains whitespace. From rongqing.li at windriver.com Fri May 20 14:39:40 2011 From: rongqing.li at windriver.com (Rongqing Li) Date: Fri, 20 May 2011 12:39:40 +0800 Subject: [patch] the memory which is allocated by matchpathcon should be freed after it is useless Message-ID: <4DD5F08C.2090205@windriver.com> the memory which is allocated by matchpathcon should be freed after it is useless Signed-off-by: Roy Li --- openbsd-compat/port-linux.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) --- a/openbsd-compat/port-linux.c +++ b/openbsd-compat/port-linux.c @@ -217,8 +217,10 @@ ssh_selinux_setfscreatecon(const char *p setfscreatecon(NULL); return; } - if (matchpathcon(path, 0700, &context) == 0) + if (matchpathcon(path, 0700, &context) == 0) { setfscreatecon(context); + freecon(context); + } } #endif /* WITH_SELINUX */ From djm at mindrot.org Fri May 20 18:47:53 2011 From: djm at mindrot.org (Damien Miller) Date: Fri, 20 May 2011 18:47:53 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <4DD002A0.6040406@fifthhorseman.net> <4DD5BE41.600@zip.com.au> Message-ID: Here's the latest version of the diff. It fixes a couple more bugs, particularly around config passing between privsep monitor and child. Index: auth-rsa.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth-rsa.c,v retrieving revision 1.79 diff -u -p -r1.79 auth-rsa.c --- auth-rsa.c 3 Dec 2010 23:55:27 -0000 1.79 +++ auth-rsa.c 20 May 2011 08:44:32 -0000 @@ -157,44 +157,27 @@ auth_rsa_challenge_dialog(Key *key) return (success); } -/* - * check if there's user key matching client_n, - * return key if login is allowed, NULL otherwise - */ - -int -auth_rsa_key_allowed(struct passwd *pw, BIGNUM *client_n, Key **rkey) +static int +rsa_key_allowed_in_file(struct passwd *pw, char *file, + const BIGNUM *client_n, Key **rkey) { - char line[SSH_MAX_PUBKEY_BYTES], *file; + char line[SSH_MAX_PUBKEY_BYTES]; int allowed = 0; u_int bits; FILE *f; u_long linenum = 0; Key *key; - /* Temporarily use the user's uid. */ - temporarily_use_uid(pw); - - /* The authorized keys. */ - file = authorized_keys_file(pw); debug("trying public RSA key file %s", file); - f = auth_openkeyfile(file, pw, options.strict_modes); - if (!f) { - xfree(file); - restore_uid(); - return (0); - } - - /* Flag indicating whether the key is allowed. */ - allowed = 0; - - key = key_new(KEY_RSA1); + if ((f = auth_openkeyfile(file, pw, options.strict_modes)) == NULL) + return 0; /* * Go though the accepted keys, looking for the current key. If * found, perform a challenge-response dialog to verify that the * user really has the corresponding private key. */ + key = key_new(KEY_RSA1); while (read_keyfile_line(f, file, line, sizeof(line), &linenum) != -1) { char *cp; char *key_options; @@ -232,7 +215,10 @@ auth_rsa_key_allowed(struct passwd *pw, } /* cp now points to the comment part. */ - /* Check if the we have found the desired key (identified by its modulus). */ + /* + * Check if the we have found the desired key (identified + * by its modulus). + */ if (BN_cmp(key->rsa->n, client_n) != 0) continue; @@ -261,11 +247,7 @@ auth_rsa_key_allowed(struct passwd *pw, break; } - /* Restore the privileged uid. */ - restore_uid(); - /* Close the file. */ - xfree(file); fclose(f); /* return key if allowed */ @@ -273,7 +255,33 @@ auth_rsa_key_allowed(struct passwd *pw, *rkey = key; else key_free(key); - return (allowed); + + return allowed; +} + +/* + * check if there's user key matching client_n, + * return key if login is allowed, NULL otherwise + */ + +int +auth_rsa_key_allowed(struct passwd *pw, BIGNUM *client_n, Key **rkey) +{ + char *file; + u_int i, allowed = 0; + + temporarily_use_uid(pw); + + for (i = 0; !allowed && i < options.num_authkeys_files; i++) { + file = expand_authorized_keys( + options.authorized_keys_files[i], pw); + allowed = rsa_key_allowed_in_file(pw, file, client_n, rkey); + xfree(file); + } + + restore_uid(); + + return allowed; } /* Index: auth.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth.c,v retrieving revision 1.92 diff -u -p -r1.92 auth.c --- auth.c 11 May 2011 04:47:06 -0000 1.92 +++ auth.c 20 May 2011 08:44:32 -0000 @@ -241,7 +241,7 @@ auth_root_allowed(char *method) * * This returns a buffer allocated by xmalloc. */ -static char * +char * expand_authorized_keys(const char *filename, struct passwd *pw) { char *file, ret[MAXPATHLEN]; @@ -265,12 +265,6 @@ expand_authorized_keys(const char *filen } char * -authorized_keys_file(struct passwd *pw) -{ - return expand_authorized_keys(options.authorized_keys_file, pw); -} - -char * authorized_principals_file(struct passwd *pw) { if (options.authorized_principals_file == NULL) Index: auth.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth.h,v retrieving revision 1.68 diff -u -p -r1.68 auth.h --- auth.h 11 May 2011 04:47:06 -0000 1.68 +++ auth.h 20 May 2011 08:44:32 -0000 @@ -145,7 +145,7 @@ struct passwd * getpwnamallow(const char char *get_challenge(Authctxt *); int verify_response(Authctxt *, const char *); -char *authorized_keys_file(struct passwd *); +char *expand_authorized_keys(const char *, struct passwd *pw); char *authorized_principals_file(struct passwd *); FILE *auth_openkeyfile(const char *, struct passwd *, int); Index: auth2-pubkey.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/auth2-pubkey.c,v retrieving revision 1.28 diff -u -p -r1.28 auth2-pubkey.c --- auth2-pubkey.c 11 May 2011 04:47:06 -0000 1.28 +++ auth2-pubkey.c 20 May 2011 08:44:32 -0000 @@ -435,7 +435,7 @@ user_cert_trusted_ca(struct passwd *pw, int user_key_allowed(struct passwd *pw, Key *key) { - int success; + u_int success, i; char *file; if (auth_key_is_revoked(key)) @@ -447,9 +447,12 @@ user_key_allowed(struct passwd *pw, Key if (success) return success; - file = authorized_keys_file(pw); - success = user_key_allowed2(pw, key, file); - xfree(file); + for (i = 0; !success && i < options.num_authkeys_files; i++) { + file = expand_authorized_keys( + options.authorized_keys_files[i], pw); + success = user_key_allowed2(pw, key, file); + xfree(file); + } return success; } Index: monitor.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/monitor.c,v retrieving revision 1.112 diff -u -p -r1.112 monitor.c --- monitor.c 20 May 2011 03:25:45 -0000 1.112 +++ monitor.c 20 May 2011 08:44:33 -0000 @@ -547,6 +547,7 @@ mm_answer_pwnamallow(int sock, Buffer *m char *username; struct passwd *pwent; int allowed = 0; + u_int i; debug3("%s", __func__); @@ -589,6 +590,10 @@ mm_answer_pwnamallow(int sock, Buffer *m if (options.x != NULL) \ buffer_put_cstring(m, options.x); \ } while (0) +#define M_CP_STRARRAYOPT(x, nx) do { \ + for (i = 0; i < options.nx; i++) \ + buffer_put_cstring(m, options.x[i]); \ + } while (0) /* See comment in servconf.h */ COPY_MATCH_STRING_OPTS(); #undef M_CP_STROPT @@ -604,7 +609,6 @@ mm_answer_pwnamallow(int sock, Buffer *m monitor_permit(mon_dispatch, MONITOR_REQ_AUTHSERV, 1); monitor_permit(mon_dispatch, MONITOR_REQ_AUTH2_READ_BANNER, 1); } - return (0); } Index: monitor_wrap.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/monitor_wrap.c,v retrieving revision 1.71 diff -u -p -r1.71 monitor_wrap.c --- monitor_wrap.c 20 May 2011 03:25:45 -0000 1.71 +++ monitor_wrap.c 20 May 2011 08:44:33 -0000 @@ -202,7 +202,7 @@ mm_getpwnamallow(const char *username) { Buffer m; struct passwd *pw; - u_int len; + u_int len, i; ServerOptions *newopts; debug3("%s entering", __func__); @@ -238,6 +238,10 @@ out: #define M_CP_STROPT(x) do { \ if (newopts->x != NULL) \ newopts->x = buffer_get_string(&m, NULL); \ + } while (0) +#define M_CP_STRARRAYOPT(x, nx) do { \ + for (i = 0; i < newopts->nx; i++) \ + newopts->x[i] = buffer_get_string(&m, NULL); \ } while (0) /* See comment in servconf.h */ COPY_MATCH_STRING_OPTS(); Index: pathnames.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/pathnames.h,v retrieving revision 1.21 diff -u -p -r1.21 pathnames.h --- pathnames.h 11 May 2011 04:47:06 -0000 1.21 +++ pathnames.h 20 May 2011 08:44:33 -0000 @@ -88,6 +88,9 @@ */ #define _PATH_SSH_USER_PERMITTED_KEYS ".ssh/authorized_keys" +/* backward compat for protocol v2 */ +#define _PATH_SSH_USER_PERMITTED_KEYS2 ".ssh/authorized_keys2" + /* * Per-user and system-wide ssh "rc" files. These files are executed with * /bin/sh before starting the shell or command if they exist. They will be Index: servconf.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/servconf.c,v retrieving revision 1.218 diff -u -p -r1.218 servconf.c --- servconf.c 20 May 2011 03:25:45 -0000 1.218 +++ servconf.c 20 May 2011 08:44:33 -0000 @@ -119,7 +119,7 @@ initialize_server_options(ServerOptions options->use_dns = -1; options->client_alive_interval = -1; options->client_alive_count_max = -1; - options->authorized_keys_file = NULL; + options->num_authkeys_files = 0; options->num_accept_env = 0; options->permit_tun = -1; options->num_permitted_opens = -1; @@ -249,8 +249,12 @@ fill_default_server_options(ServerOption options->client_alive_interval = 0; if (options->client_alive_count_max == -1) options->client_alive_count_max = 3; - if (options->authorized_keys_file == NULL) - options->authorized_keys_file = xstrdup(_PATH_SSH_USER_PERMITTED_KEYS); + if (options->num_authkeys_files == 0) { + options->authorized_keys_files[options->num_authkeys_files++] = + xstrdup(_PATH_SSH_USER_PERMITTED_KEYS); + options->authorized_keys_files[options->num_authkeys_files++] = + xstrdup(_PATH_SSH_USER_PERMITTED_KEYS2); + } if (options->permit_tun == -1) options->permit_tun = SSH_TUNMODE_NO; if (options->zero_knowledge_password_authentication == -1) @@ -286,7 +290,7 @@ typedef enum { sMaxStartups, sMaxAuthTries, sMaxSessions, sBanner, sUseDNS, sHostbasedAuthentication, sHostbasedUsesNameFromPacketOnly, sClientAliveInterval, - sClientAliveCountMax, sAuthorizedKeysFile, sAuthorizedKeysFile2, + sClientAliveCountMax, sAuthorizedKeysFile, sGssAuthentication, sGssCleanupCreds, sAcceptEnv, sPermitTunnel, sMatch, sPermitOpen, sForceCommand, sChrootDirectory, sUsePrivilegeSeparation, sAllowAgentForwarding, @@ -391,7 +395,7 @@ static struct { { "clientaliveinterval", sClientAliveInterval, SSHCFG_GLOBAL }, { "clientalivecountmax", sClientAliveCountMax, SSHCFG_GLOBAL }, { "authorizedkeysfile", sAuthorizedKeysFile, SSHCFG_ALL }, - { "authorizedkeysfile2", sAuthorizedKeysFile2, SSHCFG_ALL }, + { "authorizedkeysfile2", sDeprecated, SSHCFG_ALL }, { "useprivilegeseparation", sUsePrivilegeSeparation, SSHCFG_GLOBAL}, { "acceptenv", sAcceptEnv, SSHCFG_GLOBAL }, { "permittunnel", sPermitTunnel, SSHCFG_ALL }, @@ -1197,11 +1201,22 @@ process_server_config_line(ServerOptions * AuthorizedKeysFile /etc/ssh_keys/%u */ case sAuthorizedKeysFile: - charptr = &options->authorized_keys_file; - goto parse_tilde_filename; + if (*activep && options->num_authkeys_files == 0) { + while ((arg = strdelim(&cp)) && *arg != '\0') { + if (options->num_authkeys_files >= + MAX_AUTHKEYS_FILES) + fatal("%s line %d: " + "too many authorized keys files.", + filename, linenum); + options->authorized_keys_files[ + options->num_authkeys_files++] = + tilde_expand_filename(arg, getuid()); + } + } + return 0; + case sAuthorizedPrincipalsFile: charptr = &options->authorized_principals_file; - parse_tilde_filename: arg = strdelim(&cp); if (!arg || *arg == '\0') fatal("%s line %d: missing file name.", @@ -1420,6 +1435,12 @@ parse_server_match_config(ServerOptions dst->n = src->n; \ } \ } while(0) +#define M_CP_STRARRAYOPT(n, num_n) do {\ + if (src->num_n != 0) { \ + for (dst->num_n = 0; dst->num_n < src->num_n; dst->num_n++) \ + dst->n[dst->num_n] = xstrdup(src->n[dst->num_n]); \ + } \ +} while(0) /* * Copy any supported values that are set. @@ -1464,12 +1485,14 @@ copy_set_server_options(ServerOptions *d */ if (preauth) return; + M_CP_STROPT(adm_forced_command); M_CP_STROPT(chroot_directory); } #undef M_CP_INTOPT #undef M_CP_STROPT +#undef M_CP_STRARRAYOPT void parse_server_config(ServerOptions *options, const char *filename, Buffer *conf, @@ -1583,7 +1606,18 @@ dump_cfg_strarray(ServerOpCodes code, u_ u_int i; for (i = 0; i < count; i++) - printf("%s %s\n", lookup_opcode_name(code), vals[i]); + printf("%s %s\n", lookup_opcode_name(code), vals[i]); +} + +static void +dump_cfg_strarray_oneline(ServerOpCodes code, u_int count, char **vals) +{ + u_int i; + + printf("%s", lookup_opcode_name(code)); + for (i = 0; i < count; i++) + printf(" %s", vals[i]); + printf("\n"); } void @@ -1676,7 +1710,6 @@ dump_config(ServerOptions *o) dump_cfg_string(sCiphers, o->ciphers); dump_cfg_string(sMacs, o->macs); dump_cfg_string(sBanner, o->banner); - dump_cfg_string(sAuthorizedKeysFile, o->authorized_keys_file); dump_cfg_string(sForceCommand, o->adm_forced_command); dump_cfg_string(sChrootDirectory, o->chroot_directory); dump_cfg_string(sTrustedUserCAKeys, o->trusted_user_ca_keys); @@ -1689,6 +1722,8 @@ dump_config(ServerOptions *o) dump_cfg_string(sLogFacility, log_facility_name(o->log_facility)); /* string array arguments */ + dump_cfg_strarray_oneline(sAuthorizedKeysFile, o->num_authkeys_files, + o->authorized_keys_files); dump_cfg_strarray(sHostKeyFile, o->num_host_key_files, o->host_key_files); dump_cfg_strarray(sHostKeyFile, o->num_host_cert_files, Index: servconf.h =================================================================== RCS file: /cvs/src/usr.bin/ssh/servconf.h,v retrieving revision 1.97 diff -u -p -r1.97 servconf.h --- servconf.h 20 May 2011 03:25:45 -0000 1.97 +++ servconf.h 20 May 2011 08:44:33 -0000 @@ -27,6 +27,7 @@ #define MAX_HOSTCERTS 256 /* Max # host certificates. */ #define MAX_ACCEPT_ENV 256 /* Max # of env vars. */ #define MAX_MATCH_GROUPS 256 /* Max # of groups for Match. */ +#define MAX_AUTHKEYS_FILES 256 /* Max # of authorized_keys files. */ /* permit_root_login */ #define PERMIT_NOT_SET -1 @@ -145,7 +146,8 @@ typedef struct { * disconnect the session */ - char *authorized_keys_file; /* File containing public keys */ + u_int num_authkeys_files; /* Files containing public keys */ + char *authorized_keys_files[MAX_AUTHKEYS_FILES]; char *adm_forced_command; @@ -169,8 +171,8 @@ typedef struct { M_CP_STROPT(banner); \ M_CP_STROPT(trusted_user_ca_keys); \ M_CP_STROPT(revoked_keys_file); \ - M_CP_STROPT(authorized_keys_file); \ M_CP_STROPT(authorized_principals_file); \ + M_CP_STRARRAYOPT(authorized_keys_files, num_authkeys_files); \ } while (0) void initialize_server_options(ServerOptions *); /* Do fork() after authentication. Used by "ssh -f" */ Index: sshd.8 =================================================================== RCS file: /cvs/src/usr.bin/ssh/sshd.8,v retrieving revision 1.260 diff -u -p -r1.260 sshd.8 --- sshd.8 28 Oct 2010 18:33:28 -0000 1.260 +++ sshd.8 20 May 2011 08:44:33 -0000 @@ -435,10 +435,12 @@ is run, and if that does not exist either, xauth is used to add the cookie. .Sh AUTHORIZED_KEYS FILE FORMAT .Cm AuthorizedKeysFile -specifies the file containing public keys for +specifies the file or files containing public keys for public key authentication; -if none is specified, the default is -.Pa ~/.ssh/authorized_keys . +if none is specified, the default is both +.Pa ~/.ssh/authorized_keys +and +.Pa ~/.ssh/authorized_keys2 . Each line of the file contains one key (empty lines and lines starting with a .Ql # Index: sshd_config =================================================================== RCS file: /cvs/src/usr.bin/ssh/sshd_config,v retrieving revision 1.83 diff -u -p -r1.83 sshd_config --- sshd_config 6 May 2011 01:03:35 -0000 1.83 +++ sshd_config 20 May 2011 08:44:33 -0000 @@ -42,7 +42,7 @@ #RSAAuthentication yes #PubkeyAuthentication yes -#AuthorizedKeysFile .ssh/authorized_keys +AuthorizedKeysFile .ssh/authorized_keys # For this to work you will also need host keys in /etc/ssh/ssh_known_hosts #RhostsRSAAuthentication no Index: sshd_config.5 =================================================================== RCS file: /cvs/src/usr.bin/ssh/sshd_config.5,v retrieving revision 1.131 diff -u -p -r1.131 sshd_config.5 --- sshd_config.5 8 Dec 2010 04:02:47 -0000 1.131 +++ sshd_config.5 20 May 2011 08:44:33 -0000 @@ -168,8 +168,11 @@ After expansion, .Cm AuthorizedKeysFile is taken to be an absolute path or one relative to the user's home directory. -The default is -.Dq .ssh/authorized_keys . +The default is both +.Dq .ssh/authorized_keys +and +.Dq .ssh/authorized_keys2 . +Multiple files may be listed separated by whitespace. .It Cm AuthorizedPrincipalsFile Specifies a file that lists principal names that are accepted for certificate authentication. From djm at mindrot.org Fri May 20 18:49:19 2011 From: djm at mindrot.org (Damien Miller) Date: Fri, 20 May 2011 18:49:19 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <4DD002A0.6040406@fifthhorseman.net> <4DD5BE41.600@zip.com.au> Message-ID: ... and here is the regress part (with some fixes for existing races) Index: cfgmatch.sh =================================================================== RCS file: /cvs/src/regress/usr.bin/ssh/cfgmatch.sh,v retrieving revision 1.4 diff -u -p -r1.4 cfgmatch.sh --- cfgmatch.sh 13 Dec 2006 08:36:36 -0000 1.4 +++ cfgmatch.sh 20 May 2011 08:48:09 -0000 @@ -7,21 +7,47 @@ pidfile=$OBJ/remote_pid fwdport=3301 fwd="-L $fwdport:127.0.0.1:$PORT" +echo "ExitOnForwardFailure=yes" >> ssh_config +echo "ExitOnForwardFailure=yes" >> ssh_proxy + +start_client() +{ + rm -f $pidfile + ${SSH} -q -$p $fwd "$@" somehost \ + exec sh -c \'"echo \$\$ > $pidfile; exec sleep 100"\' \ + >>$TEST_SSH_LOGFILE 2>&1 & + client_pid=$! + # Wait for remote end + n=0 + while test ! -f $pidfile ; do + sleep 1 + n=`expr $n + 1` + if test $n -gt 60; then + kill $client_pid + fatal "timeout waiting for background ssh" + fi + done +} + stop_client() { pid=`cat $pidfile` if [ ! -z "$pid" ]; then kill $pid fi + wait } cp $OBJ/sshd_proxy $OBJ/sshd_proxy_bak - echo "PermitOpen 127.0.0.1:1" >>$OBJ/sshd_config echo "Match Address 127.0.0.1" >>$OBJ/sshd_config echo "PermitOpen 127.0.0.1:$PORT" >>$OBJ/sshd_config +grep -v AuthorizedKeysFile $OBJ/sshd_proxy_bak > $OBJ/sshd_proxy +echo "AuthorizedKeysFile /dev/null" >>$OBJ/sshd_proxy echo "PermitOpen 127.0.0.1:1" >>$OBJ/sshd_proxy +echo "Match user $USER" >>$OBJ/sshd_proxy +echo "AuthorizedKeysFile /dev/null $OBJ/authorized_keys_%u" >>$OBJ/sshd_proxy echo "Match Address 127.0.0.1" >>$OBJ/sshd_proxy echo "PermitOpen 127.0.0.1:$PORT" >>$OBJ/sshd_proxy @@ -31,12 +57,8 @@ start_sshd # Test Match + PermitOpen in sshd_config. This should be permitted for p in 1 2; do - rm -f $pidfile trace "match permitopen localhost proto $p" - ${SSH} -$p $fwd -F $OBJ/ssh_config -f somehost \ - exec sh -c \'"echo \$\$ > $pidfile; exec sleep 100"\' >>$TEST_SSH_LOGFILE 2>&1 ||\ - fail "match permitopen proto $p sshd failed" - sleep 1; + start_client -F $OBJ/ssh_config ${SSH} -q -$p -p $fwdport -F $OBJ/ssh_config somehost true || \ fail "match permitopen permit proto $p" stop_client @@ -44,12 +66,8 @@ done # Same but from different source. This should not be permitted for p in 1 2; do - rm -f $pidfile trace "match permitopen proxy proto $p" - ${SSH} -q -$p $fwd -F $OBJ/ssh_proxy -f somehost \ - exec sh -c \'"echo \$\$ > $pidfile; exec sleep 100"\' >>$TEST_SSH_LOGFILE 2>&1 ||\ - fail "match permitopen proxy proto $p sshd failed" - sleep 1; + start_client -F $OBJ/ssh_proxy ${SSH} -q -$p -p $fwdport -F $OBJ/ssh_config somehost true && \ fail "match permitopen deny proto $p" stop_client @@ -61,12 +79,8 @@ cat $OBJ/rsa.pub >> $OBJ/authorized_keys echo -n 'permitopen="127.0.0.1:'$PORT'" ' >>$OBJ/authorized_keys_$USER cat $OBJ/rsa1.pub >> $OBJ/authorized_keys_$USER for p in 1 2; do - rm -f $pidfile trace "match permitopen proxy w/key opts proto $p" - ${SSH} -q -$p $fwd -F $OBJ/ssh_proxy -f somehost \ - exec sh -c \'"echo \$\$ > $pidfile; exec sleep 100"\' >>$TEST_SSH_LOGFILE 2>&1 ||\ - fail "match permitopen w/key opt proto $p sshd failed" - sleep 1; + start_client -F $OBJ/ssh_proxy ${SSH} -q -$p -p $fwdport -F $OBJ/ssh_config somehost true && \ fail "match permitopen deny w/key opt proto $p" stop_client @@ -75,12 +89,8 @@ done # Test both sshd_config and key options permitting the same dst/port pair. # Should be permitted. for p in 1 2; do - rm -f $pidfile trace "match permitopen localhost proto $p" - ${SSH} -$p $fwd -F $OBJ/ssh_config -f somehost \ - exec sh -c \'"echo \$\$ > $pidfile; exec sleep 100"\' >>$TEST_SSH_LOGFILE 2>&1 ||\ - fail "match permitopen proto $p sshd failed" - sleep 1; + start_client -F $OBJ/ssh_config ${SSH} -q -$p -p $fwdport -F $OBJ/ssh_config somehost true || \ fail "match permitopen permit proto $p" stop_client @@ -93,12 +103,8 @@ echo "PermitOpen 127.0.0.1:1 127.0.0.1:2 # Test that a Match overrides a PermitOpen in the global section for p in 1 2; do - rm -f $pidfile trace "match permitopen proxy w/key opts proto $p" - ${SSH} -q -$p $fwd -F $OBJ/ssh_proxy -f somehost \ - exec sh -c \'"echo \$\$ > $pidfile; exec sleep 100"\' >>$TEST_SSH_LOGFILE 2>&1 ||\ - fail "match override permitopen proto $p sshd failed" - sleep 1; + start_client -F $OBJ/ssh_proxy ${SSH} -q -$p -p $fwdport -F $OBJ/ssh_config somehost true && \ fail "match override permitopen proto $p" stop_client @@ -112,12 +118,8 @@ echo "PermitOpen 127.0.0.1:1 127.0.0.1:2 # Test that a rule that doesn't match doesn't override, plus test a # PermitOpen entry that's not at the start of the list for p in 1 2; do - rm -f $pidfile trace "nomatch permitopen proxy w/key opts proto $p" - ${SSH} -q -$p $fwd -F $OBJ/ssh_proxy -f somehost \ - exec sh -c \'"echo \$\$ > $pidfile; exec sleep 100"\' >>$TEST_SSH_LOGFILE 2>&1 ||\ - fail "nomatch override permitopen proto $p sshd failed" - sleep 1; + start_client -F $OBJ/ssh_proxy ${SSH} -q -$p -p $fwdport -F $OBJ/ssh_config somehost true || \ fail "nomatch override permitopen proto $p" stop_client From keisial at gmail.com Sat May 21 00:28:34 2011 From: keisial at gmail.com (=?ISO-8859-1?Q?=C1ngel_Gonz=E1lez?=) Date: Fri, 20 May 2011 16:28:34 +0200 Subject: Might a patch to ssh-agent to allow relaxing of peer euid check be accepted? In-Reply-To: <20110519152950.GA1032@jadzia.bu.edu> References: <20110518151253.GA3322@jadzia.bu.edu> <20110519132530.GA26130@jadzia.bu.edu> <20110519152950.GA1032@jadzia.bu.edu> Message-ID: <4DD67A92.6000506@gmail.com> Matthew Miller wrote: > On Thu, May 19, 2011 at 03:51:46PM +0200, Wout Mertens wrote: >> Why not simply give each user their own private key and add/remove it from >> the authorized_keys at the appropriate times? > With that model, there's a lot to keep track of. Individual users must keep > track of their keys, and the various authorized_keys files must be managed > carefully. There's no way to enforce "good" behavior with private key files: > they might have no passphrase, they might get copied around or stolen -- > potentially without the users' knowledge. And if there are a large number of > accounts on different systems accessed in this way, one needs a system to > manage those (and it's likely things will get overlooked). > > I'm open to entertaining more conversations of this nature, but I think it's > really off-topic for this list and I don't want to trouble everyone with it > -- I think it'd be better to send me such messages directly. Thanks. Maybe you could use a CA issuing short-lived certificates to your clients? From pcerny at suse.cz Sat May 21 02:00:05 2011 From: pcerny at suse.cz (Petr Cerny) Date: Fri, 20 May 2011 18:00:05 +0200 Subject: Problem SSHing to HP ILO SSH-2.0-mpSSH_0.1.0 with 5.8p1 In-Reply-To: References: <4DD4216F.5070507@basefarm.no> <4DD437C8.3070603@basefarm.no> Message-ID: <4DD69005.3070604@suse.cz> Damien Miller wrote: > On Wed, 18 May 2011, Espen Fjellv?r Olsen wrote: > >> On 18. mai 2011 23:15, Damien Miller wrote: >> > On Wed, 18 May 2011, Espen Fjellv?r Olsen wrote: >> > >> > > Hi everyone, >> > > We are recently seeing a problem with OpenSSH 5.8p1 and SSH to ILO cards >> > > running SSH-2.0-mpSSH_0.1.0. >> > > This has previously worked with OpenSSH 5.5p1 (last known version for us >> > > to >> > > work). >> > > >> > > ssh ilohost -vvv gives the following on 5.8p1: >> > Could you try >> > >> > ssh -vvv -oKexAlgorithms=diffie-hellman-group1-sha1 ilohost? >> > >> > If that doesn't work, try adding "-oServerHostkeyAlgorithms=ssh-rsa" >> > >> Aha, >> Heres something; >> -oKexAlgorithms=diffie-hellman-group1-sha1 did not work. >> -oServerHostkeyAlgorithms=ssh-rsa wasnt recognized as an option, but >> -oHostKeyAlgorithms=ssh-rsa on the other hand, did infact work! > > ok, so HP's ILO SSH implementation is junk. Harmlessly ignoring unsupported > algorithms is the very point of the initial SSH negotiation, so that the > HP code gets this really basic thing wrong is hugely worrying - if they > can't get the simple stuff right, what else have they botched? mpSSH is not really the best SSH implementation around - among other things it seems to fail when requested to set up an environment variable, so be careful not to SendEnv anything to HP iLO. Just for the record - we've had this issue reported from one of our customers and the solution found has been: $ ssh -vvv \ -o PasswordAuthentication=yes \ -o ChallengeResponseAuthentication=no \ -o GSSAPIAuthentication=no \ -o HostbasedAuthentication=no \ -o PubkeyAuthentication=no \ -o RSAAuthentication=no \ -o Compression=no \ -o ForwardAgent=no \ -o ForwardX11=no \ -o Ciphers=aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128 \ -o HostKeyAlgorithms=ssh-rsa,ssh-dss \ user at HP_iLO Along with the patch (for v 5.1p1-): date: 2008/11/03 08:20:14; - markus at cvs.openbsd.org 2008/09/11 14:22:37 [compat.c compat.h nchan.c ssh.c] only send eow and no-more-sessions requests to openssh 5 and newer; fixes interop problems with broken ssh v2 implementations; ok djm@ Kind regards Petr -- Petr Cerny Mozilla/OpenSSH maintainer for SUSE Linux From efo at basefarm.no Sat May 21 02:08:05 2011 From: efo at basefarm.no (=?ISO-8859-1?Q?Espen_Fjellv=E6r_Olsen?=) Date: Fri, 20 May 2011 18:08:05 +0200 Subject: Problem SSHing to HP ILO SSH-2.0-mpSSH_0.1.0 with 5.8p1 In-Reply-To: References: <4DD4216F.5070507@basefarm.no> <4DD437C8.3070603@basefarm.no> Message-ID: <4DD691E5.5060609@basefarm.no> On 19. mai 2011 00:24, Damien Miller wrote: > On Wed, 18 May 2011, Espen Fjellv?r Olsen wrote: > >> On 18. mai 2011 23:15, Damien Miller wrote: >>> On Wed, 18 May 2011, Espen Fjellv?r Olsen wrote: >>> >>>> Hi everyone, >>>> We are recently seeing a problem with OpenSSH 5.8p1 and SSH to ILO cards >>>> running SSH-2.0-mpSSH_0.1.0. >>>> This has previously worked with OpenSSH 5.5p1 (last known version for us >>>> to >>>> work). >>>> >>>> ssh ilohost -vvv gives the following on 5.8p1: >>> Could you try >>> >>> ssh -vvv -oKexAlgorithms=diffie-hellman-group1-sha1 ilohost? >>> >>> If that doesn't work, try adding "-oServerHostkeyAlgorithms=ssh-rsa" >>> >> Aha, >> Heres something; >> -oKexAlgorithms=diffie-hellman-group1-sha1 did not work. >> -oServerHostkeyAlgorithms=ssh-rsa wasnt recognized as an option, but >> -oHostKeyAlgorithms=ssh-rsa on the other hand, did infact work! > ok, so HP's ILO SSH implementation is junk. Harmlessly ignoring unsupported > algorithms is the very point of the initial SSH negotiation, so that the > HP code gets this really basic thing wrong is hugely worrying - if they > can't get the simple stuff right, what else have they botched? > > Anyway, adding > > Host ilo1 ilo2 omfgilo ... > KexAlgorithms diffie-hellman-group1-sha1 > HostkeyAlgorithms ssh-rsa > > to your ~/.ssh/config (replaceing the host names) should let you connect. > > Could you please file a bug with HP? I'd love to hear what they say. > This did the trick, yes. Thank you Damien. I am still awaiting to file a bug with HP. Need some of the hardware people to give me access to our service agreement details over at HP, or file a bug them selves :) Will reply back when i get an answer from HP (Altough I am sure they will tell us to upgrade last years servers to this years servers which comes with ILO3 and not this problem). -- Br Espen Fjellv?r Olsen Basefarm AS From Kent.Wick at ers.state.tx.us Sat May 21 04:14:59 2011 From: Kent.Wick at ers.state.tx.us (Kent Wick) Date: Fri, 20 May 2011 13:14:59 -0500 Subject: Possible error in coding of AllowUsers / AllowGroups in ssh 5.8p2 Message-ID: <90D3595279D731438117A083CB9635B1059BCA6C02@PEXCHMB003.ad.ers.state.tx.us> What I was trying to do: I wanted to use the AllowGroups facility to allow users in by group instead of listing individual usernames but also allow root only from a single central host. Setup actions: targetusername on target host has a secondary group entry of "staff". Updated sshd_config to add the lines: AllowUsers root at nimsrvr AllowGroups staff targertusername is NOT listed in AllowUsers Stopped and started sshd Attempted to ssh from another host as "ssh targetusername at targethost date" I always get the syslog message "user X from Y not allowed because not listed in AllowUsers. The possible error (as I see it): The man page reads as if I should be able to specify a groupname or list of groupnames without having to specify a list of usernames. (it should be treated as an "or" condition). The way that I read the code in "auth.c" is: If the AllowUsers option is present, check the targetusername against the AllowedUsers list. If it is not in the list, return false (which appears to reject the login). Only if the targetusername is in the AllowUsers list will the code then check the AllowGroups list. It appears that the code treats the AllowUsers / AllowGroups as an "and" condition rather than an "or" condition as the man page implies. So either the code needs to change or the man page needs to change to be more explicit on the processing of the parameters. :) Kent Wick 512 867 7325 Unix System Admin From djm at mindrot.org Sat May 21 17:00:57 2011 From: djm at mindrot.org (Damien Miller) Date: Sat, 21 May 2011 17:00:57 +1000 (EST) Subject: backdoor by authorized_keys2 leftovers In-Reply-To: <8EA916E7-72E3-4DB9-9D88-2597CAFB2AE5@pobox.com> References: <20110509142103.GC26185@rzssh1.informatik.uni-hamburg.de> <86B9DA49-6149-4772-B578-D1AFFC150773@doxpara.com> <87wrhxiwm5.fsf@servo.factory.finestructure.net> <4DD002A0.6040406@fifthhorseman.net> <4DD5B4D5.3010402@zip.com.au> <8EA916E7-72E3-4DB9-9D88-2597CAFB2AE5@pobox.com> Message-ID: On Thu, 19 May 2011, Jim Knoble wrote: > While we're at it, Ciphers and MACs use commas for separators. Using > spaces may prevent placing authorized_keys files in a directory whose > name contains whitespace. I think the solution for these is to make strdelim() handle quotes, because paths with spaces are broken more generally. IIRC there is already a bug for this if anyone wants to do it. -d From aris at 0xbadc0de.be Mon May 23 18:00:09 2011 From: aris at 0xbadc0de.be (Aris Adamantiadis) Date: Mon, 23 May 2011 10:00:09 +0200 Subject: Security of OpenSSL ECDSA signatures Message-ID: <4DDA1409.8090607@0xbadc0de.be> Dear OpenSSH devs, I came accross this paper yesterday. http://eprint.iacr.org/2011/232 It states that they were able to recover ECDSA keys from TLS servers by using timing attacks agains OpenSSL's ECDSA implementation. Is that known to be exploitable by OpenSSH ? (In my understanding, it's easy to get a payload signed by ECDSA during the key exchange so my opinion is that it is). There's a patch for openssl in the paper, that remove the detectable optimization away. Would you consider blacklisting openssl versions that do not implement that workaround ? Abstract follows. Kr, Aris Abstract: For over two decades, timing attacks have been an active area of research within applied cryptography. These attacks exploit cryptosystem or protocol implementations that do not run in constant time. When implementing an elliptic curve cryptosystem that provides side-channel resistance, the scalar multiplication routine is a critical component. In such instances, one attractive method often suggested in the literature is Montgomery's ladder that performs a fixed sequence of curve and field operations. This paper describes a timing attack vulnerability in OpenSSL's ladder implementation for curves over binary fields. We use this vulnerability to steal the private key of a TLS server where the server authenticates with ECDSA signatures. Using the timing of the exchanged messages, the messages themselves, and the signatures, we mount a lattice attack that recovers the private key. Finally, we describe and implement an effective countermeasure. From djm at mindrot.org Mon May 23 22:31:49 2011 From: djm at mindrot.org (Damien Miller) Date: Mon, 23 May 2011 22:31:49 +1000 (EST) Subject: Security of OpenSSL ECDSA signatures In-Reply-To: <4DDA1409.8090607@0xbadc0de.be> References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: On Mon, 23 May 2011, Aris Adamantiadis wrote: > Dear OpenSSH devs, > > I came accross this paper yesterday. http://eprint.iacr.org/2011/232 > It states that they were able to recover ECDSA keys from TLS servers by > using timing attacks agains OpenSSL's ECDSA implementation. > Is that known to be exploitable by OpenSSH ? (In my understanding, it's > easy to get a payload signed by ECDSA during the key exchange so my > opinion is that it is). There's a patch for openssl in the paper, that > remove the detectable optimization away. Would you consider blacklisting > openssl versions that do not implement that workaround This result concerns binary/GF(2m) fields only and not the prime fields that OpenSSH uses in recent versions. Unless a similar timing oracle is found for GF(p) fields then no OpenSSH-side workaround is required. -d From aris at 0xbadc0de.be Mon May 23 22:34:48 2011 From: aris at 0xbadc0de.be (Aris Adamantiadis) Date: Mon, 23 May 2011 14:34:48 +0200 Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: <4DDA5468.4000004@0xbadc0de.be> Le 23/05/11 14:31, Damien Miller a ?crit : > This result concerns binary/GF(2m) fields only and not the prime fields > that OpenSSH uses in recent versions. > > Unless a similar timing oracle is found for GF(p) fields then no > OpenSSH-side workaround is required. Thanks for your explanation, I'm not familiar enough with ECC. Regards, Aris From dan at doxpara.com Mon May 23 23:13:03 2011 From: dan at doxpara.com (Dan Kaminsky) Date: Mon, 23 May 2011 06:13:03 -0700 Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: > > This result concerns binary/GF(2m) fields only and not the prime fields > that OpenSSH uses in recent versions. > > Unless a similar timing oracle is found for GF(p) fields then no > OpenSSH-side workaround is required. > > OpenSSL has had timing attacks against most of their production ciphers (RSA, AES, etc). Has the author of the paper weighed in on whether he thinks his attack will affect GF(p)? --Dan From djm at mindrot.org Tue May 24 07:36:35 2011 From: djm at mindrot.org (Damien Miller) Date: Tue, 24 May 2011 07:36:35 +1000 (EST) Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: On Mon, 23 May 2011, Dan Kaminsky wrote: > > Unless a similar timing oracle is found for GF(p) fields then no > > OpenSSH-side workaround is required. > > > > > OpenSSL has had timing attacks against most of their production ciphers > (RSA, AES, etc). Has the author of the paper weighed in on whether he > thinks his attack will affect GF(p)? The Brumley and Tuveri attack is against a scalar multiplication algorithm that is specific to GF(2m) fields (see section 3.2 of the paper). An attack on prime fields would be a new one altogether. -d From dan at doxpara.com Tue May 24 13:43:02 2011 From: dan at doxpara.com (Dan Kaminsky) Date: Mon, 23 May 2011 20:43:02 -0700 Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: On Mon, May 23, 2011 at 2:36 PM, Damien Miller wrote: > On Mon, 23 May 2011, Dan Kaminsky wrote: > > > > Unless a similar timing oracle is found for GF(p) fields then no > > > OpenSSH-side workaround is required. > > > > > > > > OpenSSL has had timing attacks against most of their production ciphers > > (RSA, AES, etc). Has the author of the paper weighed in on whether he > > thinks his attack will affect GF(p)? > > The Brumley and Tuveri attack is against a scalar multiplication > algorithm that is specific to GF(2m) fields (see section 3.2 of the > paper). An attack on prime fields would be a new one altogether. > > -d > I asked one of the timing attack guys if they were able to run their nanosecond scale attacks against a device having a network interface enforced jitter several orders of magnitude higher than what they were looking for. Command looked something like: tc qdisc change dev eth0 root netem delay 2ms 1ms No reply. I know in theory this shouldn't help, but if OpenSSH's ECDSA implementation is in fact variable time, why not add a random usleep at 10-100x the worst case scenario for at least average hardware? From djm at mindrot.org Tue May 24 16:29:14 2011 From: djm at mindrot.org (Damien Miller) Date: Tue, 24 May 2011 16:29:14 +1000 (EST) Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: On Mon, 23 May 2011, Dan Kaminsky wrote: > The Brumley and Tuveri attack is against a scalar multiplication > algorithm that is specific to GF(2m) fields (see section 3.2 of the > paper). An attack on prime fields would be a new one altogether. > > -d > > > I asked one of the timing attack guys if they were able to run their > nanosecond scale attacks against a device having a network interface > enforced jitter several orders of magnitude higher than what they were > looking for. Command looked something like: > > tc qdisc change dev eth0 root netem delay 2ms 1ms > > No reply. > > I know in theory this shouldn't help, but if OpenSSH's ECDSA implementation > is in fact variable time, why not add a random usleep at 10-100x the worst > case scenario for at least average hardware? random delays will not help because you can sample to eliminate them. I think you would want something like the following, that rounds signing operations up to the next power of two milliseconds. Index: key.c =================================================================== RCS file: /cvs/src/usr.bin/ssh/key.c,v retrieving revision 1.97 diff -u -p -r1.97 key.c --- key.c 17 May 2011 07:13:31 -0000 1.97 +++ key.c 24 May 2011 06:28:21 -0000 @@ -36,11 +36,13 @@ #include #include +#include #include #include #include +#include #include "xmalloc.h" #include "key.h" @@ -1588,28 +1590,68 @@ key_to_blob(const Key *key, u_char **blo return len; } +/* + * Round up to nearest power of two. Bit-twiddle algorithm from + * http://graphics.stanford.edu/~seander/bithacks.html#RoundUpPowerOf2 + */ +static u_int64_t +ceil2_u64(u_int64_t n) +{ + n--; + n |= n >> 1; + n |= n >> 2; + n |= n >> 4; + n |= n >> 8; + n |= n >> 16; + n |= n >> 32; + n++; + return n; +} + int key_sign( const Key *key, u_char **sigp, u_int *lenp, const u_char *data, u_int datalen) { + int result = -1; + struct timeval start, finish, diff; + u_int64_t duration, desired; + + gettimeofday(&start, NULL); switch (key->type) { case KEY_DSA_CERT_V00: case KEY_DSA_CERT: case KEY_DSA: - return ssh_dss_sign(key, sigp, lenp, data, datalen); + result = ssh_dss_sign(key, sigp, lenp, data, datalen); + break; case KEY_ECDSA_CERT: case KEY_ECDSA: - return ssh_ecdsa_sign(key, sigp, lenp, data, datalen); + result = ssh_ecdsa_sign(key, sigp, lenp, data, datalen); + break; case KEY_RSA_CERT_V00: case KEY_RSA_CERT: case KEY_RSA: - return ssh_rsa_sign(key, sigp, lenp, data, datalen); + result = ssh_rsa_sign(key, sigp, lenp, data, datalen); + break; default: error("key_sign: invalid key type %d", key->type); - return -1; + result = -1; + break; } + + /* + * Round up perceived duration of signing operation to the nearest + * power of two milliseconds by sleeping the difference. + */ + gettimeofday(&finish, NULL); + timersub(&finish, &start, &diff); + duration = diff.tv_sec * 1000000 + diff.tv_usec; + duration = (duration + 999) / 1000; /* round up to milliseconds */ + desired = ceil2_u64(duration); /* round up to 2^n */ + if (desired > duration) + usleep((desired - duration) * 1000); + return result; } /* From djm at mindrot.org Tue May 24 16:57:28 2011 From: djm at mindrot.org (Damien Miller) Date: Tue, 24 May 2011 16:57:28 +1000 (EST) Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: On Tue, 24 May 2011, Damien Miller wrote: > random delays will not help because you can sample to eliminate them. I > think you would want something like the following, that rounds signing > operations up to the next power of two milliseconds. FYI this sleeps for around 30ms on my old IBM x40 laptop* using ECDSA in a 256 bit curve field. -d * cpu0: Intel(R) Pentium(R) M processor 1200MHz ("GenuineIntel" 686-class) 1.20 GHz From dan at doxpara.com Tue May 24 17:12:16 2011 From: dan at doxpara.com (Dan Kaminsky) Date: Tue, 24 May 2011 00:12:16 -0700 Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: > > random delays will not help because you can sample to eliminate them. I > think you would want something like the following, that rounds signing > operations up to the next power of two milliseconds. > > I could see some weird corner case where the private key determines which edge a delay falls on, causing a response in 2ms or 4ms. That's actually a really clean signal to look for. The gold standard is constant time. But making every operation constant time is hard, and getting harder, due to higher level languages (not our problem). In theory, a random delay can be teased out by looking at the peak of the distribution of results -- it'll be slightly shifted when the original delay is minimal rather than maximal. But I tell you, I sure note the people doing timing attacks at the tens of nanosecond scale aren't getting their attacks to work versus Internet level latencies...that something is possible after billions of packets does not mean that something is necessarily practical. So I'd definitely do random noise over next-power-of-two-ms. Better than both though would be to discover the worst case scenario time for a handful of keys, double that, and then sleep until twice the difference has elapsed. So, if the worst case scenario for ECDSA was calculated as 10ms, execute, see a delay of 5ms, sleep 15ms, and go on with life. This keeps burning people, might as well fix it now. From djm at mindrot.org Tue May 24 18:08:14 2011 From: djm at mindrot.org (Damien Miller) Date: Tue, 24 May 2011 18:08:14 +1000 (EST) Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: On Tue, 24 May 2011, Dan Kaminsky wrote: > > > > random delays will not help because you can sample to eliminate them. I > > think you would want something like the following, that rounds signing > > operations up to the next power of two milliseconds. > > I could see some weird corner case where the private key determines which > edge a delay falls on, causing a response in 2ms or 4ms. That's actually a > really clean signal to look for. I don't think it likely that signing operations would be reliably poised at the precise microsecond bounday to cause it to flip between 2^n and 2^n+1. In any case, the proposed patch drops the (easily*) observable resolution to milliseconds, which are well above the threshold for timing attacks. Actually, we can make the magic increment harder to hit by using clock_gettime() to give us nanosecond precision. If someone came up with a credible attack, then we could drop the resolution further to 5ms increments and it still would not be perceptible to users. I guess if you wanted to dither (pun!) then you could do something like: + duration = (duration + (arc4random() & 1) ? 999 : 0) / 1000; to make it random whether you round up or down, but I suspect that would increase the infomation leaked rather than decrease it. > The aold standard is constant time. But making every operation constant > time is hard, and getting harder, due to higher level languages (not our > problem). In theory, a random delay can be teased out by looking at the > peak of the distribution of results -- it'll be slightly shifted when the > original delay is minimal rather than maximal. > > But I tell you, I sure note the people doing timing attacks at the tens of > nanosecond scale aren't getting their attacks to work versus Internet level > latencies...that something is possible after billions of packets does not > mean that something is necessarily practical. > > So I'd definitely do random noise over next-power-of-two-ms. That these attacks require nanosecond precision is a good argument for rounding to ms and 2^n too. The rounding approach doesn't have the statistical leak either. > Better than both though would be to discover the worst case scenario time > for a handful of keys, double that, and then sleep until twice the > difference has elapsed. So, if the worst case scenario for ECDSA was > calculated as 10ms, execute, see a delay of 5ms, sleep 15ms, and go on with > life. A handful of keys under what circumstances? An unloaded machine? A server that just started with cold caches? A machine running hundreds of concurrent signing operations? A machine that is deep in swap? There is no good guide except the time it took to complete the last signing operation itself. -d * sleeping to foil timing oracles might be susceptible to attacks that observe CPU load on the target. These might be possible for an authenticated user. From aris at 0xbadc0de.be Tue May 24 18:15:10 2011 From: aris at 0xbadc0de.be (Aris Adamantiadis) Date: Tue, 24 May 2011 10:15:10 +0200 Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: <4DDB690E.7010607@0xbadc0de.be> Le 24/05/11 10:08, Damien Miller a ?crit : > > I guess if you wanted to dither (pun!) then you could do something like: > > + duration = (duration + (arc4random() & 1) ? 999 : 0) / 1000; > > to make it random whether you round up or down, but I suspect that would > increase the infomation leaked rather than decrease it. > I think that shooting in the dark in order to block unknown timing attacks is likely to worsen the problem rather to mitigate it. Wouldn't be better to "simply" analyze the upstream algorithm and try to make it O(1), whatever the situation ? As said in the thread, putting timer loops will not stop side infoleaks like cpu load, cache misses etc. Aris From dan at doxpara.com Tue May 24 18:21:26 2011 From: dan at doxpara.com (Dan Kaminsky) Date: Tue, 24 May 2011 01:21:26 -0700 Subject: Security of OpenSSL ECDSA signatures In-Reply-To: <4DDB690E.7010607@0xbadc0de.be> References: <4DDA1409.8090607@0xbadc0de.be> <4DDB690E.7010607@0xbadc0de.be> Message-ID: On Tue, May 24, 2011 at 1:15 AM, Aris Adamantiadis wrote: > Le 24/05/11 10:08, Damien Miller a ?crit : > > > > I guess if you wanted to dither (pun!) then you could do something like: > > > > + duration = (duration + (arc4random() & 1) ? 999 : 0) / 1000; > > > > to make it random whether you round up or down, but I suspect that would > > increase the infomation leaked rather than decrease it. > > > > I think that shooting in the dark in order to block unknown timing > attacks is likely to worsen the problem rather to mitigate it. Wouldn't > be better to "simply" analyze the upstream algorithm and try to make it > O(1), whatever the situation ? > As said in the thread, putting timer loops will not stop side infoleaks > like cpu load, cache misses etc. > It's the quotes around "simply" that belie everything. I had someone swear up and down that MD5 was always constant time. And it should be. MD5 in Javascript isn't. It just isn't, not even close. Other people can pretend local security works; I'm simply going to focus on the network case. My assumption is that OpenSSH simply borrows the ECDSA implementation from OpenSSL and doesn't have the degrees of freedom to learn how to make it either constant or uncorrelated (as in RSA blinding) time. It does have the freedom to delay network traffic, however. From dan at doxpara.com Tue May 24 18:38:10 2011 From: dan at doxpara.com (Dan Kaminsky) Date: Tue, 24 May 2011 01:38:10 -0700 Subject: Security of OpenSSL ECDSA signatures In-Reply-To: References: <4DDA1409.8090607@0xbadc0de.be> Message-ID: On Tue, May 24, 2011 at 1:08 AM, Damien Miller wrote: > On Tue, 24 May 2011, Dan Kaminsky wrote: > > > > > > > random delays will not help because you can sample to eliminate them. I > > > think you would want something like the following, that rounds signing > > > operations up to the next power of two milliseconds. > > > > I could see some weird corner case where the private key determines which > > edge a delay falls on, causing a response in 2ms or 4ms. That's actually > a > > really clean signal to look for. > > I don't think it likely that signing operations would be reliably poised > at the precise microsecond bounday to cause it to flip between 2^n and > 2^n+1. In any case, the proposed patch drops the (easily*) observable > resolution to milliseconds, which are well above the threshold for > timing attacks. > > Actually, we can make the magic increment harder to hit by using > clock_gettime() to give us nanosecond precision. > > If someone came up with a credible attack, then we could drop the > resolution further to 5ms increments and it still would not be > perceptible to users. > > I guess if you wanted to dither (pun!) then you could do something like: > > + duration = (duration + (arc4random() & 1) ? 999 : 0) / 1000; > > to make it random whether you round up or down, but I suspect that would > increase the infomation leaked rather than decrease it. > Eh, the thing is that you're creating a _really_ strong signal (a multi millisecond delay) thats dependent on microsecond timings. The whole game is in making it so the attacker can't differentiate the slight microsecond differences derived from noise, from the slight differences derived from the underlying crypto functions. According to Nate Lawson, he got all sorts of interesting data watching the quantized distribution of network traffic off EC2 nodes. He was totally doing stuff involving a packet sent at 1.9999ms vs. 2.0000ms. I do think that's viable. > > > The aold standard is constant time. But making every operation constant > > time is hard, and getting harder, due to higher level languages (not our > > problem). In theory, a random delay can be teased out by looking at the > > peak of the distribution of results -- it'll be slightly shifted when the > > original delay is minimal rather than maximal. > > > > But I tell you, I sure note the people doing timing attacks at the tens > of > > nanosecond scale aren't getting their attacks to work versus Internet > level > > latencies...that something is possible after billions of packets does not > > mean that something is necessarily practical. > > > > So I'd definitely do random noise over next-power-of-two-ms. > > That these attacks require nanosecond precision is a good argument for > rounding to ms and 2^n too. The rounding approach doesn't have the > statistical leak either. > It does, it's just at the edge finding layer. Think of it like you have certain inputs that suddenly get super high jitter. > > > Better than both though would be to discover the worst case scenario time > > for a handful of keys, double that, and then sleep until twice the > > difference has elapsed. So, if the worst case scenario for ECDSA was > > calculated as 10ms, execute, see a delay of 5ms, sleep 15ms, and go on > with > > life. > > A handful of keys under what circumstances? An unloaded machine? A > server that just started with cold caches? A machine running hundreds > of concurrent signing operations? A machine that is deep in swap? There > is no good guide except the time it took to complete the last signing > operation itself. > So scale the delay to 2x-10x the longest signature of the last 6 hours, with some maximum, and some minimum too. I think we agree roughly that there's an intermediate mitigation between O(1) and nothing, heretical as that statement may be. > > -d > > * sleeping to foil timing oracles might be susceptible to attacks that > observe CPU load on the target. These might be possible for an > authenticated user. > Lets deal with the network case for now. From alon at nolaviz.org Fri May 27 17:17:52 2011 From: alon at nolaviz.org (Alon Ziv) Date: Fri, 27 May 2011 10:17:52 +0300 Subject: Security of OpenSSL ECDSA signatures Message-ID: <4DDF5020.4080809@nolaviz.org> Isn't it enough to limit the range of $k$ (the random value selected during signature generation) to values with the top bit set? (Or--if there are too few of those--only use values with "01" in the top bits, still leaving 2^{158} possible values?) -a From cal.leeming at simplicitymedialtd.co.uk Tue May 31 06:30:09 2011 From: cal.leeming at simplicitymedialtd.co.uk (Cal Leeming [Simplicity Media Ltd]) Date: Mon, 30 May 2011 21:30:09 +0100 Subject: Fwd: cgroup OOM killer loop causes system to lockup (possible fix included) In-Reply-To: <4DE3F7F9.4010709@simplicitymedialtd.co.uk> References: <4DE2CB80.8040101@simplicitymedialtd.co.uk> <4DE2E3A5.4060208@simplicitymedialtd.co.uk> <4DE39710.4050707@simplicitymedialtd.co.uk> <4DE3A8DB.2000208@simplicitymedialtd.co.uk> <4DE3D5BF.3010707@simplicitymedialtd.co.uk> <4DE3EFB4.10305@simplicitymedialtd.co.uk> <4DE3F7F9.4010709@simplicitymedialtd.co.uk> Message-ID: Hi all, Please find below a complete transcript of the emails between debian/kernel-mm mailing lists. I've had a response back from someone on the deb mailing list stating: ==================================== The bug seems to be that sshd does not reset the OOM adjustment before running the login shell (or other program). Therefore, please report a bug against openssh-server. ==================================== Therefore, I am submitting this bug to you also.. If someone would be kind enough to have a flick thru all the below debug/logs, it'd be very much appreciated. Cal ---------- Forwarded message ---------- From: Cal Leeming [Simplicity Media Ltd] < cal.leeming at simplicitymedialtd.co.uk> Date: Mon, May 30, 2011 at 9:03 PM Subject: Re: cgroup OOM killer loop causes system to lockup (possible fix included) To: debian-kernel at lists.debian.org, debian-user at lists.debian.org More strangeness.. If I keep the kernel module loaded, but disable the entry in /etc/network/interfaces for eth0, the oom_adj problem disappears. But then ofc, I'm left with no network interface. I then tried binding /etc/ssh/sshd_config to only listen on 127.0.0.1.. effectively bypassing the eth0 interface, whilst still allowing it to be loaded. But the problem still happens. As a last ditch attempt, I tried restarting the ssh daemon from the command line. I have pretty much explored almost every single option I can think of.. I now have absolutely no idea why this is happening. So, I've opted for the hacky fix of putting the following into /etc/profile and /etc/bash.bashrc: echo 0 > /proc/self/oom_adj This ensures all shell sessions start off with an oom_adj of 0. This is *EXTREMELY* hacky, and I'd certainly hope to see that the debian devs fix this issue in the future. Cal On 30/05/2011 20:27, Cal Leeming [Simplicity Media Ltd] wrote: Bit more info: root at vicky:~# ethtool -i eth0 driver: bnx2 version: 2.0.2 firmware-version: 5.0.11 NCSI 2.0.5 bus-info: 0000:02:00.0 root at vicky:~# Btw, to make 100% sure it was the bnx2 module which was causing the issue, I removed bnx2 from auto loading at boot time (on the broken server), ran the same check, and the issue had been resolved. But this meant I had no network card lol. I also tried disabling MSI by using kernel boot param: pci=nomsi, and this didn't seem to do anything. Cal On 30/05/2011 18:37, Cal Leeming [Simplicity Media Ltd] wrote: Here is an strace of the SSH process (which is somehow inheriting the -17 oom_adj on all forked user instances) (broken server - with bnx2 module loaded) [pid 2200] [ 7f13a09c9cb0] open("/proc/self/oom_adj", O_WRONLY|O_CREAT|O_TRUNC, 0666 [pid 2120] [ 7f13a09c9f00] write(7, "\0\0\2\240\n\n\n\nPort 22\n\n\n\nProtocol 2\n\nH"..., 680 [pid 2200] [ 7f13a09c9cb0] <... open resumed> ) = 9 [pid 2120] [ 7f13a09c9f00] <... write resumed> ) = 680 [pid 2120] [ 7f13a09c9e40] close(7 [pid 2200] [ 7f13a09c9844] fstat(9, [pid 2120] [ 7f13a09c9e40] <... close resumed> ) = 0 [pid 2200] [ 7f13a09c9844] <... fstat resumed> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 [pid 2120] [ 7f13a09c9e40] close(8 [pid 2200] [ 7f13a09d2a2a] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 [pid 2120] [ 7f13a09c9e40] <... close resumed> ) = 0 [pid 2200] [ 7f13a09d2a2a] <... mmap resumed> ) = 0x7f13a25a6000 [pid 2120] [ 7f13a09c9e40] close(4 [pid 2200] [ 7f13a09c9f00] write(9, "-17\n", 4 (working server - with bnx2 module unloaded) [pid 1323] [ 7fae577fbe40] close(7) = 0 [pid 1631] [ 7fae577fbcb0] open("/proc/self/oom_adj", O_WRONLY|O_CREAT|O_TRUNC, 0666 [pid 1323] [ 7fae577fbf00] write(8, "\0\0\2\217\0", 5 [pid 1631] [ 7fae577fbcb0] <... open resumed> ) = 10 [pid 1323] [ 7fae577fbf00] <... write resumed> ) = 5 [pid 1323] [ 7fae577fbf00] write(8, "\0\0\2\206\n\n\n\nPort 22\n\n\n\nProtocol 2\n\nH"..., 654 [pid 1631] [ 7fae577fb844] fstat(10, [pid 1323] [ 7fae577fbf00] <... write resumed> ) = 654 [pid 1631] [ 7fae577fb844] <... fstat resumed> {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 [pid 1323] [ 7fae577fbe40] close(8) = 0 [pid 1631] [ 7fae57804a2a] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 [pid 1323] [ 7fae577fbe40] close(9 [pid 1631] [ 7fae57804a2a] <... mmap resumed> ) = 0x7fae593d9000 [pid 1323] [ 7fae577fbe40] <... close resumed> ) = 0 [pid 1323] [ 7fae577fbe40] close(5 [pid 1631] [ 7fae577fbf00] write(10, "0\n", 2 The two servers are *EXACT* duplicates of each other, completely fresh Debian installs, with exactly the same packages installed. As you can see, the working server sends "0" into the oom_adj and the broken one sends "-17". On 30/05/2011 15:25, Cal Leeming [Simplicity Media Ltd] wrote: To eliminate the possibility of this being hardware related, or ISO related, I performed two new fresh installs. One on the (previously working) server, and another one (using the firmware modified ISO - from the debian 'unofficial' downloads page) into a VirtualBox machine. I then installed openssh-server on both servers, rebooted, and checked the results of both. Firmware modified ISO + openssh-server - OOM bug not present. Non firmware modified ISO + openssh-server - OOM bug not present. So, a fresh install on a VirtualBox machine (using the exact same install CD ISO as the broken server), and the old machine (using the official release), shows that the bug is not happening.. The only thing that I can think is different, is that the broken server uses 'firmware-bnx2'.. so I purged the bnx2 package, removed the bnx*.ko files from /lib/modules, ran update-initramfs, and then rebooted (i then confirmed it was removed by checking ifconfig and lsmod). And guess what.. IT FUCKING WORKED. So, this problem seems to be caused by the firmware-bnx2 module being loaded.. some how, that module is causing -17 oom_adj to be set for everything.. WTF!!!!!!!!!!!!!!!!!!!!!!! On 30/05/2011 14:09, Cal Leeming [Simplicity Media Ltd] wrote: Please note, the way I am detecting if the oom_adj bug is present, is by checking if the sshd user processes are -17. If they are, then the bug is present. ATTEMPT 1: apt-get install hashalot cryptsetup make g++ cpp automake ncurses-dev bison flex curl build-essential kernel-package locales locales-all - oom problem not found (pre-built from previous install) dpkg -i linux-headers-2.6.32.41-grsec_2.6.32.41-grsec-10.00.Custom_amd64.deb linux-image-2.6.32.41-grsec_2.6.32.41-grsec-10.00.Custom_amd64.deb - oom problem not found (booted into new kernel) shutdown -r now - oom problem DETECTED!!! (booted into old kernel) dpkg --purge linux-image-2.6.32.41-grsec linux-headers-2.6.32.41-grsec shutdown -r now - oom problem DETECTED!!! Hmmmm.... apt-get purge hashalot cryptsetup make g++ cpp automake ncurses-dev bison flex curl build-essential kernel-package locales locales-all shutdown -r now - oom problem STILL DETECTED!!! WTF!!!!!!!!!!!!!!! So, it would appear that either the kernel install or the "apt-get" caused irreversible changes on the system... REINSTALL ATTEMPT 2: (this time, I'm going to use tripwire to detect filesystem changes between each reboot) Installed openssh-server and configured tripwire. Made test modification to /usr/sbin and ran tripwire, to ensure its logging correctly. - oom problem not found rebooted - oom problem DETECTED!! Purged openssh-server - oom problem not found Reinstalled openssh-server - oom problem not found rebooted - oom problem DETECTED So, we've now pinpointed it down to the openssh package... This makes sense, because the oom_adj is inherited from whatever binary the process forks from.. In this case, the LXCs are started up from an ssh session, which is why it inherits the ssh -17 oom_adj. Here is what tripwire is reporting has changed: Modified: "/usr/sbin" "/usr/sbin/sshd" "/usr/lib" "/usr/lib/openssh" "/usr/lib/openssh/sftp-server" "/usr/lib/sftp-server" "/var/run/acpid.pid" "/var/run/acpid.socket" "/var/run/atd.pid" "/var/run/crond.pid" "/var/run/crond.reboot" "/var/run/exim4/exim.pid" "/var/run/mdadm/monitor.pid" "/var/run/portmap.pid" "/var/run/portmap_mapping" "/var/run/rpc.statd.pid" "/var/run/rsyslogd.pid" "/var/run/sm-notify.pid" "/var/log/dmesg.0" "/etc" "/etc/adjtime" "/etc/default" "/etc/default/ssh" "/etc/lvm/cache" "/etc/lvm/cache/.cache" "/etc/mtab" "/etc/network/if-up.d" "/etc/network/if-up.d/openssh-server" "/etc/network/run" "/etc/network/run/ifstate" "/etc/pam.d" "/etc/pam.d/sshd" "/etc/passwd-" "/etc/shadow-" "/etc/ssh" "/etc/ssh/ssh_host_dsa_key" "/etc/ssh/ssh_host_dsa_key.pub" "/etc/ssh/ssh_host_rsa_key" "/etc/ssh/ssh_host_rsa_key.pub" "/etc/ssh/sshd_config" "/etc/tripwire" "/etc/ufw" "/etc/ufw/applications.d" "/etc/ufw/applications.d/openssh-server" "/etc/init.d" "/etc/init.d/.depend.boot" "/etc/init.d/.depend.start" "/etc/init.d/.depend.stop" "/etc/init.d/ssh" "/etc/rc2.d" "/etc/rc2.d/S18ssh" "/etc/rc3.d" "/etc/rc3.d/S18ssh" "/etc/rc4.d" "/etc/rc4.d/S18ssh" "/etc/rc5.d" "/etc/rc5.d/S18ssh" "/etc/passwd" "/etc/shadow" "/root" "/root/.nano_history" Added: "/var/log/dmesg.1.gz" "/var/log/dmesg.2.gz" "/root/.bash_history" Removed: "/etc/nologin" I thought maybe the reason this is happening is because ssh (when being started from rc.d) is inheriting the -17 from init.. but it appears init doesn't have -17: root at vicky:~# cat /proc/1/oom_adj 0 Then I thought maybe its because the ssh versions are diff, but they are not: root at vicky:/home/foxx# dpkg -l | grep openssh-server ii openssh-server 1:5.5p1-6 secure shell (SSH) server, for secure access from remote machines root at courtney.internal [/home/foxx] > dpkg -l | grep openssh-server ii openssh-server 1:5.5p1-6 secure shell (SSH) server, for secure access from remote machines foxx at vicky:~$ md5sum /usr/sbin/sshd f8c11462e8f2a7bf80e212e06041492b /usr/sbin/sshd root at courtney.internal [/home/foxx] > md5sum /usr/sbin/sshd f8c11462e8f2a7bf80e212e06041492b /usr/sbin/sshd Then I made sure that the sshd_config's matched, and that I was using the same login process for both (shared key only)... they both matched, yet the problem still happens.. Then I thought maybe it's inheriting from the sshd server process, but turns out that isn't it either: (broken server) root 1583 0.0 0.0 49168 1140 ? Ss 12:42 0:00 /usr/sbin/sshd root at vicky:~# cat /proc/1583/oom_adj -17 (working server) root 2105 0.0 0.0 49184 1152 ? Ss 00:47 0:00 /usr/sbin/sshd root at courtney.internal [/home/foxx] > cat /proc/2105/oom_adj -17 So, I looked through the process tree to see where it was inheriting from at what stage.. (working server) root at courtney.internal [/home/foxx] > ps faux | grep sshd (-17) root 2105 0.0 0.0 49184 1152 ? Ss 00:47 0:00 /usr/sbin/sshd (0) root 4735 0.0 0.0 76668 3356 ? Ss 12:47 0:00 \_ sshd: foxx [priv] foxx 4740 0.0 0.0 76668 1644 ? S 12:47 0:00 \_ sshd: foxx at pts/0 root 4757 0.0 0.0 112344 876 pts/0 S+ 12:48 0:00 \_ grep sshd (broken server) foxx at vicky:~$ ps faux | grep sshd (-17) root 1583 0.0 0.0 49168 1140 ? Ss 12:42 0:00 /usr/sbin/sshd (-17) root 1616 0.0 0.0 70488 3376 ? Ss 12:43 0:00 \_ sshd: root at pts/0 (-17) root 1685 0.2 0.0 70488 3292 ? Ss 12:50 0:00 \_ sshd: foxx [priv] foxx 1688 0.0 0.0 70488 1576 ? S 12:50 0:00 \_ sshd: foxx at pts/1 foxx 1715 0.0 0.0 7544 840 pts/1 S+ 12:50 0:00 \_ grep sshd As you can see, the line where it says "sshd: foxx [priv]" is causing the -17 oom_adj. Accoridng to the documentation, this appears to be where the privilege seperation takes place. So, now I started to check the ssh packages themselves, and make sure the repos are exactly the same on both servers. At this point, I realise that the working server is slightly out of date on the following packages: root at courtney.internal [/home/foxx] > md5sum /etc/apt/sources.list 00bcf3cf28e2994f9b512f0a8ffb0765 /etc/apt/sources.list root at vicky:/etc# md5sum /etc/apt/sources.list 00bcf3cf28e2994f9b512f0a8ffb0765 /etc/apt/sources.list root at courtney.internal [/home/foxx] > apt-get upgrade The following packages will be upgraded: bind9-host dnsutils exim4 exim4-base exim4-config exim4-daemon-light host isc-dhcp-client isc-dhcp-common libbind9-60 libdns69 libisc62 libisccc60 libisccfg62 liblwres60 linux-base linux-image-2.6.32-5-amd64 linux-libc-dev sd-agent The one that springs immediately to my attention is linux-base. (working server) root at courtney.internal [/home/foxx] > dpkg -l | grep linux-base ii linux-base 2.6.32-31 Linux image base package (broken server) root at vicky:/etc# dpkg -l | grep linux-base ii linux-base 2.6.32-34squeeze1 Linux image base package Sooooo, I bite the bullet, and perform an upgrade of linux-base on the working server... root at courtney.internal [/home/foxx] > apt-get install linux-base Setting up linux-base (2.6.32-34squeeze1) ... I then re-run the dryrun upgrade command this to make sure its upgraded: bind9-host dnsutils exim4 exim4-base exim4-config exim4-daemon-light host isc-dhcp-client isc-dhcp-common libbind9-60 libdns69 libisc62 libisccc60 libisccfg62 liblwres60 linux-image-2.6.32-5-amd64 linux-libc-dev sd-agent (as you can see its disappeared from the list) I then reboot the server.. and wait for the longest 3 minutes of my life.. But guess what... it didn't break :/ So, I bite another bullet, and upgrade the remaining packages on the server: bind9-host dnsutils exim4 exim4-base exim4-config exim4-daemon-light host isc-dhcp-client isc-dhcp-common libbind9-60 libdns69 libisc62 libisccc60 libisccfg62 liblwres60 linux-image-2.6.32-5-amd64 linux-libc-dev sd-agent I then make sure both servers are running the exact same stock kernel from Debian (as the working server was using an old kernel). root at vicky:/etc# dpkg -l | grep linux | grep image ii linux-base 2.6.32-34squeeze1 Linux image base package ii linux-image-2.6-amd64 2.6.32+29 Linux 2.6 for 64-bit PCs (meta-package) ii linux-image-2.6.32-5-amd64 2.6.32-34squeeze1 Linux 2.6.32 for 64-bit PCs root at courtney.internal [/home/foxx] > dpkg -l | grep linux | grep image ii linux-base 2.6.32-34squeeze1 Linux image base package ii linux-image-2.6-amd64 2.6.32+29 Linux 2.6 for 64-bit PCs (meta-package) ii linux-image-2.6.32-5-amd64 2.6.32-34squeeze1 Linux 2.6.32 for 64-bit PCs root at vicky:/etc# uname -a Linux vicky 2.6.32-5-amd64 #1 SMP Wed May 18 23:13:22 UTC 2011 x86_64 GNU/Linux root at courtney.internal [/home/foxx] > uname -a Linux courtney.internal 2.6.32-5-amd64 #1 SMP Wed May 18 23:13:22 UTC 2011 x86_64 GNU/Linux After another long 3 minute wait, I test for the oom bug... and guess what.. it's STILL NOT DOING IT!!! :( So now I check for differences in the /etc/pam.d... I notice pam_cap.so is missing in common-auth on the broken server: root at courtney.internal [/etc/pam.d] > dpkg -l | grep cap ii libcap2 1:2.19-3 support for getting/setting POSIX.1e capabilities ii libcap2-bin 1:2.19-3 basic utility programs for using capabilities Broken server: ? PAM profiles to enable: ? ? ? ? [*] Unix authentication ? Working server: ? [*] Unix authentication ? ? [*] Inheritable Capabilities Management ? So, I install 'libcap2-bin' on the broken server, reboot.. and still no god damn luck. At this point /etc/pam.d on both servers are matching (md5sum matches up on all files).. So now, I decide to check all files relating to openssh-server.. again, all matches up fine. Then I start to get really pissed off, and check the md5sum for all files in /etc: (left is working server, right is broken) root at courtney.internal [/etc/pam.d] > diff /tmp/etcmd5-courtney /tmp/etcmd5-vicky -y --suppress-common-lines a81fbd39142e18d5ed1fb8a7b3ecce71 /etc/adjtime | fa9192c6cdaab85ec952576ab3139fd1 /etc/adjtime 7fcee51274f69cdf5d4c8b7be799637b /etc/apt/trustdb.gpg | 1319acca28ae6475a915ca0684d0cd62 /etc/apt/trustdb.gpg a3710991fcce0b1574586450c81095e1 /etc/apt/trusted.gpg | d802712c9255f13bea3bea87b83180b1 /etc/apt/trusted.gpg 366d165a9f5de024d3a21b9dc51de057 /etc/bash.bashrc | 5b3c3bc73d236e4e1b6f9b6c1ed5964e /etc/bash.bashrc 109789e71a8cf8e21302bf44b5b716f7 /etc/blkid.tab | aa0de4c3c85ae212f6c59b6b89b21b9a /etc/blkid.tab 2de357d9da09d738c179e8d269696b9c /etc/blkid.tab.old | aa0de4c3c85ae212f6c59b6b89b21b9a /etc/blkid.tab.old 22133b5bd4023d48b50863d2b1c7b25e /etc/console-setup/cached.k | bdd92d16a8172f9e7ea3f866b59b4fc4 /etc/console-setup/cached.k b88b0f0a4d3b4beec0cc4b09b6c2aaea /etc/cron.daily/ntp < 4e5aa59f38b520c5a45d3fdc7cdec46c /etc/cron.daily/sysstat | d1e8b20b26a33a6a0781d59bc312442e /etc/cron.daily/tripwire 455c3c071b6daabb4e4490828975034c /etc/cron.d/sysstat < 1cffe509bba095a0f7ece99a971e6e9a /etc/crypttab < 756141f7eacf1a272a2f6e51646b3aa4 /etc/default/cryptdisks < 6bba39eb6c39aef755f1fadb48ded5a5 /etc/default/lxc < cd7a62fbb18fa8fe5893dee93064b328 /etc/default/ntp < e0d7efac23e911c65f44b08de446e837 /etc/default/rsync < 21614b7a3d91ee06750feedbfdaec898 /etc/default/sysstat < fbc234ecd0f7e8bc1c394bbde5867be1 /etc/dhcp/dhclient-exit-hoo < 1a2b9d0a869e2aa885ae3621c557fb95 /etc/dpkg/shlibs.default < 84b1e69080569cc5c613a50887af5200 /etc/dpkg/shlibs.override < 297521889d690871ec9d89c5eeff745a /etc/emacs/site-start.d/50a < c10e4cfb6fb21c281f04105d9482736a /etc/exim4/update-exim4.con | 2109f7e59c5d7ab2431aad0f095e2e34 /etc/exim4/update-exim4.con 0bec0044c716f14083f72c42af543d16 /etc/fstab | ab91b08889eb01fa6cd2364eba136ae8 /etc/fstab c9457cf5b2196da67d5ac816d1c86a4f /etc/fuse.conf < e030dc891a3af1f7779429b5c0554c98 /etc/gdb/gdbinit < 5d151dd5c443ed7b2a5ded95740bf00d /etc/glusterd/glusterd.info < f8ab4b0d63d43e8385e8e0a7b0b0fdba /etc/group | 6c8ccd77ad88953d80a0c8230feb43b0 /etc/group b2eb12a6eb86aa16730fcc78f3856f99 /etc/group- | f8366cb252dab0be81d315dbb0bfd54d /etc/group- 1721fcab19363ee66aa66829b9876f0e /etc/gshadow | 9a53c91ec405b6c2589b257d1f610e11 /etc/gshadow b43f2208e196045e2e4eff32a32a43cb /etc/gshadow- | 5db46dc414e73d989833f1718646ec40 /etc/gshadow- 289108902ba56c6d3d10392b994f5063 /etc/hostname | be7724203e323a7c97fe531e3662521c /etc/hostname f6c39850a5646ce96a62f8bbfadcab12 /etc/hosts | e7358f34e94f27ce975c2beb64a5fd31 /etc/hosts e9d8dadacde9e17f0a9b19951109bd15 /etc/init.d/cryptdisks | 94c8893c0233f51b5b35c44afcc9d064 /etc/init.d/.depend.boot a544f8db0b5b71722ebf28cd29d5c99f /etc/init.d/cryptdisks-earl | 018bfdbf3ce7d5000d4771861558084c /etc/init.d/.depend.start 47f49c3084a87495a4b21b16d62f08ce /etc/init.d/.depend.boot | efcaecb9b1dbf7ee6999ce6d7fe6cbce /etc/init.d/.depend.stop c32f1bac3bf2ead96ef4328f8fa8b6a4 /etc/init.d/.depend.start < 7a5306deeb6f58cf9fddc44176e944b2 /etc/init.d/.depend.stop < 0a911b5d7bdf62f4a27f59544859a25c /etc/init.d/enc < d0b8cce6d932e1cd90812ce32c3f81a4 /etc/init.d/fuse < 88def01f8a173be24e1347d215d713f1 /etc/init.d/github01 < a8e1b7caac5f373ae3ee68cfc6703c4c /etc/init.d/lxc < 1593209e2edaef7930940759b07caee1 /etc/init.d/ntp < 9d74671cca3077de30a6cbed26d4cd0e /etc/init.d/rsync < 7702ad8bd63cbe13b8bb455199435191 /etc/init.d/screen-cleanup < ee350831ec30475b16b8bda31a3f24de /etc/init.d/sd-agent < 8fb5289db2c7f67aa9347ae7e8b445dd /etc/init.d/sws01 < e3cf21d607c6852e2e5013524c657c6e /etc/init.d/sysstat < bc93dd93f82749814f2bde70d9428c0d /etc/initramfs-tools/conf.d < 98fc27159733746de9890831d51d95d4 /etc/iptables.up.rules < 02e24efb09d5343d249780465b59bfd6 /etc/ld.so.cache | ac8d6701fefa22e490c2c956a047f224 /etc/ld.so.cache 3459aad5fab462e1c126997b4ac449bb /etc/logcheck/ignore.d.serv < 32d3f9199b3438cd41ed3cb1122135b7 /etc/lvm/backup/default | c0adb70d988e7ad895b42aa23644dce0 /etc/lvm/backup/mech a5ab01460cb3dba4daedd86002bdba67 /etc/lvm/cache/.cache | 4436f6d9c98cb918c4757f48446ddefc /etc/lvm/backup/ssd > a96425623ae854ba979f7fd8b901bd21 /etc/lvm/cache/.cache f51730e7056489397b8b2a9a9b99662c /etc/mailname | 1cc22bdbd9f0dd61c3dbdf484f5e1954 /etc/mailname 2b0e1a3f52757145d65c353ca49d7756 /etc/mdadm/mdadm.conf | 75ddc14817703ef468326041acd8bfb1 /etc/mdadm/mdadm.conf 2ed9e1f60da7f3f87faca9879d0a6531 /etc/mtab | d6eb5404b14026e3cb6377fdbfa36a07 /etc/mtab 0a38058aafd42b7b4105505379194e1b /etc/nanorc | fc57b93c907fefbccf12317d40b4a204 /etc/nanorc d8e3272886cfc3ca3e103840f17c28b3 /etc/network/interfaces | 11aed0887d5bd14a426e2a4f7c7d4f4a /etc/network/interfaces 0925a154d88cf0bc2d0f588ab7c670c9 /etc/network/run/ifstate | 40a731c20283789c9916b1978b8b56b8 /etc/network/run/ifstate 1e47bfedf84da1cdfaa45ee016fb3609 /etc/networks | 5293c479ab40a68b4fa7a6eeff755256 /etc/networks 3e250ecaf470e1d3a2b68edd5de46bfd /etc/ntp.conf < a3bf39d554578558648717be72405bb4 /etc/passwd | ec13c66df3dee36481a8c3d432e54d8f /etc/passwd c495660bf88840c35e5c3ede628e5e5d /etc/passwd- | ec13c66df3dee36481a8c3d432e54d8f /etc/passwd- b08c4faf56551a861d7ae6858ac52b2e /etc/profile | b94c2e3df2a779ac12080942df4d86ea /etc/profile fe0b86955e4eb444f17f54d086580b1f /etc/resolv.conf | 2bc8c1c0361ac0fae5581bcaf8d7f136 /etc/resolv.conf c22ef5f592ae97d3152e1d58657e2c8a /etc/rssh.conf < f4aa40956bb6f150815b4d60a505760c /etc/screenrc < 78b737784042d973d6bed47e7667b1bb /etc/sd-agent/config.cfg < 4eccd6267f438812bfa1d4eb8ac05217 /etc/shadow | e2f45652caa1cbb84c778adc75f7545b /etc/shadow 676a49b9dbe67ce8be7a2921f7e10570 /etc/shadow- | e2f45652caa1cbb84c778adc75f7545b /etc/shadow- 3c1144bd2727cf00af012021fa3de4c5 /etc/shells | 0e85c87e09d716ecb03624ccff511760 /etc/shells 9fa92b39192a027af603fbff3d2f42eb /etc/siege/siegerc < fb778297a8e612868e41225cf4db7c9d /etc/siege/urls.txt < 813856cf9d8c29095b3a4e19d92d3da0 /etc/ssh/ssh_host_dsa_key | 3f4beaeb582ce81b42cca475e65dc75a /etc/ssh/ssh_host_dsa_key 75d221c8d4abe42699ff813e5a1e8cc7 /etc/ssh/ssh_host_dsa_key.p | 55e34345f7a2e1ac5ec7ce78543487e7 /etc/ssh/ssh_host_dsa_key.p b85a52219856a7ecf95d625a1bee5068 /etc/ssh/ssh_host_rsa_key | 70af0ef16b661edd96c317058ef55a78 /etc/ssh/ssh_host_rsa_key 3aea4190a19facc76222e69c5700f5ac /etc/ssh/ssh_host_rsa_key.p | b8524bc48e4d5c71c69888b452e8d6ae /etc/ssh/ssh_host_rsa_key.p 16e9567a6298125264967d276e6a139f /etc/sudoers | c5dab0f2771411ed7e67d6dab60a311f /etc/sudoers a86605ae7354f25d8060bcb5ad83edf7 /etc/sysctl.conf | 2c6f89fdb09aeac5735144497a261782 /etc/sysctl.conf e52dbe02e5da26d9be965373676e9355 /etc/sysstat/sysstat < fa92b01baa2130e26822c30fb27ac56e /etc/sysstat/sysstat.ioconf < > d87271b624ab8e93c2e51cd59bade631 /etc/tripwire/site.key > 8f6ebb12f511b46fbf0203d978c3bf01 /etc/tripwire/tw.cfg > 1821c7a0d207a168f1d7c766f238e816 /etc/tripwire/twcfg.txt > 717b4afa4f0f8614f3947441a3ddf422 /etc/tripwire/tw.pol > 92c9b38e95c90eebf1d746633a81909c /etc/tripwire/tw.pol.bak > d08d31fa833b50d2fb9a37f97b07cbd0 /etc/tripwire/twpol.txt > fdbfa3e0879f0d959bbdfd5601ef4d4f /etc/tripwire/vicky-local.k aeb6fe5dcfc873b0632ba92345ed16e2 /etc/udev/rules.d/70-persis | 0fdf03b558e118edcf8ce29abaf296f1 /etc/udev/rules.d/70-persis 24cc33b9f96e3189b7e34cf5484cb99f /etc/udev/rules.d/70-persis | 4a49e7ddeacbb3ded8bb3968f219362c /etc/udev/rules.d/70-persis I patched up the /etc/init.d/.depend.* files so the 'ssh' entries are matching.. still. no. luck It's now 24 hours of trying to fix this issue, and I'm getting extremely pissed off :@ If anyone could offer up some advice, it would be VERY much appreciated. On 30/05/2011 01:24, Cal Leeming [Simplicity Media Ltd] wrote: Another quick update on this... Something really really really strange has happened this time.. I just did a completely fresh re-install (exactly like last time) and the problem went away. So, I'm going to build up the lxc / kernel support step by step (and check for the oom_adj problem after each change) and will report back once I've found the step it kicks in at. Hopefully, my hours of frustration will prevent someone else from going through the same thing :/ Cal On 29/05/2011 23:41, Cal Leeming [Simplicity Media Ltd] wrote: After submitting this problem to Kernel mailing list, I found a strange (and even more curious problem). I attempted to run both kernels on two seperate installs of Debian (both 6.0), but slightly different in packages.. One install did not have the OOM problem, yet the other one did.. And they both ran the same kernel, the same LXC container etc.. I have a feeling this may be a Debian bug..? Further to back up these claims: Working server: root at courtney.internal [/mnt/encstore/lxc/github01.rootfs] > for x in `find /proc -iname 'oom_adj' | xargs grep "\-17" | awk -F '/' '{print $3}'` ; do ps -p $x u --no-headers ; done grep: /proc/3090/task/3090/oom_adj: No such file or directory grep: /proc/3090/oom_adj: No such file or directory root 804 0.0 0.0 16992 1112 ? S for x in `find /proc -iname 'oom_adj' | xargs grep "\-17" | awk -F '/' '{print $3}'` ; do ps -p $x u --no-headers ; done grep: /proc/4597/task/4597/oom_adj: No such file or directory grep: /proc/4597/oom_adj: No such file or directory root 889 0.0 0.0 16788 860 ? S Organization: Simplicity Media Ltd To: linux-kernel at vger.kernel.org, linux-rt-users at vger.kernel.org Some further logs: ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.369927] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.369939] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.399285] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.399296] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.428690] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.428702] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.487696] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.487708] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.517023] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.517035] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.546379] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:38 vicky kernel: [ 2283.546391] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.310789] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.310804] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.369918] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.369930] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.399284] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.399296] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.433634] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.433648] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.463947] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.463959] [] ? oom_kill_process+0x82/0x283 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.493439] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=-17 ./log/syslog:May 30 07:44:43 vicky kernel: [ 2288.493451] [] ? oom_kill_process+0x82/0x283 On 29/05/2011 22:50, Cal Leeming [Simplicity Media Ltd] wrote: > First of all, my apologies if I have submitted this problem to the > wrong place, spent 20 minutes trying to figure out where it needs to > be sent, and was still none the wiser. > > The problem is related to applying memory limitations within a cgroup. > If the OOM killer kicks in, it gets stuck in a loop where it tries to > kill a process which has an oom_adj of -17. This causes an infinite > loop, which in turn locks up the system. > > May 30 03:13:08 vicky kernel: [ 1578.117055] Memory cgroup out of > memory: kill process 6016 (java) score 0 or a child > May 30 03:13:08 vicky kernel: [ 1578.117154] Memory cgroup out of > memory: kill process 6016 (java) score 0 or a child > May 30 03:13:08 vicky kernel: [ 1578.117248] Memory cgroup out of > memory: kill process 6016 (java) score 0 or a child > May 30 03:13:08 vicky kernel: [ 1578.117343] Memory cgroup out of > memory: kill process 6016 (java) score 0 or a child > May 30 03:13:08 vicky kernel: [ 1578.117441] Memory cgroup out of > memory: kill process 6016 (java) score 0 or a child > > > root at vicky [/home/foxx] > uname -a > Linux vicky 2.6.32.41-grsec #3 SMP Mon May 30 02:34:43 BST 2011 x86_64 > GNU/Linux > (this happens on both the grsec patched and non patched 2.6.32.41 kernel) > > When this is encountered, the memory usage across the whole server is > still within limits (not even hitting swap). > > The memory configuration for the cgroup/lxc is: > lxc.cgroup.memory.limit_in_bytes = 3000M > lxc.cgroup.memory.memsw.limit_in_bytes = 3128M > > Now, what is even more strange, is that when running under the > 2.6.32.28 kernel (both patched and unpatched), this problem doesn't > happen. However, there is a slight difference between the two kernels. > The 2.6.32.28 kernel gives a default of 0 in the /proc/X/oom_adj, > where as the 2.6.32.41 gives a default of -17. I suspect this is the > root cause of why it's showing in the later kernel, but not the earlier. > > To test this theory, I started up the lxc on both servers, and then > ran a one liner which showed me all the processes with an oom_adj of -17: > > (the below is the older/working kernel) > root at courtney.internal [/mnt/encstore/lxc] > uname -a > Linux courtney.internal 2.6.32.28-grsec #3 SMP Fri Feb 18 16:09:07 GMT > 2011 x86_64 GNU/Linux > root at courtney.internal [/mnt/encstore/lxc] > for x in `find /proc > -iname 'oom_adj' | xargs grep "\-17" | awk -F '/' '{print $3}'` ; do > ps -p $x --no-headers ; done > grep: /proc/1411/task/1411/oom_adj: No such file or directory > grep: /proc/1411/oom_adj: No such file or directory > 804 ? 00:00:00 udevd > 804 ? 00:00:00 udevd > 25536 ? 00:00:00 sshd > 25536 ? 00:00:00 sshd > 31861 ? 00:00:00 sshd > 31861 ? 00:00:00 sshd > 32173 ? 00:00:00 udevd > 32173 ? 00:00:00 udevd > 32174 ? 00:00:00 udevd > 32174 ? 00:00:00 udevd > > (the below is the newer/broken kernel) > root at vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41] > uname -a > Linux vicky 2.6.32.41-grsec #3 SMP Mon May 30 02:34:43 BST 2011 x86_64 > GNU/Linux > root at vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41] > for x in > `find /proc -iname 'oom_adj' | xargs grep "\-17" | awk -F '/' '{print > $3}'` ; do ps -p $x --no-headers ; done > grep: /proc/3118/task/3118/oom_adj: No such file or directory > grep: /proc/3118/oom_adj: No such file or directory > 895 ? 00:00:00 udevd > 895 ? 00:00:00 udevd > 1091 ? 00:00:00 udevd > 1091 ? 00:00:00 udevd > 1092 ? 00:00:00 udevd > 1092 ? 00:00:00 udevd > 2596 ? 00:00:00 sshd > 2596 ? 00:00:00 sshd > 2608 ? 00:00:00 sshd > 2608 ? 00:00:00 sshd > 2613 ? 00:00:00 sshd > 2613 ? 00:00:00 sshd > 2614 pts/0 00:00:00 bash > 2614 pts/0 00:00:00 bash > 2620 pts/0 00:00:00 sudo > 2620 pts/0 00:00:00 sudo > 2621 pts/0 00:00:00 su > 2621 pts/0 00:00:00 su > 2622 pts/0 00:00:00 bash > 2622 pts/0 00:00:00 bash > 2685 ? 00:00:00 lxc-start > 2685 ? 00:00:00 lxc-start > 2699 ? 00:00:00 init > 2699 ? 00:00:00 init > 2939 ? 00:00:00 rc > 2939 ? 00:00:00 rc > 2942 ? 00:00:00 startpar > 2942 ? 00:00:00 startpar > 2964 ? 00:00:00 rsyslogd > 2964 ? 00:00:00 rsyslogd > 2964 ? 00:00:00 rsyslogd > 2964 ? 00:00:00 rsyslogd > 2980 ? 00:00:00 startpar > 2980 ? 00:00:00 startpar > 2981 ? 00:00:00 ctlscript.sh > 2981 ? 00:00:00 ctlscript.sh > 3016 ? 00:00:00 cron > 3016 ? 00:00:00 cron > 3025 ? 00:00:00 mysqld_safe > 3025 ? 00:00:00 mysqld_safe > 3032 ? 00:00:00 sshd > 3032 ? 00:00:00 sshd > 3097 ? 00:00:00 mysqld.bin > 3097 ? 00:00:00 mysqld.bin > 3097 ? 00:00:00 mysqld.bin > 3097 ? 00:00:00 mysqld.bin > 3097 ? 00:00:00 mysqld.bin > 3097 ? 00:00:00 mysqld.bin > 3097 ? 00:00:00 mysqld.bin > 3097 ? 00:00:00 mysqld.bin > 3097 ? 00:00:00 mysqld.bin > 3097 ? 00:00:00 mysqld.bin > 3113 ? 00:00:00 ctl.sh > 3113 ? 00:00:00 ctl.sh > 3115 ? 00:00:00 sleep > 3115 ? 00:00:00 sleep > 3116 ? 00:00:00 .memcached.bin > 3116 ? 00:00:00 .memcached.bin > > > As you can see, it is clear that the newer kernel is setting -17 by > default, which in turn is causing the OOM killer loop. > > So I began to try and find what may have caused this problem by > comparing the two sources... > > I checked the code for all references to 'oom_adj' and 'oom_adjust' in > both code sets, but found no obvious differences: > grep -R -e oom_adjust -e oom_adj . | sort | grep -R -e oom_adjust -e > oom_adj > > Then I checked for references to "-17" in all .c and .h files, and > found a couple of matches, but only one obvious one: > grep -R "\-17" . | grep -e ".c:" -e ".h:" -e "\-17" | wc -l > ./include/linux/oom.h:#define OOM_DISABLE (-17) > > But again, a search for OOM_DISABLE came up with nothing obvious... > > In a last ditch attempt, I did a search for all references to 'oom' > (case-insensitive) in both code bases, then compared the two: > root at annabelle [~/lol/linux-2.6.32.28] > grep -i -R "oom" . | sort -n > > /tmp/annabelle.oom_adj > root at vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41] > grep -i -R > "oom" . | sort -n > /tmp/vicky.oom_adj > > and this brought back (yet again) nothing obvious.. > > > root at vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41] > md5sum > ./include/linux/oom.h > 2a32622f6cd38299fc2801d10a9a3ea8 ./include/linux/oom.h > > root at annabelle [~/lol/linux-2.6.32.28] > md5sum ./include/linux/oom.h > 2a32622f6cd38299fc2801d10a9a3ea8 ./include/linux/oom.h > > root at vicky [/mnt/encstore/ssd/kernel/linux-2.6.32.41] > md5sum > ./mm/oom_kill.c > 1ef2c2bec19868d13ec66ec22033f10a ./mm/oom_kill.c > > root at annabelle [~/lol/linux-2.6.32.28] > md5sum ./mm/oom_kill.c > 1ef2c2bec19868d13ec66ec22033f10a ./mm/oom_kill.c > > > > Could anyone please shed some light as to why the default oom_adj is > set to -17 now (and where it is actually set)? From what I can tell, > the fix for this issue will either be: > > 1. Allow OOM killer to override the decision of ignoring oom_adj == > -17 if an unrecoverable loop is encountered. > 2. Change the default back to 0. > > Again, my apologies if this bug report is slightly unorthodox, or > doesn't follow usual procedure etc. I can assure you I have tried my > absolute best to give all the necessary information though. > > Cal > From cal.leeming at simplicitymedialtd.co.uk Tue May 31 06:56:23 2011 From: cal.leeming at simplicitymedialtd.co.uk (Cal Leeming [Simplicity Media Ltd]) Date: Mon, 30 May 2011 21:56:23 +0100 Subject: Fwd: Re: Fwd: cgroup OOM killer loop causes system to lockup (possible fix included) - now pinpointed to openssh-server Message-ID: <4DE40477.9000407@simplicitymedialtd.co.uk> Just did some testing.. root at vicky:~# cat /var/log/auth.log | grep "Set" May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17 May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17 root at vicky:~# ps faux | grep 1574 root 1574 0.0 0.0 70488 3404 ? Ss 21:41 0:00 \_ sshd: root at pts/1 root at vicky:~# ps faux | grep "1568" root 1568 0.0 0.0 49168 1152 ? Ss 21:41 0:00 /usr/sbin/sshd In sshd.c there seems to be: static int oom_adj_save = INT_MIN; root at courtney:~/openssh-5.5p1# grep -R "Set %s to %d" . ./openbsd-compat/port-linux.c: verbose("Set %s to %d", OOM_ADJ_PATH, oom_adj_save); Then I tried on a server with different network card hardware (as shown below), and got this from the logs: root at courtney:~/openssh-5.5p1# cat /var/log/auth.log | grep "Set" May 30 21:50:15 courtney sshd[4821]: Set /proc/self/oom_adj from 0 to -17 May 30 21:50:26 courtney sshd[4848]: Set /proc/self/oom_adj to 0 root at courtney:~/openssh-5.5p1# ps faux | grep "4848" root 4848 0.0 0.0 70488 3372 ? Ss 21:50 0:00 \_ sshd: root at pts/1 root at courtney:~/openssh-5.5p1# ps faux | grep "4821" root 4821 0.0 0.0 49168 1160 ? Ss 21:50 0:00 /usr/sbin/sshd root at courtney:~/openssh-5.5p1# cat /var/log/auth.log | grep -e "Set" -e "oom_adjust_restore" May 30 21:50:15 courtney sshd[4821]: Set /proc/self/oom_adj from 0 to -17 May 30 21:50:26 courtney sshd[4848]: debug3: oom_adjust_restore May 30 21:50:26 courtney sshd[4848]: Set /proc/self/oom_adj to 0 On 30/05/2011 21:30, Cal Leeming [Simplicity Media Ltd] wrote: > Hi all, > > Please find below a complete transcript of the emails between > debian/kernel-mm mailing lists. > > I've had a response back from someone on the deb mailing list stating: > > ==================================== > The bug seems to be that sshd does not reset the OOM adjustment before > running the login shell (or other program). Therefore, please report a > bug against openssh-server. > ==================================== > > Therefore, I am submitting this bug to you also.. If someone would be > kind enough to have a flick thru all the below debug/logs, it'd be > very much appreciated. > > Cal From cal.leeming at simplicitymedialtd.co.uk Tue May 31 07:32:24 2011 From: cal.leeming at simplicitymedialtd.co.uk (Cal Leeming [Simplicity Media Ltd]) Date: Mon, 30 May 2011 22:32:24 +0100 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: <4DE40477.9000407@simplicitymedialtd.co.uk> References: <4DE40477.9000407@simplicitymedialtd.co.uk> Message-ID: <4DE40CE8.6020609@simplicitymedialtd.co.uk> So I modified the code to try and repair this oom_adj problem... port-linux.c: line 235: //static int oom_adj_save = INT_MIN; line 236: static int oom_adj_save = 0; line 277: verbose("Set %s to %d - sleepycal", OOM_ADJ_PATH, oom_adj_save); I then ran compiled the package, ran SSHd, and yet we still have -17 in oom_adj_save. Wtf? Now, I'm not much of a C coder, but this is weird even in my books... May 30 22:18:19 vicky sshd[12825]: Set /proc/self/oom_adj to -17 - sleepycal So, I went all out crazy, and did the following patch: static int sleepycal_oom_adj_save = 0; verbose("sleepycal_oom_adj_save=%d", sleepycal_oom_adj_save); if (fprintf(fp, "%d\n", sleepycal_oom_adj_save) <= 0) verbose("error writing %s: %s", OOM_ADJ_PATH, strerror(errno)); else verbose("Set %s to %d - sleepycal", OOM_ADJ_PATH, sleepycal_oom_adj_save); And it worked!!! :) May 30 22:27:12 vicky sshd[2532]: sleepycal_oom_adj_save=0 May 30 22:27:12 vicky sshd[2532]: Set /proc/self/oom_adj to 0 - sleepycal root at vicky:~/openssh-5.5p1# cat /proc/2532/oom_adj 0 So, it turns out that it is actually OpenSSH which is broken, after almost 3 days of frustrating digging through millions of lines of code lol. Anyways, would appreciate if someone could get this merged into master (obv rename the vars if you want). Attached is the appropriate patch file as of openssh-5.5p1 Cal On 30/05/2011 21:56, Cal Leeming [Simplicity Media Ltd] wrote: > Just did some testing.. > > root at vicky:~# cat /var/log/auth.log | grep "Set" > May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17 > May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17 > > root at vicky:~# ps faux | grep 1574 > root 1574 0.0 0.0 70488 3404 ? Ss 21:41 0:00 \_ > sshd: root at pts/1 > > root at vicky:~# ps faux | grep "1568" > root 1568 0.0 0.0 49168 1152 ? Ss 21:41 0:00 > /usr/sbin/sshd > > In sshd.c there seems to be: > static int oom_adj_save = INT_MIN; > > root at courtney:~/openssh-5.5p1# grep -R "Set %s to %d" . > ./openbsd-compat/port-linux.c: verbose("Set %s to %d", > OOM_ADJ_PATH, oom_adj_save); > > Then I tried on a server with different network card hardware (as > shown below), and got this from the logs: > > root at courtney:~/openssh-5.5p1# cat /var/log/auth.log | grep "Set" > May 30 21:50:15 courtney sshd[4821]: Set /proc/self/oom_adj from 0 to -17 > May 30 21:50:26 courtney sshd[4848]: Set /proc/self/oom_adj to 0 > > root at courtney:~/openssh-5.5p1# ps faux | grep "4848" > root 4848 0.0 0.0 70488 3372 ? Ss 21:50 0:00 \_ > sshd: root at pts/1 > > root at courtney:~/openssh-5.5p1# ps faux | grep "4821" > root 4821 0.0 0.0 49168 1160 ? Ss 21:50 0:00 > /usr/sbin/sshd > > root at courtney:~/openssh-5.5p1# cat /var/log/auth.log | grep -e "Set" > -e "oom_adjust_restore" > May 30 21:50:15 courtney sshd[4821]: Set /proc/self/oom_adj from 0 to -17 > May 30 21:50:26 courtney sshd[4848]: debug3: oom_adjust_restore > May 30 21:50:26 courtney sshd[4848]: Set /proc/self/oom_adj to 0 > > > > > On 30/05/2011 21:30, Cal Leeming [Simplicity Media Ltd] wrote: >> Hi all, >> >> Please find below a complete transcript of the emails between >> debian/kernel-mm mailing lists. >> >> I've had a response back from someone on the deb mailing list stating: >> >> ==================================== >> The bug seems to be that sshd does not reset the OOM adjustment before >> running the login shell (or other program). Therefore, please report a >> bug against openssh-server. >> ==================================== >> >> Therefore, I am submitting this bug to you also.. If someone would be >> kind enough to have a flick thru all the below debug/logs, it'd be >> very much appreciated. >> >> Cal > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: oom_patch_for_openssh-5.5p1_by_sleepycal.patch URL: From gert at greenie.muc.de Tue May 31 17:25:40 2011 From: gert at greenie.muc.de (Gert Doering) Date: Tue, 31 May 2011 09:25:40 +0200 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: <4DE40CE8.6020609@simplicitymedialtd.co.uk> References: <4DE40477.9000407@simplicitymedialtd.co.uk> <4DE40CE8.6020609@simplicitymedialtd.co.uk> Message-ID: <20110531072540.GU8496@greenie.muc.de> Hi, On Mon, May 30, 2011 at 10:32:24PM +0100, Cal Leeming [Simplicity Media Ltd] wrote: > So, it turns out that it is actually OpenSSH which is broken, after I would not second this. To me, this very much looks like: > On 30/05/2011 21:56, Cal Leeming [Simplicity Media Ltd] wrote: > > Just did some testing.. > > > >root at vicky:~# cat /var/log/auth.log | grep "Set" > >May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17 > >May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17 ... it's reading out the old value, saving it, setting it to "-17" (for the sshd listener process, that one is not to be killed), and later on *restoring* the old value (for all child processes). See the comments in platform.c The log messages look weird because the value is -17 already when sshd starts - so it's adjusting "-17 to -17" and later on "restoring -17" - looks stupid, but that's computers for you. But what it tells you is that the value isn't set by sshd to "-17" but that sshd inherited that from whoever started it. The question here is why sshd is sometimes started with -17 and sometimes with 0 - that's the bug, not that sshd keeps what it's given. (Ask yourself: if sshd had no idea about oom_adj at all, would this make it buggy by not changing the value?) Anyway, as a workaround for your system, you can certainly set oom_adj_save = 0; in the beginning of port-linux.c / oom_adjust_restore(), to claim that "hey, this was the saved value to start with" and "restore" oom_adj to 0 then - but that's just hiding the bug, not fixing it. gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany gert at greenie.muc.de fax: +49-89-35655025 gert at net.informatik.tu-muenchen.de From cal.leeming at simplicitymedialtd.co.uk Tue May 31 21:11:13 2011 From: cal.leeming at simplicitymedialtd.co.uk (Cal Leeming [Simplicity Media Ltd]) Date: Tue, 31 May 2011 12:11:13 +0100 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: <20110531072540.GU8496@greenie.muc.de> References: <4DE40477.9000407@simplicitymedialtd.co.uk> <4DE40CE8.6020609@simplicitymedialtd.co.uk> <20110531072540.GU8496@greenie.muc.de> Message-ID: Hi Gert, Thank you for taking the time to respond! Further comments below: On Tue, May 31, 2011 at 8:25 AM, Gert Doering wrote: > Hi, > > On Mon, May 30, 2011 at 10:32:24PM +0100, Cal Leeming [Simplicity Media Ltd] wrote: >> So, it turns out that it is actually OpenSSH which is broken, after > > I would not second this. ?To me, this very much looks like: > >> On 30/05/2011 21:56, Cal Leeming [Simplicity Media Ltd] wrote: >> > Just did some testing.. >> > >> >root at vicky:~# cat /var/log/auth.log | grep "Set" >> >May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17 >> >May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17 > > ... it's reading out the old value, saving it, setting it to "-17" (for > the sshd listener process, that one is not to be killed), and later on > *restoring* the old value (for all child processes). ?See the comments > in platform.c > > The log messages look weird because the value is -17 already when sshd > starts - so it's adjusting "-17 to -17" and later on "restoring -17" - > looks stupid, but that's computers for you. ?But what it tells you is > that the value isn't set by sshd to "-17" but that sshd inherited that > from whoever started it. Could you point out the line of code where oom_adj_save is set to the original value, because I've looked everywhere, and from what I can tell, it's only ever set to INT_MIN, and no where else is it changed. (C is not my strongest language tho, so I most likely have overlooked something). This is where I got thrown off. > > The question here is why sshd is sometimes started with -17 and sometimes > with 0 - that's the bug, not that sshd keeps what it's given. > > (Ask yourself: if sshd had no idea about oom_adj at all, would this make > it buggy by not changing the value?) This was what I was trying to pinpoint down before. I had came to this conclusion myself, sent it to the Debian bug list, and they dismissed on the grounds that it was an openssh problem... So far, the buck has been passed from kernel-mm to debian-kernel, to openssh, and now back to debian-kernel lol. The most annoying thing, is that you can't get this bug to happen unless you physically test on a machine which requires the bnx2 firmwire, so I get the feeling this won't get resolved :/ > > > Anyway, as a workaround for your system, you can certainly set > > ?oom_adj_save = 0; > > in the beginning of port-linux.c / oom_adjust_restore(), to claim that > "hey, this was the saved value to start with" and "restore" oom_adj to 0 > then - but that's just hiding the bug, not fixing it. I'm disappointed this wasn't the correct fix, I honestly thought I had patched it right :( But, on the other hand, ssh users should really never have a default oom_adj of -17, so maybe 0 should be set as default anyway? If this is not the case, could you give reasons why?? > > gert > -- > USENET is *not* the non-clickable part of WWW! > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? //www.muc.de/~gert/ > Gert Doering - Munich, Germany ? ? ? ? ? ? ? ? ? ? ? ? ? ? gert at greenie.muc.de > fax: +49-89-35655025 ? ? ? ? ? ? ? ? ? ? ? ?gert at net.informatik.tu-muenchen.de > From cal.leeming at simplicitymedialtd.co.uk Tue May 31 21:26:52 2011 From: cal.leeming at simplicitymedialtd.co.uk (Cal Leeming [Simplicity Media Ltd]) Date: Tue, 31 May 2011 12:26:52 +0100 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: References: <4DE40477.9000407@simplicitymedialtd.co.uk> <4DE40CE8.6020609@simplicitymedialtd.co.uk> <20110531072540.GU8496@greenie.muc.de> Message-ID: Bug report submitted to Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=628690 Should be interesting to see what happens with this lol. On Tue, May 31, 2011 at 12:11 PM, Cal Leeming [Simplicity Media Ltd] wrote: > Hi Gert, > > Thank you for taking the time to respond! Further comments below: > > On Tue, May 31, 2011 at 8:25 AM, Gert Doering wrote: >> Hi, >> >> On Mon, May 30, 2011 at 10:32:24PM +0100, Cal Leeming [Simplicity Media Ltd] wrote: >>> So, it turns out that it is actually OpenSSH which is broken, after >> >> I would not second this. ?To me, this very much looks like: >> >>> On 30/05/2011 21:56, Cal Leeming [Simplicity Media Ltd] wrote: >>> > Just did some testing.. >>> > >>> >root at vicky:~# cat /var/log/auth.log | grep "Set" >>> >May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17 >>> >May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17 >> >> ... it's reading out the old value, saving it, setting it to "-17" (for >> the sshd listener process, that one is not to be killed), and later on >> *restoring* the old value (for all child processes). ?See the comments >> in platform.c >> >> The log messages look weird because the value is -17 already when sshd >> starts - so it's adjusting "-17 to -17" and later on "restoring -17" - >> looks stupid, but that's computers for you. ?But what it tells you is >> that the value isn't set by sshd to "-17" but that sshd inherited that >> from whoever started it. > > Could you point out the line of code where oom_adj_save is set to the > original value, because I've looked everywhere, and from what I can > tell, it's only ever set to INT_MIN, and no where else is it changed. > (C is not my strongest language tho, so I most likely have overlooked > something). This is where I got thrown off. > >> >> The question here is why sshd is sometimes started with -17 and sometimes >> with 0 - that's the bug, not that sshd keeps what it's given. >> >> (Ask yourself: if sshd had no idea about oom_adj at all, would this make >> it buggy by not changing the value?) > > This was what I was trying to pinpoint down before. I had came to this > conclusion myself, sent it to the Debian bug list, and they dismissed > on the grounds that it was an openssh problem... > > So far, the buck has been passed from kernel-mm to debian-kernel, to > openssh, and now back to debian-kernel lol. The most annoying thing, > is that you can't get this bug to happen unless you physically test on > a machine which requires the bnx2 firmwire, so I get the feeling this > won't get resolved :/ > >> >> >> Anyway, as a workaround for your system, you can certainly set >> >> ?oom_adj_save = 0; >> >> in the beginning of port-linux.c / oom_adjust_restore(), to claim that >> "hey, this was the saved value to start with" and "restore" oom_adj to 0 >> then - but that's just hiding the bug, not fixing it. > > I'm disappointed this wasn't the correct fix, I honestly thought I had > patched it right :( > > But, on the other hand, ssh users should really never have a default > oom_adj of -17, so maybe 0 should be set as default anyway? If this is > not the case, could you give reasons why?? > >> >> gert >> -- >> USENET is *not* the non-clickable part of WWW! >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? //www.muc.de/~gert/ >> Gert Doering - Munich, Germany ? ? ? ? ? ? ? ? ? ? ? ? ? ? gert at greenie.muc.de >> fax: +49-89-35655025 ? ? ? ? ? ? ? ? ? ? ? ?gert at net.informatik.tu-muenchen.de >> > From dtucker at zip.com.au Tue May 31 22:11:01 2011 From: dtucker at zip.com.au (Darren Tucker) Date: Tue, 31 May 2011 22:11:01 +1000 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: References: <4DE40477.9000407@simplicitymedialtd.co.uk> <4DE40CE8.6020609@simplicitymedialtd.co.uk> <20110531072540.GU8496@greenie.muc.de> Message-ID: On Tue, May 31, 2011 at 9:11 PM, Cal Leeming [Simplicity Media Ltd] wrote: [...] > Could you point out the line of code where oom_adj_save is set to the > original value, because I've looked everywhere, and from what I can > tell, it's only ever set to INT_MIN, It's read from /proc/self/oom_adj at startup time in oom_adjust_setup(): if ((fp = fopen(oom_adj_path, "r+")) != NULL) { if (fscanf(fp, "%d", &oom_adj_save) != 1) This is the reason for the "Set /proc/self/oom_adj from -17 to -17" probably why Gert commented on it. Basically, sshd sets the listening process to -17, then restores whatever was previously set for all forked processes. If oom_adj was previously -17, sshd will restore that. [...] > This was what I was trying to pinpoint down before. I had came to this > conclusion myself, sent it to the Debian bug list, and they dismissed > on the grounds that it was an openssh problem... I'd suggest "grep -rl oom_adj /etc" and see if one of your system startup scripts sets it. Failing that, I'd try cold booting your machine without your problem module, modprobe it and check /proc/self/oom_adj and see if the modprobe or module loading somehow changes that (I can't imagine that it would, but you seem to have a really strange case here...). -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4? 37C9 C982 80C7 8FF4 FA69 ? ? Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement. From cal.leeming at simplicitymedialtd.co.uk Tue May 31 22:18:06 2011 From: cal.leeming at simplicitymedialtd.co.uk (Cal Leeming [Simplicity Media Ltd]) Date: Tue, 31 May 2011 13:18:06 +0100 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: References: <4DE40477.9000407@simplicitymedialtd.co.uk> <4DE40CE8.6020609@simplicitymedialtd.co.uk> <20110531072540.GU8496@greenie.muc.de> Message-ID: On Tue, May 31, 2011 at 1:11 PM, Darren Tucker wrote: > On Tue, May 31, 2011 at 9:11 PM, Cal Leeming [Simplicity Media Ltd] > wrote: > [...] >> Could you point out the line of code where oom_adj_save is set to the >> original value, because I've looked everywhere, and from what I can >> tell, it's only ever set to INT_MIN, > > It's read from /proc/self/oom_adj at startup time in oom_adjust_setup(): > > ? ? ? ? ? ? ? if ((fp = fopen(oom_adj_path, "r+")) != NULL) { > ? ? ? ? ? ? ? ? ? ? ? ?if (fscanf(fp, "%d", &oom_adj_save) != 1) > > This is the reason for the "Set /proc/self/oom_adj from -17 to -17" > probably why Gert commented on it. > > Basically, sshd sets the listening process to -17, then restores > whatever was previously set for all forked processes. ?If oom_adj was > previously -17, sshd will restore that. > > [...] >> This was what I was trying to pinpoint down before. I had came to this >> conclusion myself, sent it to the Debian bug list, and they dismissed >> on the grounds that it was an openssh problem... > > I'd suggest "grep -rl oom_adj /etc" and see if one of your system > startup scripts sets it. ?Failing that, I'd try cold booting your > machine without your problem module, modprobe it and check > /proc/self/oom_adj and see if the modprobe or module loading somehow > changes that (I can't imagine that it would, but you seem to have a > really strange case here...). Oh trust me, I looked *everywhere*. Even to the extent of running tripwire from a bare bones system, and looking manually at every change made. I also looked for loads of different keywords (-17, oom, proc, self) etc. Spent hours on it :/ As for the comment about the modprobe, I already did all this (full debug can be found at http://www.debianhelp.org/content/cgroup-oom-killer-loop-causes-system-lockup-possible-fix-included ), and found that when the bnx2 module isn't loaded, the problem goes away.. When it is loaded, the problem comes back. This is what I mean by it being a very VERY strange problem. My guess is that the bnx2 firmware does some sort of kernel-to-userspace weirdness, which causes user land apps (which have to go through the bnx2 in the network stack) to somehow inherit the -17 that all kernel processes get... Sadly, I don't know enough about how the kernel works to even begin to debug the problem.. plus (from what I can tell) the firmware (*.fw) is closed source.. I have a very strong feeling that the buck will probably just get passed around until it eventually gets forgotten about :( I wish I knew more about kernel development to try and fix this issue! > > -- > Darren Tucker (dtucker at zip.com.au) > GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4? 37C9 C982 80C7 8FF4 FA69 > ? ? Good judgement comes with experience. Unfortunately, the experience > usually comes from bad judgement. > From gert at greenie.muc.de Tue May 31 22:25:10 2011 From: gert at greenie.muc.de (Gert Doering) Date: Tue, 31 May 2011 14:25:10 +0200 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: References: <4DE40477.9000407@simplicitymedialtd.co.uk> <4DE40CE8.6020609@simplicitymedialtd.co.uk> <20110531072540.GU8496@greenie.muc.de> Message-ID: <20110531122509.GX8496@greenie.muc.de> Hi, On Tue, May 31, 2011 at 12:11:13PM +0100, Cal Leeming [Simplicity Media Ltd] wrote: > Could you point out the line of code where oom_adj_save is set to the > original value, because I've looked everywhere, and from what I can > tell, it's only ever set to INT_MIN, and no where else is it changed. > (C is not my strongest language tho, so I most likely have overlooked > something). This is where I got thrown off. oom_adjust_setup() does this: if ((fp = fopen(oom_adj_path, "r+")) != NULL) { if (fscanf(fp, "%d", &oom_adj_save) != 1) verbose("error reading %s: %s", oom_adj_path, strerror(errno)); the "fscanf()" call will read an integer ("%d") from the file named, and write that number into the variable being pointed to (&oom_adj_save). The loop is a bit tricky to read as it takes different paths into account, and will exit after the first successful update. fscanf() will return the number of successful conversions, so if it was able to read "one number", the return value is "1", and it will jump to the else block else { rewind(fp); if (fprintf(fp, "%d\n", value) <= 0) verbose("error writing %s: %s", oom_adj_path, strerror(errno)); else verbose("Set %s from %d to %d", oom_adj_path, oom_adj_save, value); } where it will overwrite what is in that file with the new value ("value"), and then print the "Set ... from -17 to -17" message that you saw. > > The question here is why sshd is sometimes started with -17 and sometimes > > with 0 - that's the bug, not that sshd keeps what it's given. > > > > (Ask yourself: if sshd had no idea about oom_adj at all, would this make > > it buggy by not changing the value?) > > This was what I was trying to pinpoint down before. I had came to this > conclusion myself, sent it to the Debian bug list, and they dismissed > on the grounds that it was an openssh problem... I must admit that I have no idea what is causing it, but from the logs, it very much looks like sshd is started with "-17" in there - but only in the problem case. > So far, the buck has been passed from kernel-mm to debian-kernel, to > openssh, and now back to debian-kernel lol. The most annoying thing, > is that you can't get this bug to happen unless you physically test on > a machine which requires the bnx2 firmwire, so I get the feeling this > won't get resolved :/ And *that* strongly points to a bug in the bnx2 stuff - if other programs change their behaviour based on the existance of a given driver, that does not smell very healthy. [..] > > Anyway, as a workaround for your system, you can certainly set > > > > ?oom_adj_save = 0; > > > > in the beginning of port-linux.c / oom_adjust_restore(), to claim that > > "hey, this was the saved value to start with" and "restore" oom_adj to 0 > > then - but that's just hiding the bug, not fixing it. > > I'm disappointed this wasn't the correct fix, I honestly thought I had > patched it right :( Well, that's the short hand - "just ignore everything that happened at init / save time, and forcibly write back '0', no matter what was there before". > But, on the other hand, ssh users should really never have a default > oom_adj of -17, so maybe 0 should be set as default anyway? If this is > not the case, could you give reasons why?? Well, I would say "the default value in there is a matter of local policy", so what if someone wants to make sure that whatever is run from sshd is always privileged regarding oom, even if a local firefox etc. is running amock and you need to ssh-in and kill the GUI stuff... One might opt to run sshd (and all its children) at "-5" (slightly special treatment), or "0" (no special treatment), or even at "-17" - but that's local policy. Mmmh. Since this seems to be inherited, it might even work if you just change the sshd startup script, and insert echo 0 >/proc/self/oom_adj in there, right before it starts the sshd... "local policy at work". gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany gert at greenie.muc.de fax: +49-89-35655025 gert at net.informatik.tu-muenchen.de From dtucker at zip.com.au Tue May 31 22:37:37 2011 From: dtucker at zip.com.au (Darren Tucker) Date: Tue, 31 May 2011 22:37:37 +1000 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: References: <4DE40477.9000407@simplicitymedialtd.co.uk> <4DE40CE8.6020609@simplicitymedialtd.co.uk> <20110531072540.GU8496@greenie.muc.de> Message-ID: On Tue, May 31, 2011 at 10:18 PM, Cal Leeming [Simplicity Media Ltd] wrote: [...] > Oh trust me, I looked *everywhere*. Even to the extent of running > tripwire from a bare bones system, and looking manually at every > change made. I also looked for loads of different keywords (-17, oom, > proc, self) etc. Spent hours on it :/ > > As for the comment about the modprobe, I already did all this (full > debug can be found at > http://www.debianhelp.org/content/cgroup-oom-killer-loop-causes-system-lockup-possible-fix-included > ), and found that when the bnx2 module isn't loaded, the problem goes > away.. When it is loaded, the problem comes back. Did you check /proc/self/oom_adj before and after loading the module? I don't see that in there, and it it *does* change it would eliminate sshd as a variable. As a workaround, you could add "echo 0 >/proc/self/oom_adj" to /etc/default/ssh. It's a bit ugly, but at least you wouldn't need to recompile anything. -- Darren Tucker (dtucker at zip.com.au) GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4? 37C9 C982 80C7 8FF4 FA69 ? ? Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement. From cal.leeming at simplicitymedialtd.co.uk Tue May 31 22:42:26 2011 From: cal.leeming at simplicitymedialtd.co.uk (Cal Leeming [Simplicity Media Ltd]) Date: Tue, 31 May 2011 13:42:26 +0100 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: <20110531122509.GX8496@greenie.muc.de> References: <4DE40477.9000407@simplicitymedialtd.co.uk> <4DE40CE8.6020609@simplicitymedialtd.co.uk> <20110531072540.GU8496@greenie.muc.de> <20110531122509.GX8496@greenie.muc.de> Message-ID: <4DE4E232.1070902@simplicitymedialtd.co.uk> On 31/05/2011 13:25, Gert Doering wrote: > Hi, > > On Tue, May 31, 2011 at 12:11:13PM +0100, Cal Leeming [Simplicity Media Ltd] wrote: >> Could you point out the line of code where oom_adj_save is set to the >> original value, because I've looked everywhere, and from what I can >> tell, it's only ever set to INT_MIN, and no where else is it changed. >> (C is not my strongest language tho, so I most likely have overlooked >> something). This is where I got thrown off. > oom_adjust_setup() does this: > > if ((fp = fopen(oom_adj_path, "r+")) != NULL) { > if (fscanf(fp, "%d",&oom_adj_save) != 1) > verbose("error reading %s: %s", oom_adj_path, > strerror(errno)); > > the "fscanf()" call will read an integer ("%d") from the file named, > and write that number into the variable being pointed to (&oom_adj_save). > > The loop is a bit tricky to read as it takes different paths into > account, and will exit after the first successful update. > > fscanf() will return the number of successful conversions, so if it > was able to read "one number", the return value is "1", and it will > jump to the else block > > else { > rewind(fp); > if (fprintf(fp, "%d\n", value)<= 0) > verbose("error writing %s: %s", > oom_adj_path, strerror(errno)); > else > verbose("Set %s from %d to %d", > oom_adj_path, oom_adj_save, value); > } > > where it will overwrite what is in that file with the new value > ("value"), and then print the "Set ... from -17 to -17" message that > you saw. Ah, thank you for explaining this. Makes a lot more sense now :) > >>> The question here is why sshd is sometimes started with -17 and sometimes >>> with 0 - that's the bug, not that sshd keeps what it's given. >>> >>> (Ask yourself: if sshd had no idea about oom_adj at all, would this make >>> it buggy by not changing the value?) >> This was what I was trying to pinpoint down before. I had came to this >> conclusion myself, sent it to the Debian bug list, and they dismissed >> on the grounds that it was an openssh problem... > I must admit that I have no idea what is causing it, but from the logs, > it very much looks like sshd is started with "-17" in there - but only > in the problem case. > > >> So far, the buck has been passed from kernel-mm to debian-kernel, to >> openssh, and now back to debian-kernel lol. The most annoying thing, >> is that you can't get this bug to happen unless you physically test on >> a machine which requires the bnx2 firmwire, so I get the feeling this >> won't get resolved :/ > And *that* strongly points to a bug in the bnx2 stuff - if other programs > change their behaviour based on the existance of a given driver, that > does not smell very healthy. Agreed.. I was thinking of adding some debug into the fs/proc/ code which does a kprint on every oom_adj read/write, but I couldn't figure out how to extract the pid from the task (pointer?). > [..] >>> Anyway, as a workaround for your system, you can certainly set >>> >>> oom_adj_save = 0; >>> >>> in the beginning of port-linux.c / oom_adjust_restore(), to claim that >>> "hey, this was the saved value to start with" and "restore" oom_adj to 0 >>> then - but that's just hiding the bug, not fixing it. >> I'm disappointed this wasn't the correct fix, I honestly thought I had >> patched it right :( > Well, that's the short hand - "just ignore everything that happened at > init / save time, and forcibly write back '0', no matter what was there > before". > >> But, on the other hand, ssh users should really never have a default >> oom_adj of -17, so maybe 0 should be set as default anyway? If this is >> not the case, could you give reasons why?? > Well, I would say "the default value in there is a matter of local policy", > so what if someone wants to make sure that whatever is run from sshd is > always privileged regarding oom, even if a local firefox etc. is running > amock and you need to ssh-in and kill the GUI stuff... > > One might opt to run sshd (and all its children) at "-5" (slightly special > treatment), or "0" (no special treatment), or even at "-17" - but that's > local policy. Ah, okay that's make sense. > > Mmmh. > > Since this seems to be inherited, it might even work if you just change > the sshd startup script, and insert > > echo 0>/proc/self/oom_adj > > in there, right before it starts the sshd... "local policy at work". Yeah I was going to do this, but then I thought "well if this problem is occurring for openssh, then what else could it be affecting?". As you pointed out above, having the oom_adj changed based on the existence of a driver is really not good. I will paste this convo trail into the debian ticket, and hopefully it'll help convince them this issue needs fixing. > gert Thanks again for taking the time to reply! From cal.leeming at simplicitymedialtd.co.uk Tue May 31 22:58:48 2011 From: cal.leeming at simplicitymedialtd.co.uk (Cal Leeming [Simplicity Media Ltd]) Date: Tue, 31 May 2011 13:58:48 +0100 Subject: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. In-Reply-To: References: <4DE40477.9000407@simplicitymedialtd.co.uk> <4DE40CE8.6020609@simplicitymedialtd.co.uk> <20110531072540.GU8496@greenie.muc.de> Message-ID: <4DE4E608.3040502@simplicitymedialtd.co.uk> On 31/05/2011 13:37, Darren Tucker wrote: > On Tue, May 31, 2011 at 10:18 PM, Cal Leeming [Simplicity Media Ltd] > wrote: > [...] >> Oh trust me, I looked *everywhere*. Even to the extent of running >> tripwire from a bare bones system, and looking manually at every >> change made. I also looked for loads of different keywords (-17, oom, >> proc, self) etc. Spent hours on it :/ >> >> As for the comment about the modprobe, I already did all this (full >> debug can be found at >> http://www.debianhelp.org/content/cgroup-oom-killer-loop-causes-system-lockup-possible-fix-included >> ), and found that when the bnx2 module isn't loaded, the problem goes >> away.. When it is loaded, the problem comes back. > Did you check /proc/self/oom_adj before and after loading the module? > I don't see that in there, and it it *does* change it would eliminate > sshd as a variable. > > As a workaround, you could add "echo 0>/proc/self/oom_adj" to > /etc/default/ssh. It's a bit ugly, but at least you wouldn't need to > recompile anything. Thank you for the response! I've put some replies back to Gerts email (relating to this suggestion).