Remote ssh intermittent failure with exit status 143

[Sileshi Kassa]

I use remote ssh echo command as heartbeat to probe remote server is alive.
After days of running, I get an intermittent ssh exit status 143 (143 - 128
= 15 ) which is SIGTERM.

The remote echo command over ssh runs every minute using public key
authentication:


/usr/bin/ssh -v -o ServerAliveInterval=60 -i ~/.ssh/public-key.rsa -l foo
<remote-ipaddress> echo

 Using openssh client/server version 7.1.P2, both hosts are running CentOS
5.x

I am unable to recreate it in standalone script, but happens in production.
Any help in this regard is appreciated.  I have checked that ssh session is
not killed by kernel OOM-killer, but it seems to fail when the system is
stressed.


#!/bin/bash
# secondary network interfaces: 10.0.0.1 and 10.0.0.2

if [ -z "$1" ]
then
    echo "missing required peer IP address"
    exit
fi

peeripaddr=$1

while :
do
   /usr/bin/ssh -v -o ServerAliveInterval=60 -i ~/.ssh/public-key.rsa -l
foo $peeripaddr echo

   rc=$?
   if [ $rc -gt 0 ]
   then
       echo "ssh echo command has failed rc=$rc"
       exit
   fi

   sleep 5