cipherdyne.org

Michael Rash, Security Researcher



Port Knocking and SPA    [Summary View]

A new SSH password guessing botnet: dd_ssh

dd_ssh botnet Every few months it seems that there is a new variation of a password guessing botnet. This time around, as reported by the SANS ISC and also by malwarecity.com, a botnet called "dd_ssh" has been built up around a vulnerability in phpMyAdmin. (Here is the vulnerability announcement from the Debian security list.) Once a vulnerable webserver has been compromised, the botnet starts scanning random IP addresses for SSH servers and attempts to brute force passwords for those systems that accept an SSH connection. Although software like Fail2ban and DenyHosts can offer detection and mitigation by applying thresholds to the number of failed login attempts in the SSH log and communicate offending source IP's to a set of interested users, a fundamental problem is that the targeted SSH daemons are reachable in the first place from arbitrary source IP's. A more powerful solution is to hide your SSH daemon behind a default-drop firewall policy and only grant access through Single Packet Authorization. Any botnet that is scanning for SSH servers will not be able to guess any passwords or even exploit a zero-day vulnerability in an SSH daemon that is protected in this way.

Further, SSH brute forcing botnets have shown some innovative techniques where password guessing attempts are spread out over a long period of time and failed passwords are distributed across the botnet. The end result is that each botnet node knows the population of passwords that other nodes have already tried on each targeted SSH daemon, and so new passwords can be tried while still slipping beneath common detection thresholds. The same password never has to be tried twice botnet-wide. It is not clear yet whether the dd_ssh botnet uses a similar password distribution strategy, but either way I intend to stop such guessing games at the earliest stage possible and block all SSH connections by default.

Pure C Implementation of Single Packet Authorization

Pure C Implementation of Single Packet Authorization For the past several months, the pure-C implementation of Single Packet Authorization with fwknop has been in development, and the first release candidate of fwknop-2.0.0 is ready for download. Damien Stuart is the primary developer of this code, and I wish to thank him for all of his efforts in making this release possible. The original perl version of fwknop will continue to receive bug fixes as they are found, but the primary development effort is now on the lightweight and portable C code. The Trac interface for the perl code has moved to fwknop-perl. Any release that is at fwknop-2.0.0 or above will be the C implementation, and any release in the 1.9 series or before are in perl (and will remain available here). The perl and C versions are compatible with each other, so SPA packets created by the fwknop perl client can be properly decrypted and decoded by the new fwknopd C server, and similarly SPA packets built with the fwknop C client can be interpreted by the fwknopd perl server.

The heart of the new C code is the libfko library, which is used by both the fwknop client and the fwknopd server for SPA packet encryption/decryption and interpretation. The libfko library is portable to several platforms including Linux and Windows, and the standard GNU autoconf project is used for building fwknop from sources.

Given the compact nature of the C code, the fwknop-2.0.0rc1 release candidate is already known to function on OpenWRT, and Damien Stuart (the primary developer of the fwknop C code) has built packages for the OpenWRT Kamikaze and Backfire releases. This brings SPA functionality to embedded systems, and has been tested on a Linksys WRT54GS V2.0 router. The OpenWRT packages are available here.

There will be a few release candidates of fwknop-2.0.0 to ensure proper operation on a variety of platforms, and the emphasis will be on stability and ironing out any bugs. Also, fwknop-2.0.0 will initially support iptables firewalls on the servers side, with ipfw support coming soon after. There is one nice feature in fwknop-2.0.0rc1 that deserves a mention - the fwknop client now supports the notion of a ~/.fwknoprc file that allows a set of stanzas to be defined for remote SPA servers and customize the command line arguments used to access each one. Here is an example stanza: # Example for a destination server of 192.168.1.20 to open access to
# SSH for an IP that is resolved externally
#
[myssh]
SPA_SERVER 192.168.1.20
ACCESS tcp/22
ALLOW_IP resolve
SPA_SERVER_PORT 50023
Now, with the fwknop client, we reference this stanza with the -n command line argument like so: $ fwknop -n myssh -vvv
Resolved external IP (via http://www.cipherdyne.org/cgi-bin/myip) as: 123.1.2.3
Enter encryption password:

FKO Field Values:
=================

Random Value: 8308719569638051
Username: mbr
Timestamp: 1280030806
FKO Version: 1.9.12
Message Type: 1
Message String: 123.1.2.3,tcp/22
Nat Access: <NULL>
Server Auth: <NULL>
Client Timeout: 0
Digest Type: 3

Encoded Data: 8308719569638051:bWJy:1280030806:1.9.12:1:OTYuMjQ0LjE3MS4yNTMsdGNwLzIy

SPA Data Digest: WcUEVesZDIY+CKRKYTck8zMW7HG0S7RcqmXVIMC4L58

Final Packed/Encrypted/Encoded Data:

87wQ3s171if9x8628P6k1oBrQEJEmS5c/+ECed5cfDmu8hGhC5esUCEVcI8IH5TdjWva44efebX+cfXUVPBU9c/6tJpOLwC8k/8sN87SgYkBK+m64aD4FnnndaeqsrIcO/oHcpXHk9S/8rqqypcumNtlXVZD/isS3Q729LMTa+2uI6TsGX9OQ3

Generating SPA packet:
protocol: udp
port: 50023
send_spa_packet: bytes sent: 182
On the fwknopd server side, we see the following printed to the terminal in --foreground mode as the SPA packet is received and an iptables ACCEPT rule is added for SSH access and then removed after 30 seconds. Using Digest Cache: '/usr/var/run/fwknop/digest.cache' (entry count = 93)
PCAP filter is: udp port 50023
Starting fwknopd main event loop.


SPA Packet from IP: 123.1.2.3 received.
SPA Packet: '87wQ3s171if9x8628P6k1oBrQEJEmS5c/+ECed5cfDmu8hGhC5esUCEVcI8IH5TdjWva44efebX+cfXUVPBU9c/6tJpOLwC8k/8sN87SgYkBK+m64aD4FnnndaeqsrIcO/oHcpXHk9S/8rqqypcumNtlXVZD/isS3Q729LMTa+2uI6TsGX9OQ3'

SPA Decode (res=0):
SPA Field Values:
=================
Random Value: 8308719569638051
Username: mbr
Timestamp: 1280030806
FKO Version: 1.9.12
Message Type: 1
Message String: 123.1.2.3,tcp/22
Nat Access: <NULL>
Server Auth: <NULL>
Client Timeout: 0
Digest Type: 3
Encoded Data: 8308719569638051:bWJy:1280030806:1.9.12:1:OTYuMjQ0LjE3MS4yNTMsdGNwLzIy
SPA Data Digest: WcUEVesZDIY+CKRKYTck8zMW7HG0S7RcqmXVIMC4L58

Added Rule to FWKNOP_INPUT for 123.1.2.3, tcp/22 expires at 1280030886
RES=0, CMD_BUF: /sbin/iptables -t filter -L FWKNOP_INPUT --line-numbers -n 2>&1
RULES LIST: Chain FWKNOP_INPUT (1 references)
num target prot opt source destination
1 ACCEPT tcp -- 123.1.2.3 0.0.0.0/0 tcp dpt:22 /* _exp_1280030886 */

Removed rule 1 from FWKNOP_INPUT with expire time of 1280030886.

holisticinfosec.org on SPA Ghost Services

holisticinfosec.org on fwknop Russ McRee of holisticinfosec.org has written the January Toolsmith issue from the ISSA Journal about fwknop and the ability to create ghost services with Single Packet Authorization. In his Toolsmith paper, Russ emphasizes the possibility of using the ghost services concept to bypass strict outbound network filtering rules on a local network in order to access an external service that is bound to a port that is filtered by the local firewall. That is, the service is made accessible by having the SPA packet created by the fwknop client request that the remote fwknopd server create iptables DNAT rules to forward connections to a port that the local network actually allows out to the port where the service is bound. Russ uses this concept to access a file that is piped through a netcat listener on TCP port 6543, but do it from the heavily filtered network over TCP port 110 (normally associated with pop3).

Here is a link to the Toolsmith PDF entitled "Single Packet Authorization: The Ghost in the Machine".

Creating Ghost Services with Single Packet Authorization

Creating Ghost Services with Single Packet Authorization Most usages of Single Packet Authorization focus on gaining access to a service such as sshd that is behind a default-drop packet filter. The point being that anyone who is scanning for the service cannot even tell that it is listening - let alone target it with an exploit or an attempt to brute force a password. This is fine, but given that firewalls such as iptables offer well designed NAT capabilities, can a more interesting access model be developed for SPA? How about accessing sshd through a port where another service (such as Apache on port 80) is already bound? Because iptables operates on packets within kernel space, NAT functions apply before there is any conflict with a user space application such as Apache. This makes it possible to create "ghost" services where a port switches for a short period of time to whatever service is requested within an SPA packet (e.g. sshd), but everyone else always just sees the service that is normally bound there (e.g. Apache on port 80).

To illustrate this concept, let's use fwknop from the spaclient system below to access sshd on the spaserver system, but request the access be granted through port 80. Further, on the spaserver system, let's verify that Apache is running and from the perspective of any scanner out on the Internet this is the only service that is accessible. That is, sshd and all other services are firewalled off by iptables. We'll assume that the spaclient system has IP 1.1.1.1, the spaserver system has IP 2.2.2.2, and the scanner system has IP 3.3.3.3.

First, let's scan the spaserver system from the scanner system and verify that only port 80 is accessible: [scanner]# nmap -P0 -n -sV 2.2.2.2

Starting Nmap 5.00 ( http://nmap.org ) at 2009-11-29 15:14 EST
Interesting ports on 2.2.2.2:
Not shown: 999 filtered ports
PORT STATE SERVICE VERSION
80/tcp open http Apache httpd 2.2.8 ((Ubuntu) mod_python/3.3.1 Python/2.5.2)

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 90.94 seconds
Good. Now, from the spaclient system, let's fire up fwknop and request access to sshd on the spaserver system. But instead of just gaining access to port 22, we'll request access to port 22 via port 80 - the fwknopd daemon will build the appropriate DNAT iptables rules to make this work: [spaclient]$ fwknop -A tcp/22 --NAT-access 2.2.2.2:80 --NAT-local -a 1.1.1.1 -D 2.2.2.2

[+] Starting fwknop client (SPA mode)...
[+] Enter an encryption key. This key must match a key in the file
/etc/fwknop/access.conf on the remote system.

Encryption Key:

[+] Building encrypted Single Packet Authorization (SPA) message...
[+] Packet fields:

Random data: 3829970026924871
Username: mbr
Timestamp: 1259526613
Version: 1.9.12
Type: 5 (Local NAT access mode)
Access: 0.0.0.0,tcp/22
NAT access: 2.2.2.2,80
SHA256 digest: Om/GsIVQIRyAp6UWyqjXVqlEQhxz+lVsQhCl1rFBfuI

[+] Sending 182 byte message to 2.2.2.2 over udp/62201...
Requesting NAT access to tcp/22 on 2.2.2.2 via port 80
With the receipt of the SPA packet, the fwknopd daemon has reconfigured iptables to allow an ssh connection through port 80 from the spaclient IP 1.1.1.1 (note the "-p 80" argument on the ssh command line): [spaclient]$ ssh -p 80 -l mbr 2.2.2.2
mbr@2.2.2.2's password:
[spaserver]$
If we can the spaserver again from the scanner system, we still only see Apache: [scanner]# nmap -P0 -n -sV 2.2.2.2

Starting Nmap 5.00 ( http://nmap.org ) at 2009-11-29 15:29 EST
Interesting ports on 2.2.2.2:
Not shown: 999 filtered ports
PORT STATE SERVICE VERSION
80/tcp open http Apache httpd 2.2.8 ((Ubuntu) mod_python/3.3.1 Python/2.5.2)

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 89.22 seconds
So, the IP 1.1.1.1 has access to sshd, but all the scanner can ever see is 'Apache httpd 2.2.8'.

A legitimate question at this point is 'why is this useful?'. Well, I have been on networks before where local access controls only allowed outbound DNS, HTTP and HTTPS traffic, so this technique allows ssh connections to be made in a manner that is consistent with these access controls. (This assumes that HTTP connections are not made through a proxy, and that the fwknopd daemon is configured to sniff SPA packets over port 53.) Further, to anyone who is able to sniff traffic, it can be hard to figure out what is really going on in terms of SPA packets and associated follow-on connections. This is especially true when other tricks such as port randomization are also applied.

I demonstrated the service ghosting technique at DojoCon a few weeks ago, and the video of this is available here (towards the end of the video).

Speaking at DojoCon

speaking at DojoCon UDPATE: The talk slides can be downloaded here and a video of the talk is available too.

On Friday November 6th I will be giving a talk on Single Packet Authorization with fwknop at the upcoming DojoCon conference at Capitol College in Laurel, MD. This talk will focus on some recent developments in the world of fwknop development and the resulting enhancements to SPA capabilities. Topics to be emphasized include:

  • The new libfko C implementation and what it means for SPA deployments.
  • Interface monitoring for administrative control changes and increasing packet counts. This adds an extra dimension of reliability for fwknopd server operations.
  • HTTP proxy support, as well as the development of an SSL web gateway for sending SPA packets on behalf of client systems that cannot construct SPA packets themselves.
  • Development efforts for future fwknop features (such as additional client integration, and deployments on embedded Linux distributions).

Software Release - fwknop-1.9.12

software release fwknop-1.9.12 The 1.9.12 release of fwknop is ready for download. This is a significant release that moves by default to the new libfko and the new FKO module for SPA encryption and decryption. Other new features include interface monitoring for the fwknopd daemon so it can survive administrative changes due to things like a DHCP address changes, and the ability to send SPA packets through HTTP proxies.

An excerpt from the 1.9.12 ChangeLog appears below:

  • Fully integrated the FKO module that is part of the libfko library for all SPA routines - encryption/decryption, digest calculation, replay attack detection, etc. The default is to always use the FKO module if it has been installed, but the original perl code remains intact as well just in case FKO does not exist on the local system. The libfko code can be viewed with Trac here
  • Added the ability to recover from interface error conditions, such as when fwknopd sniffs a ppp interface (say, associated with a VPN) that goes away and then is recreated. In previous versions of fwknop, this would result in the fwknopd daemon no longer able to receive SPA packets. This new functionality is controlled by five new configuration variables in the fwknop.conf file: ENABLE_INTF_CHECKS, INTF_CHECKS_INTERVAL, ENABLE_INTF_EXISTS_CHECK, ENABLE_INTF_RUNNING_CHECK, and ENABLE_INTF_BYTES_CHECK. By default, all of these checks are enabled and are run every 20 seconds by the knoptm daemon. If any check fails, then knoptm stops the fwknopd daemon once the error condition is corrected (such as when the interface comes back) so that knopwatchd will then restart it. The fwknopd daemon cannot receive packet data until the error condition is cleared (most likely except perhaps for the "RUNNING" check, but restarting the fwknopd daemon is better than not being able to access a service).
  • Updated the fwknop client to include the SPA destination before DNS resolution when sending an SPA packet over an HTTP request. This allows more flexible Apache configurations with virtual web hosts to function properly with HTTP requests that contain SPA packet data. Also updated the fwknop client to include a leading "/" in SPA packets over HTTP, and updated the fwknopd server to strip this out before attempting SPA packet decryption.
  • Updated the fwknop client to resolve external IP addresses (with the -R argument) here by default.
  • (Jonathan Bennett): Submitted patch to the fwknop client to add HTTP proxy support when sending SPA packets over HTTP. The result is a new --HTTP-proxy option that expects the proxy host to be given as "http://HOST", and it also supports the "http://HOST:PORT" notation as well.

Morpheus-fwknop Windows UI update

The speed at which the open source software movement develops new code is frequently quite impressive, and people crank out higher levels of production when they do something they really like to work on. This remains true for the new Morpheus-fwknop Windows UI written by Daniel Lopez, and a new release (0.7) of Morpheus is now available from Sourceforge. The release notes for 0.7 read as follows:

  • Implement PGP Encryption.
  • Allow to Open multiple port on Forward/Nat Local mode.
  • New UI with a little more color.
  • Launch an Application after sending the SPA Packet.
Here are a couple of screenshots of Morpheus in action. The look and feel has changed quite a bit from the previous release. morpheus-fwknop 0.7 morpheus-fwknop 0.7

Software Releases - libfko and morpheus-fwknop

libfko and morpheus-fwknop This past week one a good one for Single Packet Authorization with two software releases. First, an initial version of libfko, the all C implementation of SPA developed by Damien Stuart can be downloaded along with documentation. The libfko library allows other programs to easily implement the SPA protocol, and a new C client is bundled with fwknop-c-0.62 as well as a new perl module "FKO" that implements a perl XS extension of libfko functions. Once the fwknopd server piece is also developed (a new replacement fwknop C client is included, but not a replacement for fwknopd yet), the libfko code will allow SPA to easily be extended to systems where perl is either not installed or cannot be run (due to hardware constraints such as small routers running OpenWRT).

fwknop-c follows the standard autoconf method of installing open source software, so just: $ ./configure --prefix=/usr && make
$ su
# make install
The new fwknop-c client can be found at /usr/bin/fwknop once you have installed per the above, and all important options are supported similarly to the perl fwknop client. So, the familiar commands like: $ fwknop -A tcp/22 -R -D <host_or_ip> should work just the same. A few of the command line arguments have been changed in the C version, and by default the output on stdout is reduced (just use -v to change this).

Second, a new project called morpheus-fwknop was developed and released by Daniel Lopez. This project is a Windows UI written in .NET to send SPA packets that fwknopd can decrypt. With morpheus-fwknop, there is no need to install Cygwin in order to access services protected by SPA from Windows. This is the first viable UI to succeed Sean Greven's UI developed in Delphi (which still works too!), and morpheus-fwknop is released under the GPL. Naturally, source code can be downloaded, and here are two screenshots that show the look and feel: morpheus-fwknop in action morpheus-fwknop in action It is excellent to see work going on in the world of user interfaces to fwknop and SPA. Without an effective UI on Windows, a large user base is effectively cut off.

OpenSSH Vulnerability Rumor

Protect OpenSSH with fwknop UPDATE: 08/07/09 So, Defcon 17 has come and gone, and I have not heard anything credible to indicate there is a new vulnerability in OpenSSH. That said, I still run all of my SSH daemons behind SPA/fwknop. My original post appears below.

There are rumors of a new vulnerability in OpenSSH that may affect several versions previous to the current OpenSSH-5.2 release. IF these rumors are true - and there is a claim that details will be provided in time for the upcoming Black Hat and Defcon conferences three weeks from now - then it may be a good idea to deploy either a port knocking or SPA solution to protect SSH daemons. Although solid proof has yet to be released, it is a good bet that if there really is a remotely exploitable vulnerability, then it will have nothing to do with brute force password guessing.

Although log parsers that wrap thresholds around failed login attempts can be useful, what I worry about are vulnerabilities where password guessing is completely unnecessary for successful compromises. This is where SPA comes in because it makes a service essentially impossible to identify via scans, and so when an attacker (who may be armed with a 0-day exploit) goes looking for SSH servers to attack, your server will not be listed. There have been remote exploits against OpenSSH before that allow arbitrary code execution, and password guessing was nowhere in sight. It is only a matter of time before the next one is found (hopefully by the whitehats first) in OpenSSH or any other SSH implementation that you are running.

In the meantime, perhaps the Metasploit framework will come up with an exploit before Black Hat. Or, perhaps this will always stay as just a rumor. Either way, there is little harm in protecting your SSH daemon behind SPA and a default-drop firewall policy.

Presentation on Single Packet Authorization at ENSOL

SPA at ENSOLOn June 19th Leandro Almeida gave a presentation entitled "Single Packet Authorization - Increasing the security in SSH" at the ENSOL conference in João Person, Brazil. ENSOL is an open source conference that goes by the title "Freedom in the Extreme", and given that Brazil is highly supportive of Linux, I'm sure that it is a good conference. Leandro has posted an English translation of his slides here. It is good to see some additional presentations on the SPA concept at open source conferences, and Leandro emphasizes the usage of the fwknop SPA implementation to protect SSH.