cipherdyne.org

Michael Rash, Security Researcher



Port Knocking and SPA    [Summary View]

WebKnock.org Single Packet Authorization Proxy

Vasilis Mavroudis has developed a web proxy called WebKnock.org that allows anyone to generate an fwknop SPA packet on their behalf with just a web browser. Although fwknop client portability has improved quite a bit in anticipation of the upcoming fwknop-2.0 release, it is a nice addition to the SPA world to not need the fwknop client installed at all. There are probably several platforms where the native client might not function but can run a web browser.

Using the webknock.org proxy requires that the user provide the SPA key over SSL to webknock.org, but this is a necessary step in exchange for not having to install the fwknop client. As of this writing, SPA via gpg keys is not yet supported, but there are plans to support this in the future. All requests to generate an SPA packet are protected by a captcha.

Behind the scenes, webknock.org executes the fwknop client on behalf of users, and Vasilis informed me that he's using the latest client code (written in C) instead of the older perl client. This is good since all recent development is done on the C version of fwknop in order to make it as small and lightweight as possible.

The service is free, and will hopefully be open-sourced at some point as well. If there are any issues, please either email me or open a ticket on the fwknop github interface. Here is a screenshot of the current webknock.org site: webknock.org SPA proxy

fwknop in the OpenWrt and Pentoo Linux Distributions

fwknop in OpenWrt The C version fwknop has now made it into the OpenWrt Linux distribution for embedded devices. Jonathan Bennett made this possible by contributing a Makefile for OpenWrt, and it was picked up the OpenWrt maintainers. It is good to see progress made towards the integration of Single Packet Authorization into operating systems that are designed to function as secure gateway devices between multiple networks.

So far, fwknop is available in the OpenWrt trunk packages feed, but will eventually become available via the opkg package manager too. Fortunately, OpenWrt makes everything available via git: $ git clone git://nbd.name/packages.git openwrt_packages.git
Initialized empty Git repository in /home/mbr/src/openwrt_packages.git/.git/
remote: Counting objects: 56118, done.
remote: Compressing objects: 100% (21342/21342), done.
remote: Total 56118 (delta 29694), reused 54875 (delta 29054)
Receiving objects: 100% (56118/56118), 11.85 MiB | 2.57 MiB/s, done.
Resolving deltas: 100% (29694/29694), done.
$ cd openwrt_packages.git
$ git ls-files |grep fwknop
net/fwknop/Makefile
$ git log net/fwknop/Makefile
commit 89475e5d6136833fa3b59c3d47c4f2be02718c7a
Author: florian <florian@3c298f89-4303-0410-b956-a3cf2f4a3e73>
Date: Wed Aug 17 10:13:20 2011 +0000

[package] add fwknopd

Signed-off-by; Jonathan Bennett <[email removed]>

git-svn-id: svn://svn.openwrt.org/openwrt/packages@28030 3c298f89-4303-0410-b956-a3cf2f4a3e73

In other news, both the perl and C versions of fwknop are also available in the Pentoo Linux distribution thanks to ozmart and the Pentoo maintainers. Pentoo is a live-cd distribution that is focused on security and derived from Gentoo. ozmart wrote a description of the use case for fwknop on Pentoo from a pentration testing perspective:

"...This is a useful script when combined with iptables and sshd. Configuration can accommodate pgp and replay attack checks. It allows the box to appear silent when running daemons if your box is deployed in say, a hostile environment.

It can also allow commands to be run without actually having to log into the box, say if you wanted to trigger something interesting from a remote location..."

fwknop in BackTrack 5

fwknop has made it into BackTrack Linux distribution. Well, it didn't make it into the default image, but it is in the BackTrack software repository so installing it is a snap with apt-get or aptitude. Below is a screenshot of the BackTrack 5 VM installed under VMware Player running on Ubuntu, and running a search shows both the fwknop-client and fwknop-server packages that are maintained by Franck Joncourt for the Debian distribution. fwknop in BackTrack 5
Update 06/24/2011: Thanks to Sebastien Jeanquier for calling this to my attention and helping to support the inclusion of fwknop in BackTrack Linux.

Single Packet Authorization on Android

fwknop on Android Kicking off 2011 with a bang is easy this year - Max Kastanas has ported the fwknop client to the Android mobile operating sytem. This brings Single Packet Authorization to Google's smart phones, and was accomplished using Damien Stuart's libfko implementation. All of the Android code can be found in the Cipherdyne Trac respository. A couple of screenshots of using the SPA app from an Android phone appear below - after the SPA packet is sent and verified passively by the fwknop daemon on the remote system, the fwknop Android app automatically launches Connectbot to access SSHD:

fwknop Android app fwknop Android app

A new SSH password guessing botnet: dd_ssh

dd_ssh botnet Every few months it seems that there is a new variation of a password guessing botnet. This time around, as reported by the SANS ISC and also by malwarecity.com, a botnet called "dd_ssh" has been built up around a vulnerability in phpMyAdmin. (Here is the vulnerability announcement from the Debian security list.) Once a vulnerable webserver has been compromised, the botnet starts scanning random IP addresses for SSH servers and attempts to brute force passwords for those systems that accept an SSH connection. Although software like Fail2ban and DenyHosts can offer detection and mitigation by applying thresholds to the number of failed login attempts in the SSH log and communicate offending source IP's to a set of interested users, a fundamental problem is that the targeted SSH daemons are reachable in the first place from arbitrary source IP's. A more powerful solution is to hide your SSH daemon behind a default-drop firewall policy and only grant access through Single Packet Authorization. Any botnet that is scanning for SSH servers will not be able to guess any passwords or even exploit a zero-day vulnerability in an SSH daemon that is protected in this way.

Further, SSH brute forcing botnets have shown some innovative techniques where password guessing attempts are spread out over a long period of time and failed passwords are distributed across the botnet. The end result is that each botnet node knows the population of passwords that other nodes have already tried on each targeted SSH daemon, and so new passwords can be tried while still slipping beneath common detection thresholds. The same password never has to be tried twice botnet-wide. It is not clear yet whether the dd_ssh botnet uses a similar password distribution strategy, but either way I intend to stop such guessing games at the earliest stage possible and block all SSH connections by default.

Pure C Implementation of Single Packet Authorization

Pure C Implementation of Single Packet Authorization For the past several months, the pure-C implementation of Single Packet Authorization with fwknop has been in development, and the first release candidate of fwknop-2.0.0 is ready for download. Damien Stuart is the primary developer of this code, and I wish to thank him for all of his efforts in making this release possible. The original perl version of fwknop will continue to receive bug fixes as they are found, but the primary development effort is now on the lightweight and portable C code. The Trac interface for the perl code has moved to fwknop-perl. Any release that is at fwknop-2.0.0 or above will be the C implementation, and any release in the 1.9 series or before are in perl (and will remain available here). The perl and C versions are compatible with each other, so SPA packets created by the fwknop perl client can be properly decrypted and decoded by the new fwknopd C server, and similarly SPA packets built with the fwknop C client can be interpreted by the fwknopd perl server.

The heart of the new C code is the libfko library, which is used by both the fwknop client and the fwknopd server for SPA packet encryption/decryption and interpretation. The libfko library is portable to several platforms including Linux and Windows, and the standard GNU autoconf project is used for building fwknop from sources.

Given the compact nature of the C code, the fwknop-2.0.0rc1 release candidate is already known to function on OpenWRT, and Damien Stuart (the primary developer of the fwknop C code) has built packages for the OpenWRT Kamikaze and Backfire releases. This brings SPA functionality to embedded systems, and has been tested on a Linksys WRT54GS V2.0 router. The OpenWRT packages are available here.

There will be a few release candidates of fwknop-2.0.0 to ensure proper operation on a variety of platforms, and the emphasis will be on stability and ironing out any bugs. Also, fwknop-2.0.0 will initially support iptables firewalls on the servers side, with ipfw support coming soon after. There is one nice feature in fwknop-2.0.0rc1 that deserves a mention - the fwknop client now supports the notion of a ~/.fwknoprc file that allows a set of stanzas to be defined for remote SPA servers and customize the command line arguments used to access each one. Here is an example stanza: # Example for a destination server of 192.168.1.20 to open access to
# SSH for an IP that is resolved externally
#
[myssh]
SPA_SERVER 192.168.1.20
ACCESS tcp/22
ALLOW_IP resolve
SPA_SERVER_PORT 50023
Now, with the fwknop client, we reference this stanza with the -n command line argument like so: $ fwknop -n myssh -vvv
Resolved external IP (via http://www.cipherdyne.org/cgi-bin/myip) as: 123.1.2.3
Enter encryption password:

FKO Field Values:
=================

Random Value: 8308719569638051
Username: mbr
Timestamp: 1280030806
FKO Version: 1.9.12
Message Type: 1
Message String: 123.1.2.3,tcp/22
Nat Access: <NULL>
Server Auth: <NULL>
Client Timeout: 0
Digest Type: 3

Encoded Data: 8308719569638051:bWJy:1280030806:1.9.12:1:OTYuMjQ0LjE3MS4yNTMsdGNwLzIy

SPA Data Digest: WcUEVesZDIY+CKRKYTck8zMW7HG0S7RcqmXVIMC4L58

Final Packed/Encrypted/Encoded Data:

87wQ3s171if9x8628P6k1oBrQEJEmS5c/+ECed5cfDmu8hGhC5esUCEVcI8IH5TdjWva44efebX+cfXUVPBU9c/6tJpOLwC8k/8sN87SgYkBK+m64aD4FnnndaeqsrIcO/oHcpXHk9S/8rqqypcumNtlXVZD/isS3Q729LMTa+2uI6TsGX9OQ3

Generating SPA packet:
protocol: udp
port: 50023
send_spa_packet: bytes sent: 182
On the fwknopd server side, we see the following printed to the terminal in --foreground mode as the SPA packet is received and an iptables ACCEPT rule is added for SSH access and then removed after 30 seconds. Using Digest Cache: '/usr/var/run/fwknop/digest.cache' (entry count = 93)
PCAP filter is: udp port 50023
Starting fwknopd main event loop.


SPA Packet from IP: 123.1.2.3 received.
SPA Packet: '87wQ3s171if9x8628P6k1oBrQEJEmS5c/+ECed5cfDmu8hGhC5esUCEVcI8IH5TdjWva44efebX+cfXUVPBU9c/6tJpOLwC8k/8sN87SgYkBK+m64aD4FnnndaeqsrIcO/oHcpXHk9S/8rqqypcumNtlXVZD/isS3Q729LMTa+2uI6TsGX9OQ3'

SPA Decode (res=0):
SPA Field Values:
=================
Random Value: 8308719569638051
Username: mbr
Timestamp: 1280030806
FKO Version: 1.9.12
Message Type: 1
Message String: 123.1.2.3,tcp/22
Nat Access: <NULL>
Server Auth: <NULL>
Client Timeout: 0
Digest Type: 3
Encoded Data: 8308719569638051:bWJy:1280030806:1.9.12:1:OTYuMjQ0LjE3MS4yNTMsdGNwLzIy
SPA Data Digest: WcUEVesZDIY+CKRKYTck8zMW7HG0S7RcqmXVIMC4L58

Added Rule to FWKNOP_INPUT for 123.1.2.3, tcp/22 expires at 1280030886
RES=0, CMD_BUF: /sbin/iptables -t filter -L FWKNOP_INPUT --line-numbers -n 2>&1
RULES LIST: Chain FWKNOP_INPUT (1 references)
num target prot opt source destination
1 ACCEPT tcp -- 123.1.2.3 0.0.0.0/0 tcp dpt:22 /* _exp_1280030886 */

Removed rule 1 from FWKNOP_INPUT with expire time of 1280030886.

holisticinfosec.org on SPA Ghost Services

holisticinfosec.org on fwknop Russ McRee of holisticinfosec.org has written the January Toolsmith issue from the ISSA Journal about fwknop and the ability to create ghost services with Single Packet Authorization. In his Toolsmith paper, Russ emphasizes the possibility of using the ghost services concept to bypass strict outbound network filtering rules on a local network in order to access an external service that is bound to a port that is filtered by the local firewall. That is, the service is made accessible by having the SPA packet created by the fwknop client request that the remote fwknopd server create iptables DNAT rules to forward connections to a port that the local network actually allows out to the port where the service is bound. Russ uses this concept to access a file that is piped through a netcat listener on TCP port 6543, but do it from the heavily filtered network over TCP port 110 (normally associated with pop3).

Here is a link to the Toolsmith PDF entitled "Single Packet Authorization: The Ghost in the Machine".

Creating Ghost Services with Single Packet Authorization

Creating Ghost Services with Single Packet Authorization Most usages of Single Packet Authorization focus on gaining access to a service such as sshd that is behind a default-drop packet filter. The point being that anyone who is scanning for the service cannot even tell that it is listening - let alone target it with an exploit or an attempt to brute force a password. This is fine, but given that firewalls such as iptables offer well designed NAT capabilities, can a more interesting access model be developed for SPA? How about accessing sshd through a port where another service (such as Apache on port 80) is already bound? Because iptables operates on packets within kernel space, NAT functions apply before there is any conflict with a user space application such as Apache. This makes it possible to create "ghost" services where a port switches for a short period of time to whatever service is requested within an SPA packet (e.g. sshd), but everyone else always just sees the service that is normally bound there (e.g. Apache on port 80).

To illustrate this concept, let's use fwknop from the spaclient system below to access sshd on the spaserver system, but request the access be granted through port 80. Further, on the spaserver system, let's verify that Apache is running and from the perspective of any scanner out on the Internet this is the only service that is accessible. That is, sshd and all other services are firewalled off by iptables. We'll assume that the spaclient system has IP 1.1.1.1, the spaserver system has IP 2.2.2.2, and the scanner system has IP 3.3.3.3.

First, let's scan the spaserver system from the scanner system and verify that only port 80 is accessible: [scanner]# nmap -P0 -n -sV 2.2.2.2

Starting Nmap 5.00 ( http://nmap.org ) at 2009-11-29 15:14 EST
Interesting ports on 2.2.2.2:
Not shown: 999 filtered ports
PORT STATE SERVICE VERSION
80/tcp open http Apache httpd 2.2.8 ((Ubuntu) mod_python/3.3.1 Python/2.5.2)

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 90.94 seconds
Good. Now, from the spaclient system, let's fire up fwknop and request access to sshd on the spaserver system. But instead of just gaining access to port 22, we'll request access to port 22 via port 80 - the fwknopd daemon will build the appropriate DNAT iptables rules to make this work: [spaclient]$ fwknop -A tcp/22 --NAT-access 2.2.2.2:80 --NAT-local -a 1.1.1.1 -D 2.2.2.2

[+] Starting fwknop client (SPA mode)...
[+] Enter an encryption key. This key must match a key in the file
/etc/fwknop/access.conf on the remote system.

Encryption Key:

[+] Building encrypted Single Packet Authorization (SPA) message...
[+] Packet fields:

Random data: 3829970026924871
Username: mbr
Timestamp: 1259526613
Version: 1.9.12
Type: 5 (Local NAT access mode)
Access: 0.0.0.0,tcp/22
NAT access: 2.2.2.2,80
SHA256 digest: Om/GsIVQIRyAp6UWyqjXVqlEQhxz+lVsQhCl1rFBfuI

[+] Sending 182 byte message to 2.2.2.2 over udp/62201...
Requesting NAT access to tcp/22 on 2.2.2.2 via port 80
With the receipt of the SPA packet, the fwknopd daemon has reconfigured iptables to allow an ssh connection through port 80 from the spaclient IP 1.1.1.1 (note the "-p 80" argument on the ssh command line): [spaclient]$ ssh -p 80 -l mbr 2.2.2.2
mbr@2.2.2.2's password:
[spaserver]$
If we can the spaserver again from the scanner system, we still only see Apache: [scanner]# nmap -P0 -n -sV 2.2.2.2

Starting Nmap 5.00 ( http://nmap.org ) at 2009-11-29 15:29 EST
Interesting ports on 2.2.2.2:
Not shown: 999 filtered ports
PORT STATE SERVICE VERSION
80/tcp open http Apache httpd 2.2.8 ((Ubuntu) mod_python/3.3.1 Python/2.5.2)

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 89.22 seconds
So, the IP 1.1.1.1 has access to sshd, but all the scanner can ever see is 'Apache httpd 2.2.8'.

A legitimate question at this point is 'why is this useful?'. Well, I have been on networks before where local access controls only allowed outbound DNS, HTTP and HTTPS traffic, so this technique allows ssh connections to be made in a manner that is consistent with these access controls. (This assumes that HTTP connections are not made through a proxy, and that the fwknopd daemon is configured to sniff SPA packets over port 53.) Further, to anyone who is able to sniff traffic, it can be hard to figure out what is really going on in terms of SPA packets and associated follow-on connections. This is especially true when other tricks such as port randomization are also applied.

I demonstrated the service ghosting technique at DojoCon a few weeks ago, and the video of this is available here (towards the end of the video).

Speaking at DojoCon

speaking at DojoCon UDPATE: The talk slides can be downloaded here and a video of the talk is available too.

On Friday November 6th I will be giving a talk on Single Packet Authorization with fwknop at the upcoming DojoCon conference at Capitol College in Laurel, MD. This talk will focus on some recent developments in the world of fwknop development and the resulting enhancements to SPA capabilities. Topics to be emphasized include:

  • The new libfko C implementation and what it means for SPA deployments.
  • Interface monitoring for administrative control changes and increasing packet counts. This adds an extra dimension of reliability for fwknopd server operations.
  • HTTP proxy support, as well as the development of an SSL web gateway for sending SPA packets on behalf of client systems that cannot construct SPA packets themselves.
  • Development efforts for future fwknop features (such as additional client integration, and deployments on embedded Linux distributions).

Software Release - fwknop-1.9.12

software release fwknop-1.9.12 The 1.9.12 release of fwknop is ready for download. This is a significant release that moves by default to the new libfko and the new FKO module for SPA encryption and decryption. Other new features include interface monitoring for the fwknopd daemon so it can survive administrative changes due to things like a DHCP address changes, and the ability to send SPA packets through HTTP proxies.

An excerpt from the 1.9.12 ChangeLog appears below:

  • Fully integrated the FKO module that is part of the libfko library for all SPA routines - encryption/decryption, digest calculation, replay attack detection, etc. The default is to always use the FKO module if it has been installed, but the original perl code remains intact as well just in case FKO does not exist on the local system. The libfko code can be viewed with Trac here
  • Added the ability to recover from interface error conditions, such as when fwknopd sniffs a ppp interface (say, associated with a VPN) that goes away and then is recreated. In previous versions of fwknop, this would result in the fwknopd daemon no longer able to receive SPA packets. This new functionality is controlled by five new configuration variables in the fwknop.conf file: ENABLE_INTF_CHECKS, INTF_CHECKS_INTERVAL, ENABLE_INTF_EXISTS_CHECK, ENABLE_INTF_RUNNING_CHECK, and ENABLE_INTF_BYTES_CHECK. By default, all of these checks are enabled and are run every 20 seconds by the knoptm daemon. If any check fails, then knoptm stops the fwknopd daemon once the error condition is corrected (such as when the interface comes back) so that knopwatchd will then restart it. The fwknopd daemon cannot receive packet data until the error condition is cleared (most likely except perhaps for the "RUNNING" check, but restarting the fwknopd daemon is better than not being able to access a service).
  • Updated the fwknop client to include the SPA destination before DNS resolution when sending an SPA packet over an HTTP request. This allows more flexible Apache configurations with virtual web hosts to function properly with HTTP requests that contain SPA packet data. Also updated the fwknop client to include a leading "/" in SPA packets over HTTP, and updated the fwknopd server to strip this out before attempting SPA packet decryption.
  • Updated the fwknop client to resolve external IP addresses (with the -R argument) here by default.
  • (Jonathan Bennett): Submitted patch to the fwknop client to add HTTP proxy support when sending SPA packets over HTTP. The result is a new --HTTP-proxy option that expects the proxy host to be given as "http://HOST", and it also supports the "http://HOST:PORT" notation as well.