cipherdyne.org

Michael Rash, Security Researcher



ShmooCon 2013 Talk on fwknop

ShmooCon 2013 Update 03/24/2013: All presentation videos have been uploaded to shmoocon.org, and here is the full video for the talk discussed below: video.

This past weekend at ShmooCon 2013 I gave a talk entitled "Generalized Single Packet Authorization for Cloud Computing Environments" (slides, video demo), and in this blog post I thought it appropriate to give an introduction to the talk in blog format. The main point of the talk was to show that it is completely feasible to integrate SPA as implemented by fwknop with Cloud Computing environments that conform to the Infrastructure as a Service (IaaS) model. In particular, it was shown that SPA can be used to protect and access both SSH and RDP services running on EC2 instances within a Virtual Private Cloud (VPC) in Amazon's AWS environment. Accessing SSHD via SPA by itself is nothing new and has been a primary use case supported by fwknop for a long time, but maintaining a default-drop packet filtering stance for RDP is a different story due to the fact that fwknop does not support a Windows firewall. However, through the use of both SNAT and DNAT capabilities within iptables, the fwknop daemon is able to convert an Ubuntu EC2 instance running within Amazon's Cloud into an "SPA gateway" that is able to then allow access to any service running on any other system within the VPC network - including RDP on a separate Windows host. Further, such access is granted through a single externally routable Amazon Elastic IP that is associated with the Ubuntu system. The Windows host doesn't even need to have a default route back out to the Internet - it always just sees incoming RDP connections from the Ubuntu host even though they really originate out on the Internet from whatever system is allowed via the SPA protocol. Here is a graphic to illustrate this scenario: fwknop + Amazon integration If you want a video demonstration of this all in action, this demo was given during the ShmooCon talk:


Now, for more explanation, the following systems are depicted in the graphic above:

  • The SPA client system that runs the fwknop client to send SPA packets and from which SSH and RDP connections originate.
  • The Amazon border firewall.
  • An Ubuntu host on the VPC network where the fwknop daemon runs.
  • A Windows system on the VPC network where RDP lives.
At the same time, the Ubuntu VPC host is also running a webserver. This is to illustrate that everyone out on the Internet can scan the externally routable Elastic IP (107.23.221.41 in this case) associated with the Ubuntu host and see that Apache is listening on port 80. (Note that this IP is not currently associated with my Amazon account.) But, no other services are visible to a scanner on the Ubuntu system because iptables is deployed in a default-drop stance for everything except for port 80. Even so, fwknop is able to create access through the iptables policy for SSH and RDP through the Ubuntu system over port 80 (i.e. SPA "ghost services"). These two scripts that are shown in the demo video illustrate gaining access to SSH and RDP:
$ cat ssh_access.sh
#!/bin/sh -x

fwknop -A tcp/80 --server-port 40001 -R -D 107.23.221.41
sleep 2
ssh -p 80 -i ~/.ssh/cdyne-vpc-testing.pem ubuntu@107.23.221.41
exit

$ cat rdp_access.sh
#!/bin/sh -x

fwknop -A tcp/80 --server-port 40001 -N 10.0.0.223,3389 -R -D 107.23.221.41
sleep 2
rdesktop -u Administrator 107.23.221.41:80
exit

To see all of the above in action just watch the video. Hopefully the talk itself will be posted online soon, and when it is I'll post a link. More information can also be found in the talk slides.

Finally, I'd like to thank the ShmooCon organizers for putting on a great conference as usual, and I'm looking forward to next year.

Software Release - psad-2.2.1

psad-2.2.1 released The 2.2.1 release of psad is available for download. This release adds new support for detecting a type of scan that psad has been previously blind to - IP protocol scanning (nmap -sO). Also in this release is the ability to detect Topera IPv6 scans and differentiate them from Nmap IPv6 scans. Other important changes include a new test suite, email throttling, and auto blocking timeouts that can be set for each danger level. Here is the complete ChangeLog:

  • Added IP protocol scan detection (nmap -sO). A new psad.conf variable PROTOCOL_SCAN_THRESHOLD defines the minimum number of different IP protocols (default = 5) that must be scanned before an alert is triggered.
  • Added detection for Topera IPv6 scans when --log-ip-options is used in the ip6tables logging rule. When this option is not used, the previous psad-2.2 release detected Topera scans. An example TCP SYN packet generated by Topera when --log-ip-options is used looks like this (note the series of empty IP options strings "OPT ( )":
    Dec 20 20:10:40 rohan kernel: [  488.495776] DROP IN=eth0 OUT=
    MAC=00:1b:b9:76:9c:e4:00:13:46:3a:41:36:86:dd
    SRC=2012:1234:1234:0000:0000:0000:0000:0001
    DST=2012:1234:1234:0000:0000:0000:0000:0002 LEN=132 TC=0 HOPLIMIT=64
    FLOWLBL=0 OPT ( ) OPT ( ) OPT ( ) OPT ( ) OPT ( ) OPT ( ) OPT ( )
    OPT ( ) OPT ( ) PROTO=TCP SPT=61287 DPT=1 WINDOW=8192 RES=0x00 SYN
    URGP=0
    
  • Bug fix in --Analyze mode when IP fields are to be searched with the --analysis-fields argument (such as --analysis-fields "SRC:1.2.3.4"). The bug was reported by Gregorio Narvaez, and looked like this:
    Use of uninitialized value $_[0] in length at
    ../../blib/lib/NetAddr/IP/UtilPP.pm (autosplit into
    ../../blib/lib/auto/NetAddr/IP/UtilPP/hasbits.al) line 126.
    Use of uninitialized value $_[0] in length at
    ../../blib/lib/NetAddr/IP/UtilPP.pm (autosplit into
    ../../blib/lib/auto/NetAddr/IP/UtilPP/hasbits.al) line 126.
    Bad argument length for NetAddr::IP::UtilPP::hasbits, is 0, should be
    128 at ../../blib/lib/NetAddr/IP/UtilPP.pm (autosplit into
    ../../blib/lib/auto/NetAddr/IP/UtilPP/_deadlen.al) line 122.
    
  • Added --stdin argument to allow psad to collect iptables log data from STDIN in --Analyze mode. This makes it easier to run an iptables logs through psad from arbitrary files like so:
    # grep "IN=.*OUT=" /var/log/kern.log | psad -A --stdin
    
  • Added the ability to acquire Snort rule 'msg' fields from fwsnort if it's also installed. A new variable FWSNORT_RULES_DIR tells psad where to look for the fwsnort rule set. This fixes a problem reported by Pui Edylie to the psad mailing list where fwsnort logged an attack that psad could not map back to a descriptive 'msg' field.
  • Added the ability to set per-danger level timeouts when psad is configured to run in auto-blocking mode. These timeouts are implemented with new AUTO_BLOCK_DL*_TIMEOUT variables - one for each of the five possible danger levels that may be assigned to a scanning IP address.
  • Added the ability to throttle emails generated by psad via a new EMAIL_THROTTLE variable which is implemented as a per-IP threshold. That is, if EMAIL_THROTTLE is set to "10", then psad will only send 1/10th as many emails for each scanning IP as it would have normally.
The complete psad-2.2.1 ChangeLog can also be found here via the psad gitweb interface.

Software Release - fwsnort-1.6.3

fwsnort-1.6.3 released The 1.6.3 release of fwsnort is available for download. This release adds a new test suite in the test/ directory that sends fwsnort through its paces for both iptables and ip6tables firewalls, speeds up iptables/ip6tables capabilities testing, and fixes a few bugs. In addition, one of the more significant changes is to ensure that Snort rules with HOME_NET=any -> EXTERNAL_NET=any are placed into the OUTPUT chain instead of the INPUT chain. This bug was reported by Dwight Davis. I would also like to thank Franck Joncourt for his support on the Debian side. Other changes were contributed by the open source community, and these are acknowledged in the complete fwsnort-1.6.3 ChangeLog below:

  • Bug fix to ensure that !, <, >, and = chars in content strings are converted to the appropriate hex equivalents. All content strings with characters outside of [A-Za-z0-9] are now converted to hex-string format in their entirety. This should also fix an issue that results in the following error when running /var/lib/fwsnort/fwsnort.sh:
        Using intrapositioned negation (`--option ! this`) is deprecated in
        favor of extrapositioned (`! --option this`).  Bad argument `bm'
        Error occurred at line: 64
        Try `iptables-restore -h' or 'iptables-restore --help' for more information.
        Done.
    
  • Bug fix to set default max string length in --no-ipt-test mode where iptables capabilities are not tested.
  • (Andrew Merenbach) Bug fix to properly honor --exclude-regex filtering option.
  • Added fwsnort test suite to the test/ directory. This mimics the test suites from the psad and fwknop projects, and it designed to examine many of the run time results of fwsnort.
  • Added the ability to easily revert the fwsnort policy back to the original iptables policy with "/var/lib/fwsnort/fwsnort.sh -r". Note that this reverts back to the policy as it was when fwsnort itself was executed.
  • Implemented a single unified function for iptables match parameter length testing, and optimized to drastically reduce run time for iptables capabilities checks (going from over 20 seconds to less than one second in some cases).
  • (Dwight Davis) Contributed patches for several bugs including not handling --exclude-regex properly, not ignoring the deleted.rules file, not handling --strict mode operations correctly, and more. These issues and the corresponding patch were originally reported here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=693000
  • Bug fix for Snort rules with HOME_NET(any) -> EXTERNAL_NET(any) to ensure they go into the OUTPUT chain instead of the INPUT chain. This bug was reported by Dwight Davis.
  • Updated to bundle the latest Emerging Threats rule set.
The complete fwsnort-1.6.3 ChangeLog can also be found here via the fwsnort gitweb interface.

Software Release - fwknop-2.0.4

fwknop-2.0.4 released The 2.0.4 release of fwknop is available for download. This release includes new support for fuzzing the SPA protocol powered by the perl FKO module along with a set of patches designed to break libfko in subtle ways, and also includes several bug fixes and small enhancements. Among the most important of these include a new check for fwknop chains on iptables firewalls after the initial fwknopd start up (so the fwknop chains can be deleted out from under the fwknop daemon without affecting SPA operations), and a new 'upstart' config for systems running the upstart daemon. Many other changes were contributed by the open source community, and these are acknowledged in the complete fwknop-2.0.4 ChangeLog below:

  • [client] Misc fixes and the addition of save_args and last command (.fwknop.last) support on the Windows platform.
  • [client] Fixed bug in username determination code where a valid value could be overrwritten in certain circumstances.
  • [server] Added upstart config at extras/upstart/fwknop.conf. This allows the fwknopd to easily be managed with upstart via commands like "service fwknop start" and "service fwknop stop".
  • [server] (Vlad Glagolev) Submitted a patch to fix ndbm/gdbm usage when --disable-file-cache is used for the autoconf configure script. This functionality was broken in be4193d734850fe60f14a26b547525ea0b9ce1e9 through improper handling of #define macros from --disable-file-cache.
  • [server] (Vlad Glagolev) Submitted a patch to fix command exec mode under SPA message type validity test. Support for command exec mode was also added to the test suite.
  • (Vlad Glagolev) Submitted an OpenBSD port for fwknop-2.0.3, and this has been checked in under the extras/openbsd/ directory.
  • [server] Bug fix to allow GPG_ALLOW_NO_PW to result in not also having to specify a Rijndael key.
  • [Android] Added new .properties files to allow the fwknop client to build and function properly on the latest Android release (4.1.2).
  • [client] Added '-P udpraw' to allow the client to send SPA packets over UDP with a spoofed source IP address. This is in addition to the original 'tcpraw' and 'icmp' protocols that also support a spoofed source IP.
  • [libfko] Bug fix to check b64_decode() return value to ensure that non-base64 encoded data is never used. Even though other validation routines checked decoded results, it is important to discard invalid data as early as possible. Note too that such invalid data would only be provided to b64_decode() after proper decryption, so the client must provide authentic SPA data.
  • [libfko] Added validation of NAT access strings in the various NAT modes.
  • [libfko] Restricted usernames embedded in SPA packets to be alpha-numeric along with "-" chars.
  • [client] (Franck Joncourt) Contributed a patch to allow the fwknop client to be stopped during the password entry prompt with Ctrl-C before any SPA packet is sent on the wire.
  • [client+server] Applied patch from Franck Joncourt to remove unnecessary chmod() call when creating client rc file and server replay cache file. The permissions are now set appropriately via open(), and at the same time this patch fixes a potential race condition since the previous code used fopen() followed by chmod().
  • [server] Bug fix to accept SPA packets over ICMP if the fwknop client is executed with '-P icmp' and the user has the required privileges.
  • [test suite] Applied patch from Franck Joncourt to have the perl FKO module link against libfko in the local directory (if it exists) so that it doesn't have to have libfko completely installed in the /usr/lib/ directory. This allows the test suite to run FKO tests without installing libfko.
  • [test suite] Significant update to include a set of fuzzing SPA packets that are built using a patched version of libfko. These packets are located in the test/fuzzing/bogus_spa_packets file, and are designed to ensure proper validation of SPA packet data. This validation is performed in --enable-perl-module-checks mode via the perl FKO module.
  • [client] Added --icmp-type and --icmp-code arguments so the user can control the icmp type/code combination for spoofed SPA packets ('-P icmp') mode.
  • [client] Updated default TTL value to 64 for spoofed SPA packets. This is closer to more OS default TTL values than the previous 255.
  • Updated build CFLAGS and LDFLAGS to conform to the Debian hardening-includes file for PIE support (e.g. '-fPIE' for CFLAGS and '-fPIE -pie' for LDFLAGS).
  • [test suite] For GnuPG tests that require a passphrase associated with a gpg key, added a pinentry check to see if the local gpg engine requires it. If so, the gpg test that require a key are excluded since.
  • [server] Added a new '--pcap-file <file>' option to allow pcap files to be processed directly by fwknopd instead of sniffing an interface. This feature is mostly intended for debugging purposes.
  • [server] Added chain_exists() check to SPA rule creation so that if any of the fwknop chains are deleted out from under fwknopd they will be recreated on the fly. This mitigates scenarios where fwknopd might be started before a system level firewall policy is applied due to init script ordering, or if an iptables policy is re-applied without restarting fwknopd.
The complete fwknop-2.0.4 ChangeLog can also be found here via the fwknop gitweb interface.

Tutorial on Single Packet Authorization with fwknop

Single Packet Authorization: The fwknop Approach One thing that has been requested a few times is a tutorial on how to use fwknop, so here it is: "Single Packet Authorization: A Comprehensive Guide to Strong Service Hardening with fwknop". This document attempts to cover most usages of fwknop to wrap an SPA layer around arbitrary services, though some less used scenarios such as SPA over Tor and SPA packet spoofing are covered as well. Please contact me if you find any errors, or if you would like to see material added on a particular fwknop use case. The complete table of contents appears below:

Single Packet Authorization: The fwknop Approach

Single Packet Authorization: The fwknop Approach There are many different port knocking implementations out there, and a few that take things to the next level with Single Packet Authorization. All PK/SPA implementations have three primary goals: 1) the deployment of a default-drop firewall policy for a service (such as SSHD) in order to protect against scans, potential vulnerabilities, and brute force password guessing attempts, 2) the passive monitoring of specially constructed authentication information from the PK/SPA client, and 3) the dynamic reconfiguration of the firewall to allow temporary access to the protected service.

The PK/SPA server never acknowledges any data from the client, so it is infeasible to scan not only for any properly implemented PK/SPA server, but also for any protected service. Effectively, services become invisible to the Internet - except for those that can produce the necessary authentication information, or those that are in a privileged position to sniff traffic between an authenticated PK/SPA client and the target service.

Users of firewalls find value in the idea that traffic to a service can be blocked from all but a few pre-defined networks according to the firewall policy. Few people question whether this is valuable from a security perspective - firewalls generally enhance security (the occasional firewall vulnerability not withstanding). The PK/SPA strategy extends the notion of filtering traffic for a set of services by adding a lightweight crypto layer to allow temporary access from networks that cannot be anticipated when the firewall policy is written. This provides concealment by default, and the SPA strategy asserts that there is value in this. This can apply to all sorts of services from SSHD and OpenVPN to mail protocols like POP and IMAP and even to HTTP. Anyone scanning for services to target will never see a service protected by SPA.

If the above describes unifying aspects of PK/SPA software, this is where the similarities in PK/SPA software ends.

Below are the design choices that guide fwknop development. Other PK/SPA software make different design choices, and whether you prefer fwknop vs. another implementation depends at least partially on whether you agree with the following:

  • No heavyweight interpreted languages:
    Many people don't want to install perl, python, or ruby on a firewall or network gateway device. Such languages are large and complex, and are implemented by similarly complex binaries that are usually themselves written in a language like C. Firewalls are frequently stripped down systems that are designed to just filter network traffic, provide administrative interfaces and sometimes VPN services, and not do much else. Both the fwknop client and server are entirely written in C - there is no requirement for perl, python, or any other interpreted language.

  • Support embedded devices:
    A consequence of fwknop being developed in C is that it only uses minimal system resources and can therefore support embedded devices that don't have a lot of computing power or main memory. For example, where the older perl version of fwknop could not run on Linksys routers supported by OpenWRT, these same routers can run the newer C version of fwknop.

  • Don't require admin access to run the SPA client:
    There are many computing environments where users don't have privileged accounts. This should not present a barrier to using the SPA client. In fwknop, SPA packets are (by default) sent over a regular UDP socket and therefore require no special privileges.

  • Don't require the manipulation of raw packet headers:
    There are some SPA implementations that communicate information within specially modified fields within IP or TCP headers. Such manipulation requires that the user have the ability to acquire a raw socket from the OS, and this requires admin level privileges. In addition, depending on how the SPA client manipulates packet headers when building an SPA packet, other monitoring systems such as an IDS or a passive OS fingerprinter may produce event data that unnecessarily calls attention to the SPA communications. This is not to say that that is impossible to detect SPA packets generated by fwknop - it is just that a monitoring system is more likely to flag communications that involve manipulated packet headers than to generate an event for packets produced by fwknop. For example, intrusion detection systems track TCP connections, and spoofed TCP ACK's that are not part of a proper connection (assuming non-asymmetric routing) may potentially be flagged. Also, manipulated TCP options fields that don't conform to OS defaults will cause an OS to appear to change under the observation of things like p0f. While sometimes this is an expected behavior such as if a VM is being run or a system is actually a NAT device with other systems behind it, there are plenty of deployment scenarios where this is not expected. Addressing this issue at length could be a separate blog post in itself.

  • Don't trust the IP header:
    Any SPA implementation that trusts the source IP address in the IP header of the SPA packet is vulnerable to a MITM attack. An inline device that can intercept an SPA packet can hold the original packet and retransmit it but with the source IP changed to whatever the attacker wants. If the SPA server relies on the source IP in the IP header, then it has no way to know that it isn't legitimate. If an SPA implementation is going to go to the trouble of leveraging cryptography - certainly important since replay attacks among other problems can't be prevented without it - then the IP should be made part of an encrypted payload. This is exactly what fwknop does with the -a or -R arguments on the fwknop command line. An attacker can intercept an SPA packet produced by fwknop, change the source IP and retransmit, but SPA server will only allow access to the IP that was originally encrypted within the SPA payload.

  • Support server-side NAT:
    There are plenty of networks with a border firewall where a remote user actually wants access to a service that is running on an internal system and not on the firewall itself. Typical ways of accessing such a service involve running VPN software, but with an SPA implementation that can manipulate NAT rules on the border firewall it is possible to transparently grant SPA users access to internal services through the firewall. With fwknop, the capability to leverage server-side NAT has been built in for a long time.

  • Support multiple firewalls:
    Because fwknop does not rely on specialized logging infrastructure or link against libraries that are tied to one firewall architecture or another, it can easily support multiple firewalls just by executing the local firewall admin command line interface. It currently supports iptables on Linux, ipfw on FreeBSD and Mac OS X, and pf on OpenBSD.

  • Minimize library dependencies:
    Given the design decisions made by fwknop above, it is important to minimize library dependencies and to audit the source code. Here are all library dependencies in the fwknopd daemon including GnuPG support (this is optional):
    $ ldd ./server/.libs/fwknopd
        linux-vdso.so.1 =>  (0x00007ffeebf820e0)
        libfko.so.0 => /usr/lib/libfko.so.0 (0x00007f1a6ae930e0)
        libpcap.so.0.8 => /usr/lib/x86_64-linux-gnu/libpcap.so.0.8 (0x00007e1a6a85c0e0)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e1a6a49e0a0)
        libgpgme.so.11 => /usr/lib/libgpgme.so.11 (0x00007f1aeaee800e)
        /lib64/ld-linux-x86-64.so.2 (0x00007e1a6aede0e0)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x0000ef1a6a06e0e0)
    
    The libfko library is developed by the fwknop project for SPA encryption, decryption, and validation. libpcap is a standard C library that is leveraged by a lot of security projects including intrusion detection systems and more. The gpgme libraries are only used if you want GnuPG support, and the remaining libraries are standard system libraries that typical Linux binaries need. That's it. One may certainly question the use of libpcap, and in defense of this choice note that: 1) fwknopd by default does not put the sniffing interface into promiscuous mode, 2) fwknopd by default only pays attention to UDP packets to port 62201, 3) if you examine Metasploit you'll see that there only a few exploits for pcap-based software and they target what rides on top of libpcap vs. libpcap itself, and 4) even if there is a vulnerability in libpcap, the exploit model is different than it is for normal server software that you can easily scan for, and being different in this game is good.

  • Support both symmetric and asymmetric ciphers:
    Some users prefer the security properties that asymmetric crypto systems have over those provided by symmetric algorithms like Rijndael, or vice versa. fwknop supports both Rijndael and GnuPG for SPA encryption/decryption. On another note, HMAC-SHA256 support is coming soon to fwknop.

Software Release - fwknop-2.0.3

fwknop-2.0.3 released The fwknop-2.0.3 release of fwknop is available for download. This is an important release that tightens up the code in several areas based on excellent research performed by Fernando Arnaboldi of IOActive. I wish to thank Fernando for this work, and also Erik Gomez of IOActive as well for making this research possible. Fernando's research turned up potential DoS and/or code execution vulnerabilities in the fwknopd server for malicious authenticated clients, insecure filesystem permissions handling, and more. All of these issues have been fixed in the 2.0.3 release.

Here is the complete fwknop-2.0.3 ChangeLog:

  • [server] Fernando Arnaboldi from IOActive found several DoS/code execution vulnerabilities for malicious fwknop clients that manage to get past the authentication stage (so a such a client must be in possession of a valid access.conf encryption key). These vulnerabilities manifested themselves in the handling of malformed access requests, and both the fwknopd server code along with libfko now perform stronger input validation of access request data. These vulnerabilities affect pre-2.0.3 fwknop releases.
  • [server] Fernando Arnaboldi from IOActive found a condition in which the server did not properly validate allow IP addresses from malicious authenticated clients. This has been fixed with stronger allow IP validation.
  • [client+server] Fernando Arnaboldi from IOActive found that strict filesystem permissions for various fwknop files are not verified. Added warnings whenever permissions are not strict enough, and ensured that files created by the fwknop client and server are only set to user read/write.
  • [client] Fernando Arnaboldi from IOActive found a local buffer overflow in --last processing with a maliciously constructed ~/.fwknop.run file. This has been fixed with proper validation of .fwknop.run arguments.
  • [server] Fernando Arnaboldi from IOActive found several conditions in which the server did not properly throw out maliciously constructed variables in the access.conf file. This has been fixed along with new fuzzing tests in the test suite.
  • [test suite] Added a new fuzzing capability to ensure proper server-side input validation. Fuzzing data is constructed with modified fwknop client code that is designed to emulate malicious behavior.
  • Fixed RPM builds by including the $(DESTDIR) prefix for uninstall-local and install-exec-hook stages in Makefile.am.
The fwknop-2.0.3 ChangeLog can also be found here via the fwknop gitweb interface.