Recently a psad user notified me via email that the psad-2.1.7.tar.gz tarball
was flagged by ClamAV as being infected with Exploit.HTML.MHTRedir-8.
After checking a few things, it turns out that ClamAV is triggering on a Snort rule in the
Emerging Threats rule set which is bundled in both
psad and fwsnort. The following analysis shows
exactly what ClamAV is detecting and why, and also provides some guidance for how to avoid
this for any software projects that distribute Snort rules. Similar logic would apply to
other software engineering efforts - including commercial intrusion detection systems - that
are (by their nature) looking for malicious artifacts on the filesystem or within network
traffic.
First, let's download psad-2.1.7.tar.gz and check the gpg signature (just to make sure we're
talking about exactly the same file):
$ wget --quiet http://www.cipherdyne.org/psad/download/psad-2.1.7.tar.gz
$ wget --quiet http://www.cipherdyne.org/psad/download/psad-2.1.7.tar.gz.asc
$ gpg --verify psad-2.1.7.tar.gz.asc
gpg: Signature made Wed 14 Jul 2010 06:01:42 PM EDT using DSA key ID 0D3E7410
gpg: Good signature from "Michael Rash (Signing key for cipherdyne.org projects) <mbr@cipherdyne.org>"
Ok, now here is what clamscan says about the psad-2.1.7.tar.gz tarball:
$ clamscan psad-2.1.7.tar.gz
psad-2.1.7.tar.gz: Exploit.HTML.MHTRedir-8 FOUND
----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 6.42 MB
Data read: 1.16 MB (ratio 5.55:1)
Time: 7.169 sec (0 m 7 s)
Let's see which file within the psad-2.1.7 tarball matches the Exploit.HTML.MHTRedir-8
signature:
$ tar xfz psad-2.1.7.tar.gz
$ clamscan -r -i psad-2.1.7
psad-2.1.7/deps/snort_rules/emerging-all.rules: Exploit.HTML.MHTRedir-8 FOUND
----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 41
Scanned files: 405
Infected files: 1
Data scanned: 12.55 MB
Data read: 6.41 MB (ratio 1.96:1)
Time: 8.446 sec (0 m 8 s)
Intuitively, this makes sense. That is, given that ClamAV is out to identify nasty things
within files, and given that Snort rules are designed to identify nasty things as they
communicate over the network, it stands to reason that there might be some overlap. This
overlap is not an indication of something wrong in either the Snort rules or in ClamAV.
Now, let's find out specifically which Snort rule within the emerging-all.rules
file is triggering the ClamAV match. We first take a look at the Exploit.HTML.MHTRedir-8
signature:
$ cp /var/lib/clamav/main.cvd .
$ sigtool --unpack main.cvd
$ grep Exploit.HTML.MHTRedir-8 main.ndb
Exploit.HTML.MHTRedir-8:3:*:6d68746d6c3a66696c653a2f2f{1-20}2168
The last line above is the entire ClamAV signature, and the pattern 6d68746d6c3a66696c653a2f2f
is the key. The ":3:" part identifies the signature as type "normalized HTML", so ClamAV
matches the pattern 6d68746d6c3a66696c653a2f2f against the "normalized HTML" representation
of each processed file. We can decode the pattern as follows:
echo 6d68746d6c3a66696c653a2f2f | xxd -r -p
mhtml:file://
So, within the emerging-all.rules file, we are interested in any Snort rule that
contains the string mhtml:file://. There is also the "{1-20}2168" criteria
which says to match the hex bytes 2168 anywhere from 1 to 20 bytes after the first
pattern match.
$ grep mhtml psad-2.1.7/deps/snort_rules/emerging-all.rules
alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET MALWARE Bundleware Spyware CHM Download"; flow: to_server,established; content:"Referer\: ms-its\:mhtml\:file\://C\:counter.mht!http\://"; nocase; content:"/counter/HELP3.CHM\:\:/help.htm"; nocase; classtype: trojan-activity; sid: 2001452; rev:4;)
Sure enough, sid:2001452 "ET MALWARE Bundleware Spyware CHM Download" has the
keyword content:"Referer\: ms-its\:mhtml\:file\://C\:counter.mht!http\://". Even
though there are escaping backslashes, the normalized HTML processing in ClamAV takes
this into account and matches the pattern anyway from the ClamAV signature.
So, how can we keep the original Snort rule, but change it so that ClamAV not longer
flags it?
Fortunately, ClamAV does not interpret the Snort rules convention of specifying non-printable
bytes between "|" characters within content fields, so we simply need to change one of
characters to hex notation. Snort will still offer the same detection if network traffic
matches the rule, and ClamAV won't flag it. So, let's just change the "m" in "mhtml\:file\://"
to its hex equivalent, like so: "|6d|html\:file\://". Once we make this change and save the
psad-2.1.7/deps/snort_rules/emerging-all.rules file, we rerun clamscan:
$ clamscan -r -i psad-2.1.7
----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 41
Scanned files: 405
Infected files: 0
Data scanned: 17.23 MB
Data read: 6.41 MB (ratio 2.69:1)
Time: 10.024 sec (0 m 10 s)
That's better. Over the next few days I'll re-release all affected versions of psad and
fwsnort with the above change to ensure that there aren't any additional ClamAV matches.
In conclusion, if you are involved in any software engineering effort that distributes
or makes use of Snort rules, it is probably a good idea to run distribution packages
through ClamAV and see if there are any matches. If so, it may be possible to take
advantage of Snort rule syntax options to still achieve the same signature coverage while
not having ClamAV flag anything.
Every few months it seems that there is a new variation of a password guessing botnet.
This time around, as reported by the SANS ISC
and also by malwarecity.com,
a botnet called "dd_ssh" has been built up around a vulnerability in phpMyAdmin.
(Here is the vulnerability announcement
from the Debian security list.) Once a vulnerable webserver has been compromised, the
botnet starts scanning random IP addresses for SSH servers and attempts to brute force
passwords for those systems that accept an SSH connection. Although software like
Fail2ban and DenyHosts can offer detection and mitigation by applying thresholds to
the number of failed login attempts in the SSH log and communicate offending source IP's
to a set of interested users, a fundamental problem is that the targeted SSH daemons
are reachable in the first place from arbitrary source IP's. A more powerful
solution is to hide your SSH daemon behind a default-drop firewall policy and only
grant access through Single Packet Authorization.
Any botnet that is scanning for SSH servers will not be able to guess any
passwords or even exploit a zero-day vulnerability in an SSH daemon that is protected
in this way.
I'm a big fan of the Smokeping
project developed by Tobi Oetiker, and I use it to graph ICMP, DNS, and HTTP round trip
times for my local internet connections and also for cipherdyne.org. Since the beginning
of 2010, I've switched around where cipherdyne.org was hosted from a system at a hosting
provider in Canada to a system running on my home network (serviced by a Verizon FiOS
connection). The drop in RTT's in mid-February was expected and quite large - going
from an average of around 200ms down to about 20ms. Then, in June, I switched to a
different Verizon DSL connection and moved the cipherdyne.org webserver system to a
newer Verizon FiOS connection at a different location, and the RTT's went up a bit to
about 50ms. Here is the graph:
The fwsnort project is now available directly through the
Fedora RPM repositories
(for Fedora 12 and 13) thanks to
Guillermo Gomez.
The version that is currently bundled is fwsnort-1.0.6. Once fwsnort-1.2 is
released, the implementation of large iptables rule sets that are derived from
Snort rules will become a lot faster. This is because fwsnort is going to
support the iptables-save format by integrating the complex rules built by
fwsnort with any existing iptables policy that is instantiated in the kernel.
This is made possible by interpreting the local policy and splicing in all
of the fwsnort rules in the right places - each iptables chain is built from
scratch upon an iptables-restore (including the built-in chains), so
integrating with a running policy is not as easy as just adding each fwsnort
rule into a set of custom chains. Compatibility with the iptables-save format
has largely been completed with this patch
in the fwsnort-1.2 development effort.
For the past several months, the pure-C implementation of Single Packet Authorization
with fwknop has been in development, and the first release candidate of
fwknop-2.0.0 is ready for download.
Damien Stuart is the primary developer of this code, and I wish to thank him for
all of his efforts in making this release possible. The original perl version of
fwknop will continue to receive bug fixes as they are found, but the primary
development effort is now on the lightweight and portable C code. The Trac
interface for the perl code has moved to
fwknop-perl. Any release
that is at fwknop-2.0.0 or above will be the C implementation, and any release in
the 1.9 series or before are in perl (and will remain available
here). The perl and C versions are compatible with
each other, so SPA packets created by the fwknop perl client can be properly
decrypted and decoded by the new fwknopd C server, and similarly SPA packets
built with the fwknop C client can be interpreted by the fwknopd perl server.
The 2.1.7 release of psad is available for
download. This release adds some
relatively minor functionality to switch whois lookups to the destination IP
whenever the source IP is part of a directly connected subnet. This change gives
better reporting data in normal psad email alerts. The complete
ChangeLog entries are displayed below. The psad-2.1.6 release was also made
a few days ago, and fixes a bug that caused psad to die on some Ubuntu systems
when using the Date::Calc Decode_Month() function, and it also added the
--Override-config feature so that alternate configuration variables can
be specified via config files besides psad.conf (implemented by Franck Joncourt).
Russ McRee of holisticinfosec.org
has written the January Toolsmith
issue from the ISSA Journal about fwknop and the ability to create
ghost services with Single Packet Authorization. In his Toolsmith paper, Russ emphasizes
the possibility of using the ghost services concept to bypass strict outbound network
filtering rules on a local network in order to access an external service that is bound
to a port that is filtered by the local firewall. That is, the service is made accessible
by having the SPA packet created by the fwknop client request that the remote fwknopd server
create iptables DNAT rules to forward connections to a port that the local network actually
allows out to the port where the service is bound. Russ uses this concept to access a file
that is piped through a netcat listener on TCP port 6543, but do it from the heavily
filtered network over TCP port 110 (normally associated with pop3).
At the upcoming
SANS Incident Detection Summit on December 9th and 10th I will be participating
in two panel discussions. The first is entitled "Enterprise Network Detection
Tools and Tactics" and is described by Richard Bejtlich
(who has organized the whole conference) as a venue where "speakers with large-scale
experience will share their tools and tactics for identifying suspicious and malicious
activity". The second, "Detection Using Logs", focuses on the usage of platform,
operating system, and application logs to detect intrusions, and Security Information
Management and log aggregation and search systems will be discussed.
UDPATE: The talk slides can be downloaded
here and a video of the
talk is available too.
There has been a Trac repository available on trac.cipherdyne.org
since 2006, and in that time I've collected quite a lot of Apache log data. Today, the
average number of hits per day against the fwknop, psad, fwsnort, and gpgdir Trac
repositories combined is over 40,000. These hits come mostly from search engine
crawlers such as Googlebot/2.1, msnbot/2.0b, Baiduspider+,
and Yahoo! Slurp/3.0;. However, not all such crawlers are created equal. It
turns out that Google is far and away the most dedicated indexer of the
cipherdyne.org Trac repositories, and to me this is somewhat surprising considering
the reputation Google has for efficiently crawling the web. That is, I would have
expected that the non-Google crawlers would most likely hit the Trac repositories
on average more often than Google. But perhaps Google is extremely interested in
getting the latest code made available via Trac indexed as quickly as possible, and
for svn repositories that are made accessible only via Trac (such as the
cipherdyne.org repositories), perhaps brute force indexing of all possible links in
Trac is better. Or, perhaps the other search engines are simply not as interested
in code within Trac repositories so they don't bother to aggressively index it, or
maybe they are just not very thorough when compared to Google.
So, the average number of hits goes from 803 starting out in May and jumps rapidly
to nearly 17,000 in December. Here are the top five User-Agents and associated hit
counts:
The month of May was certainly an aberration with over 56,000 hits per day, and August
topped out at 42,000 hits per day. In May, the top five crawlers were:
Fyodor recently released
Nmap-5.00, and this release marks a major
milestone in Nmap development as it is now quite mature, has a large
following, and is feature rich. With Nmap, service discovery and
interrogation has never been easier or more automated, and for me
Nmap provided much of the inspiration to develop psad (scan
detection via iptables log messages) and fwknop (hide services
from all types of scans behind a default-drop packet filter with
Single Packet Authorization).
It is excellent to see work going on in the world of user interfaces to fwknop
and SPA. Without an effective UI on Windows, a large user base is effectively
cut off.
Recently, Bobby Krupczak, a reader of "Linux Firewalls"
pointed out to me that the iptables script used in the book does
not log traffic over the loopback interface, and that such traffic is also blocked
because of the INPUT and OUTPUT policies of "DROP" (instead of having a separate DROP rule). This
should be made more clear in the book. Quite right - all logging is excluded for traffic that
is sent or received over the loopback interface, and the iptables policy also
drops loopback traffic because no ACCEPT rule exists. The lack of a logging rule
is mostly because logging traffic generated locally and restricted to the loopback interface
is usually a distraction from logging more important (and potentially malicious)
traffic from remote networks. However, if a local process seems to have
connectivity issues, then making sure that loopback traffic flows unimpeded is
important. The iptables.sh script has been updated to ACCEPT all loopback
traffic handled by the INPUT and OUTPUT chains.
UPDATE: 08/07/09 So, Defcon 17 has come and gone, and I have not heard anything credible
to indicate there is a new vulnerability in OpenSSH. That said, I still run all of my SSH
daemons behind SPA/fwknop. My original post appears below.