Botnet Detection and Removal: Methods & Best Practices

November 3, 2015 | Garrett Gross

Get the latest security news in your inbox.

Subscribe via Email

No thanks. Close this now.

Botnet Detection and Removal Structure of a Botnet

Botnet 101

Before we get to best practices in botnet detection, let’s do a quick review of exactly what a botnet is.

One of the most powerful ways to pursue any computationally challenging task is to leverage the untapped processing power of a very large number of everyday endpoints. This is the idea behind the modern botnet: a collection of compromised workstations and servers distributed over the public Internet, which jointly serve the agenda of a malicious or criminal entity.

Once infiltrated with malware in a variety of ways, these compromised systems (“bots”) typically link back to a command and control (C&C) server and wait for instructions. The botnet can then be used for tasks ranging from distributed denial of service (DDOS) attacks, to spam-marketing on a mass scale, to collecting sensitive credit card/financial data… leading in short order to identity theft and fraud.

Want an example? The Gameover ZeuS botnet malware package that runs on Microsoft OS, originally discovered in 2007, operated for over three years in just this fashion, eventually leading to an estimated $70 million in stolen funds and the arrest of over a hundred individuals by the FBI in 2010. And it wasn’t until March 2012 that Microsoft announced it had succeeded in shutting down the “majority” of C&C servers.

As you might guess from the length of Gameover ZeuS’ tenure — still ongoing! — organizations that own compromised workstations often aren’t even aware this is happening until considerable damage has been done. And of course over time the number of botnets has grown significantly in number and value while becoming more sophisticated in their targets, infiltration, anti-detection, and attack techniques. So today, it’s increasingly important for IT professionals to be well-versed in botnet detection techniques and tools.

Botnet detection: Ferreting out one or more bots on your network

Initial signs and symptoms

There are several symptoms which often manifest shortly after botnet infiltration as the compromised machine begins executing its instructions. Awareness of these symptoms can aid in early botnet detection. They include

  • Linking to established C&C servers for instructions
  • Generating Internet Relay Chat (IRC) traffic via a specific range of ports
  • Generating simultaneous identical domain name system (DNS) requests
  • Generating Simple Mail Transfer Protocol (SMTP) traffic/e-mails
  • Reducing workstation performance/Internet access to the point it’s obvious to end users

As you can see, these issues manifest both at the level of individual, compromised workstations and the network as a whole. For network managers, that means there are different botnet detection tactics that can be used at both of these levels.

Botnet detection at the endpoint

Host-based botnet detection begins with client-side anti-viral solutions, since the infiltration itself nearly always happens via malware. Unfortunately, antiviral technology often simply fails to spot an infection, so administrators should also be on the lookout for additional issues.

Host-based botnet detection include things like rootkit installations, unexpected popups while browsing over HTTP (though this may simply be spyware), or any sudden change to the Windows Hosts file, which can be used (or abused) to restrict outbound server access. Also, of course, if the default DNS servers have been modified, that’s likely a sign that traffic is going places the organization doesn’t want it to go.

Botnet detection on the network

Network-based botnet detection is a bit more complex. One approach lies in detecting and monitoring internet relay chat (IRC) traffic, which probably shouldn’t exist on a company network at all. IRC traffic is also sent unencrypted, meaning keywords can be detected with a packet sniffer. The default IRC port is 6667, but the entire port range (from 6660-6669 and 7000) might be utilized by bots.

As suggested earlier, if many endpoints are suddenly and simultaneously hitting one or more external sites, that’s a sign that a botnet-driven DDOS attack is being launched from your network. Similarly, mass outbound traffic happening over SMTP indicates spam-mailing may be an issue. Include rules for these symptoms in your network-based security tools to tune them for botnet detection.

Botnet detection via honeypot

Especially ambitious security professionals may consider creating a honeypot (false infiltration opportunity) and seeing if it, indeed, becomes infiltrated — and if so, how. If you use Suricata, the free open-source intrusion detection solution, you may be able get a list of botnet recognition signatures for it. And, of course, always look for any attempt to connect to known C&C servers.

Host vs. Network Botnet Detection Best Practices Checklist

1.Deploy both host- and network-based botnet detection tools, neither will find every instance every time by themselves. 
2.Ensure your host-based IDS or an anti-malware solution that is capable of detecting the common endpoint signs of botnet infection and is frequently updated with the last known C&C server information. Not catching the easy, obvious infections can be used as a sign of negligence. 
3.Implement a honeypot (or several) if you are protecting reasonably valuable information, have a lot of brand equity in your company’s name, or make for a particularly juicy target for a lawsuit by a victim of a botnet-based attack originating from your network. 

Static vs. behavioral botnet detection

Botnet detection falls into two categories: Static Analysis and Behavioral Analysis. Static analyses are simplistic, fast, and resource friendly. Behavioral analyses are more thorough but also more resource intensive.

Static analysis in botnet detection: your first line of defense

Static techniques — basically, looking for a highly specific match to something like a malware signature or specific executable or C&C connection address (see above) — are fast and, when they work, effective.

Unfortunately, they simply don’t always work; botnet managers (“herders”) are getting increasingly sophisticated about evading such simple giveaways, using counters such as file polymorphism to alter the executables in unpredictable ways, URL obfuscation to hide the targets of DDOS attacks, server proxies, and even rapidly changing the IP addresses of their own C&C servers. Botnet detection via Static Analysis alone simply isn’t enough.

Add behavioral analysis to your botnet detection arsenal to be sure

That’s why behavioral analysis is virtually always an essential approach to botnet detection as well. For instance, the timing of attacks is often a dead giveaway; a C&C server usually issues blanket orders for bots to take specific actions, generating enormous network activity at one point in time (usually, of the types described above under network-based detection).

The average interval of time between an endpoint connecting to different outbound servers will generally be low for bots simply because there isn’t a human driving that network activity. There will be more failed connection attempts for the same reason and those connection attempts are more likely to involve numerical IP addresses than server names. And, of course, port-scanning the local network for new infiltration opportunities is classic behavior for a bot. All of these behaviors can be detected with SIEM / Network IDS rules to expand an organizations botnet detection capabilities.

One slightly newer wrinkle for botnets is a P2P management architecture. This works in a decentralized way, such that there is no central C&C server and commands are issued from peers. Such a botnet is harder to detect, though infected bots will usually act in much the same ways otherwise because the bot herder has the same goals.

Also, botnets are now being designed to go after targets considered “not worth it” in the past – Linux systems, including embedded systems like WiFi routers, CCTV cameras, and more.

Static vs. Behavioral Analysis Botnet Detection Best Practices

1.Use static analysis at a minimum, but organizations should focus botnet detection on behavioral analysis if at all possible, as it is much more effective. 
2.Talk to in-house and external experts about P2P botnet detection techniques. 
3.Ensure the rules for your behavioral, network-based botnet detection systems take into account less common systems. 

Botnet tools and the future of botnet detection

The news isn’t all bad. As botnets have evolved, so have the tools to detect and eradicate them. Today, focused open-source solutions like Snort and more comprehensive, integrated security intelligence offerings from providers like AlienVault are available to:

  • Determine when network activity is unusual in predefined ways
  • Identify its network origin
  • Analyze its nature and impact
  • Directly quarantine, limit, or eradicate local bots

And going forward, such solutions are only getting smarter — fast. This is happening in a variety of ways, some tech-centric (such as machine learning as implemented for botnet pattern recognition), some human-centric and some that combine the two.

For instance, AlienVault’s Unified Security Management (USM) platform integrates expert threat intelligence (correlation rules, IDS signatures, etc.) from AlienVault Labs into all of it’s security controls: SIEM, intrusion detection, vulnerability assessment and compliance reporting. You also get additional context in the form of Indicators of Compromise (known malicious IPs/domains, malware, etc.), generated by threat researchers in the Open Threat Exchange (OTX) community.

Garrett Gross

About the Author: Garrett Gross
Garrett Gross has always had an insatiable appetite for technology and information security, as well as an underlying curiosity about how it all works. Garrett has over 15 years of professional experience in information technology, filling several roles: systems administration, network engineering, product marketing, technical support, and helpdesk. In his current role in field enablement, he uses his experience to help managed security service providers be successful in evangelizing and operationalizing AlienVault USM.
Read more posts from Garrett Gross ›