Botnet Detection: Understanding the Need, Techniques, and Challenges

Botnets are a coordinated group of malware instances that are controlled via command and control (C&C). They can perform multiple kinds of activities, making them difficult to detect using traditional security measures. In this article, we’ll explore the need for botnet detection, techniques used for botnet detection in enterprise networks and on the internet, and the challenges associated with botnet detection.

The Need for Botnet Detection

The goal of network monitoring is to detect and prevent attack traffic. However, traditional firewalls and network intrusion detection systems (IDS) are becoming less effective due to the wide use of mobile devices. Additionally, attack traffic is becoming more subtle and often looks like normal traffic. For example, a botnet’s HTTP-based command and control traffic would look like normal legitimate web traffic. Therefore, we need more advanced network monitoring systems to detect this new generation of attacks.

Botnet Detection in Enterprise Networks

To detect botnets in enterprise networks, a botnet detection system can be deployed at a gateway or router. The system should look for correlated events across a time horizon, even if a bot has multiple activities in its life cycle. In horizontal correlation, we are looking for similar or coordinated behaviors across multiple bots. In cause-and-effect correlation, we inject traffic to play with the bot to confirm that the traffic is generated by a bot versus a human.

Two systems that can be used for botnet detection in enterprise networks are BotHunter and Botminer. BotHunter is an IDS-based system that performs vertical correlation of multiple events that belong to the life cycle of a bot. It uses a table-driven correlation algorithm to correlate evidence to decide if a host is compromised as a bot. For each bot, BotHunter outputs a description of its behavior profile. The main idea behind BotHunter is to analyze network traffic to detect patterns that suggest any of these activities belonging to the botnet lifecycle. BotHunter uses a table to keep track of the evidence that it collects for each host. The BotHunter architecture consists of three main components: SCADE, SLADE, and Signature Engine. SCADE is for scan detection, SLADE is for anomaly detection in network payloads, and Signature Engine can detect known exploits and known patterns of command control.

On the other hand, Botminer is a botnet detection system that is independent of the C&C protocol and structure. Botminer’s goal is to have a botnet detection system that focuses on the intrinsic properties of botnets, such as long-term use, similar or coordinated communication and activities. Botminer’s architecture consists of three components: C-Plane, A-Plane, and Cross-Plane Correlation. The C-Plane monitor is for monitoring command and control traffic, and the A-Plane is for monitoring malicious activities because these are malware instances. On both planes, clustering is performed to detect groups that are in correlated or similar ways. In cross-plane correlation, the system looks for the intersection between clusters produced by the A-Plane and C-Plane.

BotHunter and Botminer are both effective botnet detection systems, but they have different approaches. BotHunter is based on IDS technologies and uses a table-driven correlation algorithm to correlate evidence to decide if a host is compromised as a bot. Botminer, on the other hand, is independent of the C&C protocol and structure and focuses on the intrinsic properties of botnets. Both systems have their strengths and weaknesses, and they can be used in combination to provide a more comprehensive botnet detection solution.

Botnet Detection on the Internet

Many botnets use DNS for command control because using DNS for command control will not stand out easily, and DNS is always allowed in a network. The DNS service providers preferred by botnets are Dynamic DNS providers because they allow frequent changes of the mapping between DNS domain name and IP address. If we can detect that a domain is used for botnet command-and-control, then we can detect which machine connects to this domain, and this machine is a bot.

For example, a botnet C&C is looked up by hundreds of thousands of machines across the Internet. And yet, it is so-called unknown according to Google search, and that is an anomaly. We can use anomaly detection at the Dynamic DNS service provider by examining queries in DNS domains to identify botnet CNC domains. Once we identify a domain is used for botnet C&C, then a number of responses are available. One is for the service provider to disable a domain, and so when a bot looks up the domain, it gets no IP address back. Another option is for the provider to set a mapping of a domain to a single address. So that instead of connecting to the botnet command-and-control server, the bots are now connected to a sinkhole.

Botnet Detection in Large Networks such as ISPs

In large networks such as ISPs, we can focus on the DNS queries that can be observed by the recursive DNS server and the ISP. We can detect any abnormal growth of the popularity of a domain name. Intuitively, the reason that a botnet will grow is that more machines become infected and become bots. Therefore, botnets are forced to use very random-looking domain names. We need to analyze the growth pattern of these suspicious domain names. Therefore, we record these domains in a Bloom filter. A Bloom filter is a very efficient representation of a set. That is, we use the Bloom filter to record a set of domains observed within these N days. After this baseline, whenever we observe a new domain, that is, a domain that is not in our Bloom filter on a set of recorded domains, if this domain does not fit our Markov model, that means it is spelled in a different way and looks suspicious.

Challenges Associated with Botnet Detection

The latest threats are more targeted and more advanced. For example, they use custom-built malware on zero-day exploits, their activities are low-and-slow, and they move within the network and cover their tracks. Even the existing botnet detection systems are not effective against these targeted advanced threats. In order to counteract these targeted and advanced threats, we need multifaceted monitoring and analysis. That is, we need malware analysis, host-based monitoring, forensics, and recovery, network monitoring, Internet monitoring, threat analysis, and attribution.

In conclusion, botnet detection is a critical aspect of network security. Traditional security measures are becoming less effective, and botnets are becoming more sophisticated. However, with advanced network monitoring systems and techniques, we can detect and prevent botnet attacks.