How to Detect Botnet Traffic with Behavioral Analysis

Bot Behavior – Distributed Attacks

Distributed Attack Behavior

Botnets enable an attacker to avoid basic volumetric detection of their malicious traffic by spreading the same overall number of malicious requests across multiple bots under their control. To a web server with no capability to aggregate or group these bots (because each reports a unique source IP address), a distributed attack can look very similar to legitimate user traffic.

As a security analyst or site administrator, this simply means you can’t ask “Why is this IP address sending us 1000 times the requests of a regular user?” Or, better yet, you can’t ask: “This source IP address has failed login 100 times in the past few minutes, can we block it?”, leaving your web application open to several classes of distributed attacks illustrated below:

Case #1: Distributed Password Guessing/Brute Force Attack

The following example traffic shows a number of HTTP requests to a WordPress login page. Note that each request comes from a different IP address but sends the same log=Administrator username field.

Case #2: Distributed Parameter Fuzzing (local file inclusion)

Our second example demonstrates several bot actors attempting to use a debugging script to access sensitive system data using a potential local file inclusion vulnerability.

Gaining Visibility Into Malicious Traffic

Blocking individual bots or source IP addresses is a losing battle against a distributed attack. If there are any particularly unique characteristics about the attack traffic, it may make sense to block requests matching those (Maybe a User-Agent HTTP header, or the Request URL is very unique). Ultimately, however, we need a more robust approach.

So how can you detect and block a distributed attack? 

To detect and block these kinds of distributed attacks we need to instead look for trends in the distribution of traffic among webpages on a given site. Simple traffic baselining allows us to know how much traffic a page typically receives as well as the number of requests a typical client may send to it. A basic heuristic for alerting, then, might be: “We usually receive 10 login requests a minute, suddenly we’re seeing 1000.”

Behavioral Analysis – Grouping Bot Actors

Tracking Behavior Across Multiple Actors

As we saw in the previous post, Bot Behavior – Distributed Attacks, bots can be used to evade basic volumetric attack detection. Because each bot may use a unique source IP address, aggregating the traffic for alerting, reporting, or blocking can be difficult.

We can try to work around this by baselining traffic patterns for a given URL within a web application, but developing rules around these traffic norms is not always sufficient to distinguish botnet traffic and take appropriate action for offenders.

To complement web application baselining we also need a method for grouping attackers by their behavior. Below is an example heuristic for clustering attacks based on common characteristics: the first step in using behavior to identify likely botnets.

Clustering Attacks – Analysis

For the purpose of this analysis, the overall profile for an attacker or “threat actor” is made up of all their  attacks, which are just a subset of HTTP requests (those which we’ve already decided are malicious in some way).

If the attacker’s attacks are similar in enough ways to another threat actor’s, we can group them and call the combined entity a ‘botnet’, ascribing control of each of the threat actors to a single entity or individual, i.e. they’re bots and each of their individual risks, behaviors, etc. make up some component of the whole entity’s.

Example attacks.json

The following 25 records are a stripped down and sanitized set of  attack logs from the ThreatX WAF, containing only enough information for the clustering example in the next section. In practice, these attacks log over 15 fields each on which we could potentially make clustering decisions. 

Clustering Attacks – Python Example

The example code below performs clustering of the attach json provided in the previous section. In order to cluster attacks we’re using a very basic set of rules:

1. Set a likeness score for each attack

2. If the attack hostname (HTTP  Host:) header matches an existing cluster, add 0.25 to likeness

3. If the attack path ( resource path portion of the URL in the request) header matches an existing cluster, add 0.50 to likeness

4. If the attack timestamp is within ~16 hours (60000s) of the last timestamp on an existing cluster, add 0.25 to likeness

5. If the attack’s total likeness is >= 0.75 for an existing cluster, add the attack to that cluster (and stop checking likeness against the rest of the clusters)

6. If the attack’s total likeness is create a cluster for the attack, using the attack’s hostname, path, and timestamp as metadata for the new cluster

Clustering based primarily on resource path gives us a pretty simple approximation of intent:

The actor intended to perform some kind attack against the resource located at this path.

At this point, we don’t really know if the actor was successful, the type of attack (we’ve stripped those parts out), or who actually controls the actor, but adding the other scores and clustering based on total likeness allows us to at least state:

These actors intended to perform some attacks against this resource path maybe somewhere around the same time, and possibly against the same host.

Which, though a big leap, is enough to say for the purpose of this example:

These actors could be coordinated, let’s consider them to be controlled by the same entity (botnet).

Example Code

Example Clustering Results

Below are selected results from our basic clustering algorithm. It was able to successfully identify the behavior seen in Part II: Case #1 – Distributed Password Guessing  /wp-login.php and Part II: Case #2 – Distributed Parameter Fuzzing  /path/to/debug/script.php as related even though each attack came from multiple source IP addresses, and, in the case of the debug script, also targeted different hosts.

We also see examples of one-off attacks from the example data set; these were sorted into clusters of their own.

Taking Action on the Entire Botnet

What does grouping attackers get us?

The clustering example included in this post is just the beginning of the techniques that can be applied to identify like attack traffic. By clustering attacks and grouping attackers we gain the ability to make decisions based on the entire group’s behavior. For botnet traffic, this means quickly detecting, tarpitting, and/or blocking malicious traffic from all identified members of the botnet, potentially before the individual member is able to fully participate in the attack.

ThreatX is continuously updating our capability to perform this kind of clustering based on attacker behavior. This, combined with active interrogation of suspicious actors and dynamic site profiling, ensures malicious bots are quickly identified and stopped.