In computer security, it quickly becomes apparent that preventing computer attacks is much more challenging than attacking computers. A good example of an easy technique to prevent a website from functioning is a distributed denial of service, or DDoS, attack in which a number of compromised computers around the internet make web (or other protocol) requests on some poor server. If the web page requested is one that requires lots of server-side processing, the resulting load from the combined requests prevents the web server from responding to legitimate requests, thus denying the service. As Tech-Recipes.com was subjected to such an attack recently, we felt it might be beneficial to others if we described the steps we took in our response.
Note: The following information is relevant to UNIX-based servers running Apache (although other platforms and software may be applicable). A prerequisite for this approach requires the use of iptables which likely means you need root access to the server, so this will probably not help you if you are using shared hosting. Sorry! Your best bet in this case is to contact your ISP for their help (good luck) as it is in their best interest to prevent high loads on their shared servers, although it is likely that they will temporarily disable your domain services, which is definitely not the best solution from your perspective.
To minimize the impact of a denial of service attack, you should make yourself aware of problems with your web server in near real-time. Several server monitoring services are available. We use both Pingdom and SiteUptime. Using these services, we have redundant notifications of any service disruption sent to our cell phones via SMS and to several email addresses (none of which are handled by our monitored servers, of course). Using these services, we are notified within a minute of a failed request to our production servers.
Before you implement specific anti-DDoS techniques or seek help from a DDos Protection service, it is wise to make sure that the problem is actually a DDoS attack. Services may fail to answer a request for a number of reasons among them the executable running the service dying unexpectedly, the ISP hosting the server experiencing a network or server outage, or some other self-correcting glitch in the matrix. To determine if the service failure is due to a DDoS and to collect information necessary to take actions against the attack, you need to dig around first.
You need access to your access_log, the log file which contains a text entry for every request made to your web server. There are many places that this file can hide and it is dependent on your server setup. When in doubt, you can check the ISP’s documentation (search for access log) — the log data may be accessible through a web console, but it is optimal if you have shell access to your server. If your ISP’s online help is not so helpful in finding the access log, you can use the UNIX find command.
Once you have found your access log, cd into the directory containing the log and run the tail command on it with the -f option:
tail -f access_log
The tail command on its own will display the last 10 lines of the specified file and then quit. The -f option will tell tail to keep running after the last 10 lines are displayed and keep displaying subsequent lines that are added to the file. You’re likely to see a flurry of entries. If you don’t see any log messages after the default 10, you’re either looking at the wrong file or the web server is dead or otherwise not seeing any traffic. It is possible that one server can host many websites and each can have a separate access_log file, so be certain that you’re tailing the right one. If the service is dead, resuscitate it through whatever mechanism is appropriate. We’re going to assume from here on out that you’re seeing a ton of access log messages that look very similar (probably hitting the same URL) that don’t have a valid or any referrer (if the referrer is slashdot.org or digg.com, well, then you can be both happy and sad that the traffic, though server crippling, is legitimate) and have a bunch of different IP addresses. During our recent siege, there were so many bogus requests that very few legitimate requests made it to the access_log file. Here are a few lines from our access_log during the attack:
220.255.7.204 - - [07/Jul/2008:19:28:18 -0700] "GET /modules.php?name=Forums&file=index HTTP/1.1" 200 0 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
220.255.7.209 – – [07/Jul/2008:19:28:18 -0700] “GET /modules.php?name=Forums&file=index HTTP/1.1” 200 0 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)”
220.255.7.208 – – [07/Jul/2008:19:28:18 -0700] “GET /modules.php?name=Forums&file=index HTTP/1.1” 200 0 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)”
71.232.78.53 – – [07/Jul/2008:19:28:19 -0700] “GET /modules.php?name=Forums&file=index HTTP/1.1” 200 14313 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”
If you aren’t familiar with the default structure of Apache’s access_log, the first value is the source IP address of the request, then the date and time of the request is in square brackets, and the first string in double quotes is the HTTP request made. In our example, the request was always the same: “GET /modules.php?name=Forums&file=index HTTP/1.1” — a GET request of the URL (/modules.php?….) and the HTTP protocol used (1.1). The portion of the URL used is important since it is the one thing we can use to identify the offending IP addresses. The second string in quotes is the referring URL, in our case there isn’t one so it shows up as “-” which lends credence to the suspicion that it is a DDoS attack. The remaining string describes the platform and software that has made the request and isn’t very meaningful or helpful to us as it can easily be forged.
Now that we know there is a DDoS attack underway, we need a list of the IP addresses involved in the attack in order to block their subsequent requests with iptables. In the example access_log above, I know that the requested URL component “/modules.php?name=Forums&file=index” can be used to search through the access_log. Because it contains special characters like & and ? which most UNIX shells interpret as something very different, it is a good idea to put the search string in double quotes in all commands that use it. The following command will grab only those lines in access_log that match the URL component above and will return a list of IP addresses that requested it and a count of how many times each IP address made the request, sorted by the count.
fgrep "/modules.php?name=Forums&file=index" access_log | cut -d\ -f1 | sort -n | uniq -c | sort -rn
The fgrep command is a fast grep which doesn’t look for regular expressions and can tear through huge log files much more efficiently. If your system doesn’t have it for some unfathomable reason, substitute plan old grep for fgrep. Substitute the name of your access log if it is different than access_log. Note that there are two spaces after the backslash in the cut -d\ option. This string of commands will pull out the matching lines in access_log (fgrep), extract the first field separated by spaces (cut), sort them numerically (sort -n), group them to make a list of unique IP addresses and count the occurences (uniq), and sort them numerically in reverse order so that the IP addresses with the highest counts come first. Here’s an example of the output:
21889 71.232.78.53
17181 220.255.7.208
16162 220.255.7.209
16142 220.255.7.204
You are likely to see two groups, some IP addresses with a large number (hundreds, thousands) of requests and some with just a few, maybe tens of requests. When picking the number of requests that separates the two groups, be careful as you don’t want to block legitimate requests. Chances are as you start blocking IP addresses from the top of the list, you’ll know when the numbers change and become legitimate seeming values, like a jump from many hundreds of hits to a few tens of hits. If you can, copy the output of the command above to some form of text editor to keep handy. One note, if your access_log is not rotated frequently (its contents copied to another location and possibly compressed), it can get huge. You can use the tail command in a different way to make this command run quicker, which your beleaguered server will probably appreciate — the “tail -10000 access_log” command will take the last 10,000 lines from access_log and continue processing them (you can change 10000 to a number of your liking):
tail -10000 access_log | fgrep "/modules.php?name=Forums&file=index" | cut -d\ -f1 | sort -n | uniq -c | sort -rn
Now that you are armed with the knowledge of which IP addresses you need to block, it’s high time to lay down the iptable law. The iptables system is capable of many different operations and, fortunately for us, blocking a single IP address is one of the easier configurations. To block all requests from one IP address like 71.232.78.53, use the command:
iptables -A INPUT -s 71.232.78.53 -j DROP
If you are not logged into the server as root, just add a ‘sudo’ in front of this command and provide your password when prompted. This one command adds a temporary rule to the iptables system to drop all packets coming from the specified IP address. This effect is temporary in that it will not persist following a reboot. If you want to make these changes permanent (probably not a great idea, but, not that bad, either, if you plan to reboot soon to liberate any leaked memory), you can try running
/etc/init.d/iptables save
This command will either work and let you know that it worked, or will error out. In the latter case, you can repeat the iptables commands, one per IP address, to reblock them.
If you have requests coming from a number of IP addresses within the same subnet, you can block the subnet in one iptables command. In our case, we did have a bunch of IP addresses from the 220.255.7.0 subnet sending requests. Rather than repeat the iptables command a couple hundred times, the subnet shorthand for the iptables command makes this task much simpler:
iptables -A INPUT -s 220.255.7.0/24 -j DROP
I wouldn’t block more than a class C subnet at a time, as shown above. Essentially, if you see a bunch of requests from IP addresses in which the first three numbers in the IP address are the same, then they are all in the same class C subnet and you can block them all using the first three period-separated numbers followed by a .0/24 as above.
Since a DDoS attack may be ongoing and more machines may take part in the attack over time, it might be worth your effort to make a quick script to block an IP address simply. I wrote the following script and placed it in /usr/sbin/block
#!/bin/bash
iptables -A INPUT -s $1 -j DROP
Once the file is in place, make it executable with chmod +x /usr/sbin/block and then use it like:
block 71.232.78.53
Every system is unique, has different applications and versions of applications installed in different places, so your mileage may vary with the instructions above.
