Layer 4 and Layer 7 attcks
Those who experienced DDOS attacks knows how hard it is to mitigate them. Well this tutorial is not going to concentrate on mitigating a DDOS attack completely, but blocking Layer 7 attacks on application layer with upto 100000 simultaneous connections. When you receive a layer 4 attack that counts in 10s of Gbps, you will need to mitigate them at your Router level or you may consider buying a hardware firewall from your provider.
But layer 4 attacks with more that 10Gbps is not something an attacker would love to chose as it cost too much money and effort to it. And even if you have a firewall that block layer 4 attacks, is it Good enough.? Unfortunately its not! The hardware firewalls are typically designed to mitigate attacks at network layer and it cant recognize and block attacks at application layer. Some Hardware firewalls with Layer 7 capabilities are available today, but they cost you hundreds of thousand Dollars which small companies cant afford.
How to block Layer 7 DDOS attack with 100,000 connections.?
This tutorials concentrates on blocking Layer 7 attacks upto 100000+ connections per second using Haproxy and Nginx and distributing the traffic among multiple servers.
Various Benchmark results says the webservers available today are not stable with over 10000 connections per second with latest Xeon processors(There are few exceptions such as Nginx, Litehttpd etc). Scaling up the servers vertically by adding more CPU and RAM is not going to help much.
Let us consider a scenario where you’re getting an application level attack with 100000 connections per seconds. And lets says your webserver can sustain 10000 connections. You will then need 10 webservers with the traffic distributed among them using a Loadbalancer. So you need a total of 11 servers – 1 loadbalancer and 10 webservers to handle the traffic. If you go with a cloudhosting such as Amazon, a cloud instance with 2 CPU and 4GB of RAM cost you about $75 a month. So the total cost per month is going to be 11X75=825. Yeah, $825 is hell of a saving if you compare it with prices of a hardware firewall with DDOS capabilities.
Well can these costs be reduced even further.? They can. Lets say your loadbalancer has layer-7 DDOS capabilities and it blocks most of the DDOS traffic at the loadbalancer level and send only remaining connections to webserver cluster. Consider the above example where the Loadbalancer receive 100000 connections, identify the attack traffic and block 75% of them (with proper tuning, you can easily reach 90% or above). Then only the remaining 25000 connections need to go to application server cluster. And yes, to handle remaining connections, you need less number of servers in the application cluster. With 3 servers in the cluster and one loadbalancer, the total cost is gonna be 4*75=$300
Ok, now the obvious question is what loadbalancer can handle more than 100,000 connections and identify which of them are attack traffic.?
Haproxy to mitigate DDOS
HaProxy is the best opensource loadbalancer on the market which has too many features in terms of rate limiting and cookie capabilities. All these features when tuned properly can act as a refined DDOS firewall. Also it is knows for handling huge number of simultaneous connections and benchmark results are quite amazing! = > http://www.haproxy.org/10g.html
Haproxy has following features to detect and block DDOS.
– It can create a table for tracking connections per IPs – the connection rate and also the request size. Then you can add certain rules to block the source IP based on their connection rate or request size over past few minutes. This feature can be effectively used to block slow rise attacks as well.
– It has cookie capabilities which maybe used to check and identify user agents. 99% of the bots will not have cookie capabilities so querying the user agents to identify attacker bots and blocking them is possible. Well, don’t worry about valid users being blocked who doesn’t have cookie capabilities on their browsers, because almost no user has cookies totally disabled in modern browsers. And if you still find any that does, let them go away, they don’t have access to anything on the net anyway. Besides, keeping your website UP and RUNNING is what we care more about.
– And lot others such as Syn Flood blocking, connection limits etc.
I am not going deep into configuration side as its beyond the scope of this tutorial. Please download haproxy 1.5 and start experimenting. => http://www.haproxy.org/
Nginx webserver capabilities to block DDOS:
So now we have a n effective loadbalancer which can distribute the loads across webserver cluster. Is that all about blocking DDOS.?
Nope, there are a lot more which your webserver Nginx can do with all its HTTP capabilities. We recommend moving to Nginx if you’re already not using it. Litehttpd is a good candidate too but its not free.
Ok, lets take a look the DDOS features which your Nginx offers.
– Nginx is a webserver with all HTTP capabilities so there are a lot more which you can do by modularizing and customizing it.
– Rate limits. Nginx identifies the number of concurrent IP addresses and also additional modules are available such as ngx_http_access_module to limit access based on client IP addresses. ngx_http_geo_module is an another module which can be used to block access based on client locations. Most effective when the attacks seems to be generating from a Country where you do not have many valid users.
So combining both haproxy and nginx and using best of their features to prevent DDOS is going to save you a lot.
Is this all you can do against DDOS.?
No, there are a lot more. Failban is a good tool that can query logs of nginx and haproxy and block the IP addresses of attackers. Also many people hardcode their pages which is being attacked with a meta refresh, then only the legitimate user are forwarded to the website. Also connection tracking using iptables and blocking them is a good strategy if the number of bots that attack your server is small in number and connections per IP is more. Varnish is another great tool which runs in front of webserver and act as a caching server and handle large number of simultaneous connections.
Haproxy doesn’t recommend running iptables, failban or any other softwares in its box. Well, I wouldn’t do that even if they did! One loadbalancer is all you got so its important that you dont load any other software to same box. All additional tools that you chode should go to webservers running Nginx, and not the loadbalancer.
What if the attacks reach 200,000 or more connections.?
Yes you still can sustain attacks with 200,000 or more connections. You just need to add more servers. Instead of one Loadbalancer, you will need to run 2 (or more) loadbalancers and use DNS in roundrobin fashion. DNS providers such as DNSMADEEASY and AWS provide DNS servers that can use persistent connections so that your users will be always sent to same loadbalancer keeping end user persistent connections. Nowadays, all Cloud hosting providers gives auto scaling options for servers which means new servers will be deployed in real time when an attack starts and decommissioned as the attack ends. All attackers tend to give up in less than an hour if they cant succeed. So a clustering with autoscaling option is going to save you a lot more as you are paying only for the usage.
Hope this gives you a basic idea of preventing the application layer attacks. Good luck with your experiments and remember no attackers are better than a sysadmin.
(Put your questions if any as comments).