How do we confirm that the server is under DDOS attack?
We can confirm it by checking the result of netstat command:
netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
This will show the states and number of connections at that time. The different states that are visible mostly in servers are:
1. ESTABLISHED - This will be legitimate connections established to the server 2. SYN_SENT - The client will be actively attempting to establish a connection. 3. SYN_RECV - A connection request has been received from the network. 4. FIN_WAIT - The socket is closed, and the connection is shutting down. 5. TIME_WAIT - The socket is waiting after close to handle packets still in the network. 6. LISTEN - The socket is listening for incoming connections. 7. LAST_ACK - The remote end has shut down, and the socket is closed. Waiting for acknowledgement.
If the number of connections in SYN_SENT, SYN_RECV, TIME_WAIT, FIN_WAIT are very large in the rate of 1000s then the server is surely under attack.
As a first step we can tweak the values set for SYN_SENT, SYN_RECV, TIME_WAIT, FIN_WAIT in the file /etc/sysctl.conf. Reduce the value of net.ipv4.tcp_fin_timeout to 3 or 5. Normally it will be set to 120 as default. Make the following changes in /etc/sysctl.conf
# Enable TCP SYN cookie protection net.ipv4.tcp_syncookies = 1
# Decrease the time default value for tcp_fin_timeout connection net.ipv4.tcp_fin_timeout = 3
# Turn off the tcp_window_scaling net.ipv4.tcp_window_scaling = 0
# Turn off the tcp_sack net.ipv4.tcp_sack = 0
Then execute the command :
sysctl -p
Then we will have to find out how the attack is being performed, is it from any particular IP or from large number of IP addresses to the server. If it is from any particular IP to the server, then we can fix it by blocking the IP in the firewall. If it is from a large number of IP with one or 2 connections then we will have to find more details to stop it. But will will not be able to completely stop the DDOS attack, we will have to tweak some settings in the server so that the number of connections can be reduced.
Once we reach the result that the server is under attack by checking the number of connections in different state, we need to find to which port the attack is being done. Suppose the number of connections in state SYN_RECV is large. Then we can get the details using the following command:
netstat -lpan | grep SYN_RECV | awk '{print $4}' | cut -d: -f2 | sort | uniq -c | sort -nk 1
The result will be the number of connections and the port open in the server. If the second field is 80 then the attack is to apache port.
In addition to the netstat command, you can use tcpdump command to find out if there is dos attack to a particular port.
tcpdump -nn -tttt -i any port 80
Similarly you can give different ports to find out to which port attack is being done. For example, port 53, 25 etc.
Once you understand the port you need to figure out is the attack done on a particular domain or IP. Suppose the attack is done on port 80, then we can tweak the apache settings as follows:
1. Increase the MaxClients so that we can prevent the condition of apache reaching its limit, since apache could not serve new requests. MaxClients can be set to a max value of the limit set in ServerLimit
2. Set KeepAlive on to set the KeepAliveTimeout
3. KeepAliveTimeout value to be reduced to 3 or 5
So the settings will be as follows:
MaxClients 500 KeepAlive On KeepAliveTimeout 3
/etc/init.d/httpd restart
In order to narrow down the issue, we need to find out if the attack is on any particular IP in the server. This can be found using the following command:
netstat -lpan | grep SYN_RECV | awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -nk 1
After confirming the attack to the IP, we need to find out if the attack is made to a particular domain in that IP or to the IP as a whole. For that, you can check the apache error logs or top command. If in the apache error logs, you are finding the errors for a particular domain, then you will have to perform steps to prevent attack to the domain. For that we can perform the following steps:
1. We can block the connections to the domain using modsecurity. CSF is connected to modsecurity so that if we write rule to block a domain, the IP from whcih connections to the domain are made will be blocked. Since it is DDOS attack, there will be many IPs connecting to the server and blocking high number of IP addresses can cause load in the server and thus server can go down. In order to prevent that, you will have to first block the checking of modsecurity in lfd.
In /etc/csf/csf.conf, set the following:
LF_MODSEC = "0"
csf -r
Then, in the modsecurity configuration file, you can add the following:
SecRule REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS|!REQUEST_HEADERS:Referer "domain\.com"
2. You can block the acesses to port 80 of the domain in the firewall using the following command:
iptables -I INPUT -p tcp --dport 80 -m string --string "domain.com" --algo bm -j DROP
3. If the connections are still not getting reduced, then you can limit the number of connections to the domain using bandwidth module as follows:
/scripts/setbwlimit --domain=domain.com --limit=256000
By executing the above command, a file named /usr/local/apache/conf/userdata/std/2/account/domain.com/cp_bw_all_limit.conf will be created. The content of the file will be :
<IfModule mod_bw.c> ForceBandWidthModule On BandWidthModule On BandWidth all 256000 </IfModule> <IfModule mod_bandwidth.c> ForceBandWidthModule On BandWidthModule On BandWidth all 256000 </IfModule>
Add a line “MaxConnection all 1” such that the number of connections will be limited to 1. So the contents will be as follows:
<IfModule mod_bw.c> MaxConnection all 1 ForceBandWidthModule On BandWidthModule On BandWidth all 256000 </IfModule> <IfModule mod_bandwidth.c> MaxConnection all 1 ForceBandWidthModule On BandWidthModule On BandWidth all 256000 </IfModule>
4. If nothing helped, you can nullroute the IP using the command:
iptables -I INPUT -d XX.XX.XX.XX -j DROP
If the domain is having dedicated IP, then there is no need of above steps, you can directly make the IP down, by deleting the IP from the /etc/ips and restarting ipaliases. But in case of main shared IP, this cannot be done. We will have to reduce the TTL of the domains and change all the domains except the domain to which attack is being made to a free IP after 4 hours and then make the IP down after that so that the attack will be there for only 4 hours. But in such cases there will be issue with cpanel license etc. We will also have to make sure of the name server setting of the domain to which attack is being made. If the domain is using remote name servers, then we cant change any DNS setting of the domain in the server.
In order to prevent this in future, you can add the following commands:
iptables -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP iptables -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j DROP iptables -A INPUT -p tcp --tcp-flags FIN,RST FIN,RST -j DROP iptables -A INPUT -p tcp --tcp-flags ACK,FIN FIN -j DROP
You can also use iftop to watch your network and determine what the target maybe – http://pkgs.repoforge.org/iftop