Ten steps for combating DDoS in real time
To the uninitiated, a distributed denial-of-service (DDoS) attack can be a scary, stressful ordeal. But don’t panic. Follow these steps by David Holmes, senior technical marketing manager: Security, F5 Networks, to successfully fight an attack:
If you appear to be suffering a volumetric attack, it helps to have a historical sense of your own traffic patterns. Keep a baseline of normal traffic patterns to compare against. If you have determined that you are under a DDoS attack, record the estimated start time in your attack log. Monitor volumetric attacks. Remember to keep a monitoring web page open to indicate when the attack may be over (or mitigated). You will need to follow (up to) 10 steps for your DDoS mitigation:
Step 1: Verify the attack
Not all outages are caused by a DDoS attack. DNS misconfiguration, upstream routing issues, and human error are also common causes of network outages. You must first rule out these types of non-DDoS attacks and distinguish the attack from a common outage.
· Rule out common outages: The faster you can verify the outage is a DDoS attack, the faster you can respond. Even if the outage was not caused by a misconfiguration or other human error, there may still be other explanations that resemble a DDoS attack.
· Check outbound connectivity: Is there outbound connectivity? If not, then the attack is so severe that it is congesting all inbound and outbound traffic. Check with your usual diagnostic tools (such as traceroute, ping, and dig) and rule out all such possibilities.
· Rule out global issues: Check Internet weather reports, such as Internet Health Report and the Internet Traffic Report, to determine if the attack is a global issue.
· Check external network access: Attempt to access your application from an external network. Services and products that can perform this kind of monitoring include: Keynote testing and monitoring, HP SiteScope agentless monitoring, SolarWinds NetFlow Traffic Analyzer, and Downforeveryoneorjustme.com.
· Confirm DNS response: Check to see if DNS is responding for your website. The following UNIX command resolves a name against the OpenDNS project server: % dig @22.214.171.124 yourdomain.com
Step 2: Contact team leads.
Once the attack is verified, contact the leads of the relevant teams. If you have not filled out any quick reference sheets or a contact list, create one now or use our templates. When an outage occurs, your organisation may hold a formal conference call including various operations and applications teams. If your company has such a process in place, use the meeting to officially confirm the DDoS attack with team leads.
· Contact your bandwidth service provider: One of the most important calls you can make is to the bandwidth service provider. List the number for your service provider in your contact sheet. The service provider can likely confirm your attack, provide information about other customers who might be under attack, and sometimes offer remediation.
· Contact your fraud team: It is especially important to invoke the fraud team as soon as the attack is verified. DDoS attacks can be used as cover to hide an infiltration. Logs that would normally show a penetration may get lost during a DDoS attack. This is why high-speed, off-box logging is so important.
Step 3: Triage applications
Once the attack is confirmed, triage your applications. When faced with an intense DDoS attack and limited resources, organisations have to make triage decisions. High-value assets typically generate high-value online revenue. These are the applications you will want to keep alive. Low-value applications, regardless of the level of legitimate traffic, should be purposefully disabled so their CPU and network resources can be put to the aid of higher-value applications. You may need the input of team leads to do this.
Ultimately, these are financial decisions. Make them appropriately. Create an application triage list; it takes only a few minutes to fill one out, and will greatly assist in making tough application decisions while combating an actual DDoS event. Decide which applications are low priority and can be disabled during the attack. This may include internal applications.
Step 4: Protect partners and remote users.
· Whitelist partner addresses: Very likely you have trusted partners who must have access to your applications or network. If you have not already done so, collect the IP addresses that must always be allowed access and maintain that list. You may have to populate the whitelist in several places throughout the network, including at the firewall, the Application Delivery Controller (ADC), and perhaps even with the service provider, to guarantee that traffic to and from those addresses is unhindered.
· Protect VPN users: Modern organisations will whitelist or provide quality-of-service for remote SSL VPN users. Typically this is done at an integrated firewall/ VPN server, which can be important if you have a significant number of remote employees.
Step 5: Identify the attack
Now is the time to gather technical intelligence about the attack. The first question you need to answer is “What are the attack vectors?” There are four types of DDoS attack types, these are
· Volumetric: flood-based attacks that can be at layers 3, 4, or 7;
· Asymmetric: designed to invoke timeouts or session-state changes;
· Computational: designed to consume CPU and memory; and
· Vulnerability-based: designed to exploit software vulnerabilities.
By now you should have called your bandwidth service provider with the information on your contacts list. If the attack is solely volumetric in nature, the service provider will have informed you and may have already taken steps at DDoS remediation. Even though well-equipped organisations use existing monitoring solutions for deep-packet captures, you may encounter cases where you have to use packet captures from other devices, such as the ADC, to assist in diagnosing the problem. These cases include: SSL attack vectors and FIPS-140.
Step 6: Evaluate source address mitigation options
If Step 5 has identified that the campaign uses advanced attack vectors that your service provider cannot mitigate (such as slow-and-low attacks, application attacks, or SSL attacks), then the next step is to consider the following question: “How many sources are there?” If the list of attacking IP addresses is small, you can block them at your firewall. Another option would be to ask your bandwidth provider to block these addresses for you.
· Geoblocking: The list of attacking IP address may be too large to block at the firewall. Each address you add to the block list will slow processing and increase CPU. But you may still be able to block the attackers if they are all in the same geographic region or a few regions you can temporarily block. The decision to block entire regions via geolocation must be made as a business decision. Finally, if there are many attackers in many regions, but you don’t care about any region except your own, you may also use geolocation as a defence by blocking all traffic except that originating from your region.
· Mitigating multiple attack vectors: If there are too many attackers to make blocking by IP address or region feasible, you may have to develop a plan to unwind the attack by mitigating “backwards”; that is, defending the site from the database tier to the application tier, and then to the web servers, load balancers, and finally the firewalls.
You may be under pressure to remediate the opposite way; for example, mitigating at layer 4 to bring the firewall back up. However, be aware that as you do this, attacks will start to reach further into the data centre.
Step 7: Mitigate specific application attacks
If you have reached this step, the DDoS attack is sufficiently sophisticated to render mitigation by the source address ineffective. Tools such as the Low Orbit Ion Cannon, the Apache Killer, or the Brobot may generate attacks that fall into this category. These attacks look like normal traffic at layer 4, but have anomalies to disrupt services in the server, application, or database tier.
To combat these attacks, you must enable or construct defences at the application delivery tier.
Once you have analysed the traffic in Step 4, if the attack appears to be an application-layer attack, the important questions are: Can you identify the malicious traffic? Does it appear to be generated by a known attack tool?
Specific application-layer attacks can be mitigated on a case-by-case basis with specific F5 counter-measures. Attackers today often use multiple types of DDoS attack vector, but most of those vectors are around layers 3 and 4, with only one or two application-layer attacks thrown in. We hope this is the case for you, which will mean you are nearly done with your DDoS attack.
Step 8: Increase application-level security posture.
If you have reached this step in a DDoS attack, you’ve already mitigated at layers 3 and 4 and evaluated mitigations for specific application attacks, and you are still experiencing issues. That means the attack is relatively sophisticated, and your ability to mitigate will depend in part on your specific applications.
Asymmetric application attack: Very likely you are being confronted with one of the most difficult of modern attacks: the asymmetric application attack. This kind of attack can be:
· A flood of recursive GETs of the entire application.
· A repeated request of some large, public object (such as an MP4 or PDF file).
· A repeated invocation of an expensive database query.
Leveraging your security perimeter: The best defence against these asymmetric attacks depends on your application. For example, financial organisations know their customers and are able to use login walls to turn away anonymous requests. Entertainment industry applications such as hotel websites, on the other hand, often do not know the user until the user agrees to make the reservation. For them, a CAPTCHA (Completely Automated Public Turning test to tell Computers and Humans Apart) might be a better deterrent.
Choose the application-level defence that makes the most sense for your application: A login wall, human detection or real browser enforcement.
Step 9: Constrain resources.
If all the previous steps fail to stop the DDoS attack, you may be forced to simply constrain resources to survive the attack. This technique turns away both good and bad traffic. In fact, rate limiting often turns away 90 to 99 percent of desirable traffic while still enabling the attacker to drive up costs at your data centre. For many organisations, it is better to just disable or “blackhole” an application rather than rate-limit it.
· Rate shaping: If you find that you must rate-limit, you can provide constraints at different points in a multi-tier DDoS architecture. At the network tier, where layer 3 and layer 4 security services reside, use rate shaping to prevent TCP floods from overwhelming your firewalls and other layer 4 devices.
Connection limits: Connection limits can be an effective mitigation technique, but they do not work well with connection-multiplexing features. Application tier connection limits should provide the best protection to prevent too much throughput from overwhelming your web servers and application middleware.
Step 10: Manage public relations
Hacktivist organisations today use the media to draw attention to their causes. Many hacktivists inform the media that an attack is underway and may contact the target company during the attack. Financial organisations, in particular, may have policies related to liability that prevent them from admitting an attack is underway. This can become a sticky situation for the public relations manager. The manager may say something like, “We are currently experiencing some technical challenges, but we are optimistic that our customers will soon have full access to our online services.”
Journalists, however, may not accept this type of hedging, especially if the site really does appear to be fully offline. In one recent case, a reporter called a bank’s local branch manager and asked how the attack was proceeding. The branch manager, who had not received media coaching, responded, “It’s awful, we’re getting killed!”
If the DDoS attack appears to be a high-profile hacktivist attack, prepare two statements:
· For the press: If your industry policies allow you to admit when you are being externally attacked, do so and be forthright about it. If policy dictates that you must deflect the inquiry, cite technical challenges but be sure to prepare the next statement.
· For internal staff, including anyone who might be contacted by the press: Your internal statement should provide cues about what to say and what not to say to media, or even better, simply instruct your staff to direct all inquiries related to the event back to the PR manager. Include a phone number.
Anton Jacobsz, managing director at Networks Unlimited, a value-adding reseller of F5 solutions throughout Africa, notes that it is the organisations focusing on a holistic security strategy that are considered forward-looking and ahead of the digital economy curve.
“In a digital age – where sensitive or personal information is at risk of being exposed, and where geo-location and sensor-based tools track movements – organisations need to be prepared for a cyber attack. It has become essential to scrutinise security throughout the entire operation and offerings in order to build the strongest cornerstones for establishing trust between company, employees and consumers,” says Jacobsz.