Beyond the Basics: AWS WAF's Lesser-Known Limitations
AWS WAF service is an L7 firewall service offered by AWS. It’s easy to set up, seamlessly integrates with other AWS services (ALB, API Gateway, etc.), and comes with a handful of managed WAF rulesets and rate limit features.
While researching AWS WAF and its latest updates, I stumbled across multiple blogs, articles, and YouTube videos talking about the basics of AWS WAF and explaining how good it is. A handful of blogs explained the disadvantages - most of them either have a product that competes with AWS WAF or give high-level information about the disadvantages.
So, I thought I would jot down my experience with AWS WAF (just like I did with Amazon GuardDuty or AWS WAF Bot Control).
TL;DR:
- AWS WAF service can be easily integrated with other AWS services
- Disadvantages include:
- Not so useful overview and dashboard. Users need to deploy a custom dashboard to visualize, get insights and query data.
- Post body inspection limits to first 8 KB. Only Cloudfront distributions can inspect up to 64 KB - which is still shorter compared to other popular commercial WAFs.
- It’s hard to manage WAF rules in multiple accounts without AWS Firewall Manager. This service inturn needs other services like AWS Config enabled - which adds to the cost of WAF deployment.
- Not so flexible rate based rules. Ratelimits are always for 5 minute rolling window. (It has another limitation described in the blog post.)
- Some more nitpicked limitations:
AWSManagedRulesAnonymousIpListmanaged rule doesn’t block AWS IPs- AWS WAF can’t protect you from DDoS
I’ve had the opportunity to run and experiment with AWS WAF on production. I’m talking about an end-to-end implementation of AWS WAF which handled 10k+ requests per minute on customer-facing subdomains.
I’m writing this technical blog to help Cloud Architects and Cloud Security Engineers understand the limitations of AWS WAF and help them make informed decisions when selecting a WAF solution.
Disadvantages of AWS WAF
Not So Useful Overview and Dashboard
Let’s start with the basic requirement of WAF - a dashboard where someone can understand traffic patterns, analyze requests from an attack, debug false positives, etc.
When you create a Web ACL, associate WAF rules, and attach it with AWS resources - you get a minimalistic dashboard containing an overview of traffic, enabled WAF rules, etc. The “Overview” tab, as the name suggests, gives an overview of the requests. You can get a high-level view of the total count of all allowed or blocked requests and the count of requests blocked by individual rules.
This Overview tab is nothing more than a metrics tab.
As you can see from the above image, there’s not much data that can be used right away. It’s hard to answer questions like “Did all the malicious traffic generate from the same IP/ASN?”
AWS folks understand this limitation pretty well. “What’s their solution?”, you ask.
It’s this - Deploy a dashboard for AWS WAF with minimal effort.
The above article helps create a decent WAF dashboard with less effort. But you must also know that the WAF dashboard’s cost (essentially the cost of OpenSearch, Kinesis Data Firehose, etc.) is separate. The cost is somewhat proportional to the amount of traffic you get to resources attached to Web ACL.
Post Body Inspection Limitation
UPDATE - On 8th March 2024, AWS WAF has released support for larger request body inspections. As per the blog post, AWS WAF supports 64 KB of request body. While this is available for services like API Gateway, App Runner and Cloudfront, it’s not available for Application Load Balancers (ALB) and App Sync.
For regional web ACLs, AWS WAF can inspect the first 8 KB of the body of a request. For CloudFront web ACLs, by default, AWS WAF can inspect the first 16 KB, and you can increase this limit in your web ACL configuration to 32 KB, 48 KB, or 64 KB.
This is a well known limitation of AWS WAF and can be bypassed by sending junk data appended with the attack payload to vulnerable endpoints.
This limitation is present with most (if not all) Cloud WAF providers. However, this 8 KB coverage of post body is a bit lesser when compared to them.
If you stick to AWS WAF for some reason and want it to inspect a larger chunk of the post body (up to 64KB), you will need to change your web app architecture by adding CloudFront. In case you do this, you will need to re-evaluate your threat model (cough caching).
AWS also offers post body oversize handling option - which allows you to log/block the request if the post body is over the 8 KB size. You need to be mindful when setting this option to ensure you don’t block genuine business-critical traffic in the name of security.
AWS Firewall Manager - yet another way to spend money
AWS recommends using multiple AWS accounts. It provides a logical separation of resources. In case you have multiple production accounts, maybe based on products, then this limitation is for you.
It’s hard to manage WAF rules for multiple accounts unless you use another service - AWS Firewall Manager. This service helps you centrally configure and manage firewall rules across your accounts and applications in AWS Organizations.
Firewall Manager needs AWS Config enabled - meaning you pay for configuration changes at least for the mentioned resources like CloudFront, ALB, etc.
Oh, did I forget to mention? If you want to manage WAF ACLs across multiple regions, AWS Firewall Manager just doesn’t support it. You will need to create a firewall manager policy in each region where you operate.
Not So Flexible Ratelimit Rules - Has A Major Limitation
UPDATE - On 4th March 2024, AWS WAF has released support for additional time windows in rate-based rules. As per the blog post, customers can now select time windows of 1 minute, 2 minutes or 10 minutes, in addition to the previously supported 5 minutes. AWS WAF also evaluates the rate of requests about every 10 seconds. This has reduced the risk - attackers can send unlimited requests per IP in a 10 second interval before ratelimit kicks in (compared to 30 seconds window earlier).
Based on my experience with WAFs, I have formed the opinion that ratelimit logic is meant for applications; not WAFs. The logic is simple - WAFs don’t have context. It doesn’t understand what a user is, how to differentiate paid users from unpaid users, etc. But your applications can differentiate.
You can make the effort to implement desired ratelimits in WAF, but you will often see it’s impossible to set up complex ratelimit logic for your business.
At the same time, ratelimit feature on WAFs can’t be ignored altogether.
During incidents like bruteforce attacks, product abuse, etc - ratelimit feature plays a major role. If you have lots of legacy code that was deployed without security threat modeling, then the ratelimit feature is a MUST HAVE. You can write a ratelimit rule to reduce the bad traffic to your backend services - giving your engineers time to make changes during incidents.
On AWS WAF, ratelimit intervals are not flexible - it’s always a rolling window of 5 minutes. And the minimum number of requests in the 5-minute interval is 100.
If you are planning to have rate rate-based rule like allowing only < 100 requests per 5 minutes or like 1000 requests in 24 hours, it’s impossible.
The biggest limitation is how AWS WAF counts the number of requests. As per the documentation:
AWS WAF checks the rate of requests every 30 seconds, and counts requests for the prior 5 minutes each time.
This means an attacker can send any number of requests in the 30-second interval before the ratelimit kicks in. In the age of nuclei and turbo intruder, this is a major limitation.
Even if I have a ratelimit rule - 100 requests per 5-minute interval, an attacker could have sent 1000s of requests in the 30-second interval before the ratelimit rule blocked it.
WAF Logs might contain sensitive data
AWS WAF has a feature to sample requests for each WAF rule. With this request sampling feature enabled, you get some requests available under the “Sampled requests” section. It doesn’t give you all the requests, but up to 100 requests that match each rule.
More often than not, you will enable the WAF logging feature to store all requests to get detailed insights about the traffic, attacks, etc., or for other reasons (like regulatory compliance).
AWS WAF logging logs most parts of the request - request method, path, query parameters, and headers. It doesn’t log the POST body.
This means sensitive headers like Authorization, WWW-Authenticate, Cookie, X- headers, etc. are logged. Also, if you have bad engineering practices like sending user or session data on GET query parameters - that will also be logged.
Enabling WAF logs with the default configuration and storing them somewhere like an S3 bucket itself hurts your organization’s security. Any leak of those logs can be a disaster.
One must take extra effort to validate traffic/logs and update Field Redaction so the sensitive values of fields (headers and query params) are redacted in logs.
These are the limitations I have found with AWS WAF after deploying on production. If you are still reading the blog, it means you are serious about understanding the limitations of AWS WAF. I have got some more bonus points (nitpicked limitations) just for you.
Bonus 1: AWSManagedRulesAnonymousIpList managed rule doesn’t block AWS IPs
If you don’t expect user traffic from public cloud providers to your customer-facing subdomains, AWS WAF has a managed rule called “AWSManagedRulesAnonymousIpList”. This rule detects Anonymous IPs (tor nodes, temporary proxies, etc.) and Hosting Provider IPs (cloud provider IPs).
Weirdly this rule doesn’t block traffic from AWS IPs.
Bonus 2: AWS WAF can’t protect you from DDoS
You can have a strictest ratelimit rules and enable most managed rules that block malicious traffic (which contains malicious payloads or originates from bad reputational IPs).
An attacker still has the advantage. This is due to the 30-second ratelimit limitation and abundant sites offering IPs for attack (say open proxies, DDoS as a service, cloud providers like DO 😜, etc.).
An attacker could send huge traffic (1000s req per second) of simple HTTP GET requests to your website in 30-second intervals from different IPs. The traffic will be blocked after a 30-second window for each IP. However, the damage will be done. Your applications would have stopped affecting your application availability to genuine users. When this happens as a DDoS over multiple IPs, then the continuous intermittent bursts of traffic will have taken down your backend.
Final Thoughts
AWS WAF has both advantages and limitations. This blog post has highlighted most limitations I found with the service. This doesn’t stop you from using AWS WAF altogether.
I still recommend AWS WAF in the following situations:
- You are trying to block script-kiddies from running their scripts against your sites
- You don’t plan to use rate-based rules or worry about the 8 KB post body limitation
- You are trying to get a feel of WAFs and block most common web attacks
In the above scenarios, AWS WAF is a very practical solution to start with. You don’t have to make changes to your infra apart from associating the ALBs, CloudFront distributions, etc. to the WAF ACL.
You will need to recheck if AWS WAF is the right solution for you if:
- You are looking for a solution that acts both as WAF and mitigates DDoS
- You need a bit more complex ratelimit rules with different intervals
- Your applications take user input in POST body and 8 KB limitation doesn’t cover most incoming traffic
If you have any doubts/ideas/suggestions, feel free to reach out on LinkedIn.