GuardDuty - the Good, the Bad and the Ugly
If you listen to anyone discussing AWS security, you probably heard about Amazon GuardDuty. It’s an intelligent “threat detection” service from AWS. It’s similar to an IDS system because it detects issues but doesn’t prevent them.
TL;DR:
- The Good
- Easy to setup
- It covers important services and detects a good number of issues
- No need to explicitly setup data sources
- Seamless integration with other AWS services
- No extra infra provisioning
- Supports custom trusted IP lists and threat lists
- The Bad
- Not a lot of options (hence not very flexible)
- It’s not foolproof and there are a few bypasses for some checks
- The Ugly
- It’s cost can get sky-high
- Should you enable it? Yes and No (check the conclusion 😉)
GuardDuty
If you ask me to describe GuardDuty’s detection capabilities in one word - I would say it’s “amazing.” It detects a variety of issues - from S3 policy allowing public read access, EC2 instances communicating with Tor nodes, and even exfiltration of data over DNS.
It detects these issues using different techniques and data sources. It’s documentation mentions:
GuardDuty combines machine learning, anomaly detection, network monitoring, and malicious file discovery, utilizing both AWS-developed and industry-leading third-party sources to help protect workloads and data on AWS. GuardDuty is capable of analyzing tens of billions of events across multiple AWS data sources…
Now, you ask me the important question: Should I enable GuardDuty?
I hope you’ll reach closer to your answer by the end of this blog post. 😉
NOTE: This blog post is based on my experience with GuardDuty on AWS accounts (enabled for almost a year now). S3 Protection got enabled by default. I HAVEN’T experimented with Kubernetes Protection and Malware Protection yet.
The Good
Easy to setup
GuardDuty is a regional service and is recommended to be enabled in every active region. Enabling it is just a few clicks away.
If you have multiple accounts under AWS Organization, then enabling it across accounts needs some additional clicks. Even if you have 100 accounts, enabling it across all doesn’t take more than 10 minutes for a single region. Also, you get the option to enable/disable certain features for each account from your AWS Org management or delegated admin account.
Unlike Amazon Inspector or AWS Systems Manager, you don’t need an agent running on EC2 instances to detect issues like malware and traffic to crypto mining servers.
Covers important services and detects a good number of issues
At the time of writing this blog post, GuardDuty detects issues/anomalies in the following services:
- Amazon IAM
- Amazon S3
- Amazon EC2 (and EBS)
- Amazon EKS
These services, in general, are used to store and process business-critical data. All of them (except EKS) are used by most AWS customers. GuardDuty has a good number of findings for these services.
No need to explicitly setup data sources
GuardDuty uses different data sources to find different types of issues. The data sources include VPC flow logs, DNS query logs, CloudTrail logs, and more. You can check the full list of data sources in the official documentation.
Once GuardDuty is enabled, these logs are generated and analyzed on the AWS side.
Note: These data sources will not be accessible to you on your AWS account. Let’s say entities on your VPC does thousands of DNS queries. You will get an alert if a DNS query relates to crypto mining servers. However, you can’t see the other DNS queries.
Seamless integration with other AWS services
If you are already invested in AWS Security Hub for CSPM, then you don’t need to make any extra configurations. Any issues GuardDuty detects are sent to Security Hub.
You can also use other AWS services to automate the actions based on the finding. Let’s say a lambda that disables an IAM user if GuardDuty raises MaliciousIPCaller finding.
No extra infra provisioning
Let’s say you don’t want to use Amazon GuardDuty and want to build it on your own. Good. Let’s give it a try.
Some findings of GuardDuty are easy to implement. Like BucketAnonymousAccessGranted and RootCredentialUsage. They are just static event-based findings. Just tap into CloudTrail management events using EventBridge and trigger a Lambda function depending on the event. These lambda functions check if the recently modified bucket allows public access, whether the root user logged in to AWS Console, etc.
The next category of findings requires a bit of computation and querying. For example: Portscan, SSHBruteForce and RDPBruteForce. These findings require you to enable data sources (like VPC flow logs), configure their storage (like sending logs to S3 in text/parquet format) and then do a continuous/periodic check for patterns (maybe using Amazon Athena).
The final category of findings requires a lot of effort to set up and get it working. Most anomalous behavior and exfiltration findings fall under this category. You will need to fetch all data sources and create a data lake. Then create and finetune ML models to find valid issues while keeping the false positive rate down and the false negative rate near zero.
*author sighs deeply*
If you successfully did it - Congrats!
You have reinvented GuardDuty. You now have greater control and understanding of its functionalities.
What was the cost of this setup? Thousands of man-hours, uncountable trials and errors, large infra setup, and a huge AWS bill.
Now, compare it with enabling GuardDuty with a few clicks.
Zero infra provisioning. Logs get generated and analyzed behind the scenes. ML models and threat intel data are regularly updated behind the scenes.
Supports custom trusted IP lists and threat lists
GuardDuty has the option to whitelist/blacklist certain IPs using trusted IP lists and threat lists. Findings for these lists are generated if the IPs were found in VPC Flow logs and CloudTrail logs.
You can add a trusted IP list to make sure no GuardDuty alerts are generated for those IPs. One common whitelisting use case is with your network scanners running on EC2 instances. You can have only one trusted IP list per AWS account per region.
Threat lists are the exact opposite of a trusted IP list. These lists contain IP addresses of threat actors, malware CNC servers, and more. If there’s any communication to these IPs, a finding gets generated. You can add up to 6 threat lists per AWS account per region.
The Bad
Not a lot of options
GuardDuty options don’t allow flexibility. You have few options, and that’s it. In some cases, it’s just yes or no.
There’s no option to:
- add a DNS-based threat list
- enable S3 protection on a subset of buckets containing sensitive data
- reset the baseline for GuardDuty machine learning models
It’s good from a beginner’s standpoint - fewer options mean lesser chances of insecure (mis)configuration.
It’s bad from an experienced user standpoint. I don’t have flexibility even if I know what I’m doing.
It’s not foolproof
Its detection capabilities are great, but that doesn’t mean it can’t be bypassed.
There are a few public (and active) bypasses:
- AWS GuardDuty Exfiltration Bypass with VPC Endpoints - describes how UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS can be bypassed using VPC endpoints
- Evading AWS GuardDuty and Network Firewall using Privacy Enhancing tech - Dhruv AHUJA - describes how GuardDuty exfiltration findings can be bypassed using public DNS resolvers.
The Ugly
Cost can get sky-high
GuardDuty’s pricing looks innocent at the first glance. The CloudTrail logs, S3 Data event logs, and EKS audit logs are priced per one million events per month. The VPC flow logs, DNS query logs, and EBS data volume are priced per GB per month.
In oversimplified terms, the price of GuardDuty is proportional to the number of resources and the logs they generate.
If you read the statement carefully, you will understand the devil in the details. 😈
You can always know the number of resources in your account while enabling GuardDuty. The mysterious part is the number of events/logs it generates. If you haven’t explicitly set up VPC flow logs, S3 access logs, etc you cannot estimate the amount of logs generated.
GuardDuty’s cost estimation page doesn’t give a simple calculator to estimate costs. Instead, it recommends enabling the GuardDuty 30-day trial and checking the average daily cost. But keep in mind that the estimate is just for the existing resources and logs generated. It need not be the same one month, six months, or one year down the line.
If the resources and/or the amount of logs generated increase, GuardDuty price can go sky-high.
I learned this the hard way after enabling GuardDuty with S3 protection on a data engineering account.
In that particular AWS account, S3 buckets were used as data lake storage (as per AWS recommendation). These buckets had a lot of data engineering tasks going on. Like AWS Glue ETL output saved to certain prefixes, adhoc Athena queries over the data, and daily aggregation queries from BI tools (using Athena connectors).
In 6 months, the business grew, the data generated and analyzed grew, and so did the GuardDuty pricing.
This graph represents the cost of GuardDuty in a single AWS account over 6 months. More than 85% of the cost was because of S3 data events.
Conclusion
AWS GuardDuty is an amazing service that intelligently detects different threats across important AWS services. It’s very easy to set up on single and multi-account environments doesn’t need any infra provisioning or existing infra changes, and integrates seamlessly with other AWS services.
While it’s great on some fronts, it has its fair share of issues - not very flexible, not foolproof, and can cost a bomb if the amount of logs generated or the number of resources increase over time.
GuardDuty (without S3 protection) has a huge bang for your buck. Adding S3 protection to the equation, in general, will have good coverage but comes with a cost.
Enough explanation so far. Now the important question - should you enable GuardDuty?
Here’s my recommendation:
Enable GuardDuty in all active regions across all accounts. Disable access to services in all non-active regions using SCPs. Enabling GuardDuty enables S3 protection by default. Disable the S3 protection in accounts that doesn’t contain business-critical data (like nonprod accounts). If you find S3 protection costs a lot in prod accounts during GuardDuty trial, disable S3 Protection and try to solve the issue from a different angle - maybe stringent S3 access policies, additional protection to sensitive S3 buckets, and more.
Please feel free to reach out to me on Twitter or LinkedIn if you have any queries. Follow me to get more insights on AWS security and other experiments.
If you have experimented with GuardDuty’s EKS and malware protection, I’d love to discuss it.