Beyond S3: Exposed Resources on AWS
Almost every week a new story comes out about a freshly discovered S3 bucket on Amazon Web Services (AWS) accidentally exposing sensitive information. S3 buckets are private by default when they are created, but for various reasons, AWS users will make them public, perhaps mistakenly assuming that they won't be found. S3 buckets are not the only service on AWS that can be accidentally exposed, however. In this blog post, we'll explore other services on AWS that can be made public, and provide stats on what has been found.
Some resources are made public on purpose. For example, S3 buckets, which are just collections of files, can be made public as part of the content for a public website. AWS enables resources to be made public because sometimes people want that data to be public. It is difficult to determine if there was a genuine desire to expose a resource publicly, so for the purpose of this post, we'll just point out stats for resources that have been exposed, without attempting to determine if this was on purpose.
Summary of Findings
To help understand the magnitude of this problem, the following summarizes how many of each resource were found exposed to the internet.
- 116,386 public Elastic Block Store (EBS) snapshots from 3,213 accounts
- 373 public Relational Database Service (RDS) snapshots from 227 accounts
- 711,598 public Amazon Machine Images (AMIs) from 20,952 accounts
- 16,000 public IPs of exposed AWS managed ElasticSearch clusters that could have their contents stolen or data possibly deleted - this means 17% of AWS-managed ElasticSearch servers with public IPs were misconfigured
For more details on what we found and how to check your own AWS configuration, keep reading.
Snapshots and AMIs
There are three resource types on AWS that have commands that let you list all public resources of that type. These are Elastic Block Store (EBS) snapshots, Relational Database Service (RDS) snapshots, and Amazon Machine Images (AMIs). Unlike S3 buckets where attackers try to brute-force the names of the resource in order to find them, AWS provides commands for these resources to list all of the public resources for all AWS accounts.
For each region, you can find all of the public EBS snapshots of any AWS account by running the following command. The other resources have similar commands.
aws ec2 describe-snapshots --no-paginate
EBS Snapshots
Virtual machines that run on AWS are called EC2 instances. These often use what is called Elastic Block Store (EBS) volumes as their virtual hard-drives. In order to make backups of these EC2s, you can take EBS snapshots. Once you've made a backup of something, you often want to move it outside of the blast radius of potential problems, or simply share it somewhere else so it can be used elsewhere. There are secure ways that this can be done, but sometimes AWS users have elected to simply make them public to more easily get access to them from somewhere else.
We decided to take a look at how many public EBS instances are exposed to the internet and found the following:
- 116,386 public EBS snapshots from 3,213 accounts
- 97%* have less than 10 public EBS volumes
- 79% have just one public EBS volume
*All percentages are percent of accounts with at least one of the resources in question made public. So 97% means 97% of 3213 accounts, not of all AWS accounts.
As mentioned, sometimes there are reasons for resources to be made public. For example, Ubuntu makes a number of EBS volumes public for people to use. Ubuntu's account (099720109477) is responsible for 53,636 of the public EBS snapshots, or roughly half of them.
Checking Your Account
You can identify any public EBS snapshots in your account by running the following command in each region:
aws ec2 describe-snapshots --owner-id self --restorable-by-user-ids all --no-paginate
RDS Snapshots
Relational Database Service (RDS) snapshots are the database version of EBS snapshots. We found:
- 373 public RDS snapshots from 227 accounts
- 99% have less than 10 public RDS snapshots
- 89% have just one public RDS snapshot
Checking Your Account
You can identify any public RDS snapshots in your account by running the following command in each region:
aws rds describe-db-snapshots --snapshot-type public --no-paginate
Amazon Machine Images (AMIs)
Amazon Machine Images (AMIs) are similar to EBS snapshots, but are used to start new EC2s. As such, these are more commonly shared on purpose. A couple of accounts, including Ubuntu again, are sharing over 100,000 public AMIs. You can see in the log scale graph below that although some accounts are sharing over a hundred public AMIs, there is a long tail of accounts with less than 10 AMIs publicly shared.
Chart of public AMI counts by account with logarithmic vertical scale
- 711,598 public AMIs from 20,952 accounts
- 90% have less than 10 public AMIs
- 51% have just one public AMI
Checking Your Account
You can identify any public AMIs in your account by running the following command in each region:
aws ec2 describe-images --owners self --executable-users all
Services That Support Resource Policies
S3 buckets can be made public because they support resource-based policies to allow them to be exposed. The services that support resource policies can be found on the page AWS Services That Work with IAM.
This page contains a number of tables with a list of services on the left, and a column for "resource-based policies" to identify the services of interest to us.
Unlike the previous resources, these services do not have commands to list all public instances. They need to be guessed or scanned for.
S3 Buckets
S3 buckets can be guessed because their names are user chosen. For example, if you name your bucket "backups," an attacker only needs to guess that name and try to access http://backups.s3.amazonaws.com. Similarly, if the bucket was named "files", the attacker would try to visit files.s3.amazonaws.com. In both those cases, the access is denied, but in the case of a bucket named "flaws.cloud," you can see that it is world-readable by visiting http://flaws.cloud.s3.amazonaws.com. In addition to being public to anyone, buckets can also be made public only to other AWS users, so an attacker has to log into an AWS account and then make similar requests.
Glacier
Glacier is similar to S3, as it is used for data storage, but is used for long-term backups that will rarely be accessed. Like S3, Glacier vaults can be made public. In fact, the docs describe how to do do this in a few places, for example:
Example documentation on making Glacier vaults public
From an attacker perspective, they need to know both the account ID and the vault name. So unlike S3 buckets where an attacker can just guess "files," "backups," etc., an attacker needs to try both every AWS account ID along with those terms.
ElasticSearch
AWS provides a service for managed ElasticSearch clusters. ElasticSearch is commonly used to store logs and text documents. Technologies such as ElasticSearch, MongoDB, Memcached and Redis have appeared periodically in the news when researchers have discovered tens of thousands of instances publicly exposed. Some of these are hosted on AWS, but many are on various other hosts.
At the start of last year, attackers were finding public databases and encrypting (or more often simply deleting) the data in them and posting a ransom note. Much of that research and what was found by others were for ElasticSearch clusters hosted on the default ElasticSearch port 9200.
AWS-managed clusters are hosted on port 443, which makes them blend in with other HTTPS servers. Their host name is a concatenation of a user-specified name along with an AWS generated random string, such as mydomain-6w5y8xjt5ydwsrubmdk4m5kcpa.us-west-2.es.amazonaws.com. That random string is equivalent to 2**169 bits of entropy, and may have led some customers into a mistaken sense of security that their instances could not be found if they were made public. However, using scanning tools like Shodan, and a few other details about these services, Duo discovered it was possible to identify all of the AWS-managed ElasticSearch clusters that had public IPs.
We discovered 95K public IPs associated with AWS-managed ElasticSearch clusters. Many of these are locked down in some way and will return an HTTP code of 403, meaning Forbidden, but by filtering on HTTP 200 codes, we found 16K public IPs of exposed AWS managed ElasticSearch clusters that could have their contents stolen or possibly data deleted. This means 17% of AWS-managed ElasticSearch servers with public IPs were misconfigured.
Duo also discovered that even for the AWS-managed ElasticSearch clusters that had not been made public, their index names could be learned.
Disclosure Timeline
We contacted the AWS security team and provided them with all of our data and intent to publish this blog post. AWS asked that we coordinate with them so that they can notify impacted customers and resolve issues before we make them public.
These issues include fixing the information disclosure of the index names for private AWS-managed ElasticSearch clusters, enforcing host header authentication to make the public clusters more difficult to access, and AWS has taken steps to reach out to customers with public ElasticSearch clusters.
The following outlines important disclosure dates:
- 01/30/2018 - Initial contact with AWS Security.
- 02/07/2018 - Response from AWS Security. Phone call requested.
- 02/08/2018 - Phone conversation with AWS and Duo teams. Duo agrees to delay blog post publication while AWS team contacts potentially impacted customers and investigates other potential changes.
- 03/08/2018 - Update provided by AWS team.
- 03/14/2018 - Draft blog post provided to AWS team.
- 04/06/2018 - Update provided by AWS team.
- 04/25/2018 - Requested update from AWS team.
- 05/08/2018 - Feedback provided by AWS team. AWS team is ready for us to publish.
- 05/15/2018 - Blog post published by Duo.
Fixing Your Elasticsearch Configuration
AWS users may have misconfigured their clusters by granting access to any IP, by attempting to use a policy similar to the "IP-based Policies" shown on Amazon Elasticsearch Service Access Control, but without an IP restriction. This screenshot from the AWS documentation shows the correct way to restrict an ElasticSearch cluster to an IP.
A better solution is to use identity-based policies instead of IP-based policies to avoid potential risks of an adversary getting access to a system within your IP range and abusing the trust relationship it would have. The best solution is to use the capabilities AWS announced on October 17, 2017 to run the AWS-managed ElasticSearch clusters within your own VPC such that they are only accessible there.
Checking Your Account
Looking for exposed ElasticSearch clusters and the many other services are more difficult to do without specialized tools. One such tool that can be used for this is Netflix's Security Monkey: https://github.com/Netflix/security_monkey
Conclusion
We hope this post helps you identify some of the other potential mistakes that can be made on AWS and helps you avoid them.
We also want to thank the AWS security team for fixing the issues we reported to them around ElasticSearch and for taking our concerns about the other public AWS resources into consideration in reaching out to customers that had exposed RDS snapshots and other resources.
We hope that this post provides security benefit to you and your organization. If you’re interested in the intersection between security and running a highly-available service on AWS, please contact Duo's Production Engineering team at prodeng-ext@duo.com.