If you’re into web security, you have hopefully heard of SecLists. It’s an amazing repository of keywords, indicators, payloads, passwords and more. It’s great not just for SecOps, but also developers and QA who want to step up their security game.
As part of a project I’m working on, I wanted to be able to quickly compare strings in the Discovery/Web_Content files against logs I have regularly synched to AWS S3 (specifically, ELB logs for my SaaS platform). In order to find interesting data in those logs, I’ve already created Athena tables, so I just need a new table for this content. So I wrote a quick script that fetches the SecLists repo, copies it up to S3, then generates an Athena table.
This gist shows how to make the whole repo searchable, but it’s worth noting that there are README files and other content in there you don’t want to query (including GIFs and other binaries). So it’s a good idea to restrict your queries to subfolders using the $path metavariable, or CREATE your table using that subfolder in the LOCATION path. (For example, since I’m only interested in web content, I gave that full path in my CREATE TABLE statement.)
What’s rad about this is that (a) it’s searchable using standard SQL, (b) I can compare strings to other data files using Athena, and (c) I only incur access/query charges when I run my queries, rather than having an always-on database instance.
I’m kind of surprised I couldn’t easily find something like this elsewhere. After all the recent news about unsecured (or very poorly secured) AWS S3 buckets, I wanted to find a quick and easy way of checking my own buckets. Between the several AWS accounts I manage, there are hundreds.
AWS sent out an email to account owners listing unsecured buckets a while back. Read more about it from A Cloud Guru, where they also discuss how to secure your buckets. But that doesn’t necessarily help with quick auditing. AWS provides some tools like Inspector to help find issues, but setting it up can take some time (though it’s totally worthwhile in the long run). I’m impatient, and I want to know stuff right now.
My solution was to write a quick script that scans my buckets for glaring issues. Namely, I want to know if any of my buckets have the READ permission set for “everyone” or “any AWS account”. If READ is allowed for “everyone” – anyone can list or download files in that bucket. If it’s allowed for “any AWS account”, a trivial barrier is set – a user just has to have an AWS account to review your bucket contents.
So here’s my script.
It requires the AWS CLI and jq, which is an awesome utility, and can be downloaded here. It’ll check top-level bucket ACLs for public-read settings, and just alert you to those bucket names. From there, I’ll leave it to you to secure your buckets.
If you just want to take the nuclear option and update your buckets to private-only, you can do that with this AWS CLI command: