Last Week In AWS Podcast

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/the-trials-and-travails-of-aws-sso/

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links Referenced:

• CanaryTokens: https://www.canarytokens.org/
• Found a solid way to avoid that sneaky method: https://blog.thinkst.com/2022/02/a-safety-net-for-aws-canarytokens.html?m=1
• The folks at Orca found a vulnerability around OCI’s handling of Server Side Request Forgery (SSRF) Metadata: https://orca.security/resources/blog/Oracle-server-side-request-forgery-ssrf-attack-metadata/
• S3 Bucket Negligence Award: https://techcrunch.com/2022/02/08/ottawa-trucker-freedom-convoy-exposed-donation/
• Only 22% of enterprise customers: https://therecord.media/microsoft-says-mfa-adoption-remains-low-only-22-among-enterprise-customers/
• Modified their hypervisor: https://www.bleepingcomputer.com/news/security/google-cloud-hypervisor-modified-to-detect-cryptominers-without-agents/
• Amazon CloudTrail: https://aws.amazon.com/cloudtrail/
• Amazon API Gateway CORS Configurator: https://cors.serverlessland.com/

Transcript

Corey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.

Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They’ve also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That’s S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.

Corey: So, last week was fairly tame and—no. I’m not going to say that because the last time I said that, all hell broke loose with Log4J and I can’t go through that again.

So, let’s see what happened last week in AWS Security. I like this one very much. Thinkst Canary provides, for free via CanaryTokens.org, an AWS credential generator that spits out IAM credentials with no permissions. The single thing they do is scream bloody murder if someone attempts to use them because those credentials have been stolen. There are some sneaky ways to avoid having the testing of those tokens show up in CloudTrail logs, but they’ve just found a solid way to avoid that sneaky method. It’s worth digging into.

I’ve been a fan of Oracle Cloud for a while, which has attracted some small amount of controversy. I stand by my opinion. That said, there’s been some debate over whether they’re a viable cloud provider at scale. There are certain things I look for as indicators that a cloud provider is a serious contender, and one of them has just been reached: the folks at Orca found a vulnerability around OCI’s handling of Server Side Request Forgery (SSRF) Metadata. It sounds like I’m kidding here, but I’m not. When third-party researchers find a vulnerability that is non-obvious to most of us, that’s an indication that real companies are using services built on top of the platform. Onward.

A donation site raising funds for the Ottawa truckers’ convoy nonsense that’s been going on scored itself an S3 Bucket Negligence Award. No matter how much I may dislike an organization or its policies, I maintain that cybersecurity needs to be available to all.

Corey: You know the drill: you’re just barely falling asleep and you’re jolted awake by an emergency page. That’s right, it’s your night on call, and this is the bad kind of Call of Duty. The good news is, is that you’ve got New Relic, so you can quickly run down the incident checklist and find the problem. You have an errors inbox that tells you that Lambdas are good, RUM is good, but something’s up in APM. So, you click the error and find the deployment marker where it all began. Dig deeper, there’s another set of errors. What is it? Of course, it’s Kubernetes, starting after an update. You ask that team to roll back and bam, problem solved. That’s the value of combining 16 different monitoring products into a single platform: you can pinpoint issues down to the line of code quickly. That’s why the Dev and Ops teams at DoorDash, GitHub, Epic Games, and more than 14,000 other companies use New Relic. The next late-night call is just waiting to happen, so get New Relic before it starts. And you can get access to the whole New Relic platform at 100 gigabytes of data free, forever, with no credit card. Visit newrelic.com/morningbrief that’s newrelic.com/morningbrief.

I knew MFA adoption was struggling among consumers, but I was stunned by Microsoft’s statement that only 22% of enterprise customers have adopted an additional security factor. Please, if you haven’t enabled MFA in your important accounts—and yes, your cloud provider is one of those—please go ahead and do it now.

An interesting security advancement over in the land of Google Cloud, they’ve modified their hypervisor to detect cryptocurrency mining without needing an agent inside of the VM. This beats my usual method of ‘looking for instances with lots of CPU usage because most of the time the fleet is bored.’

Over in AWS-land, they didn’t have anything particularly noteworthy that came out last week for security, so I want to talk a little bit about a service that gets too little love: Amazon CloudTrail. Think of this as an audit log for all of the management events that happen in your AWS account. You’re going to want to secure where the logs live, ideally in another account for your AWS organization. To AWS’s credit, they made the first management trail free a few years ago and enabled it across all accounts by default as a result. This is going to help someone out there, I suspect. Remember, if you haven’t heard about it before, it’s new to you.

And I found a fun tool that’s just transformative because if the bully who beat you up and stole your lunch money in middle school were a technology, they would undoubtedly be CORS, or ‘Cross-Origin Resource Sharing.’ The

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/are-aws-account-ids-sensitive-information/

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• CodeBuild to exfiltrate data from an AWS VPC: https://awsteele.com/blog/2022/02/03/aws-vpc-data-exfiltration-using-codebuild.html
• Thousands of Open Databases: https://InfoSecwriteups.com/how-i-discovered-thousands-of-open-databases-on-aws-764729aa7f32
• “Why do Amazon S3 Data Breaches Keep Happening?”: https://markn.ca/2022/why-do-amazon-s3-data-breaches-keep-happening/
• You’re going to be placed on a public list of shame: https://Twitter.com/0xdabbad00/status/1489305680490106880?s=12
• How to report security issues in other people’s software: https://Twitter.com/notdurson/status/1489350457730469888
• S3 Bucket Negligence Award: https://www.zdnet.com/article/unsecured-aws-server-exposed-airport-employee-records-3tb-in-data/
• “Security Practices in AWS Multi-Tenant SaaS Environments”: https://aws.amazon.com/blogs/security/security-practices-in-aws-multi-tenant-saas-environments/
• Stratus Red Team: https://github.com/Datadog/stratus-red-team

Transcript

Corey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.

Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They’ve also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That’s S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.

Corey: Hello there. Another week, another erosion of the perception of AWS’s hard security boundaries. I don’t like what 2022 is doing to my opinion of AWS’s security track record. Let’s get into it.

We start this week with a rather disturbing post from Aidan Steele, who talks about using CodeBuild to exfiltrate data from an AWS VPC. We’re increasingly seeing increased VPC complexity, which in turn means that most of us don’t have a full understanding of where the security boundaries and guarantees lie.

Someone decided to scan a bunch of public AWS IP ranges and lo and behold, an awful lot of us suck at security. Specifically, they found Thousands of Open Databases. This is clearly not an exclusively AWS problem seeing as how it falls fairly on the customer side of the Shared Responsibility Model, but it does have the potential to be interpreted otherwise by folks with a less nuanced understanding.

Mark Nunnikhoven has a blog post up that asks the question “Why do Amazon S3 Data Breaches Keep Happening?” I’ve often wondered the same thing. The vector has been known for years, the console screams at you if you attempt to configure things this way, and at this point, there’s really little excuse for a customer making these mistakes. And yet they keep happening.

Scott Piper has had enough. He’s issued a simple warning: If you’re a vendor who offers a solution that deploys EC2 instances to customer environments, and you don’t support IMDSv2, you’re going to be placed on a public list of shame. He’s right: His first shame example is AWS themselves with a new feature release. For those who aren’t aware of what IMDSv2 is, it’s the instance metadata service. Ideally, you have to authenticate against that thing before just grabbing data off of it. This is partially how Capital One wound up getting smacked a couple years back.

Corey: You know the drill: You’re just barely falling asleep and you’re jolted awake by an emergency page. That’s right, it’s your night on call, and this is the bad kind of Call of Duty. The good news is, is that you’ve got New Relic, so you can quickly run down the incident checklist and find the problem. You have an errors inbox that tells you that Lambdas are good, RUM is good, but something’s up in APM. So, you click the error and find the deployment marker where it all began. Dig deeper, there’s another set of errors. What is it? Of course, it’s Kubernetes, starting after an update. You ask that team to roll back and bam, problem solved. That’s the value of combining 16 different monitoring products into a single platform: You can pinpoint issues down to the line of code quickly. That’s why the Dev and Ops teams at DoorDash, GitHub, Epic Games, and more than 14,000 other companies use New Relic. The next late-night call is just waiting to happen, so get New Relic before it starts. And you can get access to the whole New Relic platform at 100 gigabytes of data free, forever, with no credit card. Visit newrelic.com/morningbrief that’s newrelic.com/morningbrief.

Corey: AWS’s Dan Urson has a thread on how to report security issues in other people’s software. Something about it’s been nagging at me, and I think I’ve figured out what it is. Ignore the stuff about, “Have a coherent report,” and, “Demonstrate a reproduction case;” it gets into following the vendor’s procedures and whatnot around disclosure. I think it has to do with where I’m coming from. I generally don’t find security problems, or other bugs, by actively exploiting vendor systems; instead, I trip over them as a customer trying to get something done. The idea that I owe that vendor much of anything when I’m in that position rankles a bit. I get that this is a nuanced topic.

And of course, 3TB of airport employee records were exposed in this week’s S3 Bucket Negligence Award. I hate to sound like I’m overly naive here, but what exactly is in the employee records that makes them take up that much space? I’m a big believer in not storing information you don’t need, and that just seems like an enormous pile of data to have lying around awaiting compromise.

AWS themselves had an interesting post go out: “Security Practices in AWS Multi-Te...

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/guardduty-for-eks-and-why-security-should-be-free

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Three vulnerabilities: https://blog.wiz.io/black-hat-2021-aws-cross-account-vulnerabilities-how-isolated-is-your-cloud-environment/
• Embarrassingly long time: https://Twitter.com/christophetd/status/1486610249045925890
• “Companies Leave Vast Amounts of Sensitive Data Unprotected”: https://www.propublica.org/article/identity-theft-surged-during-the-pandemic-heres-where-a-lot-of-the-stolen-data-came-from?token=pIt-Qx8lrKMcPei_lM3rFDQpHXkkcxXQ
• Google Drive started mistakenly flagging files as infringing copyright: https://www.theregister.com/2022/01/25/google_drive_copyright_infringement/
• “How to deploy AWS Network Firewall to help protect your network from malware”: https://aws.amazon.com/blogs/security/how-to-deploy-aws-network-firewall-to-help-protect-your-network-from-malware/
• “How to use tokenization to improve data security and reduce audit scope”: https://aws.amazon.com/blogs/security/how-to-use-tokenization-to-improve-data-security-and-reduce-audit-scope/
• “Ransomware-resistant backups with S3”: https://www.franzoni.eu/ransomware-resistant-backups/

Transcript

Corey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.

Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment. They’ve also gone deep in-depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That’s S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense.

After the content for this episode was effectively laid out, AWS did a late Friday night announcement of a new GuardDuty enhancement that would automatically opt people in to a chargeable service unless they explicitly opted each account out. This obviously doesn’t thrill me or other affected customers. so, as I record this, the situation is still evolving, but rest assured I’m going to have further thoughts on this next week.

Now, let’s see what happened last week in AWS security. so, last year, Wiz found three vulnerabilities that allowed attackers to read or write into other customers’ AWS accounts. This flew beneath the radar at the time, but they’re all coming out of the woodwork now, and AWS’s security reputation, more or less, lies in tatters, replaced by a reputation for clamming up and admitting nothing. I’m already wincing at this summer’s re:Inforce keynote. if they try their usual messaging line, it’s not going to end well for them.

There was apparently a serious vulnerability within the Linux polkit library. It took Amazon Linux an embarrassingly long time to acknowledge it and put out a release. Now, I’m not a fan of single-vendor Linux installs; any bets on how many non-Amazonians have commit rights to the distribution?

Failing to learn from experience is never a great look, but as per ProPublica, “Companies Leave Vast Amounts of Sensitive Data Unprotected” despite decades of breaches. Please, please, please, if you’re listening to this, don’t be one of them. There’s no value in buying the latest whiz-bang vendor software to defend against state-level actors if you’re going to leave the S3 bucket containing the backups open to the world.

And an uncomfortable reminder that we might not be the only parties perusing our “private” files stored within various cloud providers, Google Drive started mistakenly flagging files as infringing copyright. Now, amusingly the files in question tended to consist entirely of a single character within the file, but the reminder isn’t usually something that cloud providers want dangled in front of us. Once again we are, in fact, reminded that Google considers privacy to be keeping information between you and Google.

Corey: You know the drill: you’re just barely falling asleep and you’re jolted awake by an emergency page. That’s right, it’s your night on call, and this is the bad kind of Call of Duty. The good news is, is that you’ve got New Relic, so you can quickly run down the incident checklist and find the problem. You have an errors inbox that tells you that Lambdas are good, RUM is good, but something’s up in APM. So, you click the error and find the deployment marker where it all began. Dig deeper, there’s another set of errors. What is it? Of course, it’s Kubernetes, starting after an update. You ask that team to roll back and bam, problem solved. That’s the value of combining 16 different monitoring products into a single platform: you can pinpoint issues down to the line of code quickly. That’s why the Dev and Ops teams at DoorDash, GitHub, Epic Games, and more than 14,000 other companies use New Relic. The next late-night call is just waiting to happen, so get New Relic before it starts. And you can get access to the whole New Relic platform at 100 gigabytes of data free, forever, with no credit card. Visit newrelic.com/morningbrief that’s newrelic.com/morningbrief.

AWS had a couple interesting blog posts. One of them was “How to deploy AWS Network Firewall to help protect your network from malware”. and I’m torn on this service, to be honest, because On the one hand, it extends the already annoying pricing model of the Managed NAT Gateway, but On the other, it provides a lot more than simple address translation and is cost-competitive with a number of 
other solutions in this space. I think I’m going to land on, “use it if it makes sense for you, but don’t expect it to be cheap.”

And a great blog post from AWS security folks—which is, honestly, something I have said a lot in the past, and I look forward to saying a lot more of in the future—

Want to give your ears a break and read this as an article? You’re looking for this link.

https://www.lastweekinaws.com/blog/going-out-to-play-with-the-cdk

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill