Last Week In AWS Podcast

Links:

• “Tokyo police lose 2 floppy disks containing personal info on 38 public housing applicants”: https://mainichi.jp/english/articles/20211227/p2a/00m/0na/072000c
• LastPass may have suffered a breach: https://news.ycombinator.com/item?id=29705957
• “Worst AWS Data Breaches of 2021”: https://securityboulevard.com/2021/12/worst-aws-data-breaches-of-2021/
• D.W. Morgan: https://www.hackread.com/logistics-giant-d-w-morgan-exposed-clients-data/
• SEGA Europe: https://vpnoverview.com/news/sega-europe-suffers-major-security-breach/
• “Identity Guide–Preventive controls with AWS Identity–SCPs”: https://aws.amazon.com/blogs/mt/identity-guide-preventive-controls-with-aws-identity-scps/
• Log4j scanner: https://github.com/google/log4jscanner

Transcript

Corey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.

Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.

Corey: The first security round-up of the year in Last Week in AWS: Security. This is relatively light, just because it covers the last week of the year, where people didn’t really “Work” so much as “Get into fights on Twitter.” Onward.

So, from the community, ever see a data breach announcement that raises oh so very many more questions than it answers? I swear this headline is from a week or so ago, not 1998: “Tokyo police lose 2 floppy disks containing personal info on 38 public housing applicants”. Yes, I said floppy disks.

The terrible orange website, also known as Hacker News, reports that LastPass may have suffered a breach. At the time I write this, the official LastPass blog has a, “No, it’s just people reusing passwords.” Enough people I trust have seen this behavior that I’d be astounded if that were true. If you can’t trust your password manager, ditch them immediately.

Security Boulevard had a roundup of the “Worst AWS Data Breaches of 2021”, and it’s the usual run-of-the-mill S3 bucket problems, but my personal favorite’s the Twitch breach because it’s particularly embarrassing, given that it is, in fact, an Amazon subsidiary.

First one goes to D.W. Morgan by leaking 100GB of client data. And they’re a logistics company that serves giant enterprises, so these are companies with zero sense of humor, so I would not want to be in D.W. Morgan’s position this week.

And the other is a little funnier. It goes to SEGA Europe, after Sonic the Hedgehog forgets to perform due diligence on his AWS environment.

Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That’s goteleport.com.

AWS had only a single thing that I found interesting: “Identity Guide–Preventive controls with AWS Identity–SCPs”. I’ve been waiting for a while for a good explainer on SCPs to come out for a while, and this looks like it actually is a thing that I want. I’ve been playing around with SCPs a lot more for the past couple of weeks. If you’re unfamiliar, it’s a way to override what the root user can do in an organization’s member accounts. It’s super handy to constrain people from doing things that are otherwise foolhardy.

And lastly, an interesting tool came out from Google—which I should not have to explain what that is to you folks; they turn things off, like Reader—they also released a log4j scanner. This one scans files on disk to detect the bad versions of log4j—which is most of them—and can replace them with the good version—which is, of course, print statements. And that’s what happened last week in AWS security. Hopefully next week will be… well, I don’t want to say less contentful, but I do want to say it’s at least not as exciting as the last month has been. Thanks for listening.

Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Please follow AWS Morning Brief on Apple Podcast, Spotify, Overcast—or wherever the hell it is you find the dulcet tones of my voice—and be sure to sign
up for the Last Week in AWS newsletter at lastweekinaws.com.

Announcer: This has been a HumblePod production. Stay humble.

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/the-aws-service-i-hate-the-most

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• “Cloud Security Breaches and Vulnerabilities”: https://blog.christophetd.fr/cloud-security-breaches-and-vulnerabilities-2021-in-review/
• S3 Bucket Negligence Award: https://mytechdecisions.com/audio/sennheiser-responds-after-customer-data-from-2018-was-exposed-online/
• Granted the role its support teams use to access customer accounts access to S3 objects: https://Twitter.com/0xdabbad00/status/1473448889948598275?s=12
• S3 Bucket Negligence Award: https://www.modernghana.com/news/1127205/report-ghana-government-agency-exposes-100000s.html
• “Simplify setup of Amazon Detective with AWS Organizations”: https://aws.amazon.com/blogs/security/simplify-setup-of-amazon-detective-with-aws-organizations/
• “AWSSupportServiceRolePolicy Informational Update”: https://aws.amazon.com/security/security-bulletins/AWS-2021-007/
• aws-sso-cli: https://github.com/synfinatic/aws-sso-cli

Transcript

Corey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.

Corey: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That’s goteleport.com.

Corey: Well, we’re certainly ending 2021 with a whirlwind in the security space. Log4J continues to haunt us, while AWS took not only an 
outage but also a bit of a security blunder that they managed to turn into a messaging win. Listen on.

But first, the Community. A depressing review of 2021’s “Cloud Security Breaches and Vulnerabilities.” Honestly, it seems like there are just so damned many ways for bad security to set the things we care about on fire. The takeaways are actionable though. Stop using static long-lived credentials and start with the basics before you get fancy.

Sennheiser scores itself an S3 Bucket Negligence Award, and of all the countries in which to suffer a data breach, I’ve got to say that Germany is at the bottom of the list. They do not mess around with data protection there.

And, Holy hell, AWS inadvertently granted the role its support teams use to access customer accounts access to S3 objects. It lasted for ten hours, and while there are mitigations out there, this is far from the first time that AWS has biffed it with regard to an unreviewed change making it into a managed IAM policy. This needs to be addressed. If you’ve got specific questions about how those things are handled, reach out to your account team; but it’s a terrible look. But there’s more to come in a second here.

Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special for you folks: If you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is, good news, they’ve opened up their Black Friday promotion for a very limited time. Same deal: $100 off a yearly plan, 249 bucks a year for the highest quality cloud and tech skills content. Nobody else is going to get this, and you have to act now because they have assured me this is not going to last for much longer. Go to cloudacademy.com, hit the ‘Start Free Trial’ button on the homepage and use the promo code, ‘CLOUD’ when checking out. That’s C-L-O-U-D. Like loud—what I am—with a C in front of it. They’ve got a free trial, too, so you’ll get seven days to try it out to make sure it really is a good fit. You’ve got nothing to lose except your ignorance about cloud. My thanks to Cloud Academy once again for sponsoring my ridiculous nonsense.

A bit off the beaten path, this week’s S3 Bucket Negligence Award goes to the government of Ghana. This one is pretty bad. I mean, you can’t exactly opt out of doing business with your government, you know?

Now, AWS has two things I want to talk about. The first is that they offer a way to “Simplify setup of Amazon Detective with AWS Organizations.” I’m actually enthusiastic about this one because there’s a significant lack of security tooling available to folks at the lower end of the market. A bunch of companies seem to start off targeting this segment, but soon realize that there’s a better future in selling things to bigger companies for $200,000 a month instead of $20.

Now, “AWSSupportServiceRolePolicy Informational Update.” Now, you heard a minute ago, I was initially extremely unhappy about this mistake. That said, I am such a fan of this notification that I can’t even articulate it without sounding like I’m fanboying. Because mistakes happen and talking about those mistakes and why defense in depth mitigates the harm of those mistakes goes a long way. This affirms my trust in AWS rather than harming it. Meanwhile Azure has absolutely nothing to say about why their tenant separation is aspirational at best.

And lastly a bit of tooling story here. To end up the year, I’ve been kicking the tires on aws-sso-cli over on GitHub, which is a tool for using AWS SSO for both the CLI and web console. I don’t know why the native SSO tooling is quite as trash as it is, but it’s a problem. There’s a lot of value to using SSO but AWS hides it as if the entire thing were under NDA. Thank you for listening. It’s been a heck of a year as we’ve launched the security portion of this weekly nonsense. I’ll talk to you more in 2022. Stay safe.

Corey: Thank you f...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/last-year-in-aws 

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• Has its own vulnerability that’s actively under exploit: https://arstechnica.com/information-technology/2021/12/patch-fixing-critical-log4j-0-day-has-its-own-vulnerability-thats-under-exploit/
• Google Project Zero deep dive into the NSO group’s iMessage exploit: https://googleprojectzero.blogspot.com/2021/12/a-deep-dive-into-nso-zero-click.html
• Three flaws: https://thehackernews.com/2021/12/hackers-begin-exploiting-second-log4j.html
• How to customize behavior of AWS Managed Rules for WAF: https://aws.amazon.com/blogs/security/how-to-customize-behavior-of-aws-managed-rules-for-aws-waf/
• Using AWS security services to protect against, detect, and respond to the Log4j vulnerability: https://aws.amazon.com/blogs/security/using-aws-security-services-to-protect-against-detect-and-respond-to-the-log4j-vulnerability/
• Update for Apache Log4j2 Issue: https://aws.amazon.com/security/security-bulletins/AWS-2021-006/
• An innocent question: https://Twitter.com/QuinnyPig/status/1473382549535662082?s=20

Transcript

Corey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.

Announcer: Are you building cloud applications with a distributed team? Check out Teleport, an open-source identity-aware access proxy for cloud resources. Teleport provides secure access for anything running somewhere behind NAT: SSH servers, Kubernetes clusters, internal web apps, and databases. Teleport gives engineers superpowers. Get access to everything via single sign-on with multi-factor, list and see all of SSH servers, Kubernetes clusters, or databases available to you in one place, and get instant access to them using tools you already have. Teleport ensures best security practices like role-based access, preventing data exfiltration, providing visibility, and ensuring compliance. And best of all, Teleport is open-source and a pleasure to use. Download Teleport at goteleport.com. That’s goteleport.com.

Corey: The burning yule log that is the log4j exploit and its downstream issues continues to burn fiercely. Meanwhile the year winds down, and it’s certainly been an eventful one. I’ll talk to you next week because that is what I do.

Now, let’s see from the community what happened. The patch to fix the log4j vulnerability apparently has its own vulnerability that’s actively under exploit. Find your nearest InfoSec friend and buy them a beer or forty because this is going to suck for a long time and basically ruin everyone’s holiday.

Also, I’ve seen the most hair-raising thing I can remember in InfoSec-land, which is the Google Project Zero deep dive into the NSO group’s iMessage exploit. Seriously, this thing requires no clicks on the part of the victim, the exploit uses a bug in the GIF processing inherent to iMessage to build a virtual CPU and assembly instruction set. There is no realistic defense against this short of hurling your phone into the sea, which I heartily recommend at this point as a best practice.

Oh, and everything is on fire and somehow worse. There are now at least three flaws in the log4j library that we’re counting, so far. Everything is terrible and we clearly should never log anything again.

Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special for you folks: If you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is, good news, they’ve opened up their Black Friday promotion for a very limited time. Same deal: $100 off a yearly plan, 249 bucks a year for the highest quality cloud and tech skills content. Nobody else is going to get this, and you have to act now because they have assured me this is not going to last for much longer. Go to cloudacademy.com, hit the ‘Start Free Trial’ button on the homepage and use the promo code, ‘CLOUD’ when checking out. That’s C-L-O-U-D. Like loud—what I am—with a C in front of it. They’ve got a free trial, too, so you’ll get seven days to try it out to make sure it really is a good fit. You’ve got nothing to lose except your ignorance about cloud. My thanks to Cloud Academy once again for sponsoring my ridiculous nonsense.

Now, AWS had a few things to say. The most relevant of them are How to customize behavior of AWS Managed Rules for WAF. So, if you’re a 
WAF vendor and you don’t link to this blog post as part of your, “Why should I pay you?” sales material, you’re missing a golden opportunity. Every time I dig into AWS’s Web Application Firewall offering, I end up regretting it, and with a headache.

There was also a post on Using AWS security services to protect against, detect, and respond to the Log4j vulnerability. I’m disappointed to see AWS starting to use the log4nonsense stuff to pitch a dizzying array of expensive security services that require customers to do an awful lot of independent work to get stuff configured properly. This kind of isn’t the time for that.

And they have an update page that they continue to update called Update for Apache Log4j2 Issue, and this post has more frequent updates than AWS’s “What’s new” RSS feed. It really drives home the sheer scope of the issue, how pervasive it is, and just how much empathy we should have for the AWS security team. Their job has pretty clearly been not fun for the last couple of weeks.

And lastly, the tip of the week is more of a request for help, honestly. I asked what I thought was an innocent question on Twitter: “What are people using to read and consume CloudTrail logs?” The answers made it clear that the answer was basically, “A bunch of very expensive enterprise grade things,” or, “Nothing.” This feels like a missed opportunity for some enterprising company out there. If you’ve got a better a...

Want to give your ears a break and read this as an article? You’re looking for this link. https://www.lastweekinaws.com/blog/overstating-awss-free-tier-generosity 

Never miss an episode

• Join the Last Week in AWS newsletter
• Subscribe wherever you get your podcasts

Help the show

• Leave a review
• Share your feedback
• Subscribe wherever you get your podcasts

What's Corey up to?

• Follow Corey on Twitter (@quinnypig)
• See our recent work at the Duckbill Group
• Apply to work with Corey and the Duckbill Group to help lower your AWS bill

Links:

• The internet is now on fire:https://www.engadget.com/log4shell-vulnerability-log4j-155543990.html
• Blog post:https://blog.cloudflare.com/exploitation-of-cve-2021-44228-before-public-disclosure-and-evolution-of-waf-evasion-patterns/
• Expecting to be down for weeks:https://www.darkreading.com/attacks-breaches/kronos-suffers-ransomware-attack-expects-full-restoration-to-take-weeks-
• Update for the Apache Log4j2 Issue:https://aws.amazon.com/security/security-bulletins/AWS-2021-006/
• Log4Shell Vulnerability Tester at log4shell.huntress.com:https://log4shell.huntress.com/

Transcript

Corey: This is the AWS Morning Brief: Security Edition. AWS is fond of saying security is job zero. That means it’s nobody in particular’s job, which means it falls to the rest of us. Just the news you need to know, none of the fluff.

Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key or a shared admin account isn’t going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open-source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport’s unique approach is not only more secure, it also improves developer productivity. To learn more, visit goteleport.com. And no, that’s not me telling you to go away; it is, goteleport.com.

Corey: I think I owe the entire internet a massive apology. See, last week I titled the episode, “A Somehow Quiet Security Week.” This is the equivalent of climbing to the top of a mountain peak during a violent thunderstorm, then waving around a long metal rod. While cursing God.

So, long story short, the internet is now on fire due to a vulnerability in the log4j open-source logging library. Effectively, if you can get an arbitrary string into the logs of a system that uses a vulnerable version of the log4j library, it will make outbound network requests. It can potentially run arbitrary code.

The impact is massive and this one’s going to be with us for years. WAF is a partial solution, but the only real answer is to patch to an updated version, or change a bunch of config options, or disallow affected systems from making outbound connections. Further, due to how thoroughly embedded in basically everything it is—like S3; more on that in a bit—a whole raft of software you run may very well be using this without your knowledge. This is, to be clear, freaking wild. I am deeply sorry for taunting fate last week. The rest of this issue of course talks entirely about this one enormous concern.

Corey: This episode is sponsored in part by my friends at Cloud Academy. Something special for you folks: if you missed their offer on Black Friday or Cyber Monday or whatever day of the week doing sales it is, good news, they’ve opened up their Black Friday promotion for a very limited time. Same deal: $100 off a yearly plan, 249 bucks a year for the highest quality cloud and tech skills content. Nobody else is going to get this, and you have to act now because they have assured me this is not going to last for much longer. Go to cloudacademy.com, hit the ‘Start Free Trial’ button on the homepage and use the promo code, ‘CLOUD’ when checking out. That’s C-L-O-U-D. Like loud—what I am—with a C in front of it. They’ve got a free trial, too, so you’ll get seven days to try it out to make sure it really is a good fit. You’ve got nothing to lose except your ignorance about cloud. My thanks to Cloud Academy once again for sponsoring my ridiculous nonsense.

Cloudflare has a blog post talking about the timeline of what they see as a global observer of exploitation attempts of this nonsense. They’re automatically shooting it down for all of their customers and users—to be clear, if you’re not paying for a service you are not its customer, you’re a marketing expense—and they’re doing this as part of the standard service they provide. Meanwhile AWS’s WAF has added the ruleset to its AWSManagedRulesKnownBadInputsRuleSet—all one word—managed rules—wait a minute; they named it that? Oh, AWS. You sad, ridiculous service-naming cloud. But yeah, you have to enable AWS WAF, for which there is effectively no free tier, and configure this rule to get its protection, as I read AWS’s original update. I’m sometimes asked why I use CloudFlare as my CDN instead of AWS’s offerings. Well, now you know.

Also, Kronos, an HR services firm, won the ransomware timing lottery. They’re expecting to be down for weeks, but due to the log4shell—which is what they’re calling this exploit: The log4shell problem—absolutely nobody is paying attention to companies that are having ransomware problems or data breaches. Good job, Kronos.

Now, what did AWS have to say? Well, they have an ongoing “Update for the Apache Log4j2 Issue” and they’ve been updating it as they go. But at the time of this recording, AWS is a Java shop, to my understanding.

That means that basically everything internet-facing at AWS—which is, you know, more or less everything they sell—has some risk exposure to this vulnerability. And AWS has moved with a speed that can only be described as astonishing, and mitigated this on their managed services in a timeline I wouldn’t have previously believed possible given the scope and scale here. This is the best possible argument to make for using higher-level managed services instead of building your own things on top of EC2. I just hope they’re classy enough not to use that as a marketing talking point.

And for the tool of the week, the Log4Shell Vulnerability Tester at log4shell.huntress.com automatically generates a string and then lets you know when that is exploited by this vulnerability what systems are connecting to is. Don’t misuse it obviously, but it’s great for validating whether a certain code path in your environment is vulnerable. And that’s what happened last week in AWS Security, and I just want to say again how deeply, deeply sorry I am for taunting fate and making everyone’s year suck. I’ll talk to you next week, if I live.

Corey: Thank you for listening to the AWS Morning Brief: Security Edition with the latest in AWS security that actually matters. Ple...