Real World DevOps

About the Guest

Elliot Murphy is a senior executive and technologist with more than 20 years of success in critical software infrastructure, online services, and healthcare. Elliot is the founder of Kindly Ops, a cybersecurity firm bringing DevOps approaches to Governance Risk & Compliance, serving regulated industries such as biotech and fintech. His interest in Governance Risk & Compliance began when working as CTO of a healthcare startup and realizing the burden of regulatory compliance was slowing life-saving healthcare innovations from being brought to market.

Links Referenced

• Book recommendation: People-Centric Security by Lance Hayden
• Book recommendation: How to Measure Anything in Cybersecurity Risk by Douglas Hubbard and Richard Seiersen
• Book recommendation:How to Measure Anything by Douglas Hubbard

Transcript

Mike Julian: Running infrastructure at scale is hard, it's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast, you're probably also interested in better monitoring tools, and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, Influx DB. But, you may not be as familiar with their other tools: Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all this out at influxdata.com.

Mike Julian: Hi, everyone, my name's Mike Julian. I'm here with Elliot Murphy, the CEO of KindlyOps. Welcome to the show, Elliot.

Elliot Murphy: Hey, nice to be here.

Mike Julian: Why don't you tell us a bit about what you and KindlyOps does?

Elliot Murphy: Yeah, so we're a cybersecurity consulting firm, and we mostly help regulated companies, biotechs, and fintechs with governance, risk, and compliance. We take a DevOps approach, so all of the good practices from that world, and apply them into what's traditionally been a pretty boring and bureaucratic set of activities.

Mike Julian: Okay, so what are these regulations we're talking about?

Elliot Murphy: Some people have heard a thousand times before, HIPAA, PCI, SOC 2. Others are a little newer, GDPR, or more specialized like FISMA. There's regulations from the Food and Drug Administration around medications and therapeutics. Pretty much every country in the world and every industry has their own set of regulations or laws. As software technology expands ever further into different parts of business, even very old businesses that didn't have much technology in the past, there's regulations that people have to comply with.

Mike Julian: Okay, so that makes a lot of sense, but every time I've been in a company with these regulatory requirements, it's really just been a huge pain in the ass. I don't like dealing with it. No one I worked with likes dealing with it. We meet the bare minimum and hope for the best. How do you view these regulations? Do you have that same take on them?

Elliot Murphy: It's funny, a few years ago, I was working as the CTO of a healthcare startup founded by a doctor. She had just some amazing innovations, and I got to see firsthand how expensive and burdensome regulatory compliance was. Then, even more upsetting, as I was meeting people, scientists and researchers at universities, I was hearing all these stories of how they were discouraged or prevented or didn't want to bother bringing actual life-saving innovations to market because of the burden of regulatory compliance. So I'm right there with you that by default they can have a pretty high cost. Our entire mission has been to try and get the value out of those regulations but without the human toil. It's a pretty existential question, why even have rules and laws?

Mike Julian: Right.

Elliot Murphy: I think some rules and laws are good, but they don't have to be as 

miserable as they have been. We really take a human-centric approach.

Mike Julian: Okay. What does that even mean, a human-centric approach to regulation? Regulations are, to me, they're handed down from on high. It's like, "You comply or face the consequences."

Elliot Murphy: Yeah, so it's funny, we've actually developed this four-step process, and it's very, very common when you're talking with people who are excited about building products and are excited about serving patients and customers and are just excited about doing their core job, they're usually pretty annoyed at externally imposed rules that aren't quite obvious how they're beneficial. Depending on people's life experiences, they have these different perspectives. Some people, particularly, if they've been doing risk and compliance work for their whole career, they see the value, and they get frustrated that other people push back on regulations. What does it actually mean to put people first? We start with building empathy for different worldviews. One definition of culture is that it's beliefs and assumptions that drive decisions and behavior. That makes sense. But, then, if you think about that again, what is another word for beliefs and assumptions that you use to make decisions, drive behavior? Those are mental models. That's a very freeing realization that, okay, what we're actually talking about when we say culture, security culture, for example, are mental models. You and I are going to have our favorite mental models, our default mental models. They're mental models we go back to immediately when we're in a high-stress situation. But, we're perfectly capable of learning other worldviews, other mental models and deciding when it's beneficial to apply them. We can try them on for size, and we can say, "Oh, yeah, I can see. Even though this is not my favorite way to think about the problem, I can see there's some utility in this case or that case."

Mike Julian: Yeah.

Elliot Murphy: Dr. Lance Hayden actually developed this fantastic Creative Commons licensed security culture diagnostic. It's called, in his book, People-Centric Security, and so that outlines four different cultures or mental models. There's a trust culture, an autonomy culture, a compliance culture,...

About the Guest

Dr. Sherry Walling is a clinical psychologist, entrepreneur, international speaker, yoga teacher, podcaster and best-selling author. She works with leaders and entrepreneurs around the world to help them tackle the many mental health and relationship challenges that go along with building a great business.

Her best-selling book, The Entrepreneur’s Guide to Keeping Your Sh*t Together, is a handbook for navigating life as an entrepreneur.

Married to a serial entrepreneur, Sherry combines her extensive experience helping people who have high-intensity jobs with her 18 years of personal experience in the trenches of the startup world. Sherry combines the insight and warmth of a therapist with the truth-telling mirth of someone who has been there.

When she’s not in the consulting room or hopping conferences; Sherry can be found on her paddleboard, in the yoga studio, or ushering her three kiddos through an art museum in some fabulous city.

She can also be found at ZenFounder.com, SherryWalling.com or on Twitter as @ZenFounder.

Links
Managing Founder Stress Guide

“How to Stay at the Top of Your Game” talk at BoS USA 2017 

Links Referenced: 

• Book recommendation: Play it Away by Charlie Hoehn
• Book recommendation: The Happiness Advantage by Shawn Achor 
• Sherry’s “How to Stay at the Top of Your Game” talk at BoS USA 2017 
• Managing Founder Stress Guide

Transcript:

Mike Julian: Running infrastructure at scale is hard, it's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools — and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you may not be as familiar with their other tools. Telegraph for metrics collection from systems, coronagraph for visualization and capacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike Julian: Hi Folks, I'm Mike Julian, your hosts for the Real World DevOps podcast. I'm here with Dr. Sherry Walling, clinical psychologist, author and fellow podcaster at ZenFounder.com. Welcome to the show, Sherry.

Sherry Walling: Hey, it's my pleasure Mike. After a couple of reschedules, I'm glad we finally got this to come together.

Mike Julian: It was a bit of work to make that happen, but I'm really excited about this episode. So Sherry, why don't you start off by just telling us a bit about who you are and what you do.

Sherry Walling: Yeah. So like you said, I'm a clinical psychologist and I have spent my professional life working with people who have high intensity jobs. That looks like a couple of different things over the course of my career. Sometimes it's with folks in the military returning from military service. I've worked a lot with physicians, who have high intensity work either in the ER or in surgery. Then I actually work a lot with software entrepreneurs and software folks. My connection to the developer world is largely through my husband, Rob Walling, who maybe some of your audience know of MicroConf and things like that. So I work with really smart people who are trying to do really hard things, and often have a lot of pressure in their work lives.

Mike Julian: Yeah, that's some awesome stuff. The reason I'm especially interested in talking to you is as DevOps engineers, we lead extraordinarily stressful lives. A day in the life of a operations engineer… we’re project driven, and yet it's often interrupt heavy. So we're never really finishing anything, thanks to putting out fires constantly. Just so many fires is all the time, everything is always awful, everything's just a tire fire everywhere we look. So we're responsible for keeping systems of multimillion dollar, sometimes multi billion dollar companies running and available, and on call is a standard part of the job. And sometimes this is even this is really bad, like the on call rotations of some companies might be 15 pages a night for a week or two weeks at a time, and it just gets insane. This is not just a few years or one job, this is like an entire career. So to say that the role of an Ops engineer is stressed, is kind of an understatement. So I'm really excited to talk to you about how can we manage the stress? Is there something we can do about it? How can we improve our lives?

Sherry Walling: Well, I'm really empathizing with the way that you're describing this, and there's a couple of things that you said that would lead me to believe that folks who are doing this kind of work are really at a pretty high risk for burnout. Whenever there's high work demand, so lots of things to do without having appropriately accomplishable goals, goals that have a tight timeline and clear successes — those three things tend to be a combination that really causes burnout in a lot of folks. So I can empathize with the amount of stress that folks are feeling, and certainly guessing that some of your listeners are really struggling at this point with stress related difficulties, or things like burnout.

Mike Julian: Yeah, absolutely. So why don't we talk, why don't we just start with burnout? There's so many different ways that I want to tackle this conversation, but why don't we just start with burnout? What is burnout?

Sherry Walling: Yeah, so burnout is a newly recognized clinical syndrome. It has its own diagnostic code. It is a real thing.

Mike Julian: That's fantastic. It's about time.

Sherry Walling: It is about time. It exists in ICD 10, which is the International Classification of Disorders 10. It's called something like “burnout estate of vital exhaustion,” is the technical title which I think is really lovely language that paints the picture of what this feels like. Vital exhaustion. So burnout is something that was ...

About the Guest

Dave Mangot is the author of Mastering DevOps from Packt Publishing. He’s formerly the head of Site Reliability Engineering (SRE) for the SolarWinds Cloud companies and an accomplished systems engineer with over 20 years' experience. He has held positions in various organizations, from small startups to multinational corporations such as Cable & Wireless and Salesforce, from systems administrator to architect. He has led transformations at multiple companies in operational maturity and in a deeper adherence to DevOps thinking. He enjoys time spent as a mentor, speaker, and student to so many talented members of the community.

Guest Links

• Twitter: @davemangot
• Mastering DevOps course by Dave Mangot

Links Referenced: 

• Familiar Smells I’ve Detected in Your Ops Organization and How to Fix Them by Dave Mangot (LISA 2018)
• Knight Capital story
• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation book by Jez Humble and David Farley
• Book recommendation: How to Measure Anything by Douglas W. Hubbard
• Calvin and Hobbes comic on Bridge Capacity
• Dan McKinley - Choose Boring Technology
• Nicole Forsgren’s DevOps research
• Book recommendation: The Lean Startup by Eric Ries

Transcript

Mike Julian: Running infrastructure at scale is hard, it's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly and author of O’Reilly's Practical Monitoring.

Mike Julian: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you may not be as familiar with their other tools. Telegraph for metrics collection from systems, coronagraph for visualization and capacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike Julian: Hey folks, this is Mike Julian, I'm here with Dave Mangot, former head of SRE for SolarWinds Cloud. Welcome to the show, Dave.

Dave Mangot: Thanks Mike. It's great to be here.

Mike Julian: So why don't you tell us a bit about yourself and what you've been up to lately.

Dave Mangot: Sure. Um, I've recently, like you said, left SolarWinds Cloud. There I was running the global SRE organization. We started that with about two people at Librato and wind up growing that into multiple teams in multiple locations with certainly lots of products you've heard of. Before that, I was an architect and technical operations for Salesforce, working on their internal infrastructure, monitoring, automation, configuration management, all kinds of fun stuff like that.

Dave Mangot: I've been able to take a bunch of the experience of those two companies plus lots of years of experience doing stuff before that into a pretty fun conference talk that I gave a few weeks ago in Nashville at the USENIX LISA ‘18 Conference. And I was talking about familiar smells I've detected in your systems engineering organization and how to fix them. And so it was really fun to be able to use a lot of the things that I've seen in my career and all the things I've seen as we try to mature engineering organizations, both operationally and all kinds of other fun DevOps ways, and be able to give a talk about that. And pretty, I guess you could say, controversial at times, but well accepted at other times sort of venue.

Mike Julian: It's always interesting when you start pointing out the smells and people's infrastructure, and their reactions are kind of all over the place. Like some people like, “Yes, you're absolutely right.” And other times it's like, “No, no, please don't say that. Like, I'm not cool with you pointing out my flaws.”

Dave Mangot: Yeah, I think the important thing is, you know, what I was trying to get across in the talk as a message is like nobody's perfect, we all have to get better. If I've seen those things in other organizations, there's a chance that I was involved in those things. It's not like I just kind of wandered in from Mars and I knew everything and everything was all good, like I've been growing throughout my career, and I try to continue to keep growing throughout my career. And if people feel a little bit uncomfortable, I think that's good, right? That's a sign of growth, if we're feeling uncomfortable. It's just, what do we do with that? If we get defensive, then we could double down on something that maybe we know in deep inside isn't really working for us as opposed to maybe we go home and sleep on it and wake up the next day and say like, "Well maybe there's a couple of things here that we could actually get some incremental improvement on." And that's ultimately how we make all this successful changes to our infrastructure. 

Mike Julian: I've had people come find me where they've taken my place in their previous job, and they'll be like, "Why did you do this really dumb thing?" And my answer is, "Yeah, it was really dumb. I'm sorry about that." Like that that was a technical smell that I created, and I'm really sorry.

Dave Mangot: Yeah.

Mike Julian: These are things that we did, like they're not necessarily ... we didn't do them because they're good ideas, we did them for all sorts of reasons.

Dave Mangot: Yeah. And also I think that's important with like John [Allspaw 00:04:53] things that it's easy to come back later and say, why did you do that? That was a bad idea. But we make the best decisions that we can with the information that we have at the time.

Mike Julian: Absolutely.

Dave Mangot: It turns out that was a bad idea.

Mike Julian: Yup. So this talk that you gave at LISA, sadly I couldn't make it to LISA to see it live, but you and I have been talking...

About the Guest

Corey is a Cloud Economist at the Quinn Advisory Group. He has a history as an engineering director, public speaker, and cloud architect. Corey specializes in helping companies address horrifying AWS bills, hosts the Screaming in the Cloud and curates LastWeekinAWS.com, a weekly newsletter summarizing the latest in AWS news, blogs, and tips, sprinkled with snark.

• Corey’s newsletter: Last Week in AWS
• Corey’s professional site: quinnadvisory.com
• Corey’s podcast, Screaming In the Cloud
• Corey’s Twitter

Links Referenced: 

• The Story of a Serverless Startup by Sam Kroonenburg
• Corey’s talk, Myth of Cloud Agnosticism
• Dan McKinley (@mcfunley)’s Choose Boring Technology

Transcript

Mike: Running infrastructure at scale is hard, it's messy, and it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're gonna talk about the rough edges. We're gonna talk about what it's really like running infrastructure at scale.

Mike: Welcome to the Real World DevOps podcast. I'm your host, Mike Jillian, editor and analyst for Monitoring Weekly, and author of O'Reilly’s Practical Monitoring.

Mike: This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their Time Series Database InfluxDB, but you may not be as familiar with their other tools. Telegraf for Metrics Collection from systems, Chronograf for visualization, and Kapacitor for Real-Time Streaming. All of this is available as open-source. They also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike: Hi folks, welcome to the Real World DevOps podcast. I'm here with Corey Quinn the editor of Last Week in AWS. Welcome to the show, Corey.

Corey: Thanks, Mike. It's always a pleasure to hear myself talking.

Mike: I'm sure it is. So for those who don't know, Corey is one of my closest friends. So, this might get a little off the wall and banter-y. But hopefully everyone will enjoy it. So for those who don't have the pleasure of having met Corey yet, Corey, what is it that you do?

Corey: A lot of things is probably the best and most honest response to that. But what I'm best know for is either shit-posting on Twitter and/or writing Last Week in AWS, which is a newsletter that gathers information from the AWS ecosystem every week, discards the crap that no one cares about, takes what's left, and then makes fun of it.

Mike: It is pretty fucking funny. So by day, you are also an AWS consultant?

Corey: Yes, but in a more directed sense. Specifically, I start and I stop professionally at fixing the horrifying AWS bill. It's one of those areas whereas a consultant, I find that I'm more effective when I'm direct. And so I'll aim at a very specific, very expensive problem.

Mike: Absolutely. So you and I were talking a while back and this is something I have repeatedly come into around isn't Amazon just a single point of failure in my infrastructure? Shouldn't I really be focused on trying to mitigate the failure of AWS going down? What happens if you us-east-1 explodes again? Like, my website's offline, and now I have huge problems. So shouldn't I at that point maybe start thinking about multi-region or maybe multi-provider or any number of other really dumb ideas?

Corey: The answer to all of that is generally it depends, which is accurate and completely useless. The fact of the matter is is depending on what your business is and what your constraints look like, you're the best person to wind up saying that this either an unacceptable risk or ultimately this is something that you absolutely need to address and focus on. The example here is if your application has people's' lives depending on it, then yeah, you need to be able to withstand everything up to and possibly including a nuclear event. Whereas if you're running my side project of “Twitter For Pets,” if your site is down for four hours because AWS region as an issue, maybe it's okay. Maybe the internet is better for it.

Mike: Yeah, that's a good point. It always seemed to me to be an incredible amount of over-engineering, of people trying to get their applications and infrastructure to be completely ... completely capable of surviving basically everything. It's like, "Hey, wait a second. You run a very small social media application that no one's going to care about."

Corey: We saw a fair bit of this a couple years back with the S3 Apocalypse wound up hitting. This is not me trying to bag on Amazon in any meaningful way. They run an incredible amount of complexity at a stupendous scale that boggles the mind. Things break. It's what they do. That's the nature of how working with computers plays out. And there was a knee jerk reaction that we saw from a lot of infrastructure types after this happened where they immediately want to turn on replication for every S3 Bucket, so it's now multi-region in one or more locations. I understand the reflex reaction to that. There's no one better than Ops people at fighting the last war. But there also needs to be a rationed, measured response to this.

Corey: One area that seems to get lost along the way somewhere is you're going to be doubling or tripling in some cases your infrastructure costs from a raw perspective. Plus, the additional complexity, plus people's time to set all of that up, plus data transfer to get things from one place to another. Is this a reasonable response for your business constraints or is it a knee jerk reaction to something that is very unlikely to ever occur again? If we look back, we don't see a litany of individual websites that were called out for things breaking. It was called out as Amazon's failure and things like Instagram were mentioned or American Airlines or Amazon's own status page as examples of things that were impacted. But that was the day the internet broke for a little bit. So, most of us just kind of shrugged and went outside. And eight hours later, things are working again and life went on.

Corey: If you're an ad tech company and you need to be able to sustain that type of outage, and maybe it makes sense to do that. People are not going to come back and look at an ad again. But conversely, I was buying, I think, a pair of socks I want to say five years ago on Amazon. It threw a 500 error and suddenly I'm staring at a dog, which is fascinating. That's amazing. To hear the way some infrastructure people talk about outages, that would ...

About the Guest

Steve Mushero is CEO of Ops Platform provider Siglos.io, and CEO of ChinaNetCloud, China's first Internet Managed Service Provider, AWS Partner, and manager of hundreds of large-scale (up to hundreds of millions of users each) systems. He's previously been CTO in a variety of organizations in Silicon Valley, Seattle, New York, and around the world. You can follow along with his work and insights on LinkedIn, Medium and Twitter.

Links Referenced

• ChinaNetCloud
• Siglos
• Taking Over & Managing Large Messy Systems (LISA 2018)
• Incidents as we Imagine Them Versus How They Actually Are by John Allspaw (PagerDuty Summit 2018)
• Brandon Gregg’s USE Model
• Steve’s tool, runqstat
• Observability vs. Monitoring, is it about Active vs. Passive or Dev vs. Ops? By Steve Mushero
• Dao of Troubleshooting
• How to Monitor the SRE Golden Signals

Transcript

Mike: Running infrastructure at scale is hard. It's messy. It's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we're going to talk about the rough edges. We're going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps Podcast. I'm your host, Mike Julian, editor and analyst for Monitoring Weekly, and author of O'Reilly's Practical Monitoring.

Mike: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools, and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you might not be as familiar with their other tools, Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out at influxdata.com.

Mike: Hi, folks. My name is Mike Julian. I'm here with Steve Mushero, CEO of ChinaNetCloud, and his new startup, Siglos. Welcome to the show, Steve.

Steve: Thank you. Glad to be here.

Mike: You and I met through Monitoring Weekly some time ago, and we've just been chatting since then, and learning all about, or me learning, primarily, about the wild world of systems in China. I believe the company you run, ChinaNetCloud is one of the largest, if not the largest Chinese AWS partner?

Steve: Probably. We're the first one. We were their first sort of operations partner here in China, and also their first MSP, or Managed Service Provider. We've been AWS here and around the world almost 10 years. Yeah, it's been an interesting time.

Mike: Yeah, I'm sure. Why don't you tell us a bit more about what your company does? What are you doing in the Chinese market?

Steve: Here in China, we're basically a managed service provider, so we started 10 years ago now, really managing systems. I was CTO for Tudou, similar to YouTube in the US, so online video and so on. Actually, older and bigger than YouTube. We saw how that market really needed operations help so we really launched this to start helping people design, build, operate, manage, monitor, all that sort of stuff, internet systems. This was before the clouds and all that. Then, we were the first cloud company in China actually doing online VMs. You could buy actual virtual servers, now, of course you have AWS, and Aliyun, and others doing that. Today, our job is really just design, build, manage, monitor, troubleshoot, and do 7-by-24 support for large-scale internet systems, so we're way in the backend, you might say, but really sort of ditch-digging operations guys and girls, really dealing with the just 24-hour systems in clouds, and hybrids, and all the interesting problems you can have. Especially with Chinese scale that we do. That's what we do.

Mike: I think it's absolutely wild that there is ... To think about the size of YouTube, like, pretty much everyone everyone in the western world thinks about YouTube, it's like, oh, wow, that's absolutely huge. To think that there's a competitor in China that's bigger than that, the scale is staggering.

Steve: Yeah. I think now YouTube is larger, but back, this is 10 years ago, now-

Mike: Oh, okay.

Steve: ... It was different, because YouTube, even now in the US, think of it more like Netflix and YouTube combined, because YouTube historically has been short video clips, right? 10 years ago, it was, hey, my cat is cute, stuff like this, you know, 90-seconds, three-minutes, five-minutes. We were running full length movies and TV, and all that, not entirely legally, you might say. It was more like Netflix.

Steve: We had a 100 million viewers a day doing sort of full length stuff, so YouTube's average thing was whatever, you know, a couple minutes, ours was an hour. Completely different infrastructure needs. This is before, especially China had good infrastructure, and BGP routing, and all kinds of stuff, so you had sort of massive CDM problems, and lots of other stuff. I think we were the world's biggest bandwidth buyer for a while.

Mike: Oh, wow.

Steve: I know. Now, everybody else is bigger, but this is, again, a long time ago.

Mike: Oh, sure.

Steve: Because, DSL had just hit China, and everybody wanted to watch Western movies and TV, and a lot of people a lot of times had nothing else to do during the day, and so they're watching this everywhere. Before smartphones too. Yes, it was quite the interesting- Actually, that company's older than YouTube, so even though a lot of things come into China are sort of copies of Western things, those companies actually had much more features, and much richer than YouTube and even Netflix today ... 10 years ago. More advanced, innovative, actually, than the Americans at this point.

Mike: Right.

Steve: It was pretty cool. I learned a lot about infrastructure here, and the craziness of data centers, and large scale systems, you know, thousands of servers, and all kinds of locations, all this kind of stuff. It's all physical servers, no virtualization, no nothing, right? This is started in 2005, and I was here in 2007. It's hundreds of racks of equipment, and all that kind of stuff.

Mike: What's it like today? Thinking back to what I was doing for systems in 2007 ....

About the Guest

Silvia is a Principal Engineer at SendGrid, a cloud email provider with household name clients like eBay, Spotify, Pandora and Airbnb. She is an avid distributed systems and databases tester and spends a lot of her day trolling her Ops team. You can hear more from Silvia on her blog https://blog.dbsmasher.com and on Twitter at @dbsmasher

Key Takeaways

• Involve your DBAs in the engineering process early on to prevent problems in the future.
• Enabling engineering teams to manage their own schema is critical to reducing day-to-day toil of the DBA team and empowering Engineering.
• If you’re wary of automating your databases, set up guard rails to add some safety to it first.

Links Referenced

• SendGrid
• Andy Jassy’s 2018 Keynote at AWS re:Invent
• Stories From the DBA Trenches - Silvia Botros (Velocity London 2018)
• Database Reliability Engineering by Laine Campbell and Charity Majors (affiliate link)
• SendGrid blog
• State of DevOps Report by DORA
• Tool: slowquerydigest (pt-query-digest)
• Tool: Anemometer
• Tool: ProxySQL
• Tool: Vitess
• Auditing Databases at The Grid by Silvia Botros

Transcript

Mike: Running infrastructure at scale is hard. It's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we are going to talk about the rough edges. We are going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor of the Monitoring Weekly Newsletter, author of O’Reilly’s Practical Monitoring, and a DevOps consultant/analyst.

Mike: Hi everyone, I'm Mike Julian, I'm here with Silvia Botros, principle DBA at SendGrid. Welcome to the show, Silvia.

Silvia: Thank you for having me.

Mike: You work at SendGrid. There's a lot of people that have no idea what SendGrid is. Monitoring Weekly is actually a customer as well. I use it for all of my contact forms. It's wonderful. But for those who don't know, what does SendGrid do?

Silvia: First, thank you for being a customer.

Mike: Absolutely.

Silvia: I like to hear that. What we do is basically an email service in the cloud. A lot of companies realized years ago that they need as part of their engagement with their customers whether it's onboarding new customers or communication with existing users, that they need to send a lot of email. It turns out that sending email is a complicated thing. There's a lot of rules around it. SPF, whitelisting IPs, dedicated IP or not. Block lists that the inbox providers maintain. There's magics of times depending on the provider, whether email looks fine or it looks spam-y, or it looks fishy. Turns out there's a lot of logistics around that, that these companies all across the board don't really have the skills to deal with and don't want to spend resources on.

Silvia: Customers of ours include a large swath of companies large and small. Some of our big high profile customers include eBay, Spotify, Uber, Pandora, so you get the gist. You sign up for an account, you need a confirmation email. We have to click a link. They want that to come to the inbox, not to the spam folder and they want it to go to the inbox fast. That is basically the service we provide to them. We also now, as you mentioned have your contact list, we also have a marketing email, a marketing campaign's product that we provide to feed into this and provide the more advanced, the more geared towards marketing and that crowd. Not just the day-to-day transaction stuff.

Mike: Yep. Yeah, it is a surprising huge pain in the ass to actually manage all that stuff. I know several companies that just use their corporate email accounts to do it and surprise, surprise everything goes to spam.

Silvia: Yes. I will fully admit I did not know this was a thing, that this was a viable business model before I got my job at SendGrid. But once I started learning about all the details it dawned on me, how much it is ... Why would you build that yourself? If you are eBay, and you're focusing on bids or getting traffic or if you're Pandora and focusing on getting people to listen to playlists, why would you worry about spam lists and how every individual inbox provider decides how to judge your emails?

Mike: Yeah. That's a rabbit hole that seemingly never ends.

Silvia: True.

Mike: You're a DBA for them which is kind of weird. Those are still a thing? I thought when we all started rubbing DevOps on everything, that DBAs just kind of went away. We didn't really need them anymore because now the developers are handling it.

Silvia: I know, like we had thought DevOps were going to eliminate the sysadmin didn't we?

Mike: Right and it turns out we still need those, so I suppose we probably still need the DBA. What does a DBA actually do? What's your role now? What does it actually look like?

Silvia: A DBA in a company that is honestly trying to apply DevOps is practices, which is whole other rabbit hole we can go into. A DBA in this environment needs to be involved in setting standards of practice, that's helping engineers debug things early on. Helping design. A lot of what I do is early cycle, early life cycle of software where they're still building a new thing and I get to give the input on whether this thing has the potential of growing really fast and what does that provide as far as implications as to the data store. Is the data store they're using appropriate for the use case? A lot of guidance, a lot of things like that. There's also still a lot of day to day things in production. For example, we are SOC2 compliant and we are now public company so we also have to do cover Sarbanes-Oxley. There's a lot of compliance needs that don't necessarily fall into any feature work but still have to be covered because every year we have odd pairs, we have fiduciary requirements that we have to cover and so the database ops team, which is what we call my team right now, is responsible for doing the work that provides the evidence that we actually are compliant with all the things that we have to be compliant with.

Mike: That's kind of interesting. What compliance stuff actually surrounds the database? I know GDPR probably has a big component in here doesn't it?

Silvia: It does. That was definitely a big thing and it covers not just relational databases, it covers a lot of data sources so in practice it involves more than just the DB ops team. To provide context, currently my team is solely focused on relational databases so we have a pretty large MySQL footprint and that's where our focus is.

Mike: Yeah, every time I think about DBA's, I think about just relational databases but once I started thinking about that more and about you and I chatting, I realized...