By Mike Julian
5
44 ratings
The podcast currently has 26 episodes available.
About Ian Sherman
Ian Sherman is Head of Software at Formant, a company building cloud infrastructure for robotics. Prior to Formant, Ian led engineering teams at Google X and Bot & Dolly. The through line of his career has been tool building, for engineers and artists alike. He’s inspired by interdisciplinary collaboration of all types; currently this takes the form of applying patterns and practices from distributed systems operations to the relatively nascent field of robotics.
Links
https://formant.io
Transcript
Mike: This is the Real World DevOps Podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing the awesome work in the world of DevOps, from the creators of your favorite tools to the organizers of amazing conferences, from the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find. This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in.
Personally I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database, Influx DB, but you may not be as familiar with their other tools. Telegraf for metrics collection from systems, Chronograf for visualization and Kapacitor for real time streaming. All of these are available as open source and as a hosted SaaS solution. You can check all of that out at Influxdata.com. My thanks to Influx Data for helping make this podcast possible.
Mike: Robots. You apparently are working at some company that does observability for robots and I'm a little confused because like what in the world is this all about? Do robots actually need observability?
Ian: Yeah. I work in a company called Formant. We are about a year and a half old and we're focused on a lot of problems in supported robots, but specifically observability for robotics is very important to us. I think it's representative of the type of concern that hasn't historically been important in robotics, but is increasing as we are shipping robots more and more to customers, deploying fleets of robot, deploying them in semi-structured environments, and generally seeing their numbers increase in the wild.
Mike: These robots, are these like Johnny 5 style robots or they're more like C3PO or The Terminator or Wall-E? Are these more Wall-E or maybe even the really terrifying stuff that General Dynamics is putting out?
Ian: Right. We like to maintain a flexible definition of a robot. I think that's maybe just a way of avoiding the definition question.
Mike: I'm sure the robots in the singularity will be very happy about your loose definition.
Ian: Yeah. The vast number of deployed robots in the world has sort of traditionally defines probably in the space of automotive manufacturing. That's where we see bolted down work cells of high payload position controls, heavy metal robots performing assembly and welding and applications like that. But the fastest growing part of the robotics market is actually in service robotics and in the deployment of robotics into less structured environments. That's environments like logistics and warehousing, retail, agriculture. This is where we have started focusing is in robots in semi-structured environments.
We do think that we have a lot to offer in industrial robotics as well, but it has some better focus to date.
Mike: I saw on your website there's this really interesting photo of a robot kind of strolling down the aisle at the grocery store.
Ian: Mm-hmm (affirmative).
Mike: Is that indicative of the kind of robots we're talking about primarily?
Ian: It is. We may have a little bit of an insight into the way things are going just from the customers we're talking to every day, but we are seeing more and more robots deployed into retail for example. It's just what that image shows. The applications at the moment are typically in things like floor cleaning, inventory scanning. Those are the front of house applications that we see the most often. Of course, in order fulfillment and logistics and warehousing, we see a lot of addition applications of robotics.
Mike: Got you. I want to take a little tangent here and ask how in the world did you get into this? I don't think anyone comes out of school and says, “You know what I'm going to do? I'm going to build robots and observability.”
Ian: I came to robotics through work at a company called Bot & Dolly about seven or eight years ago. It was focused on applying robotics to challenges in film and visual effects. I had an opportunity to get involved in novel applications of industrial robotics at that company. We were acquired into Google and that was around the time that a number of robotics companies were acquired, including Boston Dynamics we mentioned. Inside Google, I had the chance to see how all of our peers were thinking about these problems. We ultimately left Google about a year and a half ago because we were excited to ship products. The timeline for that is...
Mike: There's a very subtle danger there.
Ian: The timeline is long at Google for shipping products, but the experience was really invaluable. Personally, I was already interested in the tools and infrastructure side of robotics. Through building tools to support these teams inside Google and through seeing how people thought about problems like observability, software deployment, configuration management in the context of robotics; it became clear that there's actually a huge opportunity to bring some of the best practices that have been developed for decades in the backend distributed systems world to the robotics world.
That's where I find a lot of inspiration. The problem is similar enough that we have a lot to learn, but different enough that it does require some new thinking and some new technology.
Mike: That's a really great segue into a really good question of what is it look like to do observability in robots? You mentioned all these tools and all these techniques that infrastructure people rely on every day. I can think management and that sort of thing. How is that being applied in your work?
Ian: The fundamental requirement of observability and robotics is really no different than it is in monitoring backend systems. We want to maintain visibility into the state of the system. Use that information to allow humans to respond to changes in internal systems state and also automated systems to respond to those changes. But there's a few key differences. One is that the data types that are relevant to us in robotics are often different than they are in backend distributed systems. We have sensors generating a lot of data about the physical world. Those data types are often geometric or three-dimensional or media-based.
The infrastructure and tooling to ingest and index and visualize that type of data is different. The workflows that we used to debug issues are different. They often require making sense of a lot of that geometric and visual data. Another difference is that centralizing data is often ...
About Aaron Sachs
Aaron Sachs is a home brewer, banjo player, and also happens to like monitoring things. He helps make his customers look like monitoring badasses to their customers at Sensu, where he's a Customer Reliability Engineer.
Links Referenced
Sensu
Aaron.sachs.blog
Twitter: @asachs01
Transcript
Mike: This is the Real World DevOps podcast and I'm your host Mike Gillian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools, to the organizers of amazing conferences, from the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.
Mike: This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database, Influx DB, but you may not be as familiar with their other tools. Telegraph for metrics collection from systems, chronograph for visualization and capacitor for real time streaming. All of these are available as open source and as a hosted SAS solution. You can check all of it out at influxdata.com. My thanks to influx data for helping make this podcast possible.
Mike: Hey Aaron, welcome to the show.
Aaron: Hey Mike, thanks for having me on.
Mike: So I want to start with a a little bit of an origin story because everyone loves a good origin story.
Aaron: Oh yeah.
Mike: You and I met over hot wings many, many, many years ago.
Aaron: Hot wings, pizza and garlic knots. All good stuff.
Mike: Right? Sadly the place is closed now, but we ended up going..
Aaron: No.
Mike: I know at the time I think you were working help desk support while studying for a communications degree?
Aaron: Oh yeah, yeah. I was at a University of Tennessee's Office of Information Technology doing desktop support with a bunch of other students and yeah, working on my communication studies degree. Yep.
Mike: Yeah. So the thing I find interesting about all this is that you never intended to go into IT at all. Like you weren't planning to go into tech in any way. You were really planning to go into something communication related.
Aaron: Yeah. Yeah. So I was planning on being a communication professor and kind of aligned my career up to that point to do just that. And yeah... It was during that time, like so I was the whole reason I got out of that was I was writing my thesis on how one determines a blogger to be credible. Like what sort of behaviors do they exhibit when they were writing a blog uh and because that was nowhere near the specialty of the department I was in, there was just a lot of politics and I got so fed up with it and I was like, I'm done. And I'm already working at the Office of Information Technology to, do you know my grad assistantship, so why not just do IT?
Mike: I remember you and I went out for beers, I want to say it was like a month after we met, and you you asked me a question of, "What if I stopped doing communications and went into tech? What would that look like?" And like the immediate follow up question was, "And can you help me?"
Aaron: Yup. Yup. That's correct.
Mike: Yeah. So, me not knowing what I was getting myself into of, "Yeah, let's just totally change the course of someone's career over beer."
Aaron: By the way, for the listeners out there, if you ever want to sucker Mike into something, get him real drunk and then asking me if ask him for help. Totally works.
Mike: Yeah apparently. So…
Aaron: Haha
Mike: This kind of started an interesting thing of you and I weren't really initially friends is more of a, you were looking to me for help on how to change your career and how to get better at a thing that you had not focused on at all. We actually became friends as a result of that like going through that whole thing, but it really developed into this informal mentorship situation.
Aaron: Oh yeah. Yeah. I very fondly recall all of our a Tuesday nights at Old City Java in Knoxville, just gorging ourselves on Meg's croissants and downing gratuitous amounts of caffeine.
Mike: Right. And wishing for a whiteboard to further explain concepts. Right?
Aaron: Right.
Mike: So through all this, clearly you've, you've done well you're working for Sensu now as a Customer Reliability Engineer, helping companies with improving their monitoring using Sensu. But one of your big focuses throughout your career, many years after us working together, has been to help other people improve their careers and improve their professional lives.
Aaron: Absolutely. Yeah. It's so you're ready for that, by the way.
Mike: Yeah. And so I know you've given several talks on this and you've had several other mentees over the years as well. I believe you were involved in a pretty formalized program at Rackspace on that topic as well.
Aaron: Yeah, there was actually... I did a lot of mentorship with my team at Rackspace, so as I moved up in the admin ranks, I always took it under, rather took it upon myself to mentor the people that were coming in. And then that somehow turned into me mentoring folks that were in my wife, Ashley's department. We actually started like a meetup that would happen once a week at Rackspace and these are folks who came in as customer service technicians. They knew enough to spell DNS and grow them from just that to actually progressing into paths as like a one guy is a, I think he's a customer success manager for a startup in San Antonio and other guys as systems engineer now at Rackspace still, and then I've got another great friend of mine who's constantly blown away because she's actually taking this whole mentorship principle that we worked on at Rackspace and she's doing it with other people that she knows now. Her name's Elle, she works at I think Lennox Academy or Jupiter Broadcasting. So yeah, it's, it's been a crazy journey.
Mike: Yeah, that sounds pretty awesome. So I want to dig into that. Let's back up and say what is mentorship?
Aaron: So let's talk about what mentorship is not because I think that that's an even more useful way of discussing the concept of mentorship.
Aaron: So mentorship, in my view is not simply... It's not transactional. It's not just the like... I mean I came to you initially and thought it was a transaction. I'm going to go to Mike and Mike's going to help me, which is fine, but that's not me...
About Andrew Rogers
Andrew leads technical strategy and architecture development for ACE IoT Solutions. Andrew also leads the development of technical and research strategy at The Enterprise Center, a non-profit focused on developing the innovation ecosystem in Chattanooga, TN. When not bringing his extensive professional experience in Industrial Control Systems, Critical Infrastructure Controls, and Network Engineering to his professional endeavors, he can most commonly be found with a camera in his hand. A deep passion for photography takes him off the beaten path the world over, and serves as a convenient excuse for a variety of other means for enjoying nature, including hiking, biking, and most board sports. Andrew loves sharing his travels and photography, and keeps an instagram account updated with his most recent adventures.
Links Referenced
LinkedIn
Twitter @acedrew
ACE IoT Solutions
Personal site
Transcript
Mike: This is the Real World DevOps podcast, and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences, from authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.
Mike: This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools, and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with their other tools. Telegraf for metrics collection from systems, Chronograf for visualization, and Capacitor for real-time streaming. All of these are available as open-source and as a hosted SaaS solution. You can check all of it out at influxdata.com. My thanks to InfluxData for helping make this podcast possible.
Mike: Hi folks, I'm Mike Julian, your host for Real World DevOps podcast, and my guest this week is a friend of mine. Andrew Rogers, he's an expert in industrial control systems and co-founder of ACE IoT solutions where he helps companies with improving visibility in their operations and energy systems. Welcome to the show.
Andrew: Thanks.
Mike: Now you live in one of my favorite cities ever, which is Chattanooga, Tennessee. I can hear the Chattanooga fans in the background just, "Yeah, this is awesome." So one of the coolest things that I think there is in Chattanooga, aside from just the gorgeous weather and great food, is the low cost municipal internet.
Andrew: Yeah, it's pretty fantastic. Um it's certainly part of the reason why I moved to Chattanooga in the first place. Uh and I think that it's had that effect on a lot of people over the years. So um we have, sort of, an out-sized technical community here based on the fact that it's easy to support a remote workforce when you have gigabit or 10 gigabit Internet available ubiquitously across the community.
Mike: That was one of the things I never expected because there are actually some pretty significant companies based out of Chattanooga entirely as a result of this, highly available, municipal Internet like companies might know of Bellhops which is based there.
Andrew: Yeah. So that project or that company started, based on another company in Chattanooga exiting and the founders starting a fund and looking for startups across the southeast that could benefit from the available broadband here. And you know it's a pretty big company now, well over 100 folks in their technical workforce, and they continue to you know grow and support the community. It's certainly been a really interesting success story for Chattanooga.
Mike: You worked for a while as, I think, a technologist in residence for one of the startup incubators in the area.
Andrew: So yeah, Chattanooga has a lot of really unique resources. You talked about one of those, the fiber broadband available over a 600 square mile area. And one of the other things that I think, is actually due to the fiber but also just due to the type of community Chattanooga is, and the effort and interest in working collaboratively, is a 501c3 nonprofit organization focused on entrepreneurship, and helping startups both at a scale of coming up and building a high growth startup, but also mom-and-pop shops, and helping them grow and build sustainable businesses or get to the next step in their business plan. They did, as part of when the fiber was first launched in 2009, a group of folks in the community came together to say, "Hey, how do we get the most out of this? This is an awesome asset. We know that the future of our economy is going to be based on growing businesses in the area. How can we use this new asset to support that, not just in luring a big company into town, and sort of the traditional economic development scenario, but growing companies locally."
Andrew: And so one of the things they decided to try to do is launch a startup accelerator. It wasn't a venture funded accelerator, it was funded by this nonprofit. And it was focused on finding the companies around the country who had ideas that were only viable when ubiquitous broadband was available. Now, that brought a lot of really interesting folks to Chattanooga. And you know we did it about four years, I think. GIGTANK still exists today. It's changed a little bit, happens every summer. But yeah, one of the challenges with GIGTANK was that, you bring in a company, and you get a group of really great mentors from the business community here in Chattanooga. You surround them with professional services companies who are eager to support growing a business in Chattanooga. Then you tell them that you know you're trying to build a high growth startup, and your total addressable market is you know a million people in the US that actually have access to this high speed connectivity.
Andrew: So what tended to happen, unfortunately, is the businesses were viable, but the reliance on the broadband wasn't. And now finally, in 2019, this was back... I was involved heavily in 2012, 2013, 2014. 2019, we're sitting here, and you see fiber deployments happening around the country. Talk of 5G is running thick right now, and so those businesses actually a lot more have a lot more viability. But what did happen was, the businesses came for the fiber and kind of stuck around for Chattanooga. So we ended up accruing and amazing tech talent pool, really interesting entrepreneurs uh who have gone on to focus on you know other things. We even took a little stint at additive manufacturing because we saw that, sort of, digital manufacturing gave you the same opportunity that, sort of, was the basis for deploying the fiber in the first place, which was making a smart grid that was truly smart, and making the trade of moving photons instead of electron to achieve the same sorts of quality of life improvements, et cetera, that rural electrification had in the early 20th century.
About Jeremy Tangren
Jeremy Tangren is a Technical Program Manager specializing in infrastructure and vendor management. Throughout his 15+ years in IT program/project management, he has managed local- and global-scale multi-million dollar projects at companies like Facebook, Splunk, and Cisco. Jeremy is based in San Francisco, CA.
Transcript
Mike: This is the Real World DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences or the authors of great books to fantastic public speakers I want to introduce you to the most interesting people I can find.
This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database influxDB, but you may not be as familiar with their other tools. Telegraf for metrics collection from Systems, Chronograf for visualization and Kapacitor for real-time streaming. All of these are available as open source and as a hosted SaaS solution. You can check all of it out at influxdata.com. My thanks to InfluxData for helping make this podcast possible.
Hi, folks, I'm Mike Julian, your hosts for the Real World DevOps podcast. My guest this week is Jeremy Tangren. He's an expert on everyone's favorite topic to completely ignore, vendor management. He's been a technical program manager for companies such as Cisco and Facebook, Splunk, and a whole bunch of other really interesting places. Most interesting to me is, Jeremy, you and I have known each other for God what? Nineteen years now?
Jeremy: Yeah, 19 going on 20 it's been a while.
Mike: It's insane. Yeah, which is weird. We met each other on a Final Fantasy Forum when we were like 12.
Jeremy: Thank you Final Fantasy Seven.
Mike: Yes. Thank you. So you've been working a lot with vendors throughout most of your career and I know you kind of fell into it accidentally.
Jeremy: Yeah, totally accidentally. It started when I was just a general IT guy and I was by myself. I needed more hands and more capabilities and I had to start bringing in vendors to get work done. It was a necessity.
Mike: Now, I imagine that most people listening to this thinking like, "Oh, my vendors, my vendors are awful," and I mean vendors are kind of annoying to work with aren't they?
Jeremy: They can be challenging. I definitely get the perception that most folks view vendors as a necessary evil and they would avoid interacting with them at all costs if they could.
Mike: That's been my experience as well. But I mean, there have definitely been a lot of vendors where I'm like, "I don't really want to work with them, but they get the job done that I need done and I don't want to do it myself." But at the same time, it doesn't really need to be that way.
Jeremy: No, not at all. You don't have to have an adversarial relationship with your vendors. In fact, you should be looking for more of a partnership than anything else with a vendor.
Mike: Tell me more about that. What do you mean by this partnership?
Jeremy: Well, take for example, when going back to my original IT guy explanation, I was by myself, but I had to have help. So when I brought in a vendor to install fiber or whatever it was, I had to work very closely with them to tell them where the work needed to be done, what my expectations were, all of the major details of the project, and then I had to again work closely with them during deployment and then finally on close of the project. At no point during that process was there an opportunity for me to step away from them and just let them run and expect things to go hunky-dory, and that's what most people tend to expect with vendors is that, "I pay them, I cut them a check, they get the job done. I don't think about it anymore," but there's just so much more than just cutting a check that goes into working with a vendor.
Mike: Okay. Let's dig into that. What else is there? Well, I mean when I have a problem I'm thinking, "I'm going to go find someone who is very good at this problem, some company, and I'm going to tell them what I need, cut a check and walk away." It's like hiring an electrician or a plumber. They're just going to get the job done. I don't have to think about it. So is there more to it than that? What else goes into it?
Jeremy: There's always more to it than that. When you're bringing in a vendor, they're specialists at whatever they do, the electrician, the plumber, but they still have to be told what it is that needs to be done. "Do you need to wire this entire house with three-phase power?" "Do you need to only fix this toilet?" They need to be given clear instructions and expectations or else they're just going to do what they think is right and that might not align with what your expectations are, so you have to work closely with them to get the right results.
Mike: I imagine that gets pretty complicated when we're talking about a software and IT. I can just imagine hiring a data comm person or data comm company to come in and wire a building for Ethernet and how complicated that would be.
Jeremy: Absolutely, you can't just bring in that cabling vendor and say, "Go." They don't know where all of the gotchas are in that building. They need a blueprint of the building. They need to be working with the facilities team. There're lots of cross-functional stuff that goes on just to wire a building with cables. So you have to have that partnership with the vendor and not just expect them to cowboy it up and take care of everything magically or again, you'll end up with results that you weren't expecting or didn't desire.
Mike: So on that note, the idea of hiring a vendor, just letting them loose. It seems like the thing that everyone does, and on that same note, people just don't think about their vendors, especially ones they're using long-term. I know for a long time I didn't until you told me how wrong that was, so why are we doing that? Why are we just kind of ignoring our vendors?
Jeremy: Because it's easy to do. I think that's what it really boils down to is, especially from an engineering standpoint, it's very easy to focus on what you're working on and what you need to get done versus the dependencies and interactions with these other teams, and vendors included. So it's very easy for an engineer or even a manager to forget that they even are working with a vendor. And when a vendor comes knocking, they go, "Hey, we're doing still doing x, y, z. Is it to your satisfaction?" And manager or engineer goes, "Oh, no, that's not right actually. You didn't do it the way I wanted," because they weren't partnering with and working with the vendor to guide them to a successful conclusion.
Mike: Something you said there made me think about this one particular thing that is often overlooked in vendor relationships, which is the role of the client. So as when me, the company, is hiring a vendor, I expect certain things from them, but to make a good relationship, they actuall...
About Mary Thengvall
Mary Thengvall is a connector of people at heart, both personally and professionally. She loves digging into the strategy of how to build and foster developer communities and has been doing so for over 10 years. After building community programs at O’Reilly Media, Chef Software, and SparkPost, she’s now consulting for companies looking to build out a Developer Relations strategy. In addition to her work, she's known for being "the one with the dog," thanks to her ever-present medical alert service dog Ember. She's the author of the first book on Developer Relations: The Business Value of Developer Relations (© 2018, Apress).
Mary is founder and co-host of Community Pulse, a podcast for Developer Relations professionals. She curates DevRel Weekly, a weekly newsletter that brings you a curated list of articles, job postings, and events every Thursday. She's also a founding member and "Benevolent Queen" of the DevRel Collective Slack team.
Mary is also a member of Prompt, a non-profit that encourages people to openly talk about mental illness in tech. She speaks at various conferences and events about building and fostering technical communities as well as how to prevent burnout in yourself and your team.
Links Referenced
Transcript
Mike: This is the Real World DevOps Podcast, and I'm your host, Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools, to the organizers of amazing conferences. From the authors of great books to fantastic public speakers, I want to introduce you to the most interesting people I can find.
This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools, and that's where Influx comes in. Personally, I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database, InfluxDB, but you may not be as familiar with their other tools, Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of these are available as open-source, and as a hosted SaaS solution. You can check all of it out at Influxdata.com. My thanks to InfluxData for helping make this podcast possible.
Hi, folks, I'm Mike Julian your host of Real World DevOps Podcast. My guest this week is Mary Thengvall, she's a long-time builder of communities for companies like O'Reilly Media, and Chef. And author of the book on developer relations, The Business Value of Developer Relations, and runs a podcast, and newsletter of her own. Now she's helping companies create community-building strategy through a new consulting firm, Persea Consulting. Welcome to the show.
Mary: Thanks for having me, Mike, I appreciate it.
Mike: So, I want to start off with a really foundational topic.
Mary: Sure.
Mike: What is community in this context?
Mary: Absolutely. You start there, and I start there, they're most of my talks as well, just because I like to make sure that people have a shared definition. The definition that I always go back to with community is, and it relates to developer relations as well. But community for me is a group of people, who not only share common principles, but also develop and share practices that help individuals in the group thrive. So it's not just, "Hey, we're all on this platform." It's, "We're all on this platform, we have a specific goal in mind, and we're all helping each other."
Mike: Okay. A lot of us are pretty familiar with like the Python community, the Ruby community, or the Chef community, and then there's kind of the non-tech stuff. Like people that go to church, have a church community. Or we have the community of our neighborhoods. Are these the same community?
Mary: They have similarities.
Mike: Okay.
Mary: And it's that idea of, if you're in the Python community and someone reaches out on Twitter, or in a Slack team, or at PyCon for instance, which just happened. And they say, "Hey, I need help with these things." You'll likely find other people around you who go, "Oh, I know how to do that. Let me help. I'm more than willing to." In neighborhoods, you have things like next door, where someone will post, "Hey, I lost my cat." Or, "I need help with my garden." Or, "Help me out in these other areas." And people will jump in and say, "Oh, yeah, I'm more than willing to help out."
In those ways they can be very similar. There's obviously nuances with really just communities with technical communities, with neighborhood communities. And in some cases, they come together easier than others. Neighborhoods have the advantage of being in the same physical location, which is always really nice. Technical communities usually have forums around a particular product, or we use Twitter hashtags to kind of see what other people are talking about. And there's a big difference to me between communities of people, and community platforms.
And I know we have a lot to cover today, so I won't spend too much time on this, but just kind of a TLDR. The platforms are usually ways to bring people together to accomplish a specific purpose. A Python community platform for instance, you could have one for new Python developers, you could have one for experienced Python developers. You could have one for people who are willing to mentor folks, or people who are into Django, or whatever this specific piece of Python lore that you're interested in.
Mike: When you're talking about these platforms, are you talking about the medium on which they operate, or are you talking about kind of segmentations of that community?
Mary: A little bit of both, and those tend to go hand-in-hand. As you kind of see communities, larger communities, people who are interested in the same topics segmenting off, you'll often find software platforms that spring up that, you know, cool, the newbie Python people are hanging out on Twitter. The experienced Python people have a back channel slack team. The people, who are willing to mentor are on LinkedIn. So they're using already existing platforms to facilitate conversations around particular topics.
Mike: That seems like a bad thing.
Mary: It can be. The hard thing about that is, if there's not a specific person that's really taking the time to lead those groups or manage those groups, or build those groups of communities, that things can devolve fairly quickly with Twitter as we've seen. Or with Reddit or places like that where it's just not... It's hard to control when you don't own the platform, or when you don't own the conversations. But at the same time if you're not, owning the conversation as a company, then you aren't necessarily limiting people to what your viewpoint is. You're allowing people to talk and explore, a...
About Kelly Shortridge
Kelly Shortridge is currently VP of Product Strategy at Capsule8. Kelly is known for research into the applications of behavioral economics to information security, which Kelly has presented conferences internationally, including Black Hat, AusCERT, Hacktivity, Troopers, and ZeroNights. Most recently, Kelly was the Product Manager for Analytics at SecurityScorecard. Previously, Kelly was the Product Manager for cross-platform detection capabilities at BAE Systems Applied Intelligence as well as co-founder and COO of IperLane, which was acquired. Prior to IperLane, Kelly was an investment banking analyst at Teneo Capital covering the data security and analytics sectors.
Kelly graduated from Vassar College with a B.A. in Economics and was awarded the Leo M. Prince Prize for Academic Achievement. In Kelly's spare time, she enjoys world-building, weight lifting, reading sci-fi novels, and playing open-world RPGs.
Links Referenced:
Transcript
Mike Julian: This is the Real World DevOps Podcast and I'm your host Mike Julian. I'm setting out to meet the world's most interesting people doing always work in the world of DevOps. From the creators of your favorite tools to the organizers or amazing conferences. From the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.
This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with our other tools. Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of this is available as open source, and as a hosted SaaS solution. You can check it out InfluxData.com.
My thanks for InfluxData for helping making this podcast possible.
Hi folks I'm Mike Julian, your host for the Real World DevOps Podcast. My guest this week is Kelly Shortridge, the VP of Product at Capsule8 and an internationally known speaker on InfoSec topics.
So Kelly, welcome to the show.
Kelly Shortridge: Thank you so much for having me Mike.
Mike Julian: You know I was looking at your LinkedIn and there was something that kind of stood out to me was your FINRA license. You started your career off in finance.
Kelly Shortridge: That's true.
Mike Julian: So what in the world happened there? How does that work?
Kelly Shortridge: Yeah. How does that even happen. So one, FINRA exams are very painful so I didn't want to have to re-up those, but mostly I started my career doing mergers and acquisitions covering information security companies. And while I quite liked the finance side I noticed that security had a ton of opportunity not just as far as vendors, but the problem space is huge and it's very unsolved and the incentive problems are enormous. So for someone like myself who had studied some Behavioral Economics, just all of the messy incentive problems were kind of like catnip for me. So I knew I had to go into the industry.
Mike Julian: Yeah. The Behavioral Economics side, the study of incentives. That's absolutely fascinating to me because, especially in security and systems got everything we do is incentives based.
Kelly Shortridge: Yes.
Mike Julian: And it's often incentive that we're not even paying attention to. Things we don't even think about. Like if you-
Kelly Shortridge: That's definitely true.
Mike Julian: Yeah. Like you make it hard to use two-factor and people aren't going to use two-factor.
Kelly Shortridge: Yup, so that's something that I feel like people outside of security understand immediately, but inside security they don't always understand the fact that if you don't make something that integrates into work flows, people are going to bypass it. But you're absolutely right, there's such a web of incentives and on the one hand you have things that are explicitly stated you know, security to your privacy is important, but then you have more of a tacit goals and priorities, which are that, well security's a cost center and really what matters is being able to deliver on time and you know releasing at a certain cadence.
So those tacit assumptions also create a bunch of incentive problems and InfoSec. But I always think that InfoSec because they wish they were more relevant try to be the culture of know and ram through really annoying technologies for people to use just to show that they're still relevant.
Mike Julian: Yeah, that hits home. I've seen that way too many times.
Kelly Shortridge: Yeah. I think most developers... It's not really a love hate relationship, it's mostly just a hate relationship for the most part. Somewhat bi-directional.
Mike Julian: What really about security drew you away from finance? Have you found there's good parallels that you've been seeing?
Kelly Shortridge: It's interesting. There's certainly parallels, particularly on the risk management side, particularly anything around risk centrality because that's a huge part of financial systems, so that's not really what I did day to day. I think what's interesting in security is there's a huge lack of effective communication. Even when you go to conferences you know, there will be some 0-day that's dropped or whatever, but it's often not communicated very well, and certainly when you look at enterprises, security priorities aren't really communicated well to the rest of the business. And a lot of investment banking is quite frankly effective communication.
It's about quickly researching something in general companies and understanding it very deeply to be able to talk about it and effectively persuade acquirers that's it's worth acquiring a company and so frankly even the excel and PowerPoint skills I learned along the way are really helpful in just being able to talk to people about security. You know I can speak to CEO's, I can speak to Board members, I can speak to DevOps people about security in a way that's still understandable and that's what I feel like we're still missing a lot of in information security.
Mike Julian: Yeah that makes a ton of sense. I was reading one of your articles, I can't remember which one it is. I'll have to go find it and throw it into the show notes, but you made mention of a tax from nation states. And I thought that was pretty interesting. It definitely stood out to me. And perhaps I have a bit more security exposure to that side of things than the average DevOps person might, just from, I worked for the government for a while so I've seen it. But for the vast majority of peo...
About Christine Yen
Christine delights in being a developer in a room full of ops folks. As a cofounder of Honeycomb.io, a tool for engineering teams to understand their production systems, she cares deeply about bridging the gap between devs and ops with technological and cultural improvements. Before Honeycomb, she built out an analytics product at Parse (bought by Facebook) and wrote software at a few now-defunct startups.
Links Referenced:
Transcript
Mike Julian: This is the Real World DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences, to the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.
Mike Julian: This episode is sponsored by the lovely folks at Influx Data. If you're listening to this podcast you're probably also interested in better monitoring tools and that's where Influx comes in. Personally I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with their other tools, Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of these are available as open source and as a hosted SaaS solution. You can check all of it out at influxdata.com. My thanks to Influx Data for helping to make this podcast possible.
Mike Julian: Hi folks. Welcome to another episode of Real World DevOps podcast. I'm your host Mike Julian. My guest this week is a conversation I've been wanting to have for quite some time. I'm chatting with Christine Yen CEO and co-founder of Honeycomb, and previously an engineer at Parse. Welcome to the show.
Christine Yen: Hello. Thanks for having me.
Mike Julian: I want to start this conversation off in kind of what might sound like a really foundational question. What are we talking about when we're all talking about observability? What do we mean?
Christine Yen: When I think about observability, and I talk about observability I like to frame it in my head as the ability to ask questions of our systems. And the reason we've got that word rather than just say, "Okay well monitoring is asking questions about our system," is that we really feel like observability is about being a little bit more flexible and ad-hoc about asking those questions. Monitoring sort of brings to mind defining very constrict parameters within which to watch your systems, or thresholds, or putting your systems in a jail cell and monitoring that behavior, whereas, we're like, "Okay, our systems are going to do things, but they're not necessarily bad." But let's be able to understand what's happening and why. And let's observe and look at the data that your systems are putting out as well as thinking about how, asking more free form questions might impact how you even think about your systems, and how you even think about what to do with that data.
Mike Julian: When you say asking questions what do you mean?
Christine Yen: When I say asking questions of my system, I mean being able to proactively be able to investigate and dig deeper into data, rather than sort of passively sitting back and looking at the answers I've curated in the past. In order to illustrate this, to compare observability monitoring a little more directly with monitoring, especially traditional monitoring when we're curating these dashboards, what we're essentially doing is we are looking at sets of answers from questions that we posed when we pulled those dashboards together. All right, so if a dashboard has existed for six months the graphs that I'm looking at to answer a question like, what's going on in my system, are answers to the questions that I had in mind six months ago when I tried to figure out what information I would need to figure out whether my system was healthy or not. In contrast, an observability tool should let you say, "Oh is my system healthy?" What does healthy mean today? What do I care about today? And if is see some sort of anomaly in a graph, or I see something odd, I should be able to continue investigating that threat without losing track of where I am, or again relying on answers from past questions.
Mike Julian: So does that mean that curating these dashboards to begin with is just the wrong way to go? Like is it just a bad idea?
Christine Yen: I think dashboards can be useful, but I think that over use of them has led to a lot of really bad habits in our industry.
Mike Julian: Yeah, tell me more about the bad habits there.
Christine Yen: An analogy I like to use is, when you go to the doctor and you're not feeling well. A doctor looks at you and asks you, "What doesn't feel well. Oh, it's your head. What kind of pain are you feeling in your head? Is it acute? Is it just kind of a dull ache? Oh, it's acute. Where in your head?" They're asking progressively more detailed questions based on what they learned at each step along the way. Honestly this is kind of parallels a natural human problem solving concept. In contrast, I think the bad habits that dashboard lead us to build are things like it would be the equivalent of a doctor saying, "Oh well based on the charts from the last three times you visited, you broke your ankle and you skinned your knee." Pretend you go to the doctor to skin your knee. You know, "Oh okay, you broke your ankle last time, did you break your ankle again? No. Okay, did you ... How's your knee doing?"
With dashboards, we have built up this belief that these answers to past questions that we've asked are going to continue to be relevant today. And there's no guarantee that they are. Especially for our engineering teams that are staying on top of incidents and responding, and fixing things that they've found along the way. You're going to continue to run into new problems, new kind of strange interactions with routine components. And you're going to be able to ask new questions of what your systems are doing today.
Mike Julian: It seems like with that dashboard problem we have that same issue with alerting. I've started calling this kind of reflexive alerting strategy where it's like, "Oh God. We just had this problem. Well we better add a new alert so we catch it next time it happens." It's like well how many times is that new alert going to fire? Probably never. Like you're probably never going to see that issue again. With dashboards, dashboards are the same way. What you're describing, God I've seen this 100 billion times where someone curates a dashboard, is like, "Okay, well first thing now that we have this alert is let's go look at the dashboards and see what went wrong." I'm like, "Well no, graphs look fine." So no problem, but clearly the sites down.
Christine Yen: Yeah, there's a term that we've been playing with, dashboard blindness. Where if it doesn't exist in the dashboard it clearly hasn't happened, or you know it just can't exist, because people start to feel like, "Okay, we have so many dashboards...
About James Turnbull
James Turnbull is originally from Australia but now lives in Brooklyn, NY. He likes wine, food, and cooking (in that order) and tattoos, books, and cats (in no particular order).
He is a CTO in residence and lead startup advocacy at Microsoft. Prior to Microsoft, he was the founding CTO at Empatico. Before that, James was CTO at Kickstarter, VP of Engineering at Venmo, and in leadership roles at Docker and Puppet. He also had a long career in enterprise, working in banking, biotech, and e-commerce. James also chairs the O'Reilly Velocity conference series. In lieu of sleep, James has written eleven technical books, largely on infrastructure topics.
Links Referenced:
Transcript
Mike Julian: This is the real world DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps from the creators of your favorite tools to the organizers of amazing conferences or the authors of great books to fantastic public speakers. I want to introduce you to the most interesting people I can find.
This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with their other tools. Telegraf for metrics collection from systems, Chronograf for visualization and capacitor for realtime streaming. All of these are available as open source and as a hosted SaaS solution. You can check all of it out at influxdata.com. My thanks to InfluxData for helping make this podcast possible.
Hi folks I'm Mike Julian, your host for the Real World DevOps podcast. My guest this week is James Turnbull. You probably know James from his seeming inability to stop writing technical books such as Monitoring with Prometheus, The Art of Monitoring, The Terraform book and like a bajillion others. He has also worked for some pretty neat companies too, like Puppet, Kickstarter and Venmo and now he works at Microsoft leading a team as CTO-in-residence. Welcome to the show, James.
James Turnbull: Hi Mike.
Mike Julian: I'm really curious like what is a CTO in residence?
James Turnbull: I guess my primary mission is to make Microsoft relevant to start ups much the same way that Microsoft is shaping its relevancy towards the open source community. We're also interested in looking at other audiences that we've traditionally not been involved with and so it's just one of those.
Mike Julian: Gotcha. And you're just leading a team of people that are focused on that sort of stuff?
James Turnbull: Yeah, so most of my team is people who've come from startups or and particularly from engineering management leadership roles in startups. One of my colleagues is ... was the CTO of SwiftKey and another one, fairly famous, Duncan Davidson who wrote Tomcat and Ant and has been around engineering management for a long time — and folks like that who really are here to help sort of startups understand a bit more about how to grow and scale. And I think some of the big challenges startups have are actually not technology related at all. They're really about, you know, how do I build a recruiting process? You know, I had 10 engineers last week, I have 100 this week. How do we structure the team? So we've sort of brought together a group of folks who have fairly deep experience in those sort of problems for startups and have sort of a deep empathy for the startup community.
Mike Julian: Yeah that's quite the task ahead of you.
James Turnbull: Yeah. Look, I think, I mean Microsoft traditionally been known as an enterprise software company. You know, a lot of startups are not sure of their relevance to us. I think increasingly we're seeing traction to cover is one is that obviously Azure is one of our focuses and the cloud platform in there. And that platform is looking more broadly at not just enterprise audiences but other groups. And secondly, a lot of startups ... Microsoft's deep in the middle of most of their customers. So particularly if you're a Beta based startups, something like that and you're, you know, you're trying to sell into enterprise business, Microsoft has been doing that for 30 years. They have all the connections and account managers and sales folks and you know, multimillion dollar relationships with some of the people you want to be customers with. We can provide you with A, some of those connections, but also a lot of advice and expertise about how to sell to those customers.
James Turnbull: And having worked at both Empatico and Docker, you know, a large part of my job was attempting to sell, you know, as a small startup, as a fairly early employee at both into big companies. You know, you can't walk in the door to Wall Street financial if you're a 30 person start up in Portland, Oregon without having a pretty credible story. So I'm happy to sort of help startups and I do some of that messaging and understand how to have some of those conversations.
Mike Julian: Yeah. It's one of the interesting things about my own company is my clients are all these large companies too and I'm a two person company, but selling into a very large company is not ... it's nothing like selling it to a small company. Everything works differently. People think about their jobs differently.
James Turnbull: Yeah.
Mike Julian: Yeah. I think maybe my most favorite thing of everything you just said is this isn't your daddy's Microsoft. The Microsoft we all grew to know and hate is not today's Microsoft at all. Not by a stretch and that's just absolutely incredible to see that turnaround.
James Turnbull: Yeah. I got a LinkedIn request from somebody yesterday and the message said, you know, I've read a bunch of your books and you know, I've used a bunch of different sorts things you've worked on. I was really surprised to see you at Microsoft. And I was like, okay, this could end badly the next couple of sentences, because I've certainly had a few people of my generation who remember the bad old days and “Linux is a cancer” and things like that. And he finished with, you know, it's really interesting to see companies grow and change. And I was like, wow, okay, that's a ... I thought that was going to go really badly but I think it's a fairly accurate reflection.
Microsoft is aware of the fact that this is not a position that pragmatically that was not a good business position to be in. The world is changing. It's moving towards the cloud, you know, the stacks in people's companies, the way they manage things, the infrastructure, the software, you know, things are changing. And I hesitate to say this conclusively, but I think open source won, you know this for certain values of one, given recent sort of events, discussions about large corporates and their contribution to open source, but as a technology choice, it's pretty clear to me that open source won. And I'm kind of a bit smug about that to be honest.
Mike Julian: Speaking of your books I would just straight up say you're th...
About VM Brasseur
VM (aka Vicky) spent most of her twenty-plus years in the tech industry leading software development departments and teams, providing technical management and leadership consulting for small and medium businesses, and helping companies understand, use, release, and contribute to free and open source software in a way that's good for both their bottom line and for the community. Now, as the Director of Open Source Strategy for Juniper Networks, she leverages her nearly 30 years of free and open source software experience and a strong business background to help Juniper be successful through free and open source software.
She is the author of Forge Your Future with Open Source, the first and only book to detail how to contribute to free and open source software projects. The book is published by The Pragmatic Programmers and is now available at https://fossforge.com.
Vicky is a moderator and author for opensource.com, an author for Linux Journal, the former Vice President of the Open Source Initiative, and a frequent and popular speaker at free/open source conferences and events. She's the proud winner of the Perl White Camel Award (2014) and the O’Reilly Open Source Award (2016). She blogs about free/open source, business, and technical management at {anonymous => 'hash'};.
Links
Transcript
Mike Julian: This is the Real World DevOps podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools, to the organizers of amazing conferences. From the authors of great books, to fantastic public speakers, I want to introduce you to the most interesting people I can find.
This episode is sponsored by the lovely folks in InfluxData. If you're listening to this podcast you're probably also interested in better monitoring tools and this is where Influx comes in. Personally I'm a huge fan of their products, and I often recommend them to my own clients. You're probably familiar with our time series database, InfluxDB, but you may not be as familiar with their other tools. Telegraf for metrics collection from systems, Chronograf for visualization and Kapacitor for real time streaming. All of this is available as open source, and they also have a hosted commercial version, too. You can check all of this out at influxdata.com.
Hi folks, I'm Mike Julian your host for the Real World DevOps podcast. My guest this week is VM Brasseur otherwise known as Vicky, an expert in open source strategy and the author of the book Forge Your Future with Open Source. She's previously the Vice President of the Open Source Initiative and currently Director of Open Source Strategy at Juniper Networks. Well Vicky thanks for coming on the show.
Vicky Brasseur: Well thanks for having me Mike, I'm very happy to be here.
Mike Julian: I want to start with a seemingly simple question, but I have recently learned in the past half hour that this is more complex than it seems. What is open source?
Vicky Brasseur: Yeah, can't imagine how you learned that. No, it's a question that a lot of folks in technology think they know the answer to, but unfortunately they're usually wrong. That's because they usually don't realize that there is a legitimate definition of what it means to be open source software. It is called the open source definition. It is maintained by the Open Source Initiative. If something does not adhere to each of those 10 points on the open source definition, it isn't really open source.
Unfortunately people just sort of assume, well if my source is out there, if my source code is out there, it's open, right? Well, not really, because if you restrict it in any way or if you don't put an appropriate license on it, then people don't know it's open source. If you just put your code out there without a license for instance, it's all rights reserved. You have the copyright over that code or your company if you developed it for your company. It's all rights reserved as far as copyright and no one else can use it, unless you put a license on and that's what the license does for you. Only an open source license, one that is approved by the Open Source Initiative, that's the only kind that you can be assured actually gives you all of the things that the open source definition guarantees.
Mike Julian: What's really interesting about that is, there's always people that go around GitHub onto like the main project and say, "Hey, I noticed that you don't have this license, you should really have a license file." I'd always thought that that was just kind of an oversight, like, "Oh yeah, it's fine, it's totally open source. There's just no license. There's no license file." What you're actually telling us is that, if you don't have that, if you haven't specified what license this is under, by default it's not open source. Like, it is “all rights reserved.”
Vicky Brasseur: It is, exactly. It is all rights reserved. The best you can call it is source available. You still retain all of the copyright over that, and therefore it is all rights reserved. You retain all rights to that code, no one can use that software at all unless you give them the rights to it. That means somebody could use your software and put themselves at legal risk by violating the copyright of your software and you. If you don't put a license on it, that's what they're doing. Therefore, they are at legal risk, they can get sued and if they are running a company and they're using your software, they can't really get acquired frankly if they are using software that is encumbered by somebody else's copyright. That's why it's so important for multiple reasons to make sure you have a license on there. It really takes care of all those legalities. It's a relatively short list of OSI approved licenses, you've got the Apache and the MIT and all your GPL flavors and LGPL and AGPL and yeah. There's a bunch of them and they cover a broad swath of things. If you just use one of them, you don't have to care about the legalities, somebody has already taken the time to figure that out for you.
Professional lawyers have written these things, gotten them approved by OSI. You know they give you everything from the open source definition and you know it's legal. Just use it. It's pretty easy.
Mike Julian: You just named off a whole bunch of different open source licensing. I'm always confused when I release a project, like what should I license this under? Screw it, I'll go with MIT or Apache and call it a day, and I never really put any thought i...
About Ryn Daniels
Ryn Daniels is a staff infrastructure software engineer who got their start in programming with TI-80 calculators back when GeoCities was still cool. Their work has focused on infrastructure operability, sustainable on-call practices, and the design of effective and empathetic engineering cultures. They are the co-author of O’Reilly’s Effective DevOps and have spoken at numerous industry conferences on devops engineering and culture topics. Ryn lives in Berlin, Germany with a perfectly reasonable number of cats and in their spare time can often be found powerlifting, playing cello, or handcrafting knitted server koozies for the data center.
Links
Transcript
Mike Julian: This is the Real World DevOps Podcast and I'm your host Mike Julian. I'm setting out to meet the most interesting people doing awesome work in the world of DevOps. From the creators of your favorite tools to the organizers of amazing conferences, or the authors of great books to fantastic public speakers, I want to introduce you to the most interesting people I can find.
This episode is sponsored by the lovely folks at InfluxData. If you're listening to this podcast, you're probably also interested in better monitoring tools and that's where Influx comes in. Personally, I'm a huge fan of their products and I often recommend them to my own clients. You're probably familiar with their time series database InfluxDB, but you may not be as familiar with our other tools. Telegraf for metrics collection from systems, Chronograf for visualization, and Kapacitor for real-time streaming. All of this is available as open source, and they also have a hosted commercial version too. You can check all of this out InfluxData.com.
Mike Julian: Hi folks. I'm Mike Julian, your host with Real World DevOps. My guest this week is Ryn Daniels, co-author of O'Reilly's Effective DevOps, a public speaker and previously worked in engineering for both Etsy and Travis CI. Ryn, I hear you're working everyone's favorite infrastructure automation company now, HashiCorp is it?
Ryn Daniels: Yes, it is. I'm a working on the terraform ecosystem team. I'm going to be working on the AWS provider.
Mike Julian: You've been writing and talking a lot about this idea of resilient culture and you wrote a article for a InfoQ, which we'll link in the show notes, about crafting resilient culture, which talked about the Apache Snafu. You and I were just talking before the show about an earlier story about Postfix and Puppet and well, things exploding in your face.
Ryn Daniels: Yes, so it's a fun story with a little less of a happy ending than the Apache snafu. My first ops job I inherited two data centers that didn't even have a lonely bash script for company. I was doing everything by hand. There were a lot of dragons and nobody was really sure where are the dragons were lurking. One of the things that I was kind of put in charge of was the idea of, "What if we didn't do literally everything manually? What if we had some sort of automation?" So I got to do fun stuff like set up automated Linux installs instead of me going around carrying a USB DVD player and yeah.
Mike Julian: Definitely been there.
Ryn Daniels: Yeah, that that was ... Those were sad times. So I was starting to put together Puppet and it was mostly going pretty well. I was starting out with the what seemed like the safe stuff. And I asked the engineering team, I'm like, "So it seems like Postfix is configured a bit on these servers, but it's not running. Should it be running?" And people talked amongst themselves a little bit and they were like, "Yeah, it should definitely be running because the servers are set up to email us when something goes wrong." Okay.
Mike Julian: So clearly everything was fine because no emails were going out.
Ryn Daniels: Exactly. Exactly. So I clear this with everyone. I tell them, I'm like, "Okay, I'm going to roll out this change." And I turn on postfix everywhere. And this was my very first ops job, so we didn't have anything like a testing or a staging environment. I was really kind of playing everything by ear at that point and learning as I went. So I turn on Postfix and then a few minutes later somebody says the site's down. Like how did turning on Postfix take the site down?
Mike Julian: That's weird.
Ryn Daniels: And we kind of kind of poke a little bit on one of the servers that I was logged into and like the web server was still running. Everything looked like it should have been fine. What happened was there were eight years of emails queued up on every single server, and when Puppet turned on Postfix, those eight years of cued emails started sending all at once. And the way that networking was or wasn't configured back then, I think I just like saturated every single network link in our two data centers with all of these emails, and everyone's like, "Ryn, help, make it stop, get everything back on line." I'm like, "I don't know how to un-send eight years worth of email, folks. Like, we're just going to have to wait this out." Which is kind of what happened. And eventually, eventually all of the emails sent and shockingly, there were a lot of error emails as it turns out in this sort of environment.
Mike Julian: Surprise, surprise.
Ryn Daniels: Yeah. And after that everyone was a little twitchy anytime I mentioned making a Puppet change. So yeah, it was definitely an exciting afternoon slash couple of days trying to figure out what went wrong with automation and try and keep it from going that sideways in the future.
Mike Julian: How did your teammates react to all this? Like aside from like, "Ryn, what have you done?"
Ryn Daniels: It was, it was mostly just that kind of panic and then everyone trying to figure out what to do. People had differing amounts of visibility into what was going on. There was kind of a homegrown monitoring system that was set up that also lived in the data center, which may or may not have been very accessible during this time. Oh, I remember, I was stuck in the data center physically because nothing was configured to have a remote, out-of-band access. So most of my days were spe...
The podcast currently has 26 episodes available.