Arrested DevOps

On this episode of Arrested DevOps, Trevor is joined by Chloe Condon, Nathen Harvey, and Nell Shamrell-Harrington. Everyone on this episode has been a part of the theater community at some point in their lives. We talk about our individual journeys from theater into tech, the lessons we learned on the way, and how we leverage those lessons every day.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Bridget curated the distributed systems track at GOTO Chicago 2018 and closes it with a live panel of the five speakers: Jeff Hodges, a consultant who talked about productionizing distributed systems, Jordan Hendricks of Joyent, who talked about adding a feature to the Manta object store within its original design trade-offs, Erik St. Martin of Microsoft, who talked about Kubernetes as building blocks for distributed systems, Alena Hall, who talked about running distributed data stores and Spark on Kubernetes, and Kyle Kingsbury, who talked about testing for safety with Jepsen and recent concurrency bugs in databases. The cold open is Erik: "it's the unknown unknowns in production are what get you every time."

Bridget asks for tips on storing state safely. Kyle says do as much immutable as possible, since it solves consensus trivially: the Lamport proof says it takes two rounds only when proposals conflict, and immutable data has no conflicts. Alena adds Kyle's point not to trust the documentation, but to learn the distributed algorithms behind a system and test it against your own use case, and if it works, maybe keep running it. Jordan says to think clearly about which services are stateless and which are stateful, to keep the stateful list very short, and to keep the design simple, since "we're not very good at reasoning about these things on our own."

Bridget asks how to tell content you want to keep from content like spam. Jeff says you can accept writes without distributing them: rather than fanning out a tweet, analyze it, and if it looks suspicious by your heuristics, don't deliver it. Jeff says that's common, because "the acceptance of writing doesn't mean that you have to accept all the reads that are gonna come out of it," and there are far more reads than writes.

Bridget asks Erik how to know what to build when building systems no one has thought of. Erik says many of the pieces would have been created anyway, like schedulers, and the important thing is to build on what exists, such as Kubernetes and etcd, because that's where mistakes are made, and leveraging existing guarantees prevents repeating them and adds less complexity.

Bridget asks Kyle whether organizations learn from other people's failures. Kyle says testing under failure conditions like partitions and clock skew has been adopted rapidly, often after a customer reports a production problem. There's also a settling-in period where a system must run at scale with weird inputs to solve corner cases, which a greenfield project doesn't have. Jeff says a talk from five years earlier had to be updated, with the future idea of Docker, Mesos and data center schedulers now being reusable pieces, and "Fortunately, we still screw up the same exact things."

Alena says there are engineering errors and algorithmic errors, and that specification languages like TLA+ describe the algorithm as mathematics to catch logic errors before engineering ones. Kyle sees a continuum of safety from proof to implementation, and gives the example of a missing fsync invalidating a correct proof. Erik notes that proofs rest on known failure modes.

Jeff says feature flags still lack a robust implementation, because they integrate with your deploy processes and user models, though metrics systems like Prometheus and Stackdriver have become reasonable, and dark rollouts need both. Even Let's Encrypt had to build its own. Jordan says Joyent's team is wrestling with how Manta behaves at scale, including whether every service for an instance will fit in DNS packets, and having to think about infinite scale from the beginning. Jeff adds that there was a patch for an old Samsung mobile platform that didn't do HTTP as expected, kept for four years.

Erik is excited about chaos engineering, and notes how shared suites can teach people failure modes like exhausting file descriptors and local ports. Jordan adds running out of TCP connections, Kyle says "What up, time wait?", and Erik explains connections closed in batches can be handed back to the operating system faster than it makes them reusable. Kyle mentions a major cloud provider that occasionally swapped pages of a VM's RAM with other VMs', which Kyle says is fixed. Jeff says anyone running about 20 machines will one day stare at a TCP state machine diagram in lsof wondering why time wait is crying.

Bridget notes Alena's cautions about Spark on Kubernetes. Alena says the demos work but production may make you the first person to run it, and gives corner cases, such as a driver pod dying as a single point of failure, and a Cassandra-to-Spark connector that didn't support Spark 2.3. Erik recalls a DEF CON badge maker's comment that talks aren't exhaustive training: they give hooks to research further, so don't feel bad if you can't follow everything in 30 minutes.

Erik says "it doesn't matter how many distributed systems you build or how long you do it for, it's still hard." Erik recommends understanding what consistency guarantees your use case needs, and building in backpressure, circuit breakers and idempotency from the beginning. Jordan says "everything fails all the time," and suggests bounding request time, avoiding compound operations, and remembering that adding instances won't fix a memory leak. Alena recounts Leslie Lamport telling a TLA+ workshop, "It took me 20 years to get all of the pieces," and recommends building observability into as many pieces as possible to find new unknown unknowns. Kyle points to Jeff's post Distributed Systems for Youngbloods and Kyle's own class, and says "Assume your clocks are garbage. Assume your runtimes will pause." Jeff takes the social ground: distributed systems involve more capital, teams and organizations, and without a migration plan to get people onto a new system, it goes nowhere, since there is "a technical set of problems embedded inside of a social space." Bridget concludes that distributed systems, like Soylent Green, are made of people.

Bridget curated the distributed systems track at GOTO Chicago 2018. In the last timeslot of the day, she gathered all the speakers from the track to discuss their topics.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Matty talks with Jay Gordon, a developer advocate at MongoDB for about a year, in what Matty calls the big dudes with tattoos and beards episode. Jay started around 2000 building small websites, spent 2002 to about 2010 as a sysadmin at DataPipe, and then worked at Courses, BuzzFeed and DigitalOcean before moving to MongoDB to get out of an on-call role, starting as a technical account manager and then moving into advocacy. The cold open is Jay saying that many people in DevOps roles prefer Chipotle as a fast food option.

Jay relays a line from a coworker, Adrian Howard, and Mary Thengvall that "developer advocacy is kind of like the good fat on certain companies, like avocados." Matty notes that advocates, evangelists and DevRel folks have very different jobs, and that a survey of the field showed little consistency, with anonymous compensation numbers ranging from $5 to $1 million. Jay adds that advocacy rarely has definable metrics, unless you work at one of the biggest companies, which counts how many people each advocate talked to. Matty says the effect can be indirect, and at PagerDuty and MongoDB the audience includes developers, admins and architects. Matty calls the role a full-duplex connection between a user community and a product team, and describes hearing from meetups what people say in words that differ from the product team's.

Jay tells of an open space MongoDB held for users at its Chicago event the day before the conference, where MongoDB's VP of engineering sat in and by the end a Jira had been written and a pull request submitted and merged. For Jay, that is a sign of success: people who use the product told the company what didn't work, and a correction followed, though "just a minor, minor change."

Asked what was hard, Jay says learning marketing and writing, including grammar, and the public face of a company. Jay compares it to an auto mechanic for 20 years who decides to start a newsletter about being auto mechanics. Jay still looks at the phone waiting for something to do, though it's no longer a pager world.

Matty calls it a phantom limb and says it was a revelation to leave the phone on another floor after moving off on-call. Matty describes Nathan Harvey leaving the laptop home on a family vacation and then the phone, only telling the family once on the plane, and a summer trip to northern Minnesota with the phone turned off in a lodge and Slack uninstalled. Matty's point is "there's no such thing as a developer advocacy emergency," and that unplugging is "a muscle that you have to exercise." Jay admits to pressuring to get everything done before vacation, which doesn't work, since the work will be there no matter what.

Matty says a vacation can be planned for, finishing things or leaving them in a starting state, and a coworker declares email and Slack bankruptcy: if it was important, people will reach out again. Jay says that's a tough move at some places. Matty's answer is "you have to train the system": tell people you'll check email once a day, or that mail during vacation is deleted. Matty adds that this is a privilege not everyone has, and Jay says changing how you communicate without telling your team is the antithesis of DevOps. Matty compares it to getting a TiVo and feeling compelled to watch everything.

Jay wanted to talk about the memcached DDoS on GitHub, a 1.7 terabyte attack from UDP reflection. What troubled Jay was the reach of GitHub, for businesses, teams, students and kids learning to code, and that so many systems weren't firewalled and providers weren't blocking ports by default. Memcached, Jay says, is like a dumb protocol you can exploit, and Matty adds that you have to go out of your way to open all the ports on a cloud instance. Matty says people think a dev box doesn't matter, but compromising small things affects neighbors, in the same way two-factor on Facebook matters because of OAuth. Jay says fast and loose startups leave the security team to clean up after a unicorn shitting rainbows, and "failure is a great teacher."

Matty repeats that people punished for mistakes won't make fewer of them, "they're just going to become really good at hiding them," as with a leaked key nobody reports. Jay once took down a major site for a company and didn't hide it, which Jay credits partly to seniority and privilege. Matty says leaders set the expectation, even by making fun of a junior person in Slack. Matty recalls joking with a friend in Chef's chat about a Knife bootstrap flag, until a colleague pointed out that newer people only saw the joke as mockery: "you are, as a more experienced member of your team, and you're setting an example."

A listener asked about the role of fast food in DevOps. Jay says every topic at a conference gets compared to DevOps, then repeats the Chipotle line. Matty disagrees, since Matty dislikes cilantro, and picks In-N-Out as the most DevOps, because it's simple with lots of hacks, and you adjust for yourself without cargo culting. Jay orders double-double animal style with fries well done, and lives in Manhattan, where delis make fast food matter less.

Matty named the episode after Jay's Fugazi posts, and both give picks. Jay suggests the Void side of the Faith/Void split, Fugazi's Cassavetes, and a MongoDB metal Slack channel playlist with Entombed, Carcass, Electric Wizard, Nails, Morbid Angel and Snapcase. Matty's coding playlist has Black Flag, Misfits, Minor Threat, Dead Kennedys, Slayer and Social Distortion.

view on Spotify

Matt will be at the devops meetup in MSP on March 20 and then home for a bit. In April he'll be at DrupalCon in Nashville, Devopsdays Des Moines, and GOTO Chicago. See mattstratton.com/speaking for more.

Lots of devopsdays: https://www.devopsdays.org/speaking/

Bridget talks with Gareth Rushgrove, who curates the DevOps Weekly newsletter and is a product manager at Docker, after time at Puppet and before that the UK government. The episode uses the newsletter as a framing device and moves to conference speaking, Docker and open source business models, and why Gareth thinks context matters more than chasing technology. The cold open is Gareth: "I can force myself to think about it from your side."

DevOps Weekly is 7-plus years old, about 360 issues, with 25,000 to 26,000 subscribers, one of those "side projects that got out of hand." Gareth's colleague Dean Wilson went to the first devopsdays in Belgium and came back saying Gareth would have liked it, and Gareth went to the second, in Hamburg. Around the same time Gareth was doing Ruby and liked Peter Cooper's Ruby Weekly, a curated weekly email, and decided to collect the things already being read and share them. The first issue was issue 0, and early on it wasn't on Sundays or truly weekly. Sunday stuck, and the regular habit helped Gareth and the readers, who read it with coffee in New York or first thing Monday in Australia.

Gareth jokes you should never put the cadence in a name: "including the cadence of something in the name is not a great idea," since a fortnightly version would need a rename, though the name also creates social pressure to keep publishing. Growth has been organic, with no advertising, plus an occasional bump when someone mentions it at a conference.

Bridget asks how Gareth's varied jobs shape the opinions. Gareth has been the first employee of a founder-led company, worked at agencies, been an in-house developer at a radio station, worked for government, and at two growth-stage technology vendors. The downside is less depth in any one of those, but the perspective helps. Gareth doesn't claim to be unbiased and will link to own work, but includes things Gareth disagrees with, using "is it interesting?" as the barometer, and doesn't cover industry news, acquisitions or hires. Bridget notices the blurbs never say who wrote an article or where they work, and Gareth says that's deliberate: "the thing is the interesting part, not the personality." Containers, serverless and Kubernetes each show up in the newsletter's volume over time, which reflects Gareth's own interests.

Gareth says DevOps as a banner has been "purposefully broad from the start," and the conversations at devopsdays have moved on, which is good, and the newsletter changes as the conversation does.

Gareth did meetups before the newsletter, starting with a local mailing list in Newcastle after asking a Brighton organizer how they got theirs going, and found that organizing means you always need a speaker. Travel was a bonus Gareth couldn't otherwise afford. Gareth values that devopsdays recordings, slides and videos create a large corpus of shared content. At vendors, content from practitioners already trusted by a community is more powerful than pure marketing, and at the UK government, "Us being super visible was strategy, not tactics," for recruiting and openness. Gareth also uses talks as a design tool, since acceptance validates interest and the audience gives feedback, and Bridget notes people from Chef or Puppet who propose a configuration management topic and then let the room talk.

On capitalizing DevOps, Gareth says "Patrick didn't camelCase it, therefore no one else should do," though Gareth has lost that argument often and once restored lowercase in a UK government publication that someone later changed.

Bridget recalls the Velocity Debate Club in Amsterdam in fall 2015, arguing whether containers and microservices are good for business, and switching sides halfway through. Gareth says taking both sides builds empathy, and that for someone trying to win an argument on the internet it doesn't matter, while for someone trying to learn it does. On containers now, Gareth says it's a messy period where it isn't about containers: Docker is trying to build a business helping large organizations modernize technology, infrastructure and applications, with gnarly problems that aren't about containers. Gareth joined as a product manager to see how the pieces work better together.

Bridget asks how a vendor decides what is open source, such as Moby versus Docker. Gareth, new at Docker, says "open source is strategic, not tactical," and the question is what gives the most benefit back, sometimes direct revenue, sometimes a large user community. Gareth points to recent posts by Luke Kanies, who founded Puppet, about licensing decisions, and notes earlier open source companies like MySQL AB were acquired before the industry could learn from their data. Now Hortonworks, Cloudera and Mongo are public and will produce data about the model.

Gareth says the microservice conversation is about people, teams and independence. Technically it adds latency and operational complexity, but it reduces complexity for an independent team: "You've moved complexity around." Bridget calls that the conservation of complexity. A 20-person company changes team structure all the time, but an organization that's been around for decades, like the UK government, can't easily reorganize to use technology the way fluid companies do. So the vendor goal is getting benefits to organizations that can't adopt early. Gareth adds that "everyone is a software company" is easy to say, but Google has about 40 to 50% of headcount in R&D, and there aren't enough software engineers to staff the FTSE 100 that way. Large organizations are also spending far more getting onto virtualization than on serverless or unikernels, and both can happen at once.

Bridget asks where things are going in early 2018. Gareth: "I'll be boring and say context." If you chase Kubernetes, Docker or serverless, you'll always be chasing the next, and should ask what business problem and what context you have. Gareth would also like the industry to get better at technology that stops changing, since large organizations have far more technology running than developers working on it, and asks what it would take to build software that lasts ten years. Bridget jokes that becomes a talk called Built to Last.

Gareth is helping run an ethics track at QCon London with Anne Curry, arguing ethics for technologists is becoming a pressing issue since the industry is increasingly powerful and lacks ethical education, and that conferences are a way to reach people. Gareth's pet project is Kubeval, a local validator for Kubernetes configurations.

Bridget discusses Devops Weekly with Gareth Rushgrove.

Postbox

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Definition of a hot take - “a piece of commentary, typically produced quickly in response to a recent event, whose primary purpose is to attract attention.”

Charity Majors (Honeycomb), Jill Jubinski (Fastly), and Eric Sigler (PagerDuty) sit down with Matt to have some fun with a few great DevOps "hot takes" and talk about what everyone is doing wrong.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Bridget and Matty sit down on February 7, 2018 with Andrew Clay Shafer, whose Twitter handle is littleidea, for the first time without an audience or other guests. Andrew has been on roughly 5% of the show's episodes, including the Kelsey Hightower platforms episode, three live devopsdays Minneapolis recordings and the GOTO Chicago episode with Bryan Cantrill. Andrew notes that Bridget once had expense reports approved by Andrew. The cold open is Andrew: "It will be obvious when it's too late."

Bridget asks for Andrew's version of the Agile 2008 gathering that led to devopsdays. Andrew calls that conference formative for reasons beyond Patrick Debois: it is where Andrew met two people who influenced Andrew, one of whom introduced Lean, while Andrew was working on Puppet and talking about agile infrastructure. There was a board of index cards for topics, and Andrew posted one about Puppet and agile infrastructure, then showed up late, because a conversation about moving from Scrum sprints toward flow, Kanban and Lean was too absorbing. Patrick had already written a paper on bringing agile practices to infrastructure and sysadmins, with small chunks and standups, ideas Andrew had been articulating from a development background. Andrew says Puppet let you bring tools and practices honed on software development to infrastructure problems.

Andrew observes that people like to argue over the meanings of words, as in the new term GitOps, when "everything's been Git-centric" from a Puppet perspective for ten years. Andrew says there's no need to argue whether it's new. Andrew spent three years on a debate scholarship, where you have to disassociate your ego from arguments, since you're assigned positions arbitrarily. Andrew says people get defensive because "they're in love with their identity" and attach it to their tasks and even the definitions of words.

Matty recalls a recruiter's post asking why DevOps engineers are so hard to find, a joking tweet that upset someone, and a long LinkedIn post about it that reached about 80,000 people in four or five days. The argumentative response came from someone who built a consulting practice on DevOps engineers being a thing. Andrew says it's worse when it's livelihood, and Matty adds that ITIL takes years to implement, so telling the person who did it they're wrong is hard. Andrew adds, speaking as the attached ego, "it's also my artwork."

Andrew says the conversation about organizational learning has been sad, because it's too meta for most people, who want paint-by-the-numbers tools, and because "most organizations don't actually want to change." They get sold transformation in waves, most rooted in different metrics on top of Taylorism. Andrew has seen two archetypes of success and many failures, quoting the line that all happy families are alike: "the unhappy ones, the unsuccessful ones are all different." Success needs an impetus, either a visionary with social capital at the highest level, or an existential crisis for the business.

Andrew adds institutional theory and isomorphism, meaning organizations ending up with the same shape. Early adopters chase competitive advantage, and later adopters are motivated by legitimacy, because a trade magazine or Gartner said a buzzword is what legitimate organizations do. Andrew used to use the cargo cult metaphor, and now leans toward "they actually don't believe in the religion," and since they're not doing it for advantage they don't get one. Matty says the outcome such organizations want is to say they're doing DevOps or ITIL, not better uptime, and asks everyone, from CIO to the junior sysadmin changing backup tapes, how their company makes money. Andrew was always baffled by colleagues who focused on technology details and disregarded why the system existed, and says in an organization you either build or sell.

Bridget asks what's new to pay attention to. Andrew says not everyone needs to twiddle cgroups, and that designing an organization is like designing a web service, with inputs, outputs and throughput, noting distributed systems papers don't presuppose the nodes are computers. Matty cites Cindy Sridharan's post that everyone is not ops, and says you can't know everything about Kubernetes, and someone should know cgroups.

Andrew describes the arc from Puppet, wrangling mismatched data center boxes, to the cloud-native world that eliminates complexity by collapsing variation: identical racks, standard operating systems, fewer runtimes. Andrew recalls a talk by Jeff Hodges at Twitter arguing "polyglot is bullshit," and says people excited that Cloud Foundry and Kubernetes can patch operating systems and collect logs forget that some teams did that with Puppet and Chef years ago, only each had to solve it alone. Bridget asks if the future will be more evenly distributed, and Andrew says no: "the consolidation below the value line is going to continue ever upward," and value is created above it.

Andrew is not always a fan of foundations, which can act as kingmakers and invite projects to chase legitimacy, as with OpenStack and possibly CNCF. There is a Cambrian explosion and then a contraction, so Andrew's advice is to experiment, fix the obvious and not go all in until it settles. Bridget notes large enterprises have pockets of everything, and Andrew says sometimes one group picks one because the other didn't.

Matty says a sales leader asked why companies have salespeople, and the answer was that humans are irrational, and recalls a blog post from Michael Hedgepeth saying NCR chose Chef because of its pre-sales experience, not steak dinners. Matty, now at PagerDuty, has spent time at Chef. Andrew says it's usually legitimacy, not the best solution, and people buy a story they can see themselves in. Matty says people think they're snowflakes, and that Sasha Bates has said every snowflake has six sides.

On a morning Twitter argument about tools, Andrew says the two sides talk past each other: tools aren't enough is not tools don't matter. Andrew recommends Team of Teams, about the Joint Task Force in Iraq, where each team had the qualities you want but there was a command of teams and no horizontal collaboration. Andrew says that parallels DevOps silos: you don't want to tear down the silos or functional specialties, you want to leverage each group's context, and "The mission is not to configure servers. The mission is not to develop software." Andrew says the book argues what must change is not how workers do work but "It's how we manage people," and that one of the dimensions of organizational learning is participation in dialogue regardless of rank.

Bridget asks about Red Hat buying CoreOS and CloudBees buying CodeShip. Andrew expects more consolidation: "You're not gonna have 2 dozen Kubernetes startups that, like, survive," and thinks CoreOS's team and etcd fit Red Hat. On the CNCF chart, Andrew says it will be obvious what to adopt when it's too late, with patterns emerging and dominant ones legitimized. Bridget notes KubeCon proposals for projects with two contributors. Andrew describes gold rushes and witch hunts: people rushing for gold that may or may not be there, and tribes uniting against something. Andrew sees people saying DevOps is over because of containers and serverless, but "if you actually think about it as a systems thinking optimization problem, then DevOps will never die." The two laugh that getting expense reports approved is simple but not easy: "It's simple. It's not easy."

Andrew plans a book on a five-element model of DevOps, essentially CALMS, to make the meta ideas more actionable, and has joined a reverse book club that meets every two weeks.

Bridget and Matt discuss the past and future of tech with Andrew Clay Shafer.

Guernica by Picasso

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Matty and Trevor host a show on entering DevOps after a different first career, with guest host Sonia Gupta, who spent eight years as a lawyer in Louisiana (public defender, prosecutor, assistant attorney general) before moving to Denver, attending the Turing School of Software and Design, and starting as a software developer at Ibotta. The episode was suggested by Michael Hedgpeth, who is about to lead application operations and DevOps for NCR's hospitality division. The guests are Annie Hedgpeth, a cloud automation engineer at Tenth Magnitude, and Megan Bohl, who has just finished a DevOps and continuous integration course and is job hunting.

Annie has an art degree with an emphasis on film, started as a casting director, then stayed home with three boys for about ten years, blogging and running a home decorating business. Coming back in the late 30s, Annie had nothing of interest on a resume, didn't want to start at the bottom, and had lost the film network. Michael kept suggesting technology and Annie thought that was crazy, since "I have a freaking art degree, man." Then InSpec came along, which is written for non-coders and is human-readable, and Michael asked for a three-week experiment. InSpec, Annie says, "lowered the barrier to entry into technology," because writing controls to check that things are as they should be teaches IT in an inverted way, and then Chef teaches remediation.

The hardest part was belief. Annie had to get over imposter syndrome and the feeling of being the dumbest person in the room, and learned that not everybody knows everything: "It was just my own psychology, I guess." Annie's InSpec tutorials got buzz on Twitter, and Matty asked Annie onto the podcast at ChefConf 2016 about six weeks into technology, which Annie finds too cringy to relisten to. Trevor worked at Tenth Magnitude then, and that is how Annie got the job.

Getting the job wasn't the endgame. At a consultancy driven by utilization, Annie knew a narrow slice, cookbooks and InSpec profiles, and "I was on the bench a lot," learning on the side until a project translating ARM templates to Terraform. A colleague, Scott Nowicki, spent so many hours training Annie that Scott's own utilization took a hit that quarter, which the company treated as an investment, and Annie was much more independent afterward. Michael had warned that it takes three to six months to be effective in a new role. Annie says new people have to be scrappy and get help where they can, like the Chef community Slack. Annie adds that the scope was the surprise, since at first it was all cookbooks and InSpec, and only over two years did the context of MS SQL, PowerShell, ARM templates and Jenkins fill in.

Sonia asks how prior careers inform the new one. Annie says casting and parenting taught reading people and empathy, "the greatest asset that I have from both of those things," while home decorating taught planning projects. Michael adds that people with art degrees had to work extremely hard to succeed, unlike the business school, and that what drew Michael to suggest technology was Annie's work ethic: "There's a lot of entitlement in IT."

Megan spent 15-plus years in supply chain operations, worked as a liaison between IT and operations and loved it, but "never really felt like I was smart enough to do that kind of job," believing a computer science degree was required. After a visit where Annie talked about Ruby and Chef, and three or four conversations with Annie and Michael, Megan started self-teaching Ruby online through Udemy and Udacity. Annie then asked Megan to join the organizing board of devopsdays DFW, as sponsor liaison, which led to job interest and a scholarship from the organizers to Tech Talent South's 12-week DevOps and continuous integration course. Megan says the DevOps crowd's culture of sharing is "so open and, I don't know, inspiring," unlike the business world. Michael says "I don't think that anybody could make a career transition without a strong mentor."

Trevor asks about first degrees: graphic design, marketing, English and American literature, and management information systems, and Matty's unfinished degree was in theater directing. Matty notes that privilege helped get opportunities without a degree. Megan's course was small, and Megan worked through tutorials ahead of the class with the instructor, which Michael says is the kind of person to hire from a code school. Sonia's school was a seven-month program of 60 to 70 hours a week, in four six-week modules of Ruby, Sinatra, Rails and APIs, and Sonia says the biggest takeaway over self-teaching was working in a team. The surprise at work was that "everything is brownfield," navigating code from engineers who've left, and understanding the whole workflow from mobile app through AWS and Lambda, which school doesn't cover. Annie adds that newcomers can't tell whether confusing code is elegant or just bad.

Michael says what was hard for Annie was "being comfortable with being just okay at something," since the world of technology is too large, and Matty adds that's the T-shaped idea: nobody knows everything deeply.

Annie coined the term DevOps native: never a sysadmin, starting in 2016 and learning only automation, configuration management and working with other teams, taught by people including Michael, Christoph Hartmann, Nathan Harvey, Dominic Richter, Trevor and Scott. Michael says job postings ask for ten years of traditional IT experience, when many of those people won't code anything, and Matty jokes "Those 10 years of experience come with 8 years of doing it wrong."

Michael and Annie describe DevOps Jumpstart (devopsjumpstart.org), which grew from the devopsdays DFW experience. It aims to sponsor people from underrepresented groups in technology with mentoring and financial help, such as childcare or scholarships, and build a repeatable pattern. Annie says the goal is life-changing for single parents or people who grew up in poverty who have the drive.

Sonia's advice: "Don't be afraid. Actually, be very afraid, but still do it." Annie recommends the book Feel the Fear and Do It Anyway.

What is it like to change careers and get into tech later in life? Annie Hedgpeth and Megan Bohl tell their stories.

This episode also features special guest Sonia Gupta!

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Bridget talks Go with two teammates, Brian Ketelsen and Erik St. Martin, both cloud developer advocates at Microsoft. Brian had just helped form a new team focused almost entirely on open source, and Erik had recently joined it. Brian started in IT at an ISP in Wyoming in 1993, billing customers on 3x5 index cards until automating it in Microsoft Access, and has since been a DBA, done data warehousing and been a CIO. Erik worked on web development, spent years on Disney's e-commerce platforms, and then moved into distributed systems and databases, with a side interest in security that began with writing no-CD cracks for video games. Erik says neither of them mentioned Go in their intros, though it has been a big part of their lives for seven or eight years. The cold open is Brian: "Writing a book is very similar to having a baby."

Brian saw the Go announcement in 2009 and played with it, and six to eight months later had a problem that needed concurrency: a big Ruby on Rails monolith that wasn't meeting its SLAs while calling many data sources. Go "blew the doors off of what I expected out of concurrency," and Brian has been a fan since. Erik remembers a group at Disney in 2009 playing with it without seeing a selling point. Two years later, Erik was job hunting, interviewed at Brian's company, and was asked to maintain a service Brian had written, too busy as CIO. That meant learning Go in the days of makefiles and a language changing about weekly.

Brian explains that before 1.0, releases were numbered like R56, which was the first version they put in production. The team shipped go fix, which rewrote old code to the new syntax, and Erik recalls only one case needing manual fixing, when rune was introduced. Brian's view is "ship it with a tool like go fix." Erik says that care for the developer is part of the love. Since the 1.0 API freeze, "anything that compiles on Go 1.0 will compile on Go 1.10."

Bridget asks for the ops-audience version. Brian says Ruby and Python historically execute on one core at a time, and Erik adds the global interpreter lock, so only one thread interprets code at a time, though blocking I/O can run in parallel. Go has goroutines, which Brian calls lightweight, low-memory threads, with many running on one OS thread. Erik adds that the concurrent code is easier to reason about, since threading was added to many languages after the fact while Go designed around concurrency from the start. Bridget calls that concurrency as a first-class citizen, and Brian asks if they can record that and put it on the GopherCon website.

Brian and Erik and a third author wrote Go in Action, published in November 2015. Since the API froze at 1.0, Brian says the book isn't out of date at all after three years. Brian has time booked that afternoon to finish an O'Reilly proposal with a co-author, which Brian won't describe, and would pick an otter for the animal. Erik says the original book began with wanting to tech review a stalled Manning Go book, and being asked if anyone they knew would write it: "screw it. Let's do it. How hard could it be?" Both say writing is like a baby: after swearing never again, you forget how painful it was.

The pair co-host Go Time FM with Carlesia, which came out of a relationship with Changelog, who invited them on to talk about GopherCon and later wanted to produce other podcasts. Erik says both groups wanted a Go podcast, and they merged. They want three co-hosts each time, and when one is missing, a guest sits in, such as Ashley McNamara, Kelsey Hightower or Scott Mansfield, and when more than one is out, they skip the episode. Erik says the format is "like we're all sitting around just having a conversation at the dinner table," with only loose notes, and the show had just reached its 65th episode.

Asked how GopherCon started, Brian says "It was a dare." Brian and Erik had been saying for two years that there should be a Go conference, and someone on Twitter said they should run it, so Brian registered a domain name. That was mid-2013, and they had hoped for 200 or 300 people and sold out at 750, needing to rearrange the venue. They had reserved a hack day for people waiting for flights and expected 100 people, but far more stayed, leaving them wondering how to feed everyone. Hotel attrition in the first years meant close calls, with Erik saying "Brian, we're gonna lose our houses, man," and the second year lost about $10,000, before hiring Convention Designs in Colorado. They remember committers to the Go project stuffing 750 swag bags on a production line in the Denver Marriott.

Attendance was about 1,200 in 2015, 1,400 in 2016 and just over 1,500 the previous year including staff, with about 1,800 expected. The dates were August 27 to 30 in Colorado: a workshop day, two days of talks and a community day. The CFP runs on PaperCall, which Brian says filled a hole in CFP management, and Erik recommends the community day, with a room of electronics programmable in Go and a Go contributor room last year where the Go team helped people get first patches in. Brian mentions a post saying conferences are dead, and agrees that loosely structured time to network matters as much as talks. They also lend the GopherCon name to events outside the US, provided there's a code of conduct and no selling speaking slots.

Bridget asks whether Microsoft running a conference for a Google-born language makes it a corporate effort. Erik says from the beginning it was community-first: "we won't sell a speaking slot," and the conference has lost sponsors over it. GopherCon is run by Gopher Academy, which employs Erik and Brian legally, and Microsoft's role is letting them do it on company time. Brian says the interview made clear Microsoft wanted Brian "not in spite of the fact that I ran GopherCon, but because of it," and says "there's no forced shilling." Erik's reaction to the arrangement: "Pinch me."

The new team's first project is the Virtual Kubelet, an interface anything can implement to look like a node on a Kubernetes cluster, be it a container, a virtual machine or a bash prompt. Brian says the Hyper.sh team built one to run virtual machines under Kubernetes, and an Amazon group is also working on the spec, with the goal of tools useful beyond Azure and of patching other projects to work better on Azure. Erik wrote a blog post explaining it, and asks people with a project to come say so.

Bridget and Erik also discuss Ashley McNamara's Gopherize Me, which began with Ashley making gophers for Brian and Erik and, Erik says, became a tool built with Matt Ryer in a day.

Bridget discusses all things Go with Brian Ketelsen and Erik St. Martin.

Artwork credit: Ashley McNamara's Gophers art repo on github

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Matty, Trevor and Bridget are joined by Joe, the editor, for the 2017 year-end episode, which doubles as a listener AMA of "anything we're willing to answer." The first half covers favorite episodes, stats and what everyone did in 2017, and the second half is Joe reading questions from listeners. The cold open is Bridget, on the recurring complaint about audio: "it's not the microphone, it's my stubbornness."

Bridget's favorites are the live recordings at GOTO Chicago and the one-on-one fireside chats. Matty agrees, and recalls that the DevOps track day at GOTO Chicago meant recording six episodes in a row, which Bridget vows never to do again, along with the live call-in show with Nicole Forsgren and the fireside chat with J. Paul Reed in episode 97, an episode the two had been trying to record for years. Joe, who edits, likes live episodes because people in the same room get visual cues and there are fewer awkward pauses.

Matty's numbers: about 22,000 website visitors, up from 16,000, about half from search, and Twitter still at 6 percent. Listens were about 263,000, up roughly 30,000 a year for three years, with about 750,000 total episode downloads. Matty's usual caveat is that download counts don't say anyone listened, though Apple's podcast app will now report minutes listened. The most listened-to and most watched episode was Old Geeks Yell at Cloud with Andrew Clay Shafer and Bryan Cantrill, and Matty notes that Bryan's episode was also the most watched video last year. The hosts disabled comments on the site, since nobody used them and Twitter is the place to talk.

Bridget started a developer advocacy job at Microsoft in September, kept scaling DevOpsDays to 51 events on 6 continents, and gave far fewer talks, on more program committees. Matty became DevOps Evangelist at PagerDuty at the beginning of December, a former sponsor of the show, after realizing that much of what Matty did in spare time had become a job. Matty spoke at DevOpsDays Denver and ChefConf, visited the first DevOpsDays Hartford, and ran Chicago, calling the result acceptable. Trevor wrapped up a first year at Chef, spoke at DevOpsDays Dallas, and moved back to Chicago. Joe hit 20 years at the same employer, which has changed names and owners, and took trips to Iceland and on a cross-country drive in an electric car. Matty also recommends an escape room run by a friend.

Matty got the idea from the Go Time podcast's AMA episode and was glad nobody asked anything too personal. Joe reads the questions.

What does a developer advocate do? Matty, new to it at PagerDuty, describes a bidirectional job: communicating to the community and customers, and bringing what's learned back to product teams. That's hard for PagerDuty because "it's really hard to do feature discovery in a product that you use during a firefight." Matty also wants to develop practices, like incident command, that are good for the community whether or not they use the product. Bridget sees advocacy as being the voice of the customer inside the company, which doesn't fit well under sales or marketing. Matty adds that Mary Thengvall is writing a book on developer advocacy.

How did you come to work for Microsoft? Bridget has never owned a Windows machine, and got a Mac with a Microsoft asset tag for a job in Linux and containers advocacy on Azure. Organizations that have been around a long time change, and joining at the right time to shape that is exciting. Bridget points to Azure Container Instances, Kubernetes work and the Virtual Kubelet, with colleagues hacking together in Austin before KubeCon.

Planes. Bridget recommends Michael Cote's podcasts, since they're soothing and fine to fall asleep to, and Matty picks books already read, with a sleep timer and a bookmark, and recommends the audiobook of The Goal done like a radio play, a pick Trevor shares. Trevor plays Nintendo Switch on planes. On drinks, dehydration is a rookie move, and Matty's pick is ginger ale, where you ask for the whole can. A side trip into small conferences, including a sci-fi convention in Fargo and one in Cardiff, runs long enough that Matty predicts a drop-off in the analytics.

What is Pete ChessBot's purpose? Matty registered it under the table at dinner with Fletcher Nichol and Pete Cheslock at the Agile Conference in Orlando, then forked a Markov bot. Its purpose is to cause mirth and to confuse people: someone once talked to the bot for an entire event while trying to reach Pete.

Can you buy better microphones? The hosts have good microphones; the guests are the problem, and so is Hangouts, which Bridget keeps as the lowest common denominator while Matty wants double-ended recording, where each person records locally and uploads as they go. Matty's example is the episode with Paul Reed, at high quality, and another with a guest in China on AirPods that came out better than this one will. The catch is that double-ending can't do video or livestreaming. Matty says every episode gets about 7,000 listens and maybe 200 views, though YouTube has close to 1,000 subscribers and Bridget says those viewers come up at conferences. Joe counts 18 episodes released, 10 of them recorded away from home, and notes that the most popular episode of last year was recorded on Bridget's iPhone. Matty's summary: "compromise is when you discuss things until you reach a point when nobody is happy."

Advice for new graduates. Trevor says go to meetups and "don't blindly accept the status quo," like a two-week change freeze nobody remembers the reason for. Matty pushes back: new people should learn context first, and "challenging to me is more about asking questions," since a fresh set of eyes "should be questioning, not correcting." On mentors, Matty says to find someone who is where you want to be in ten years, to ask them to talk about their history over coffee, and to "never do anything that looks like you're asking somebody to do more work." Trevor adds asking colleagues for feedback right after projects and meetings.

SRE versus DevOps. Bridget says SRE is a job title and DevOps is a cultural practice: one is a job you do, the other is something you do at that job. To get into either, Bridget says, you'll have to learn on your own, and Matty suggests volunteer work, adding that your first job will not be SRE at Google.

Hot tools with zero ops headcount. Bridget reads hot as trendy, says picking tech because it's trendy is "great for resume-driven development and terrible for your employer," and cites Charity Majors: the best tool is the one you don't need and the next best is SaaS. There's a SaaS for canary releases, for instance, through feature flags. Matty adds that if a tool needs someone who knows ops and there's no ops headcount, don't use it. Bridget's advice is to "use the minimum viable complexity to get you there," and notes that Facebook scaled on a PHP monolith.

Ops people afraid of automation. Matty says that "automation is not a headcount reduction practice," and suggests asking what the sysadmins would rather be doing and how big their backlog is. Fear of automation is also a chance to listen: automation run through tests is less risky than manual change, since "humans will do it differently every time because we're fallible meatbags." Trevor adds small projects to show leadership the value. The last resort is "change your job or change your job," which Trevor first heard from Steve Murawski and Matty traces to Nathan Harvey's talk.

The episode closes with upcoming CFPs, the discount code, and a long aside on sticker inventory.

Matt, Trevor, Bridget, and Joe discuss 2017.

Past year-in-review eps:

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Discount codes: ADO2018 for 20% off lots of devopsdays, 10% off ChefConf

Matty and guest host Sasha Rosenbaum, who says this is probably a fourth appearance on the show, record at devopsdays Chicago 2017 about work-life balance, with two of the event's speakers. Katie opened the conference with a talk on devaluing hard work, and works at HCSC in Chicago after starting as a DevOps consultant with no technical background. Bill Weiss, a security architect at Puppet who helped run devopsdays Chicago for its first two years and now helps with Portland, spoke on security not fearing DevOps, which Matty admits had nothing to do with work-life balance, but Bill has a life. The cold open is Katie's line about coming back from a long vacation to 1,000 emails, and Matty's reply, "Sure you can" delete them.

Matty says the theme of Katie's talk was hero culture: rewarding extra hours gives people no reason to work more efficiently. Katie says organizations assume automation is attractive, and people outside the conference room don't care what you call it. Bill says people want to do what looks immediately right, like spending 16 hours beating on a problem, and that a weekend of miserable firefighting is a good thing done that shouldn't have been necessary. Sasha would go further and say it's a bad thing when recurrent, noting companies give double incentives, praising collaboration and efficiency and then a prize for the most hours worked. Sasha adds research that productivity stops after about 55 hours a week.

Matty recalls Katie's line that no one gets claps for working two hours on Friday. "You get claps for working 10 hours on Friday." Matty describes a sysadmin who worked 70 hours every week, and whose review said the person "takes our site's uptime personally," which Matty meant as praise, and now sees as putting responsibility on the human instead of the system. An old office poster asked "how does it feel to save the day every day? Not all superheroes wear capes." Sasha notes a burnout talk that same day, and that email, laptops and remote work make around-the-clock demands easier. Katie adds that a manager is likely to like whoever answers the phone on Saturday, even while everyone says to have a good weekend.

Matty recalls an interview with a CEO who said the interview process includes a Saturday call to see whether candidates answer, which Twitter destroyed. Matty thinks that direct toxicity is less common than the unintentional kind, like a manager sending email at 10 at night, even if that manager would rather you watch a show. Sasha says messaging can be direct in the healthy direction, too: a Silicon Valley startup CEO who personally called anyone who answered email on vacation, because no single person can be available 100% of the time. Sasha says at Microsoft nobody minds disappearing for a few hours mid-day as long as results are delivered.

Matty says when Chef changed to unlimited PTO, the CEO, CTO and CFO took three-week vacations first. Bill, at a previous company with unlimited vacation, told the team about time at conferences and shrugged "unlimited PTO," and before a four-week honeymoon pasted junk into Outlook's change password form so there was no getting back in. Bill likes the regulated-industry practice of surprise vacations for accountants, and wants to do that to ops teams: "If we can't survive for 2 weeks without you, it should burn." Matty calls it "a chaos monkey for your team." Katie admits to the arrogance of thinking the team can't live without you, and says at an older-style enterprise, employees may have to stand up for themselves. Matty adds that an unenforced boundary gets crossed, and if it isn't respected, getting out may be the answer. Sasha cites Sheryl Sandberg's Lean In: a McKinsey mentor was surprised how many people quit from burnout with unused vacation.

Katie began charging meetups to the company's training bucket after a speaker said it's training: "Charge this to your training bucket." Matty says flexible remote work usually means working more, since the commute demarcation is gone, and Sasha agrees. Matty blocks time for what the job owes back ("Chef owes me 2 hours from last night") and keeps 2 hours a day on the calendar for tasks, treating it like a meeting with someone else. Sasha signs up for a class that starts at 6, and Bill tracks time as if billing customers, and checks vacation quarterly: "If it's August and I've taken a week off, I need to take a break." Bill notes HR likes unlimited vacation because nothing is owed at departure and people take less, and Sasha agrees that it's often "a trick which actually works against you." Matty shares stories from Chef, where a colleague's manager insisted on choosing a vacation budget and dates, and from Travis CI, where vacation is tracked.

Sasha says working on vacation can be self-protection, to avoid coming back to 1,000 emails. Matty's answer is email bankruptcy: set an out-of-office that names who will help, then delete the backlog, and even declare Slack bankruptcy, because "If it's important, it will happen again." Matty says you have to train coworkers first and calls it maybe controversial. Katie takes a more conservative route, scheduling a full day of meetings with the IM off on the first day back so nobody asks anything while going through email.