Arrested DevOps

Jessica Kerr hosts Mark Hibberd, head of technology at Kinesis, a small company building software products that help cities with climate change. Mark has dabbled in distributed systems, security, cryptography and more recently data and machine learning systems, and the common thread is building complex systems that work: reliability, and how you change systems with many users that can't break. The episode's theme, as Jessica puts it in the opening, is making positive change in the world with DevOps.

Mark says complex systems are always failing, and failures become big problems when they cascade. If failures are independent, the probability of a combined failure drops, and change breaks independence: version 1 and version 2 of a service are coupled through their data, and clients couple through interfaces. With two versions of every client and service but one version of the data, everything is coupled to everything. So "your reliability is particularly dependent on how good your deployment process is," which Jessica promises to quote.

Mark says a deployment process needs a full feedback loop, where production tells you whether it's working and the process adapts, citing an AWS talk on closing loops and opening minds. Jessica compares it to test-driven development, asking "how will I know it works?" in terms of users, not just not crashing. Concrete examples: running old and new services in parallel, using only the old results, and comparing performance. Jessica calls it a shadow deployment, and mentions progressive delivery, a term from RedMonk. Mark says rewrites create a cliff, temporal coupling that removes independence, and that a deployment can carry its own checks, such as verifying that the first 100 queries to a spatial service return sensible shapes.

Mark's three topics are building, operating and changing systems. On building, accidental coupling happens at design time, and Mark cites Michael Nygard's entity service anti-pattern, which Mark says describes nearly every microservices tutorial, with a service per noun that makes everything chatty. A smell is a state field describing which part of the system is using the object. Using an order moving from cart to shipping to archive, Mark argues for services around lifecycle stages with handoffs, which Jessica ties to bounded contexts in Domain-Driven Design. Mark's chess example: in-flight games need a fast database for 5,000 games, while 5 million historical games suit a slower searchable one, instead of a Cassandra cluster with hundreds of nodes for terabytes of data when only a fast fraction needed speed. Toy examples in tutorials teach small systems, which isn't a problem unless people assume they scale.

On operating, Mark says health checks are commonly coupled so that one service's failure makes everything report unhealthy, and the cluster shuts production down. Jessica says as soon as you automate around diagnostics, "that's production code." Serving some requests beats serving none, which means aggressive timeouts, and Mark recalls a licensing system for a large antivirus product with 40 million clients where fast checks shared a server with 45-megabyte update downloads over dial-up.

Mark prefers a positive spin on reliability: "unreliable things are often the most valuable things," especially early in a product. Mark works with data scientists, and instead of handing their work off to programmers who take six months, ships their code to production independently, so that if it crashes it can't break the rest of the system. A similar approach works for feature spikes shipped in an afternoon, with the caveat that customers may come to expect a feature. Jessica notes that programmers take six months because of guidelines built for systems that can't fail, and calls it disposable code until it succeeds and needs hardening. Mark says reliability techniques can enable experiments as well as defend.

For change, Mark says developers who know the operational environment should write a production check with each feature. Mark cites GitHub's Scientist library and Facebook's Hack language, where a programmer adds an indicative type and ships it, production monitors whether it's ever wrong, and after weeks of consistent data it raises a change request to enforce it. Jessica summarizes it as making a prediction, injecting an observation so you can be surprised, then escalating the consequences of surprise. Mark likens it to pair programming with the user, and Jessica mentions Dark, a language and environment for changing a running system. Jessica says this works for teams evaluated on how successful users are, not on cards completed. Jessica connects Safety II, which asks what causes success, to Mark's point that we should sell these techniques for what they enable, not only what they prevent.

Kinesis brings together urban datasets, from weather and mobility to land use and electricity consumption, for analytics and predictions, such as the impact of a building standard on cost and greenhouse gas emissions. Mark describes a Sydney tower, where solar panels created excess power, which feeds a recycled water plant, which needed a way to use the water, so plants were grown up the side, which shades the building and reduces air conditioning. Kinesis found up to a 7 degree Celsius difference between parts of Sydney on a hot day, tied to tree canopy cover, and helped a government change policy on where to plant trees, correlating hot zones with at-risk populations. Competing customers share data in building partnerships that cut carbon footprints more than other buildings, and Mark says the work is "definitely better than selling ads."

Matty and Jay Gordon, on the show for the second time since the punk rock episode about a year earlier, have what Matty calls an eavesdroppable Zoom coffee about certifications, hiring, and on-call war stories. Jay works at Microsoft and recently earned an Azure certification. Matty, in Chicago, works at PagerDuty and opens with the line "I used to say that I had a lot of respect for MCSEs until I became one." Jay also hosts the On-Call Nightmares podcast.

Jay needs to speak to an Azure audience credibly, and wanted a new way to learn, so returned to the certifications that Jay used in youth, having come into technology without college or an internship. The exam covered the big picture, like governance, subscriptions and business units within a subscription, and a certification was a check on what Jay did and didn't know. Jay's view: "I like them as a way to do some self-check. I don't like them as a way to justify whether or not you really do know something." Jay adds that Microsoft lets employees take the exams for free, and suggests taking a vendor's test if you can afford it.

Matty got into the Microsoft world doing Apple field service for prepress shops whose RIP stations ran NT 3.51, then took a job at a systems integrator that needed staff to be Microsoft Certified Professionals within 60 days. Matty failed the first exam by one question, developed a method of marking questions known to be right and counting up to a passing score, and later collected an MCP+I and an MCSE of six exams, retaking only Networking Essentials, which had a slightly higher passing score. A coworker once passed all of the MCSE exams in a single day, and Matty once bet a senior sysadmin over who could pass more exams at TechEd and won, despite the sysadmin being a better engineer. Matty last certified on Microsoft in 2003 and was among the first Chef Certified Developers, an exam that was mostly lab.

Jay says between about 1999 and 2006 IT recruiters believed certifications were the bellwether of a good technologist, which Jay thinks was mostly memorization, and for-profit schools promised day-one jobs. Matty says the exams taught "the way that that vendor wanted you to answer the question," a stamp of approval as their representative, and that without a college degree the certs gave hiring managers reassurance. Matty adds that later exams, like the CCIE lab and Chef's, became hands-on. Jay notes interviews moved from paper credentials to practice.

Matty argues tests that forbid Google are flawed, since work involves API docs, a search window and an editor. Matty prepared for a coding interview by writing a little Go against the company's API, and the interviewers were happy to let Matty pull up the existing code. Matty says "if you want to understand how someone is going to work, you need to replicate the conditions upon where they will work," and praises an old interview where three team members posed a real problem and asked the candidate to lead. Jay says they should expect people to build things, not repeat information, and Matty adds that it's about being able to "synthesize information and not just memorize information."

Jay says learning new things as a 40-year-old is worth continuing, through video tutorials, better documentation, and the cert as a self-check, and that tools change, noting Corey Quinn's comment that Kubernetes won't matter in five years, even if every company keeps a legacy thing. Matty says a certificate is "a nice carrot," and what matters is that you care about it. Matty recalls rereading old Outlook email from a network engineering job about frame relay, knowledge now irrelevant.

Jay says on the podcast Jay tells old-days stories without glorifying them. Jay's worst on-call moment, around 2005 or 2006, was a Saturday-evening storm when the automatic transfer switch at a New Jersey data center failed to move to the UPS and generator, taking down about 2,600 servers, followed by a night of fsck and repairs. Jay's reflection: "I didn't learn a damn thing," except that the company hadn't made the right decisions. Matty describes patching SQL Slammer by logging into thousands of servers in different domains, which wasn't a battle scar worth being proud of. Matty tells of a night early in a career when a mail server went down and the boss and owner, who had said to leave it alone, took the server completely apart on the floor overnight, so the informal postmortem lesson was "keep the boss away from things that are broken." Matty also recalls a change request rejected for lacking a backup plan for turning a server off, where the answer was to turn it back on.

Jay hopes the episode said something about learning: some certifications are worth it, Amazon's for example, for certain roles, as a way to test yourself, not as a mark of being good at a job. Matty has been relearning incident management through PagerDuty training and is taking an incident commander rotation, carrying a pager again, and jokes about an Arrested DevOps Listener Certification, with questions about Bridget's preferred airline and Trevor's favorite anime.

Bridget talks with Jessie Frazelle and Andrew Clay Shafer in what the show notes call the unofficial pilot of their new podcast, weird trick mafia. Andrew's pitch for it came from watching Jessie have adventures: "a weekly podcast where Jess could explain computers and I could explain feelings." Jessie is between jobs, bored, and spent the week job shadowing, so the conversation wanders from the Pentagon to an operating room to organization design, with Andrew applying CAP theorem to people. The cold open is Jessie on reaching a level where "you're just one amongst the dipshits."

Jessie ran out of New York museums and went to Washington, where a friend at the US Digital Service gave about half a day's tour of the Pentagon, including an office called Protocol that reminded Jessie of Parks and Recreation. The next day Jessie shadowed a friend who is a surgical resident and watched a liver procedure, finding it less dramatic than Grey's Anatomy. The takeaways: the military is an intense authoritarian system, which the US Digital Service is trying to shake up with support from the Secretary of State, and a childhood wish to be a doctor gave way to being far more interested in computers. Jessie liked how respectful the residents, nurses and attending doctors were, and how knowledge moves between them by doing your time and getting promoted.

Andrew points to the book Team of Teams, by General McChrystal of the Joint Task Force, which has DevOps themes: empower the edge to make decisions, and avoid silos that deprive people of context. Jessie says the USDS practice of bureaucracy hacking is like cold-emailing another team across an organizational boundary.

Andrew says the CAP theorem papers say nothing about computers, only nodes passing messages, which applies to humans, except that humans will acknowledge writes that never happened. A designed organization must choose consistency or availability, and partitions get injected through acquisitions and personalities. Bridget wonders about speculative execution, with many similar initiatives running in a large organization. Jessie says weird organizational structures keep coming up, such as laptop firmware teams at big vendors that apparently don't talk to each other, and Andrew says "It's almost like Conway's Law is true."

Andrew, whose wife is a doctor, calls residency a medieval system and an extended hazing ritual not optimized for learning or patient care, with 80-hour weeks. Jessie agrees tech stakes are lower since no life is on the line. Andrew notes that experienced doctors feel less pressure because they've run the procedure a hundred times, while tech pressure comes from anomalies nobody has practiced, for which there aren't good algorithms. Bridget adds that anything easy has been automated, leaving only mysterious corner cases.

Andrew raises the geopolitics of Huawei and chips, and Bridget the difficulty of sourcing parts locally. Andrew says the price of computers and clothing rests on a chain of human suffering that cost-externalizing hides, and doesn't have an answer. Bridget says every choice has an impact and suggests looking at the carbon footprint of computing, such as cloud providers moving toward carbon-neutral data centers and the CoEd Ethics conference in London, while Andrew says to skip Bitcoin. Bridget mentions Astrid Atkinson leaving Google for a clean energy startup.

Asked to imagine being the CEO, Jessie says Jessie hates titles and career ladders, like the military's authoritarian rule, and cites a talk by Bryan Cantrill about everyone having the same title. Purpose and mission motivate people more than climbing a ladder. Jessie describes the n+1 shithead problem: the person one level up is a shithead, so why climb. Andrew and Bridget note incentive structures like OKRs get gamed, and Andrew says the industry's state of the art is Taylorism, which doesn't unlock the creative potential software needs.

Andrew describes the cube-square law: an ant, with an exoskeleton and no lungs, can lift 50 times its weight, while an elephant has the highest bone-to-mass ratio, and an ant scaled to elephant size would suffocate and crush its own organs. So "the majority of the DevOps presentations" from cat-pictures companies "are the equivalent of ants explaining how they can lift 50 times their body weight." You can't copy what someone did without the context of why, and an organization should evolve for its habitat: "if you have an undifferentiated mass in a human body, then that's a tumor," but an amoeba-sized organization looks like one, and that's fine. Andrew's goal is that "leaders should make more leaders, not leaders should have followers," and there's no magic formula, since scale breaks culture in phase shifts. Andrew recalls Brian Foote's talk on the Ball of Mud, asking what to call people who build such architectures: "Millionaires."

Andrew will speak at devopsdays Atlanta, starting with chess puzzles to argue that seeing the board, as Wardley maps emphasize, isn't enough without understanding its dynamics, with John Boyd in the mix. Jessie is speaking at QCon on Intel SGX, where new information from experts changed Jessie's mind, and at dotGo on eBPF in Linux and Go, which Jessie would like to see replace iptables, which is "just archaic," though eBPF is hard to debug. Jessie explains SGX, built first as DRM for Netflix, then used for running code in enclaves in a cloud you don't trust, which just shifts trust to the hardware provider, and Andrew says "All security starts with physical security."

Jessie: QCon London, dotGo Paris

Andrew: devopsdays Atlanta

Jessica Kerr hosts for the first time, alongside Matty, with Baron Schwartz, founder and CTO of VividCortex, whose QCon San Francisco talk opened the DevOps track Jessica chaired. Baron started as a developer into extreme programming, moved to databases and consulting around performance, and founded VividCortex, which monitors MySQL, Postgres, Redis, MongoDB and Aurora and RDS by combining database signals, operating system data and a measurement of every query from network traffic. The cold open is Jessica's line "to be present with your data in all its persistence," and Baron's "And mindful of your queries."

Baron says DevOps is something you live, hard to put in words, and that the core community, by not defining it, unintentionally gatekept while vendors like AWS, Microsoft and New Relic wrote decent definitions. Baron once wrote on O'Reilly that a manifesto might help. Matty says the official definition is CALMS, culture, automation, lean, measurement and sharing, which is a starting point for conversation and not a definition. Jessica offers one: DevOps "is not a thing, capital T. It's a situation," where operations and development responsibility sit with the same people, which gives them more options. Matty adds that a DevOps team or engineer can be "an organizational smell," since it's often an automation team, and Baron suggests calling it a team that supports DevOps outcomes.

Baron cites Charity Majors: the first age of DevOps was operations people writing infrastructure as code, and the second is developers owning things in production, accountable for performance, operability and observability. For databases, Baron sees DBAs automating away manual work, but less often developers owning database performance. In a survey Baron tweeted, developers owning database performance of their code ranked only about halfway, which surprised Baron, who would put it at the top. Customers that succeed with VividCortex are in the second age, with developers engaged daily and DBAs having moved from guarding a walled garden to running a self-service platform, while some companies declined to replace a departing DBA. VividCortex can't push a company there, but it helps when a company is going anyway.

Matty asks how to avoid assuming software engineers know everything, citing the islands and bridges metaphor from the Effective DevOps book in place of silos. Jessica says developers have respect for DBAs that they didn't always have for ops people, and Baron says the database is scary because we push statefulness down the stack so the upper layers can be stateless, and "It's terrifying down there," with logs, file systems and RAID controllers turning into distributed systems on one server. Baron says everyone needs to model data well, and 80% of indexing and schema design is reachable by developers, who then need specialists for hard cases and to "make friends with the optimizer." Jessica says making friends with the DBAs early in a career earned query permissions. Baron adds that SQL hides intent, so as a consultant, Baron would ask what a query is trying to do.

The first thing is that "you can't bring in a vendor to solve a culture problem." Baron thinks culture is "emergent from the ways that things are done," from incentives and what is praised, so changing incentives or making things easier changes culture, and no vendor can be asked to create culture change. Matty agrees a good vendor helps you see what to change, though you can't just rub DevOps on it, and uses the Switch story of a machine redesigned so both hands had to be away from the blade, the right way being the easy way. Jessica says Agile's manifesto was co-opted by vendors selling culture change, which Jessica calls garbage. Baron notes cloud providers sell a new way of life, like Google's customer reliability engineering team, and that professional services or customer success matter in early markets, so Baron wants customers to learn from each other.

Baron groups what's correlated with success in four buckets, people, culture, structure and process, and tooling, which align with CAMS and CALMS: deploy and release tooling, monitoring and observability that make you a better programmer, and shared knowledge and process, such as deploy confidence dashboards linked from deploy tooling. Baron mentions the full-cycle developer idea from Netflix, and Matty prefers full-cycle to full-stack, being involved through the whole cycle without being responsible for all of it. Baron says it's more important to be present with what you built and shipped and your customers' experience through time, not through layers of the stack.

According to...

Matty, Trevor, Bridget and Joe record the 2018 wrap-up, with Joe hosting for the one time a year Joe does, and with questions Joe wrote. They cover favorite episodes, the one million listens milestone, what each did in 2018, predictions, travel, a round of conference rants and what they're looking forward to in 2019. Matty calls 2018 "a race condition" and Bridget says it never had a starting or an ending point. The cold open is Trevor: "I am the 25th best shuffleboarder in Chicago."

Matty picked three and then noticed that all of them were recorded solo. Let's Be Careful Out There with J. Paul Reed and Mary Thengvall is about resilience of organizations, people and teams, recorded before their Redeploy conference. Punk Rock DevOps with Jay Gordon was mostly about punk and metal, and comes with a playlist for DevOpsing. Shouting at the DevOps, a crossover with Corey Quinn's podcast, is about getting started with conference talks, and Matty heard from listeners who went on to submit abstracts.

Bridget's pick is the GOTO Chicago episode, a panel of the speakers from the distributed systems track Bridget curated, which was "like fanfic": getting speakers to discuss each other's talks. Trevor, who did five episodes, liked the theater episode, the DevOpsDays Chicago episode, and the Ignite interviews with Microsoft people about the history of Git and TFS and why Azure DevOps exists in its form. Joe picks the theater episode and again favors live episodes for being easier to edit. This year's live episodes came from Amsterdam, Minneapolis, Kansas City, Chicago and Salt Lake City, with Jessica DeVita and Nicole as guest hosts.

Trevor pitches revisiting old episodes the way Alton Brown revisits Good Eats, with commentary on what was wrong, like Trevor having Googled the definition of DevOps for the first episode. Matty suggests a riff-track crossover with Software Defined Talk. Matty also mentions a two-hour crossover episode with hosts from the Ship Show, Food Fight and Software Defined Talk that proved "mostly unlistenable," and jokes about putting it behind Patreon.

The only figure Matty wants to share is that the show passed one million listens, "a big number that has two commas in it." The most listened-to episode of 2018 was Punk Rock DevOps, and second was Theatre Geeks Unite and Tech Over the World, which Matty says keeps coming up in Twitter threads from people who came to tech from acting.

Bridget joked that airline status is "the gamification of poor life choices," and after cutting back in 2017, said yes to too much and is Diamond on Delta again. The year included fewer talks but more Kubernetes workshops with Jerome Petazzoni and more involvement with the product side. Matty traveled about 135 days out of roughly 200, visited more countries for the first time than in all previous years combined, went to Australia and to Amsterdam twice, finished the first full year at PagerDuty, where the advocacy team grew from one to three with a developer advocate being hired, and moved from Chicago to San Francisco. Trevor spent the year as a Chef partner architect with Microsoft, launched the Chef Automate managed service for Azure preview on stage at Ignite, started learning guitar properly, and played the Chicago House of Blues with the Chef band. Joe followed Bridget around the world, with Norway the new country, noting that "the secret pipeline is me" for getting Bridget to say yes to a conference.

Joe asks what the talks will be after Docker and Kubernetes. Bridget expects service mesh, noting that Kubernetes, Prometheus and now Envoy are CNCF graduated projects, and hopes for unvarnished experience reports on gluing Envoy to Kubernetes. Matty says DevSecOps will be a consistent theme and that business response matters as well as technology. Trevor predicts "the minimum viable application modernization project," where an old application shoved into the cloud gets optimized without a rewrite. Bridget says it's a familiar pattern: "You put your data center in the cloud and you wonder why the cloud is expensive." Bridget expects customers to tell these stories on stage at vendors' first-party events first.

Bridget listens to Pod Save America, Matty to Hidden Brain, Trevor watches Techmoan on YouTube, and Joe's only podcast is Extra Hot Great. Matty's favorite trip is a tie between Sydney and Helsinki, where the organizers took speakers to a cabin two hours away for a sauna and dinner, and speakers warned Matty that Finns don't ask questions. Bridget recommends taking the Eurostar between Paris and London, with terrible Wi-Fi all the way. Trevor went to Tokyo for fun and to Las Vegas for Inspire, which went as badly as expected. Joe's highlight is DevOpsDays Amsterdam and HashiDays, including a dinner that grew from 10 people to about 45 and a restaurant that seemed glad to see them leave.

Trevor wants to get deeper into product ownership, do more speaking, and play in a shuffleboard tournament in St. Petersburg, Florida, having placed 25th of 64 in Chicago. Bridget plans more product work, including helping PM the Helm 3 release, and significantly less travel, which Joe doubts. Matty plans more targeted travel and more writing, including on burnout and psychological safety. Joe wants to be prepared for a fat bike race in March and for guitar lessons, and sums up the job: "I look at PowerPoint professionally, and then I get on a plane and I go look at PowerPoint recreationally."

Matty talks with Jessica Kerr, a developer for 19 years living in St. Louis, who works at Atomist and co-hosts the Greater Than Code podcast. The two met at re:Deploy, where Jessica spoke on symmathesy, and the episode follows a week when Jessica was track chair for the DevOps track at QCon San Francisco and Matty sat on a panel there. The cold open is Jessica: "I have opinions about DevOps."

Jessica wanted a steady paycheck and the ability to work in any city, and a college internship at FedEx in operations research, programming in ArcInfo to make maps, showed programming was doable, with no homework at 5:30. Grad school in physics offered a long road to job security, while a developer could earn more right away. The motives were selfish, Jessica says, but now software and automation are what Jessica thinks about in spare time.

At Atomist, which Jessica joined almost two years earlier, the work is development automation, in which Jessica studies the domain of software development itself. Jessica is reading the Domain-Driven Design book. Everyone needs their own delivery automation, but not everyone can have a team devoted to it like Netflix, Stripe or Facebook, and Atomist's service does triggering and event correlation for event-driven delivery choreography. Matty likes the word choreography, which Matty once suggested on the show in place of orchestration, and Jessica says "when the world is ready for an idea, it doesn't come to just one person."

Asked how the practice of software could be better, Jessica says "we don't know what we're doing yet," and there are no universal laws or silver bullets: not everyone is Netflix, so not everyone should use Spinnaker, but everyone needs their own automation system. Teams learn by asking better questions. From re:Deploy, Jessica takes the safety questions: not only what made this fail, but "what made this succeed? What made it not worse? What keeps this system together at all?" Also who keeps it running and what matters to them. Matty adds the question of how your company makes money, and Jessica adds that staying profitable is a necessary condition, but "Profit as an aim is not interesting or useful." Matty says deploying 13,000 times a minute is no use if the business isn't changing at that rate, and Jessica adds that software shipped as a part may be updated every few years. Jessica cautions against asking how to go faster, since "that's a destructive question," and asking how to go smoother instead.

Jessica says the industry is thinking about systems and teams, and excitedly notes non-software businesses learning from Agile: letting a team become a learning system and influence the organization turns paperwork into knowledge work. Software lets us create, modify and study very complex systems in days and weeks, not lifetimes, which makes us better at systems thinking. Jessica adds that humans are wired to track relationships, and we're learning to do that with software.

Matty asks Jessica to explain the word from the re:Deploy talk. Symmathesy, Jessica says, is more specific than system, which suggests a mechanical thing that could in principle be predicted: it is "a learning system made of learning parts," where the relationships between the parts keep changing because the parts learn from each other and the whole. An ecosystem is one, as is a team or an organization, and a learning team can't avoid influencing the organization. Jessica has a short blog post with the summary.

Jessica was track chair for QCon's DevOps track, after an earlier experience hosting the frontend track, which felt awful because Jessica is a backend developer. Jessica says "Software delivery is not a nuisance. It's not something that's in the way of your job. It is your job," and "We don't write code. We operate useful software." Jessica compares QCon, structured and practical, with Code Mesh in London, which is informal and opinionated and curated by the program committee. Matty, at a first QCon, says some of the theme language like engineers over evangelists felt exclusionary, and Jessica agrees it's the "tech rules the world thing." Matty praises the track, including the opening keynote by Nicole Forsgren and Jez Humble, and Jessica notes QCon speakers are invited, so tweeting about wanting to speak got Baron Schwartz into the track with a talk on DevOps for the database.

Jessica says the podcast is "we talk to technologists about things more important than technology." It started when the panel of a Ruby podcast lost its editor, Mandy, and started a new show around Mandy, aiming for a more diverse set of panelists. Matty notes Mandy was the first editor of Arrested DevOps. Jessica says all episodes are transcribed, and being a panelist means showing up for an hour and a half to talk to someone interesting. Matty says Arrested DevOps is more like putting on a podcast in the barn with no schedule, and Jessica calls that continuous delivery: "you make a podcast when you have a podcast."

Jessica's career took off seven years earlier with speaking. Jessica liked being on stage but didn't feel like an expert, then learned "you don't have to be an expert in anything. You just have to learn enough to talk about it for an hour." Jessica's advice is to pick several topics you'd like to learn, submit abstracts, and build only the ones accepted, giving you a deadline and incentive, or give them at user groups first. Blogging stays around and lasts longer. "Learning begets community, which begets learning and new ideas," and the sharing makes it geometric instead of logarithmic.

Greater Than Code podcast

Matty and Corey Quinn trade hosting duties in a crossover with Corey's Screaming in the Cloud and talk about how to build a good conference talk, a craft both do often. Corey says the two have had several conversations about where talk ideas come from and thought they would help more than the two of them. Matty, now a DevOps advocate at PagerDuty, says Matty has given about ten times as many talks this year as in an entire prior career. The cold open is Matty's running joke that the show doesn't name drop "except when we do."

Corey's rule: "The secret to giving a good conference talk is always to give a bunch of crappy ones first," then iterating until you learn a joke will never work and cut it. Matty cites Robert Rodriguez in Rebel Without a Crew saying everyone has 10 bad films in them, and Matty hopes the ten bad talks are behind now, and notes they can be versions of the same material. Matty also takes a tip from Jez Humble to iterate on roughly one talk a year. Corey's joke is to give the ten bad ones in a row and get fired.

Corey's pet peeve is talks where the speaker is the hero with the solution for everyone's problem, where a coworker in the audience doesn't recall the project. Corey wants more failure stories, which are "fantastic, and no one wants to give them," including how the decision was made, since "we're all in the process of making terrible decisions." If you lack a story, start there: "People want stories. They don't want you to read a man page to them." Matty adds that stories can be fiction as a narrative device, like a made-up Acme Corp with characters, and recalls Adam Jacob telling a talk through a personal story. Matty also describes a speaker at a large enterprise whose PR department asked to remove everything that said the company did it wrong, which was the point of the talk.

Corey says "no one in the audience is hoping a presenter screws up," and everyone is panicking the night before and nervous on stage. Matty adds "Remember always that nobody knows what you meant to do but you," as in the story of Hitchcock never watching finished films because they never matched what was in Hitchcock's head. Corey notes that the talks they hold up as examples have their speakers cringing too, and that the talks people love most are often ones the speaker would rather not show.

Matty admits to writing slides on the plane and a history of last-minute success, and now wants to reframe it: how much better could it be with preparation? Corey does it from poor time management, not pride. Giving a talk more than once means the first time isn't practice on an audience, though Corey warns not to show another conference's title slide. Matty's advice on practice: record yourself every time and watch it, and focus on one improvement per talk, like bowling one frame at a time, so "I am focusing on one improvement." Corey jokes that a list of 18 things to fix doesn't stick.

Corey keeps a running note on a phone, jotting ideas from conversations, and gets themes from other people's talks and consulting. Matty does best when a shower or cutting grass turns off the thinking, and does expense reports at the desk for the same effect. Giving a talk is a good way to learn: Corey's Terrible Ideas in Git came from using a tool without understanding it well, and Matty's Keeping Calm with Azure DevOps came from wanting to learn it. Both recommend a fun title and a suitably vague abstract, and waiting until it's accepted to build it out. Corey says "Submit early, submit often," about six proposals per conference, expecting one accepted. Matty adds to submit appropriately, not a shotgun, since organizers get to know you, and use the notes field to explain. Anyone who says "don't you know who I am?" has lost the fight.

Corey says new speakers choose a five-minute auto-advancing Ignite to get it over with, which is almost always a bad idea, since "it takes me somewhere between 3 to 5 times longer to build the Ignite talk than it does the 45-minute session," with no chance to back up. Matty's friend prefers Ignites because every beat can be rehearsed. Both say reading from notes is deadening, with caveats from Corey's first Ignite.

Matty says an audience can either read or listen, so a slide should be mostly empty while talking. A plain black slide makes a technical audience think "something broke," so Matty mentions Emily Freeman's approach of using a calm photo. Corey recalls Emily opening a keynote at devopsdays Indianapolis with "the dumpster is on fire" as a fire alarm went off. Both recommend a cold open with a story, because "you have about 45 seconds to grab people's interest before they get back into their phone," and the resume slide can wait a few slides, a snap from Decker Communications training. Corey says the urge for a resume slide comes from the voice saying you're a fraud, and Matty answers that "You don't need to prove that you have the validity to be there because the talk selection committee already did that by giving you the stage." Corey: "If you're holding the microphone, you deserve to be there."

Trevor records live at devopsdays Chicago 2018, the fifth year for the event, with a panel of speakers, and Matty, a founding organizer of devopsdays Chicago, appears as a guest with Trevor as host. The panel is Aaron Kalin of DNSimple, who spoke on drawing parallels between DevOps and the restaurant industry, Katie Prizy, who gave a first Ignite talk on five rules for becoming a better DevOps leader, and Jeff Smith of Centro, whose Ignite was on making on-call more humane. Matty notes the show has recorded at devopsdays Chicago every year, though last year's took about six months to release. Most of the discussion comes from an open space and talk on bias and ethics.

Katie says the conference balanced deeply technical material with emotional conversations about bias and ethics: "I didn't know what a service mesh was until yesterday," after an open space where people explained it well, and then a session on bias where people "got really emotional."

Katie describes the session starting with a video of a race where people step forward if they've never worried about where the next meal comes from, so that by the end some are near the finish and others are far behind through no choices of their own. The point was to look at who is around you, and to use an advantage to help people behind. Open spaces on bias are usually a self-selecting group in agreement, but this one had varied opinions. Katie describes one participant who graduated in a class where the only three people hired during the recession were women, which affected that person, and an immigrant woman of color who has had a hard time navigating life and work. Katie says the aim is to listen to voices not yet heard while bringing the perspectives together.

Jeff says the conversation followed a talk by a speaker named Sonia, who directly challenged white supremacy, and Jeff's table had an intense reaction. Jeff's observation: "people have a problem accepting a narrative that runs counter to their experience," and the challenge likely filled the open space with diverse opinions, more than the usual virtue signaling. Trevor says a similar circle at ChefConf in 2015 was mostly white men, so the change is notable.

Jeff says the ethics part raised whether software development should become a profession with an enforced, rigorous governing body, as systems move into safety-critical settings, such as Teslas that park themselves and may run outdated software. Katie recalls a DevOps Enterprise Summit panel with a pilot, a doctor and Gene Kim on whether engineers are responsible for how software runs in production, such as in a self-driving car, and says that without a decision "we're really just a bunch of cowboys pressing buttons and then saying, well, I didn't do it, not my fault."

Matty points out Sonia's point that a consortium publishing ethics guidance was nearly all men, with one person of color, so how can you enforce ethics from one set of experiences. Jeff says messaging matters, because someone at the table heard it as white men being incapable of being ethical, when the point was that no single group can represent the whole spectrum.

Trevor asks whether the discussion reached who uses the software. Aaron raises the reports about ICE using Microsoft's platform, and asks what an engineer does who learns their software is helping doctors and also enabling agents to do harm. Matty says that without governance that backs you up, quitting on ethical grounds is "a place of privilege" since you're alone when you do it, and compares it to lawyers who can be disbarred and so can refuse an unethical request from a firm. Jeff adds that much of the industry rests on questionable uses of user data, so a serious ethical conversation has to acknowledge the consequences for companies like Google and Facebook.

Aaron says GDPR acts as a governing body for the domain world, catching registries and registrars off guard, since transferring domains needs WHOIS information that now needs the owner's permission, and some countries and domains don't care. Jeff says GDPR is noble but technologically a nightmare, since companies use delay tactics on requests for data, and "we can't just design a rule for privacy, we have to design a system of privacy," or we end up with "a EULA 2.0." Matty says making the right thing hard means people half-ass it. Aaron says reactions to disasters tend to be poor, so voluntary action is better, citing Target's breach and Have I Been Pwned. Jeff notes people still shop at Target, and Matty calls the shrug at breaches "normalization of deviance," with Equifax as the example.

Aaron, who works for a fully remote company, heard in another open space about companies moving toward remote by allowing one day, then three, for people with families and long commutes. One company used software to keep everyone's webcam on, and made a reluctant person join, which Aaron says "would be terrifying." Aaron adds: "You don't want to see me at 3 in the morning."

Matty records at devopsdays Kansas City 2018, the third year for the event and Matty's first, with Jessica DeVita of Microsoft as guest co-host and a panel mixing attendees, organizers and a speaker. Ben Clayton is a director of DevOps who runs a local user group, Ana Medina works on chaos engineering at the startup Gremlin and gave the morning keynote, Dan Barker is chief architect at the National Association of Insurance Commissioners and an organizer of DevOps KC and devopsdays KC, and Monica Hart is a technology associate at VML Y&R at a first devopsdays. The cold open is Monica on "the amount of smart and knowledgeable people" at the conference.

Dan says the event started in a much smaller theater and has kept the theater theme, with better chairs than last year, new workshops, and an Ignite karaoke tradition where a visitor from another city does one about Kansas City, and a local does one about the visitor's home. The event has "doubled pretty much our attendees in the last 3 years," and there is a video wall and roaming cameras. Dan also notes this year was the first with volunteers, who helped with registration and other tasks.

Monica, from a non-tech background, was struck by the amount of talent and the differing views on how DevOps is used at work, plans to tinker with Kubernetes afterward, and admits to nerding out in front of speakers. Dan shares that feeling when learning Mike Julian had submitted. Monica says the DevOps community is "a really judgment-free zone," where people ask why you think so, instead of dismissing you.

Ana's keynote was an introduction to chaos engineering in a fun manner. The reasons start with scale: systems are getting more complex, and downtime is expensive, with a report of "around $300,000 for every hour that a company is down." Chaos engineering gets teams thinking about resilience first-hand, can help onboard people to on-call, and frees engineers for development by reducing incidents. Jessica says talks on the practice only recently appeared at devopsdays events, and Ana says antifragility is becoming part of the conversation, and that while Ana has done this for three years, the practice is about ten years old, coined by Netflix with Chaos Monkey, with other companies doing it quietly.

Matty says it's been a year of resilience and notes the re:Deploy videos had just been released, with talks from Nora Jones of Netflix and John Allspaw of Adaptive Capacity Labs, and adds that "our systems are always in a state of some type of degradation."

Ben says the virtualization community has realized it has to embrace DevOps culture, and tells engineers at VMUG to learn at least scripting and DevOps principles. Ben appreciates that this event is volunteer-led.

Matty says the morning talks exist to give people something to discuss in the afternoon's open spaces. Matty shares that Andrew Clay Shafer said at devopsdays Chicago that in the first devopsdays, Andrew and Patrick mainly cared about open spaces and had talks only because companies would pay for them. Monica's favorite was an open space on workforce transformation, hearing common issues that aren't usually discussed. Jessica urges people to run open spaces at their company, like a lean coffee, since "It's not PowerPoint and presentation. It's conversation." About three quarters of the room were first-time attendees, which surprised Dan.

Matty says the variety of topics made Matty better rounded, including a keynote on flintknapping that left Matty thinking "I might want to make Paleolithic tools as a hobby." Ben sees a community change, with companies once hesitant now sending many people. Ana, based in San Francisco, says it's good to see Kansas City's conversations match the ones on the coasts, about monitoring, Kubernetes and chaos engineering. Dan was surprised by the number of first-timers and of companies not historically involved in DevOps.

Asked to describe devopsdays Kansas City in three words, Ben says "Culture, fun, and networking," Ana says "that was rad," Dan says "Growth, community, and challenge," and Monica says "Mind-blowingly awesome."

Matty (with special guest host Jessica DeVita) is joined by Ana Medina, Dan Barker, Ben Clayton, and Monica Hart at DevOpsDays Kansas City 2018.

Trevor talks with Jessica Deen, a cloud developer advocate at Microsoft, at Ignite 2018. Jessica's last name is spelled with two Es, no relation to James Dean, and Jessica focuses on Azure, open source, Linux, DevOps, containers and Kubernetes. Trevor works at Chef, and Jessica's employer owns the products discussed.

Jessica is part of the DevOps advocacy team led by Donovan Brown, which the team calls the League, and whose stage motto is "rub a little DevOps on it." The team splits between dev-focused and ops-focused members, and Jessica says they try to embody the culture they talk about. Three members were at Ignite, all five at Build earlier in the year, and they keep in touch to borrow each other's knowledge even when apart. That week Jessica was in Scott Hanselman's general session showing Azure DevOps, led a 75-minute container DevOps session, recorded for Channel 9, gave an MVP interview and led another session with JFrog.

Asked for a favorite topic, Jessica says every topic goes back to DevOps, even Kubernetes, Helm and Draft, which are built on infrastructure as code and automated release, like the Jim Carrey movie where everything adds up to 23. Jessica mentions Donovan opening sessions with a pit-crew video. A technical glitch that day meant switching to a failover demo: "I work in ops and I consider redundancy."

Announcements Jessica lists: serverless in Kubernetes clusters, tying Azure Container Instances into the Kubernetes service, changes to Cosmos DB billing, and the rebrand from VSTS to Azure DevOps. Another announcement, which Corey Sanders tweeted after leaving it out of a blog post, is a virtual machine image builder based on HashiCorp's Packer, supported for Ubuntu 16.04 and 18.04 and in private preview, with Windows containers on the roadmap. Jessica also learned this week that a KubeCon talk on Windows containers with Patrick Lang was accepted. All sessions were recorded and uploaded quickly, so Jessica watched the recording of a session the next day and will queue some for a flight. Jessica says staying current is hard even inside Microsoft: someone asked about in-cluster ACI and AKS three hours after it was announced, and the answer was that it was news to Jessica too.

Jessica says "I need a Kanban board for my brain," and uses Trello for personal projects, demos and new content. Jessica is known for a dotfiles project called Badass Terminal, named by the community and popular on Reddit, which works on macOS, Windows Subsystem for Linux and Ubuntu, and opens PRs for the community. Trevor admits not having learned Kubernetes yet, and Jessica offers sessions and demos.

Jessica says this may be the largest Ignite ever, and the conference center is about half a mile end to end. The recurring complaint in Jessica's session feedback was that rooms were cold, which the speaker didn't control. The moral: "always relay your feedback to the people who can make that change," using the survey for logistics and leaving session feedback for speakers and content.

Jessica's content is mostly at the 200 to 300 level, and Jessica wants a real-world demo of a stateful application with a database: WordPress with Helm charts and MySQL on a persistent volume claim, then high availability, failover, load balancing, canary and blue-green deployments, using NGINX ingress or Azure's HTTP routing and Istio for traffic splitting. Jessica wonders whether the database belongs in Kubernetes or an external service. Trevor notes the trouble of finding an open source legacy application with a complex enough architecture, and used an old movie database sample from ASP.NET MVC 1. A working title for the session: Kubernetes for realsies.

Jessica talks about artifacts: if you use an upstream image from Docker Hub you don't control what's in it, and if you write your own, you own the vulnerabilities, such as one Alpine Linux announced the day before. A base image with vulnerabilities baked in carries them through a multi-stage build. Jessica mentions a post by Jess Frazelle that found about 700 versions of Java in one image. Jessica's practice: "build small containers." An example is cloning a private repository in one stage without baking a personal access token or SSH key into a layer, with the artifact passed to a final image. Using Debian the image is about 86 megabytes, and with Alpine about 8.

Jessica tells Trevor that as a former consultant Jessica said "good, fast, and cheap. You can only pick 2." Pulling an image from Docker Hub is cheap and fast, but you don't know if it's good. The week's big takeaway: "there's 30,000 new people that I never knew existed that are obsessed about the same technology stuff I am."

Trevor is joined by Jessica Deen at Microsoft Ignite 2018, and have a quick catch up on the event and the state of the DevOps world.