Arrested DevOps

Trevor talks with Edward Thomson, a program manager on the Azure DevOps team at Microsoft, at Ignite 2018. Edward came to the role about a year and a half earlier after writing software at Microsoft and at GitHub, and is, in Trevor's words, "that guy from Office Space," the person who takes customer requirements to the feature PMs, not even to the engineers. The episode covers decades of version control, how Git got into Microsoft's tools, and what the Azure DevOps split means for open source projects. Trevor works at Chef and Edward's employer owns the products discussed.

Edward says the role means different things for different people: feature PMs shape the direction of a component like Azure Repos, mastering the backlog, while Edward is more customer-focused and doesn't own a feature. Edward began with Git and version control customers and noticed that people now struggle more with the rest of the pipeline, so Edward looks increasingly at CI and CD.

Edward started in scientific computing, then went to SourceGear in central Illinois, where the first product was Source Offsite, a TCP/IP server layered over Visual SourceSafe to stop its habit of corrupting itself, then worked at Teamprise, building cross-platform clients for Team Foundation Server, until Microsoft bought the company. The requirement to explicitly check out files in early Team Foundation Version Control came from huge teams like Windows and Office: the Windows source tree is about 350 gigabytes, so scanning for changes would take forever. Later versions offered scanning, since most people don't have 350-gigabyte trees.

Moving Windows to Git was a challenge, since LFS pages in large files, not a giant tree. Edward's team built the Virtual File System for Git, a kernel driver built with the Windows team, because "nobody should trust me writing kernel code." A clone gets metadata only, takes about a minute or two on the Windows repository, and files are pulled from Azure Repos as they are opened. The first try at a 350-gig Git clone finished overnight, and git status took about eight minutes.

Edward says centralized version control still makes sense for game development, since TFVC lets you lock files and handles big files, that Perforce influenced TFVC, and that "Git is just dominant." Edward thinks SVN's time is past and appreciates Mercurial users.

Edward's friend Martin, who came over from Teamprise, pushed to bake Git into Team Foundation Server and what was then Visual Studio Team Services. Edward first thought it absurd, then they worked out how to put GPL code into Visual Studio. The lawyer for the developer division, who had been a software engineer, answered the pitch that it would be great, since the lawyer was tired of using GitHub for after-hours projects. Edward wrote much of the code, while Martin did the planning and "sneaky execution," including getting it past Steve Ballmer. The work used libgit2, which built a relationship with GitHub, and Edward later left Microsoft for GitHub's Git infrastructure team. Edward missed customer interaction and returned as a program manager, now writing code at night and on airplanes.

Edward says it isn't a rebrand, since "rebranding suggests that all we did was change the name," and it was split into separate products like Azure Boards, Repos and Pipelines, so a Jira and GitHub shop can adopt just Pipelines. Services are free for up to five users, and open source projects get 10 parallel pipelines with unlimited minutes for free. Edward moved libgit2's CI, which used several providers of mixed speed, some paid out of pocket, to Azure Pipelines, with Mac, Linux and Windows agents, for lower cost and faster builds. Edward's answer to what open source projects want: "make it free." Because the agents are real virtual machines, QEMU can run PowerPC and ARM images, which Edward says libgit2 is about to add. Edward credits the DevDiv lawyer, Jason Barnwell, with changing Microsoft's open source culture, and Trevor says the explanation of the name makes sense, though some people object to it, and some CIOs might hear DevOps in a box.

Edward missed most Ignite announcements, as a last-day speaker. Trevor names Azure Blueprints, a governance framework that deploys a subscription with resources and policy, and, for Chef, the announcement of a managed Chef Automate service on Azure, which Trevor says is a private preview and describes as the first product Trevor has released as product owner. Edward is looking forward to seeing open source projects adopt Azure Pipelines.

Trevor is joined by Edward Thomson at Microsoft Ignite 2018, and have a quick catch up on the event and the state of the DevOps world.

Trevor catches up with Steven Murawski at Microsoft Ignite 2018. Steven, a senior cloud ops advocate at Microsoft, leads a team focused on DevOps, site reliability and cloud-native scenarios from an ops perspective, and worked with Trevor at Chef, on the community engineering team there. Steven notes every appearance on the show has come with a different job, starting with Stack Overflow's site reliability work. The two joke about verbal tics: Trevor can't stop saying awesome and Steven says super excited, and Trevor calls it a switch statement hitting the default.

Steven's team is operations-focused advocates, started after Microsoft's developer advocacy effort, a reboot of its technical outreach about a year and a half earlier, had few people with operations backgrounds. The team became official at the end of May and spent the summer hiring and onboarding. Steven lists Jason Hand, David Blank-Edelman and Jay Gordon, with Emily Freeman joining to focus on DevOps and incident response. Steven was a developer advocate on Donovan Brown's team before that.

Microsoft's term IT pro is a huge bucket for anyone who isn't a developer, and the cloud ops advocates focus on server admins and the people moving environments into the cloud, whether on Windows or Linux. Steven's team focuses on the ops crowd around DevOps, site reliability and cloud-native, pushing practices like source control and automated delivery from a single source of truth to reduce manual intervention. All the advocates report to the same general manager, and "Our job title does not reflect the tooling and capabilities that we talk about and expose." It reflects the audience, because "the lines are blurring" between ops and developer topics. Steven likes docs.microsoft.com, which put the documentation that used to be split between TechNet and MSDN in one place, and was part of the reason to join.

Steven says the team can be found at devopsdays, SREcon, LISA, PSConf Asia, WinOps in London and Chocolatey Fest, but best online. The foundational idea is that "we exist to help be a conduit between our communities and the product engineering teams," since product teams are incentivized when people use their services, which they won't if the services don't fit existing workflows, like Terraform, Splunk or Jenkins with Azure. Problems are chances to improve documentation, bring feature requests or support bug fixes. Trevor compares it to Chef, and Steven says it's the same work as on Chef's community team. Steven is a latecomer to IT, on a third career, and says that freely shared podcasts, blogs, code samples and IRC answers made it possible, so getting paid to give back is a blessing.

Steven's session was on continuous monitoring. After CI/CD comes the feedback loops of the second and third ways from The Phoenix Project, through monitoring and instrumentation. Azure Monitor ties together App Insights, Network Watcher, container and VM monitoring, and thresholds can feed back into CI/CD pipelines as quality gates, stopping deployments if rules evaluate unfavorably and replacing manual inspection. Steven's teammate David gave a session on setting service level objectives and indicators in Azure, using the same metrics and Log Analytics tooling that Microsoft uses to run Azure DevOps.

Steven doesn't love the name Azure DevOps, which is what VSTS became, but likes the componentization into Pipelines, Repos, Boards and Artifacts, so you use what complements what you have, for example tracking issues in GitHub and not using Boards. Steven tips that people not yet moved can turn on preview features in their user settings and move at their own pace.

Favorite announcements: Chef Workstation in Cloud Shell, which Steven is increasingly using as a default place to work and can be wired into Visual Studio Code through the Azure account plugin, the public preview of a managed Chef Automate service, and the rebrand of Azure Monitor, which is run out of the Azure SRE org. Steven admits the picks skew toward personal interests, and notes the announcements came as an ebook of 50-some pages of two-sentence entries.

Steven's closing thought is that discussions of moving faster jump to automation tools, when the first step is to define what the work is and what testing and validation look like, because "you can't inspect quality into a product," a line Steven says Deming was quoting from Harold Dodge. Focus up front on what quality and done look like, and be confident what's in your environment so you can move with confidence.

Trevor is joined by Steven Murawski at Microsoft Ignite 2018, and have a quick catch up on the event and the state of the DevOps world.

Trevor records at Microsoft Ignite 2018 with Donovan Brown, who runs the Microsoft DevOps advocacy team nicknamed the League, and a last-minute co-host, Jason Hand, who had joined Microsoft about two weeks earlier and walked by the booth. Donovan has been at Microsoft just under five years, joined to sell Team Foundation Server, moved to the VSTS product team, and then built the advocacy team with Steven Murawski, Damian Brady, Abel Wang and Jessica Deen. Steven now manages Jason, and Donovan helped interview Jason. The cold open is Jason: "let's all advance together because it's going to work out best that way." Trevor is at Chef, and all three say on air that they're a little Microsoft-biased.

Donovan is a 20-year developer and notes that Jason's hiring interview collided on one topic, since developers usually tell Donovan the ops team is in the way, while Jason says it's usually developers applying the brakes. The two were brought into different floors of the same kind of organization. Donovan says ops teams can be rewarded for keeping the lights on, while developers are rewarded for changing things, so the two teams' bonuses conflict. Donovan tells companies to make the bonus depend on both teams reaching the same point, since "you literally told them to work against each other, and they're doing exactly what you told them to do." Jason adds that in large companies value to the customer gets abstracted away, and the fix is aligned goals and transparency, with dashboards showing the business, since SRE "isn't about reliability of infrastructure, it's reliability of the business." Trevor's first question to people is what they're incentivized on.

Donovan's keynote was about the rebranding of VSTS, once nearly monolithic, to Azure DevOps, whose components can now be used individually. The stage demo wired a GitHub repository to pipelines through a GitHub Marketplace extension, showing only pipelines, with no boards or repos, since customers are on GitHub. Donovan stresses it's a platform that can be extended, with the agent and tasks open source on GitHub, and Donovan learned to write a task by cloning the repo. The message is "any language, any platform," which Donovan backs with a blog post on VB6 and a demo of a Node app built on Linux and deployed to Kubernetes.

Trevor says four or five years earlier, learning C# felt like a poor choice. Jason and Donovan say the open source shift and Microsoft Learn show competition dissolving. Trevor wonders aloud whether 90-year copyrights hold society back, which Donovan says is for another show. Donovan is most excited about the announced plan to acquire GitHub, and says "I want GitHub to stay GitHub," and not be absorbed into Microsoft. Donovan cites the mission to empower every person and organization to achieve more, which "you don't do that by holding patents over them," and says it's the only Microsoft Donovan has known, joining in December 2013, just before Satya Nadella took over.

Donovan loves VS Code and once let an audience pick the language (.NET, .NET Core, Node or Java), the platform and the Azure region for a live demo, with VS Code on a Mac, a PC and Linux. Trevor notes VS Code pieces appear in Cloud Shell, and Donovan says Azure DevOps's quick edit uses Monaco, the engine behind it.

Donovan explains the catchphrase. On the VSTS team, Donovan and two other PMs were at a large customer's headquarters trying to make a big Gradle build work with Java, knocking down pain points one after another, and blurted "we're just going to rub a little DevOps on that." The room laughed, Donovan tweeted a picture of a tube labeled DevOps, and the first time saying it live was Build 2016. Some people hate it, and one who did came to a meetup and was disappointed when Donovan forgot to say it. A birthday cake carried the hashtag. Donovan means it literally: find what hurts most in your pipeline, fix it, and when the next thing becomes the bottleneck, go back to it, which is continuous improvement.

Jason plans to bring SRE and incident response to Microsoft's advocacy on the operations side. Donovan ends by pointing to the League's website and hashtag, which reach the whole DevOps advocacy team.

Trevor and Jason are joined by Donovan Brown at Microsoft Ignite 2018, and have a quick catch up on the event and the state of the DevOps world.

Matty talks with VM (Vicky) Brasseur, vice president of the Open Source Initiative, a freelance open source developer, policy and strategy consultant who helps companies use, contribute to, release and comply with licenses for free and open source software in a way that balances the bottom line with the community. VM's book, Forge Your Future with Open Source, is in early release with Pragmatic Publishers, with hard copies expected in mid-October. The cold open is VM: "For fuck's sake, doc your shit!"

Matty raises Redis's announcement of a licensing change in the previous day or two. VM explains the company posted that certain unnamed components would be relicensed under the Apache license with the Commons Clause, which prevents making money from those components. Core Redis, VM says, according to the lead developer, is and will always be BSD-3, but the Commons Clause only implies open source, since open source means the freedom to do whatever you want, including make money. VM has blogged and written on Hacker News about it, and Matty will link the post.

VM says people think open source is a business model, which is foolish: "Open source is not now and never has been a business model." Open core can be a business model, but VM says few open core companies are a success, and those were exits by acquihire, since the technology is already free. VM points to John Mark Walker's article series on the subject. VCs will fund an open core company through a runway, but it then struggles to sell support or add-ons. VM does make a living from free and open source software, but thinks it needs case-by-case reevaluation.

Asked for success stories, VM says a company can't release everything and expect payment without a good reason. But free and open source software is in anywhere from the 80s to 98% of software developed in proprietary companies, which is an amazing way to bootstrap innovation, while contributing back falls short. VM cites Nadia Eghbal's Ford Foundation study on the infrastructure of software development, which set off alarm bells and the sustainability conversation, and points to Heartbleed and OpenSSL as a tiny underfunded team. Silicon Valley's answer is to throw money at it. VM believes maintainers deserve pay but says what they need most is help, including learning to let others help: "It's not actually a business issue." Matty adds burnout and thankless maintenance, citing Jess Frazelle's post The Art of Closing on learning to say no.

VM's advice to maintainers is to document. VM gives a talk on drive-through contributors, people who give one contribution and never return, and argues they're a good metric: if someone can show up, get a patch merged and leave, the project is doing something right that makes it easier for others to stay. In VM's research, drive-through contributors found no install docs, user docs or developer environment setup. Writing is hard, but it's the best force multiplier, and "even shitty documentation is probably better than no documentation at all." Matty says a blank page is intimidating and a half-written doc can be edited. VM adds that writing docs requires knowing the project, so "just write docs" is not good advice for newcomers, but fixing docs is, and as a maintainer, starting people off with a doc or two helps.

VM's first job was at a library automation software company, where VM learned to write everything in an issue tracker, treating it "like a scientist's lab book." Matty adds that doing PRs for oneself models the workflow for other contributors.

Matty asks about lawyers getting in the way of customers contributing back. VM says the answer to almost everything is it depends, IP lawyers are rightly conservative, and VM's IP advice in the book boils down to "Don't fuck with IP law," a phrase the editors keep out of print. Some companies, VM says GitHub and GitLab among them, changed employment agreements to allow contributions to free and open source software on any device at any time, and some publish a checklist for when you don't need to ask permission. But a checklist varies with each company's risk profile, and VM says borrowing another company's is a load of hooey.

VM says contributing isn't only code, and that free and open source software has become programmer-centric, which is why usability is poor and there will never be a year of Linux on the desktop. Designers, usability and accessibility experts, marketers and finance people are needed, along with organizations like Software Freedom Conservancy. "Software is about all of the people that have to come together to make it happen." Matty raises the ops side, such as running crates.io for Rust with volunteers on call, and VM says at HPE, the team dedicated to upstream open source included many who ran OpenStack's contribution infrastructure. The book covers every way to contribute, not just code.

Matty and VM were both at re:Deploy the week before, which VM calls content-packed, relevant and well connected. VM will have reviewed over 1,000 conference proposals this year, and offers two tests: find a unique angle on a hot topic by checking YouTube for the same talk, and "tell me audience takeaways." VM says to write, "by the end of this talk, the audience will know," and to give three things people can do afterward, which beats 95% of proposals. Employers send people to learn, VM says, and a conference is not the place for story time.

Put on the spot, VM says readers will be able to find a project that fits them, not just any project, and to accept feedback in an empathetic way and give it, because communication is the key to free and open source software.

VM closes with a rant: people are still asking how to contribute, twenty years into open source and nearly forty into free software, and "we have let people down" by not making it obvious. The GitHub Octoverse shows millions of new public repositories a year, several million new open source projects even after discounting non-OSI licenses and forks, and "this is not sustainable." VM says "we're just going to crash under the weight of our own success," so contributors should contribute and maintainers should make it easier.

Matty is joined by VM (aka Vicky) Brasseur, Vice President of the Open Source Initiative, for a chat about contributing to open source software.

Matty talks with Christine Spang, CTO and co-founder of Nylas, about building a welcoming company culture, on-call, and going remote. Christine grew up in upstate New York after being born in Toronto, played the French horn, got into programming through computer games and Debian, and went to MIT, where the MIT computer club, the Student Information Processing Board, led to a first job at Ksplice, which turned kernel security patches into binary hot patches. After about three years at Ksplice, two of them at Oracle following the sale, Christine founded Nylas around August 2013. The cold open is Christine: "they don't have this trauma from a world where development and operations were super, super separate."

Nylas is an API company. Christine's thesis is that email hasn't seen much product innovation since Gmail because it has become so complicated to develop against, and Nylas exists to make it easier to build on email, contacts and calendar. Email is the lingua franca of business, and its usage keeps growing, unlike SMS.

Christine says early values went unsaid, which works with a few people in a room but not as a company grows, and the company doubled in size in the past eight to ten months. A change in the founding team led to writing and publishing a company handbook, which Christine declines to dig into but calls valuable. Christine describes a leadership style rooted in trust and listening: "as a founder, you are the leader of the company, whether you say so or not," and listening and empathy build trust over time, which is the foundation of a great culture.

To build trust, Christine says lead by example, give people responsibility without micromanaging, and be consistent. Asking someone to write a blog post and then rewriting it undermines trust by not letting them share their voice, and switching direction constantly makes it hard for a team to commit, so be upfront when you change your mind.

Christine says "our infrastructure is our product," and the team is backend heavy, so every new engineer joins the on-call rotation within about three to six months, first on a front-line rotation, and later some move to escalation. When the company hired its first full-time operations person, that person was surprised at how easy it was to ask engineers to join. Christine thinks younger engineers expect to own what they build. A past period when the company had two products, an API backend and a desktop email client, left the on-call rotation with three people on a three-week rotation, which Christine calls soul-crushing, and Christine wrote a blog post about it. Matty notes PagerDuty's incident commander rotation is three days, and that in Australia on-call pay complicates putting everyone on call.

Nylas caches a copy of the mailbox and calendar data it serves, since reconstructing a thread from IMAP can take half a dozen calls, so it runs a fleet of horizontally sharded MySQL clusters with a team of DBAs. One of them, based in Russia, volunteered to cover the nights, which helps the rotation and means fewer pages.

Nylas began fully co-located in San Francisco, but after Series A hiring, office space and housing costs limited growth, and Christine says that affects diversity, since people with families are at a disadvantage. About 80% of the team is still in San Francisco and four engineers are full-time remote, with more offers to remote candidates. Steps to include them: moving the Friday all-hands to directly after lunch West Coast time once people were on the East Coast, limiting time zones to North America for now, and putting cameras and area microphones in conference rooms. The company writes things down in Slack and uses Dropbox Paper as a wiki. Matty, who was remote for eight years before moving to San Francisco, adds to start with noise-canceling headphones and to remember time zones.

Christine says scaling from two people to many shows why company structures form: past about ten people communication breaks down, and more diverse teams need to be clearer about the words they use. That has given Christine lasting empathy for other companies' processes, which can look like a black box unless you watch them grow: "there's always a reason why things end up that way." Matty's summary: "Context is a thing."

Matty has a chat with Christine Spang of Nylas about company culture and on-call techniques and war stories.

Matty records live at devopsdays Salt Lake City 2018, the third year for the event, with Nicole Forsgren as co-host. Matty's first time in Utah came with a talk called How to Infect Your Organization with Humane Ops, about making on-call more humane when you're not the person running the whole IT organization. Nicole, a former professor at Utah State, keynotes the next day. The guests are Wes Novack, a systems engineer at Pluralsight, Chris from Qualtrics, organizer Jason Vance, and Matthew Barlocker, whose company Blue Matador is a first-time sponsor.

Wes is on a second devopsdays Salt Lake City, and says last year's takeaway was the realization "we are doing things well," while this year has been more specific. Matty likes the event's practice of asking attendees for an aha moment after a talk, and gives a model of one: "interrupts actually caused me to lose productivity for 45 minutes," as against liking the Star Wars slides. Nicole recalls Alice Goldfuss's on-call selfies as a way to make a lonely practice visible. Matty remembers on-call at a bank, when the requirement to reach a terminal within five minutes meant staying home all weekend and not walking the dog more than a couple of blocks away.

Wes says Pluralsight has autonomous product teams that own applications all the way into production and carry their own on-call, and are decoupled enough that cascading failures are rare, since each team owns a vertical slice from front end to database. Wes took away from Matty's talk the idea of having the business conversation about noisy, non-actionable alerts, turning off monitoring that isn't worth making actionable. Matty adds that people assume the answer will be no, and says to push back with information reasonable people understand, such as "we have to invest time in this flapping alert because right now it's having us fly blind."

Chris, on Qualtrics' data platform, working on automation and alerting, came because a friend at Elastic mentioned going and the manager said to expense it. It's Chris's first tech conference, after seven years at Qualtrics and a narrow echo chamber. An aha moment was the idea that "complexity leads to fragility": a system launched before it was ready took about eight months to stabilize, and the fix reduced complexity, moving from very fine-grained message handling to a per-database queue with fairness protocols. Nicole calls it a smart tradeoff, and says Qualtrics is known for data-driven releases and got its start among professors. Chris hopes to submit a talk this year.

Jason says the event improved by documenting everything last year, holding a live retrospective with the staff, and having a stable board, with most members there three years running and one person as dedicated CFO. The conference is about 70% funded by sponsors, with only two new vendors, and "repeat offenders" the rest. Jason's wish for the future: "Not plan it during a major security conference." Matty notes there are over 50 devopsdays that year, so avoiding overlap with other events is not always possible, and mentions Chicago's past conflict with VMworld and this year's with GopherCon. Matty also asks attendees to visit the sponsors, even without a demo.

Matthew describes Blue Matador as "a recommendation engine" for proactive monitoring, aimed at getting ahead of reactive alerting, and works near the venue. The story in the morning keynote minute: Matthew was in the hospital with a newborn son when a former employer called, in a very Star Wars-esque manner saying "help us, you're our only hope," and later got an alert at a family sledding outing. Matty jokes that a tool should recommend whom to page based on who is doing the least important thing.

Matthew has been to the first devopsdays Salt Lake City at a different venue, and it's the first time working a booth, as an introvert. Matty and Matthew say big conferences are "trick-or-treat," with attendees collecting swag, while devopsdays visitors want to talk, helped by a sponsor passport with a drawing for stamps. Matty recalls an old manager's spouse saying Chef was "a t-shirt design company who also sold software."

Nicole asks listeners to take the State of DevOps survey, about 20 minutes and open until June 8th, covering monitoring and observability, cloud platform, database and reliability, this year in partnership with Google Cloud. Nicole's book Accelerate, on the science of lean software and DevOps, launched March 27th, and VictorOps sponsored a book signing with free copies.

Matty chats with Nicole Forsgren, Wes Novack, a shadowy figure known only as Chris from Qualtics, Jason Vance, and Matthew Barlocker in front of a live studio audience at devopsdays Salt Lake City 2018.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Bridget and Matty record live in front of an audience at devopsdays Minneapolis 2019 with three panelists: Liz Fong-Jones, a developer advocate at Honeycomb who gave the opening keynote, Alice Goldfuss, an infrastructure engineer at GitHub who gave the closing keynote, and RJ Williams, a devopsdays Raleigh co-organizer and senior marketing specialist at ASBE, a training and coaching company. It is the first Minneapolis for all three. Matty says it's the sixth Minneapolis event Matty has attended, and that devopsdays Chicago always follows right after, so Matty brings something back from each one to borrow.

Liz's talk was about focusing on people, processes and culture to run more reliable systems, since "you can't really just buy an alphabet soup of tools." Alice's talk was four and a half years of pain running containers in production, and its theme is that containers are just a tool. You shouldn't deploy them because someone talked you into it in the hallway track, so assess whether they fix your problem.

Liz adds that doing it correctly takes serious investment, so why not pay someone else to run it. Alice says the startup habit used to be building things to save money, and encourages people to look at what their business does to generate revenue and whether what they're building is part of that core value. Liz connects it to Heidi Waterhouse's Ignite talk the night before, about which thing is the goose that lays the eggs, and Matty says to find out how your company makes money: "If you don't, go find out. I'll wait."

RJ's takeaways from the open spaces were that legacy and complexity exist everywhere, and that everyone's DevOps is different, with no need for Jenkins or any particular tool to know you're doing it. Liz says automation and tools help with things you already know how to do, but what you don't know how to do or what your culture doesn't represent has to be worked out first. Alice says it needs driving home because people think buying Docker gets them DevOps, and "DevOps is a culture. It is not tools." Matty's version: "you can't buy DevOps, but I sure as hell can sell you some."

Liz says you can't hope to succeed until you know what you're optimizing for, which is why the talk stressed service level indicators and measuring the customer's experience, and starting somewhere instead of targeting all the nines. RJ says clients often arrive after an executive announces "we're DevOps now," and ASBE coaches them through it while training. Most of ASBE's DevOps audience skews toward developers, and RJ asks whether the infrastructure side is agile too, since if not, "you're not doing DevOps."

Liz says automation exists to empower people to do more, and that you won't get better decisions by taking humans out of the loop. RJ says talks at the event kept returning to the human side of automation. Matty says automation should get the human everything they need for the judgment call about whether a spike meant anything, and jokes that Skynet will eventually run PagerDuty. Bridget says this is the golden era of AI washing, and Alice's version: "It's AI when you're fundraising. It's ML when you're hiring engineers. And it's an if-else statement when you're actually making it." Liz points to Jessica Kerr's talk debunking the idea that smarter automation requires stupider humans.

Bridget also recommends Nivea Henry's talk on how the beliefs of a system's creators, conscious or not, end up in its algorithms, with the Babbage line "if we put the wrong figures in, do the right answers come out?" Liz adds that the exposure of people's information matters too, citing a large retailer that mailed pregnancy congratulations to people's houses.

RJ would steal the closed captioning, and praises the thoughtfulness and inclusion behind it. Liz says that in 2019 they only speak at events that have live or after-the-fact captioning, and that in 2020 they'll require live captioning for the room: "you wouldn't have a conference without a code of conduct. Why are you having conferences that are not accessible?" Bridget says participants who never thought captions were for them later say they caught something they missed, and Alice, who has a history of hearing problems, says the captions help people whose need isn't visible.

Bridget describes the room for nursing mothers, which costs no extra budget: organizers who won't be in their hotel rooms between about 7 AM and 10 PM have the room cleaned and leave a key card at the info desk. A nursing mother used Bridget's room that day.

Alice says the AV, lighting and stage make Minneapolis stand out, compared with Portland's first year, when the organizers borrowed a screen from a local BSides group. Bridget credits Joe, an organizer who works for PSAV, and gives the recipe: time travel to the '90s and date the cute audiovisual tech. Liz says sponsors pay for that magic, and Minneapolis has worked hard to support them. Alice likes that the sponsor hall sits between the door and the food, so attendees have to go through "the sponsor gauntlet," which Alice loves as a sponsor. RJ chose this event by seeing how many sponsors it had. Bridget notes it depends on the venue, and that local companies may send 20 people instead of 4 to the expensive coastal conference. Matty adds that the Minneapolis meetup will show 250 RSVPs on meetup.com and 245 people will show up.

Matty says the devopsdays Chicago badges say participant, not attendee, and this year none will say speaker or sponsor. Organizers may get a different color for identifying staff, and their badges will say participant too. Matty reminds everyone that for most attendees this may be the one event they attend all year. Liz wants a more explicit New York call for papers process and a concrete guide to facilitating open spaces so newcomers raise a hand. RJ says the inclusion bar has been raised and is taking notes back to the Raleigh organizers.

Alice, as MC, normalizes inclusive behavior, and each year locates the gender-neutral bathroom at a shared conference center and announces it every day. Bridget says Minneapolis signs say "stalls and urinals" and "stalls only" instead of labeling by people, which gives information instead of assumptions, like documentation that doesn't make untoward assumptions about its end user. Bridget adds that Minneapolis participants often work in large, brownfield enterprises with a lot of legacy, and are pragmatic about building on what exists.

Alice says Bridget got their room extended when a flight was delayed three hours, and the venue televisions showed flight status for anyone waiting, small touches that make a speaker feel welcome. RJ says people were open about their transformations, which is the on-the-ground view that training clients don't usually share. Liz says that talking at the speaker dinner with Serena, who spoke about logging pipelines for security, turned up that they were fighting the same Apache Kafka problems, even though Serena is outside the startup bubble. The show ends with Matty's sign-off: "there's always DevOps in the banana stand."

Upcoming events mentioned: devopsdays Chicago, August 27th and 28th; devopsdays Portland, September 10th through 12th, with a hackathon on the first day; devopsdays Raleigh, October 1st and 2nd; and devopsdays New York in early 2020.

Bridget and Matty chat with guests Liz Fong-Jones, Alice Goldfuss, and RJ Williams, in front of a live studio audience at devopsdays Minneapolis 2019.

Liz Fong-Jones is a new organizer of devopsdays New York City. Alice Goldfuss is the MC for devopsdays Portland. RJ Williams is an organizer of devopsdays Raleigh.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Bridget and Matty chat about organizing conferences with guests Jam Leomi, Debbie Gillespie, and Christian Herro, in front of a live studio audience at devopsdays Minneapolis 2018.

Jam Leomi is a past organizer of devopsdays Silicon Valley, and they just moved to Minneapolis. Christian Herro is a founding organizer of devopsdays Madison, and Debbie Gillespie is a new organizer of devopsdays Minneapolis.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Matty talks with J. Paul Reed, who returns after the fireside chat with "Grandpa Paul" at the end of the previous year, about the thesis Paul is finishing, and then with Mary Thengvall about resilient systems, teams and people, and the re:Deploy conference they are putting on. Matty records right after ChefConf. The cold open is Paul telling Matty to cut something out.

Paul is finishing the Human Factors and System Safety program at Lund University, a two-year program founded by Sidney Dekker, author of the Field Guide to Understanding Human Error, who built it after getting type rated on the 737 and noticing odd questions about how the airline industry works. Paul's classmates include pilots, air traffic controllers, accident investigators and doctors, among them an orthopedic trauma surgeon and an investigator who looks into deaths in live-fire military exercises. John Allspaw went through the program earlier, making Paul the second IT person. Allspaw's thesis looked at decisions under high-tempo, high-stress incidents. Paul's is on what happens after: how organizations use the artifacts of postmortems and retrospectives.

Paul says classmates in other industries deal with high-tempo, high-stakes situations and keep raising technology's effect on their fields, from electronic health records to airline flight decks and the NTSB report on the Uber crash. Paul says tech people in the program bring context, such as how software gets developed and into an emergency room. Nora Jones of Netflix is in the year behind Paul, and Paul says it opens up how you think about the world when reading postmortems and NTSB reports.

Matty asks what safety means beyond redundancy. Paul says many people think they're not in a safety group since they don't write flight control or nuclear software, pointing to the old Java license that excluded certain industries, and Matty mentions Chef's similar EULA. Paul gives two meanings: the business's financial safety, which means understanding the value stream, and the fact that "we live in an increasingly interconnected world," where you don't know how technology gets used. Examples: Wi-Fi pet feeders that starved pets for about a day during a cloud outage, security hardware sold with a cable service that failed to unlocked when the network was cut, and Netflix reaching out to someone who had watched one show for about 80 hours to ask if they were okay. Matty raises Google Duplex, and Paul a report of an Alexa sending a family's conversations to an employee. Matty recalls a tweet that Black Mirror is not supposed to be a how-to, and Paul says letting humans train AI without rules means bad things happen, as with Microsoft's chatbot.

Paul says more backups and redundancy was the right answer up until Three Mile Island, and that Three Mile Island, Chernobyl and the Challenger explosion made people think about the problem differently.

Paul's research question asks how the artifacts of a post-incident review are used in a software development and operations company. There was an industry survey and a case study of a high-performing company. In the survey, postmortem was by far the most common term at about 60%, followed by retrospective at 17%, and root cause analysis came up in write-ins. The top two items collected, each at 85 to 90%, were a list of remediation items and an event timeline. One respondent wrote in blame and fault. Some organizations record luck, meaning where they got lucky. Operations engineers update documentation notably more than managers or developers, and larger organizations are less open with their retrospective reports inside the company, which Paul says means people without a conscious bias toward transparency may unconsciously default to less.

The case study company is one everyone would know, called DevOps Co. in the thesis for research protocol reasons. It calls the process an after-incident review, and "they don't focus on remediation items," and sometimes decide not to fix something. Paul found three things. First, the aim is context sharing, not action items. Second, the reviews continuously map the complex sociotechnical system, both technical connections between systems and human ones, such as an overseas team under attack that didn't know who to talk to, and small fires before they blow up. Third, the reviews curate tribal knowledge and culture: anyone can deploy at any time, within guidelines mostly generated by outages. Matty's question for any protective process, such as a change board, is "how many times in the past year has your process saved you?" and nobody can say. Paul says you can get the same outcomes by trusting people's gut feel and not heavyweight checklists.

Mary has dealt with burnout personally and looks at how to prevent it, how to treat people respectfully, and how to ensure work is valuable in context. Paul tweeted that it's an organizational anti-pattern when some people's vacations matter more than others, and Mary says you can't pay people enough to be online all the time and miss family occasions. Mary says developer relations has no real downtime, since low-season time goes to conference prep and content, and burned-out people say they can't step away because nobody else can cover. Paul describes the sharp end of the system, where work gets done and trade-offs are made without all the information, and says we assume we know how work is done there. Mary says to keep track of your work and be your own PR system to show why your work is valuable and why you need time off.

Paul and Mary describe re:Deploy, August 16 and 17 in San Francisco, where the RE stands for resilience engineering, to look at the intersection of technology, organizations and people. Mary says you can't separate resilient tech, people and teams: "You can't have resilient people without having resilient teams to support them." Paul adds that many have run disaster recovery sites that don't work when turned on, so redundancy isn't the whole answer. The conference is looking for sponsors.

Matty's checkouts include Chef Workstation and Automate 2.0 from ChefConf, and the Cardhop contact manager. Mary's pre-order book is The Business Value of Developer Relations, which Matty says Matty is in.

Guests J. Paul Reed and Mary Thengvall talk about resiliency, safety, and a great new conference - REdeploy!

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Matty and Bridget record live at devopsdays Amsterdam in the third open space slot, in a room where an open space on public speaking and conference CFPs was still going, and decided to turn it into an episode. The guests are Michael Coté, who works at Pivotal and has come to Amsterdam to live, Jessica Brown of Fastly, who led the open space and also organizes and submits to conferences, Thijs de Meester, who works for an ISP in the Netherlands and has never submitted a CFP, and Kris Buytaert, who has started three conferences, including this one, Config Management Camp and a small one called LoadDays. The cold open is Kris on the usual suspects who submit "a lot and a lot and a lot of talks."

Thijs gives workshops and talks inside the company but hasn't made the first CFP. Jessica says it took a long time for the first one, because Jessica is very critical of self and didn't know what people wanted to hear. Friends just say it will be fine. What helped was presenting a 15-minute talk about a project at small meetups, seeing how excited people were afterward, and expanding it to 30 minutes for the first accepted CFP. Bridget says people love hearing what didn't go well, and that a talk at a meetup with good video is golden for a submission.

Coté says that starting from zero, without a software project to show, talks can still be built: watch customers' presentations, aggregate smart things they say, footnote them, like a liberal arts paper. Once in the loop, it's "sort of like sourdough," building on conversations and questions. Coté also thinks of "a talk I would like to have," and assigns a deadline by having a conference accept it. Matty goes title first, and ideas come from the shower, mowing the grass, or explaining the same thing repeatedly, which is good fodder for a blog post or a talk. Matty says personal stories are key, and people would rather hear how someone did Kubernetes in the real world than a hand-wavy theory. A talk proposal can also be a way to learn something and show the journey. Matty's trick is a self-help book title with the word DevOps in it. Bridget describes the talk, Docker in Production: Reality, Not Hype at OSCON 2015, which came from a year of running it and the bugs and janky workarounds, and "People eat that stuff up." Coté adds that "a lot of people don't know a lot of things," so you have to remember when you were ignorant of the topic.

Kris says talk selection is hard: you want the best speakers, and new people whose quality is unknown, so Kris looks at whether they've spoken at meetups. When Kris helped run the DevOps track at DrupalCon, speakers needed to have spoken at a local meetup first, which raised the bar but also meant the first time wasn't at the big conference. The usual suspects submit a lot, and Bridget says that's because speaking is their job. Kris says a small topic you think is trivial, like what you did over three weeks and why, may be valuable to someone else. On tech talks, Kris says the audience is not the authors of the tool, and being an early adopter means you can say how you broke it, which Kris did in talks in 2004 and 2005 about features the Linux kernel team had just released. Jessica wants more tech talks, which are hardest to do, and Kris says "Pro tip, don't do demos."

Coté describes selecting for a Pivotal track: first eliminate talks that fit the wrong room, then ask whether Coté is interested in it, consider speaker experience and then balance the slots, such as two on a DevOps topic, three on build pipelines, some miscellaneous and "a floating one," without eight talks on empathy. Matty notes you never write a talk until it's accepted and Bridget adds not to submit three if you'll only give one.

Matty says not being accepted doesn't inherently mean the proposal was bad, since seven CI talks may have come in, though it might mean more work is needed. Looking at what got accepted helps, and if organizers offer feedback, be kind and don't litigate: a speaker once wrote back to say why the reasons were wrong. Bridget says saying who you are or where you work leads organizers to be less interested, and Jessica says they've refused a company's later submissions for that attitude. Bridget suggests asking colleagues or friends to mark up the abstract in a doc, or asking on Twitter, and Matty mentions a website of volunteer speaker mentors.

Kris jokingly says "don't," then "don't submit too many because you might actually end up speaking too much." Jessica says it's the same as applying for jobs: "Let us say no. Don't say no for us." Bridget: "Don't self-select out." Matty advises trying material on the road at small meetups, perhaps not in your hometown. Coté gives three: make sure the talk lines up with the conference's topics, make the title say what it's about with a keyword, and state in the abstract what people will learn, which also makes the organizers' fit decision faster.

On this episode of Arrested DevOps, Matty and Bridget discuss conference speaking with participants at devopsdays Amsterdam 2018.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf