Arrested DevOps

Matty hosts solo, recorded live in front of an audience on the second day of DevOpsDays Chicago, the third year the show has taped there. Matty's panel is Nell Shamrell-Harrington, a software engineer at Chef from Seattle, Jill Jubinski, a community evangelist and recruiter at IBM by way of its Blue Box acquisition, and Mike Stahnke, a director of engineering at Puppet. Matty, an organizer, says this was the smoothest of the three events, and then worries about having jinxed it. The episode's cold open is Mike saying most of the DevOps advice Mike takes comes from 90s music, and Nell adds the TLC song about not chasing waterfalls as advice for anyone getting started.

Nell saw a strong emphasis on humanity and technology and how they aren't so different. Mike saw open space after open space come down to testing: "testing all the way down." Jill says a constant theme is that "technology is hard, people are harder." Matty recalls Adam Jacob's keynote on humane systems, context and thriving together, and Jill's morning talk on DevOpsing recruitment, which included a game of guessing whether a recruiter or an engineer said each line, where the answer was always both. Matty also liked a line from Nell's talk on refactoring that what software does matters more than what it was intended to do, and Nell adds that "the only source of truth is when you execute the code itself."

Matty says test-driven infrastructure has only really been possible for about three years, and can't remember how cookbooks got written before Test Kitchen. Nell says Nathan Harvey, Nell's boss, suggested that to teach TDD to sysadmins, you point out they already test: when they configure by hand, the first thing they do is log in and check it worked. Testing is the same thing, faster and more reliable than a human. Nell sat in an open space on applying the testing pyramid to infrastructure code, where people from Chef, Puppet, Ansible, sysadmin and app dev backgrounds showed the principles are the same whatever the tool.

Matty says people at infra-code vendors are in a bubble where table stakes aren't table stakes for everyone. Mike describes conversations where people know the outline of a transformation plan, but every bullet is a journey that can take nine months or a year, and people who read about 50 deploys a day don't have 50 tests yet: "just because you know the roadmap doesn't mean you can actually drive." Nell recommends what Martin Fowler calls the strangler method, adding tests around one small part of a legacy mess until the new well-tested application strangles the old one.

Matty tells of a bridge in Louisville that was built on top of an old bridge, which let the old one fall away once the new one was strong enough, as a metaphor for keeping the old thing until the new is ready. The host connects it to shims and temporary bridges, which product owners hate, and to Jeff Smith's Ignite about Dungeons & Dragons DevOps: you fight level one monsters first, and learn to write a Git commit message before the 50-deploys-a-day dragon. Customers want to manage their Hitachi storage before they've written a recipe to install a package. Mike finds writing the tests "way, way harder than writing the software," and has mad respect for test automation engineers. Matty says writing the tests makes the code easier, since you've done half of it.

Matty ties this to time to first delight, Matty's own term, which Andrew Clay Shafer calls mean time to dopamine: people have to experience results, since you can't talk them into it, and if you're selling against something that gave them a dopamine hit, as Adam Jacob would say, they will argue with math.

Mike says it was the first recruiter talk Mike had seen at a DevOpsDays, and it was great. Jill's goal for 2016 was to speak at an engineering conference, and this was the third talk toward it, after Monitorama, where the topic was Taylor Swift and open source, and Boston. Jill wants to spread empathy for recruiters, and says the good ones are not rare. Mike says Puppet grew from 35 people when Mike arrived to 470 and couldn't have without recruiting and hiring pipelines. Nell praises Chef's in-house recruiter, who also trains interviewers, since some questions engineers are used to asking are actually illegal. Jill says to rely on recruiters for that, and for diversity, since they understand the market, while engineers assess the technical side.

Matty says engineers can be arrogant about everyone else's job, assuming sales or marketing or recruiting is easy. Mike calls it Dunning-Kruger, "The more you know, the more you know you don't know." Why do recruiters get a bad rap? Jill says there are internal recruiters who are ingrained in the culture, and external ones who work on commission and fill seats without knowing the companies. Jill's advice to companies big enough is to "get an internal recruiter and I promise you it will change your life." Jill has gotten emails pitching a DevOps engineer role, and Nell one pitching an administrative assistant job. Matty says the reason we hear about it is that we live in a bubble, and IT people get complained about on Facebook. Making fun of recruiters is lazy, and Mike loves fake internet points.

Jill says recruiters often lack effort as well, and should know their audience, since spamming LinkedIn is wrong for engineers but works for marketers. Matty sometimes replies to recruiters offering to tell them what is wrong with their job description, and about half the time gets a response saying let's talk. Jill runs messages by engineer friends for honest feedback.

Nell, who is hard of hearing, enjoyed the party at the bowling alley with card games, since it wasn't a loud bar. Jill liked the close venue compared with Boston, where a larger stage felt disconnected from the audience. Mike was thrilled with the deep dish lunch on day two, and Matty recalls that in the first year a speaker got mobbed with questions and missed the pizza.

Three first-timers in the audience share feedback. Mark liked the networking and hearing how companies integrated DevOps, and wished open spaces were organized more tightly. Yasser, a developer, preferred the non-technical talks and suggested speaker ratings. Mike says DevOpsDays went through an arc where culture talks pushed out tools talks and now a few technical talks are coming back, and non-technical talks travel better since they're relevant even if you're not on that technology. Matty recalls two people at the first Chicago, one from a smaller company saying it seemed like it was for big companies and one at a big insurance firm saying it was for small ones, after the same event. Dongmin Liu, also a first-timer, says the non-technical part is the harder one, because it's not about technology but how you package it to convince other teams.

This is the show's first experiment with a guest host. Michael Hedgpeth of NCR, who is working on DevOps in the company's hospitality division and worked with Matty on bringing Chef into NCR, takes the host chair, and Matty and Bridget are the guests, alongside Doug Ireton, a Senior AWS Chef Engineer at 1Strategy. Michael proposed the show after catching part of Doug's ChefConf talk on operationalizing open source in the enterprise, and realizing there were questions for all three of them. Bridget, who does tech advocacy for Cloud Foundry at Pivotal, opens with the line that comes back later: people contribute to open source out of passion, but they "also like to sleep. And see their families."

Doug was the unofficial go-to person for open source questions at Nordstrom, a setup that needed no formal committee or title, just people who care, as Doug tells it. Doug's main takeaway is that "culture change takes time": it took about four years to go from open source contributions being allowed to open source being the default for fast-moving teams. Doug walks through the timeline. In 2008 the only open source they used was Red Hat Enterprise Linux and nobody went to conferences. They bought Chef in spring 2012, and in fall 2012 a VP said contributing is part of using open source, and employees signed an intellectual property agreement. In 2015 developers wanted to contribute to React.js, which required a contributor license agreement, and getting Facebook's approved took about four months, a lot of back and forth from legal and two VPs' approval. By 2016 the Google corporate CLA, for Kubernetes contributions, was signed in a week.

Doug says the close relationship with legal never developed, and wishes it had. Leadership approval was in place, but the VP first approached sent the request to another VP who then left, and Doug suspects a non-technology company's legal team may not focus on differences between licenses.

Michael asks how starting with Chef led to open source in core development. Doug says Nordstrom is a big company, Doug's team never used Kubernetes or React, but leadership came to see that open source lets you move faster instead of a six-month vendor selection, and app teams co-evolved with the Chef users. Bridget says it isn't only for frontend teams, and that if a vendor tells you their platform is the only thing, you should ask whether it's built out of open source. Michael says for a Microsoft organization like NCR, the world of many tools instead of one umbrella is a big shift, and Chef helped them take baby steps toward being community members.

Matty says the same holds for IBM shops, with the old joke that nobody got fired for buying IBM. Matty describes going to a vendor with a problem and being told they don't quite have it, then buying the closest thing anyway and taking on tech debt, as with a BizTalk system that could have been a small open source tool. CIOs love "single throat to choke," but buying everything from one vendor doesn't mean it works together.

Michael says NCR wants to standardize for efficiency, which runs against the open source idea of letting teams pick the best tools. Bridget says a well-composed platform like Cloud Foundry isn't opposed to devs using small tools, as long as choices are open and compatible. Doug uses Netflix, where most teams use Java because there is a paved road, but you can choose something else and support it yourself. People resist standards from a central architecture team that reads white papers and talks to vendors, and accept them from architects embedded on dev teams who write code and support production, and Nordstrom's API teams have moved to Go and Node.

Matty says the best way to get people to do something is "to make the thing you want them to do to be the easiest way to do the thing they want to do," not "thou must." Matty advises a customer to think about the result they don't want, such as an artifact that enables remote logging, instead of policing how people write infrastructure code, and notes that in a huge enterprise, standards have to get vague as you move up. Michael's takeaway is that "standardization is great when it enables freedom," and that Chef works that way for NCR, since it made adding HashiCorp Consul easy. Bridget adds that organizations adopt Netflix OSS like Eureka and Hystrix because good, battle-tested tooling gives an incentive to choose the tried-and-true path.

Doug's talk argued that an open source stance affects hiring. Several engineers said they looked at Nordstrom's GitHub profile before interviewing, and Doug says "your GitHub organization is your organization's resume in the open source world." They were able to hire someone from Chef primarily because they contributed, and it helps retention too. Doug ran a Twitter poll asking whether people would work for a company that didn't allow open source contributions: 68% disagreed, 25% were neutral and 7% agreed. Bridget says companies building on open source can't expect it all to be written in people's spare time, and should pay employees to contribute. Doug adds that private forks become untenable quickly.

Michael asks when paying for help with open source makes sense. Michael describes the open source workflow of running it, Googling the error and running it again, and Matty says the important part is then sharing what you learned so the next person doesn't repeat it. Companies with healthy internal communities look just like public ones, behind the firewall, and Matty can tell what wiki software customers use from referral links to ADO episodes. Michael says the relationship with Chef morphed from support to consulting, which helped with thinking strategically about who talks to whom, and gives an example of a colleague in Prague who disagreed with how Michael did Chef, and Michael told the colleague to go with it, which was freeing.

Michael asks whether relying on a vendor for support of an open source tool is unhealthy. Matty says the vast majority of Chef users don't pay, and that if all a vendor sells is support, it isn't a sustainable business, because you'll eventually get better at the product than they are. Matty adds that with closed source, support might be all you get and you can't stop paying. Bridget says Cloud Foundry has contributors like IBM, HP, Swisscom and GE, and Pivotal contributes about 65% of the commits, and that a free 60-day trial of the commercial product lets people try it before paying. So "I can't install it from GitHub, so I can't cut a PO" is a straw man. Matty says if a customer has to spend a week standing up infrastructure to see what a tool does, the vendor has failed, and the goal is time to first delight. Bridget says Andrew Clay Shafer calls it "mean time to dopamine." Bridget also says look at the community around a project, like the US government's 18F building cloud.gov on open source Cloud Foundry, with Terraform scripts on GitHub. Doug pushes vendors to hand over a Terraform script or CloudFormation.

Doug says enterprises new to open source respond to bugs with "the vendor should fix that," but they have unique insight into the problem. Doug found a bug in Supermarket that could be fixed in five minutes, and it was merged within an hour, versus filing a support ticket that takes two and a half weeks. Michael says Chef support once demonstrated the same lesson by opening a GitHub issue, submitting the PR and reporting back with a link. Doug has also been on a closed-source project where the vendor said it might get to a fix in six months, after millions were paid. Bridget likes subscription models because vendors care about churn, and Matty says the dirty secret is that "Renewal time is the best time to ask for anything from your vendor."

Michael asks about the freemium model like Chef Automate. Matty says that in Chef 11 you had to reinstall everything to drop the enterprise version, whereas since Chef 12 and through Automate, the core is open source and everything around it is a wrapper, and "you want that premium thing to be sticky because it's valuable, not because it's in the way." Michael asks whether the answer is instead to build a Jenkins pipeline and Elasticsearch cluster in-house. Bridget's answer is that it depends, but the fallacy in organizations with smart people is thinking they must master everything down to chip fabrication: "you should only build the things that make a differentiating business value for you." At DramaFever, they didn't build their own CDN, and when the Akamai bill was high, they prepared to go multi-CDN and it turned out Akamai would negotiate. They did build a Docker, Packer and Chef-based platform, since it let them stand up a K-drama site or a horror site from the same image.

Matty says there's no reason to run your own email, and that "modern DevOps, we're all about coulda, not about shoulda." Pay someone to do the thing you'll never do again, like setting up a cluster once every five years, but not to write automation you'll change. Matty recalls dev leads at Apartments.com who wanted to write their own authentication for an internal customer service app when they were a Microsoft shop with Active Directory, and calls that hubris and resume-driven development. Doug adds that consulting is most useful when the consultants partner with you and you dedicate engineering hours, and that paying for Chef was worthwhile for the support and the add-ons customers expected, like a web UI. Matty ends with an example: at Chef they didn't write their own search provider, they use Solr and Elasticsearch.

Matt and Bridget chat with Michael Hedgpeth (NCR) and Doug Ireton (1Strategy) about organizations adopting open source.

Image credit: https://flic.kr/p/3V99c8

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

For any devopsdays, try the code ADO2016! It should get you 20% off.

Tim Gross (Joyent) and Adam Jacob (Chef) independently started solving the problem of how to put applications in control of their own configuration. They discuss Habitat and ContainerPilot, to the edification of Matt and Bridget.

github.com/joyent/containerpilot and example applications at github.com/autopilotpattern

habitat.sh

The Art of Closing - open source maintainer advice from Jessie Frazelle

Badass by Kathy Sierra

DevOpsDictionary.com - I’m having a bit too much fun contributing to it.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

For any devopsdays, try the code ADO2016! It should get you 20% off.

Bridget records a live panel in front of a studio audience at DevOpsDays Minneapolis, largely improvised. The panelists are Charity Majors, who founded Honeycomb, the company formerly known as Hound until a rename two or three weeks earlier, Nicole Forsgren, who is at Chef and a co-founder of DevOps Research and Assessment, Andrew Clay Shafer, and James Watters of Pivotal, who is responsible for Pivotal's products. Bridget reports to Andrew at Pivotal. Bridget admits forgetting to tell most of them that a podcast was the plan, and that they are in fact being live-streamed. Andrew reminds Bridget that last year's taping, which was billed as eating sushi with Andrew, had no sushi. Nicole opens with a line that returns later: "Teams deliver software, individuals don't. Teams perform, individuals don't."

Bridget asks what kinds of companies are realizing software matters. James mentions Merrill, a sponsor, whose executives were betting on a new software product as a global SaaS brand. Charity says platforms mean you never know who will show up: Disney signed up and created an account without talking to them, next to 13-year-olds building software on the same platforms as corporate America. Charity calls it the democratization of tools that used to be a competitive edge for big vendors with giant R&D budgets. Andrew describes a reinforcing spiral that started with open source, where the primitives for building Facebook, Google and Amazon became available, and has gone up the stack to things like TensorFlow, so you can start thinking about your domain and have neural networks. "The democracy is going up the stack," Andrew says, and James adds "abstraction levels going up democratize technologies," like a toddler using a touchscreen.

Nicole says the companies doing it right reach customers earlier, get to market and feedback faster, and reduce complexity with an MVP, and the ones that spend a year on an RFP fall behind. Charity says that's why hashtag NoOps draws a wince: operations isn't vanishing, good operations is a competitive advantage, and what is shifting is the definition.

Charity describes "the shadow self of DevOps": the industry has spent almost a decade telling ops people to write better code and tests, but hasn't told developers what they need to learn, the operational skills that let them build, ship and maintain products. Andrew dislikes labeling people as ops or dev, and prefers thinking in capabilities, since not everyone can do everything as organizations scale. Andrew says NoOps is a reaction to a world where sysadmins were responsible for the mail, and that "the systems are actually a reflection of the organization." Charity says anyone giving advice without context is selling nonsense. James asks enterprises what problem they're solving before talking about product differentiation, and asks how long deploys take. One bank told James eight weeks and 10% of their staff, a year into building their own platform. Nicole says that is solidly mid-to-low performance, and not even the worst of it.

Charity says an inherent trait of infrastructure done well is that you don't notice it, like roads, and Bridget recalls the Minneapolis freeway that fell into the river in 2007. Nicole adds "when the work is done correctly, the work disappears," which contributes to devaluing it, and points to systematic salary differences between development and ops, and Charity says Google couldn't retain SREs until it paid them more than software engineers. Andrew says leading-edge cloud-native companies don't have as much of a disparity, and that treating IT as a cost center forces bad behavior. James says offshoring tried to lower the unit cost of developers as paying less for ops did. James describes operators running thousands of containers and hundreds of apps and says raising abstraction lets an operator have leverage, so James tells the operators' bosses how valuable the operators are.

James's advice for IT professionals is that "application architecture and operations architecture are not dissimilar things," and to think about how a database is operated, not just its API. Andrew says lots of people who think they have operations problems have architecture problems, and adds, "Day 2 matters."

After James leaves, Charity says context is everything, and Andrew says continuous delivery and microservices can sound like fairy tales to someone far from them. Nicole says you have to meet people where they are and help them envision what's possible, remembering the comfort of doing it manually back at IBM. Andrew uses a marathon analogy: if you're not ready and you try it, you'll hurt yourself. Charity's version is a patient in the emergency room with a broken leg and blood spurting from their head, who shouldn't be worried about skin cancer yet. First stabilize the patient, and the top skill in startups is "ruthless prioritization." Andrew says overfunded startups fail because cash protects them from their context.

Asked how large organizations get startup nimbleness, Nicole says it happens in small teams that try something and scale, not in an organization-wide nimble process with posters. Charity says Facebook was a revelation, with a horizon 18 months out, and that what works at scale is "Outcomes-oriented and trust," define an outcome, develop trust and then go hands-off, empowering people to make mistakes as long as they won't destroy the company. Andrew says smart people solve problems, and also, "Smart people also cause problems." Charity says every workplace of Charity's had an outage caused by nearly dropping all the data.

An audience member asks about HR and career development in DevOps teams. Charity says what an org values differs, like Heroku's five nines counting toward promotions, and signaling what you value is how people are incentivized. Charity likes asking reviewers who they would most like to be paired with on call, or would call at 3 a.m., and who they least want to be paired with, since it differs from who the best engineer is. Andrew says work can be done so that you're better at it afterwards, and if not, reevaluate the work, and adds that not confronting incompetence demoralizes the rest. Nicole says university doesn't prepare you for distributed software or continuous delivery, for developers or ops.

Nicole's rant is that individual performance reviews are nonsense, because "Teams deliver software, individuals don't," and invisible work like the person who glues the team together may show the fewest commits. Andrew says "As soon as you make a measure a target, you made the measure useless," and takes a contrarian position that humans are non-fungible. Nicole says Google's study of 36,000 engineers found team dynamics, with psychological safety first, matter most, and Charity says Google still hires "as though they're hiring for Lego bricks." Andrew describes the 80 percenter persona who can prototype anything overnight and will never build anything that belongs in production. Bridget recalls Alice Goldfuss's talk on rock stars, builders and janitors. Charity says startups have the advantage of hiring for what the team needs, and that constrained resources drive creativity, and you can carve out a small budget for a team inside a big company, like a startup.

Nicole responds to the assumption that the data doesn't apply to enterprises: "there are no statistical differences among the different enterprise sizes," or by industry, including highly regulated ones. Nicole classifies teams as high, medium and low performers by throughput, meaning deploy frequency and lead time, and stability, meaning mean time to restore and change fail rate. High performers score significantly higher on the Westrum culture measure, with high trust, good information flow and messengers not shot. Practices like version control of infrastructure, application and configuration differ, and the firms' characteristics don't. Nicole also sees a 50% difference in stock price performance between high and low performers over three years. Charity summarizes: you have no excuse. Bridget notes a recent conversation with a large company excited to be getting Git this year.

An audience member asks whether high performers do stability or throughput first. Nicole doesn't have data on sequence and doesn't see trade-offs, and says gains now come in speed because "slow is the new down" and stability is near its ceiling. Andrew adds "Each nine costs 10 times more than the last one." Another audience member asks Andrew about synthesizing ideas, and Andrew says the best thing a technical person can do for their career is learn to speak and to write, helped by Andrew's time on a debate scholarship.

Charity has said no to 26 conferences over the next six months, and wants more acknowledgement at DevOpsDays that this is for software engineers too. Nicole shares Andrew's ideas from the conference: "all code is technical debt," so people should be rewarded for removing it, and "software is never the endgame," so do things for the business, the customer, not because of DevOps. Andrew adds that tests are code and thinks DevOps, microservices and continuous delivery are a single phenomenon, a cloud-native paradigm that can't be treated as separate initiatives in separate silos, and cites Deming: "change is not mandatory. Survival is not mandatory either." Bridget closes by praising Jeff Smith's talk from Grubhub about what happened when they did DevOps, with the spoiler that not everything is unicorns and rainbows.

How do large enterprises transform the way they do IT? What does it mean for every company to become a software company? Our panel of experts at devopsdays Minneapolis 2016 has worked in some of the largest orgs out there and has seen a lot of transformation first-hand.

Matty and Trevor record live from the expo floor at ChefConf 2016 in Austin, at the end of day one, with three guests who between them span all five ChefConfs. Jon Cowie, a staff ops engineer at Etsy, was on episode 11 and is at a third or fourth ChefConf. Fletcher Nichol, a longtime Chef user who now works on Habitat, has been to every one. Annie Hedgpeth is at a first-ever technical conference and has been in technology for about three months. Matty is at a third ChefConf and works at Chef, so parts of this are a report on the company's announcements from the inside. The show opens on Trevor pleading, "dear God, don't call it DevOps 2.0."

Matty notes the keynote theme of shipping delight and velocity. Jon says what has been most interesting over the years is that this rapid delivery has spread to much bigger companies that traditionally didn't work that way: the National Football League and GE spoke that day, and GE said it generates something like 40% of the world's power and needs to iterate on features. Trevor adds Alaska Airlines to the list and says the point is that they're succeeding, since so many have said they can't because they're big.

Fletcher brings an example to conversations with friends whose companies say they can't move faster: Standard Bank went from days or weeks to hours or minutes, and Disney said last year it had been at it five years, so a competitor starting now is five years behind a company that's already faster than it was two years ago. Fletcher argues the audit-every-three-years compliance theater "just does not ring true to me anymore." Matty says Alaska presented itself as an 84-year-old multi-billion-dollar startup that doesn't throw money at problems. Jon says it is easy to call startups disruptive when the cost of being wrong is slight, and it's cool to see an airline attack the idea that travel just sucks, including a demo of e-ink baggage tags that update from the mobile app.

Matty adds a customer who said Chef had started to break down their silos, because everyone in the room had never worked together until they started doing it. Matty also recalls enterprise customers saying Matty was the fifth person that day to tell them to be like Facebook, and the Etsy episode line that they're "not a unicorn, they're a sparkly horse." Trevor has just been in Singapore, running a Chef training at a PowerShell meetup where DBAs, bankers and machine automation people all had the same story.

Annie wants to show people that InSpec and Chef are accessible, and says, at three months in, that "there should be no fear." Matty says the idea behind Chef Compliance and InSpec was to communicate with code, so compliance and audit teams hand you compliant InSpec code in place of Excel sheets and PDFs, and has been asked whether they'd want to. Annie's point, which Matty loves, is that it is accessible, though not easy. Annie credits Test Kitchen, and a long conversation with Fletcher the day before, for lowering the barrier to entry, and says open source is making things more accessible to non-technical people, and "I don't think it's dumbing it down at all."

Matty says that moves the value to the content, not the arcane, and admits some veterans took protectiveness or job security from being the only one who knows. Matty's response is to let the robots do the boring stuff and use your big brain on other things.

Annie loved the Community Summit, calling it a hallway track times five. Annie proposed a security compliance topic and only about eight people put a dot on the card, which was informative, given Barry's keynote claim that compliance is the bottleneck. Fletcher's take is to give it a year or two: at the very first Opscode summit, where Fletcher got a free ticket for having contributed to Chef, conversations about testing infrastructure got mixed reactions, and it later became a real thing. Matty adds having asked Fletcher about Test Kitchen for Windows two years ago, then 45 people showed up to an open space six months later, and six months after that it shipped.

Fletcher says the DNA is the same, professionally done and with talks you want to see, but at the first one companies were half saying this is what we're thinking of doing, and by years two and three they were sharing lessons learned, from automating infrastructure to testing it to continuous delivery. This year's theme is going from someone's brain to production, which pushes people to talk about the business, and CEOs now come too, so the audience is the whole organization. Fletcher's mind-blown year was when Facebook and Disney talked.

Trevor recalls telling Jon last year, a month into using Chef, that it felt like lying to clients, and Jon replied "slow down, you're an expert because you can find the answers." Jon adds that Chef is a for-profit company whose future is in enterprises, but the open source core hasn't been lost: Facebook invests heavily in the community, Target contributes cookbooks, and the governing board and most project lieutenants aren't Chef employees.

Jon says earlier Chef web UIs were built by and for terminal people, and someone told Jon during the announcement "thank God Chef has a sane UI at last." It gives a view of converging nodes, changes deployed and compliance, and Jon hesitates to call it a single pane of glass but says it's a tightly scoped one. Matty likes that the decisions were about what belongs in a UI, meaning visualization. Automate's three pillars are Chef, Habitat and InSpec, which Matty notes are the three guests, more or less, and Fletcher corrects Matty that they're products, not projects.

Fletcher wants well-behaved services from Habitat. Startups with infinite time might build everything like Netflix components, with health checks, streamed metrics and self-clustering, but most business software, even a mainframe, doesn't, and if every service, even Postgres, behaved the same way, you could compose systems on top. People try to fit it into something they know, like Docker, and Fletcher says the confusion comes from the pieces. The guest says the seed of the idea "has legs and I can feel it in my gut." Matty says people wrapped shell scripts in execute blocks when they first met Chef, and Habitat needs the same step back: articulate what you want and let the product handle the sausage. Fletcher says there's a "Chefness," a consistent company DNA across three products that come at you at right angles.

Trevor has been hearing people ask what comes next for DevOps 2.0, and hopes that saying it on the show will bring enough ridicule that it never sees daylight. Trevor called the person after the keynote and said the thing they meant already has a better name. Jon says these labels are usually a logical evolution of what people have been doing, moving into new areas, and "we just happen to like sticking names on it that look good in headlines." Matty says jargon is "a linguistic shortcut for communication," which fails when it means different things to different people.

Annie loved the InSpec talk by Christoph Hartmann, standing room only, where the speaker tested something for compliance and remediated it with a cookbook in 20 minutes. Fletcher's moments were the GE presenter and a colleague's Habitat talk that revealed an automatic sharding approach that was new to Fletcher. Jon's favorite was Tim Smith on composable cookbooks with custom resources, using the Tomcat community cookbook as an example of one that grew to include Gentoo and systemd support until it did none well, an honest retrospective from a Chef employee. Jon calls Chef "a toolbox that gives you the tools to solve your own problems," and says that is why composable resources beat a drop-in cookbook. Trevor liked the CTO and psychologist keynote, which helped with a transformation project Trevor is leading in Singapore, and the Target talk on Chef and SharePoint, which they are working on open sourcing.

Looking ahead, Jon is more interested in organizational transformation as Jon's role moves toward management, and this is the first ChefConf where the guest isn't speaking. Fletcher is looking forward to Adam Jacob's second-day keynote, which has been a personal turning point at earlier ones, and to talking to people about whether Habitat means you need less orchestration. Annie predicts next year will be a lot more about security, since it's still siloed and a huge bottleneck.

Awesome videos from ChefConf:

Other delightful stuff:

Notes from 2016 DevOps Days DC session on DR for your career: http://e.devopsdaysdc.org/p/openspace-spades-A

Severance agreement timelines:

How to bounce back:

Use all your networks

Want to dig deeper into some of the things we discussed today?

Matty makes a triumphant return alongside Bridget to talk modern monitoring with two people who build it for a living. Jason Dixon works on Monitorama and open source projects like Graphite, was finishing up at Librato and about to move to a startup called RainTank, and Aneel Lakhani works at SignalFx, both monitoring-as-a-service providers. Aneel is nearing 20 years of full-time ops work, having started at 15 with a tech support job. The show opens on the exchange that Bridget calls serverless nonsense "because there are still servers. You just can't SSH into them," and Aneel adds, "There are always servers."

Jason describes the old Nagios model as monitoring how something is doing right now, which loses the historical context of how the service has behaved. Over six or seven years, open source projects for different functional areas produced the event stream model, with collected metrics, window threshold queries and alerts, and eventually paid services for specific areas of the architecture. Aneel says the old way only asked about one thing, whether it was up, and nothing about the cluster, the service or the user's experience, and between checks you have no idea how it is doing. That isn't enough if you measure yourself on real performance for real people.

On thresholds, Jason says monitoring works like security in depth, with layers, since no single answer says how a service is doing. Jason is not a fan of machine learning as the solution, though it is a great addition, and thresholds are fine where you know what to expect. What matters is business context: am I doing the work I'm supposed to be doing, not just returning 200s? Jason recalls a customer at OmniTI who said something to the effect that they didn't care if their servers were on fire as long as they were making money. Aneel says people think moving beyond static checks is only for ephemeral infrastructure, but what you always wanted was to know whether you were within your expected performance envelope, and now you can measure it. Matty adds a story about a sysadmin alarmed that a database was at 90% CPU when it was SQL Server doing its job.

Matty asks "how big is your pane of glass?" Jason says a single pane is specific to your role, and that tools should be dynamic, which is why Jason built dashboards that let you look for hotspots on the fly, and says horizon charts are handy for spotting problems at the surface. Aneel, who spent a long time at IBM watching single panes come and go, says each role wants one, but the systems shouldn't be walled gardens, since the CEO's may be Tableau and marketing's Mixpanel. Data has to flow in and out so everyone can compose their own view. Jason says "if you're siloing your data within your organization, I think you have bigger cultural issues." Aneel describes a company where one metric, the number of documents users open in a time period, is tracked by everyone from the CEO to network engineering, though not every business reduces to one.

Bridget cites James Turnbull's point that the operations team isn't the only customer of monitoring, and concludes that it should be self-service and composable. Jason loves self-service monitoring, and describes a Heroku tool that queried Graphite and returned an HTTP status code, so anyone could put a check into Pingdom in minutes.

Matty quotes John Sheehan, from an earlier episode, that "monitoring is just testing with a time dimension," and asks customers whether something important enough to monitor in production is being tested. Jason jokes that testing is monitoring without context. Aneel says the engineers at SignalFx bake metrics into code from day one and carry pagers, and that if you want to know whether a change had its intended effect, the same metrics have to follow it from code through test, QA, canary and production.

Jason asks about usage-based pricing, which both companies use. Aneel says it is the scale of monitoring you want to do, not the number of things, and that it gives you freedom to change rates and stop measuring things, like buying bandwidth, but it comes with the mindset of iterating on which handful of metrics matter.

Jason says the most common and hardest question is what to monitor, and the honest answer is that you're the only one who knows. Aneel adds that the approach has to be non-static. Aneel uses Kafka, which has no metric for average message size, though a change in message size affects throughput, unless the producers and consumers are designed for size to change. Bridget confesses to a cron job checking whether an Elasticsearch cluster had gone yellow, and Aneel says you should get one alert when a cluster changes status, not 40 alerts from static checks.

Jason cites an Adrian Cockcroft talk about systems moving from months to weeks to days to minutes or seconds with containers. You can't cron that, and hostnames don't matter since the system is gone before you finish a sentence, so you have to think in work units: how much work has to happen, and is it happening? Aneel says with Lambda-style systems you still measure the timing and performance of the functions and the experience downstream, and with service level objectives, changes in the underlying model change only how you measure. Aneel also says ephemeral infrastructure isn't what drives the need for metrics. A customer of Aneel's runs 100 physical machines with 20 to 30 Docker containers each across five locations, measures performance so precisely they don't need auto scaling, and still wants metrics.

Jason's advice is to start with open source until you hit the point where your time is better spent elsewhere, then outsource to someone like Librato or SignalFx, since knowing the technology makes you an informed shopper. Aneel recommends collectd as a collection agent, for its plugin ecosystem and, compared with every other agent tried in 20 years, its stability, though the warning is to watch its plugins. Jason trusts the team because many are OpenBSD developers, and says the biggest problem with agents is leaking memory and consuming CPU. Aneel adds that Graphite is a good starting point, and Jason mentions that the Graphite book is nearly done. For Windows, Aneel and Jason mention a collectd port by a team at Bloomberg, a PerfCounter reporter that SignalFx engineers wrote, and Metrics.NET.

Aneel returns to machine learning. In Aneel's experience in operations, nothing has done better than about a 50% false positive rate. The right use is as an additional tool, like a system that classifies incoming metrics as in-line or out-line and sends the outliers to another process, and "the single worst thing you could do" is turn on an algorithm and let it generate alerts.

Bridget asks about the difference between monitoring and alerting. Aneel says "there's no monitoring without alerting": in operations, you monitor to figure out what is worth alerting on, and you should page only when something takes you outside the performance envelope of your service. Everything else should be an event you can go back and interrogate. Jason says alerting is a subset of monitoring, and the same data feeds capacity planning and analytics. Aneel adds availability to the mix: "if something is available and non-performant, it might as well not be available," so capacity, availability and performance move together.

Jason's advice is to "try not to chase the shiny stuff," to start with basics like instrumenting your code and to ask why configuration management providers don't do more to emit instrumentation to storage engines. Jason also says that if a tool is painful to use, try another, since there's so much choice now. Aneel says no one will figure out what metrics matter to you: "If you don't take responsibility for figuring out what metrics matter to you, no amount of money and technology is going to do that for you." Matty's version is that the hardest part of monitoring is figuring out what you care about.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

For any devopsdays, try the code ADO2016! It should get you 20% off.

We have t-shirts now! And mugs! They are available at store.arresteddevops.com! Only unisex for now, but more styles coming! Buy one today. Or not. We’re not the boss of you.

Bridget records this one at DevOpsDays Toronto, in one of the open spaces on the second day, with a mix of people from the event. The guests are Dave Cliffe (PagerDuty) and Steve Pereira (Statflo), two of the local organizers, Sarah Kowalik of PagerDuty's operations team, representing a sponsor, Sean Walberg of the NFL, who spoke, and Joe Laha, the Arrested DevOps editor and a DevOpsDays Minneapolis organizer. Joe usually stays silent on these recordings, and here says the reason for coming to events like this is to "steal all the good ideas and implement them in Minneapolis."

Dave says this was the third Toronto event, in a couple of venues, with a diverse audience and a growing enterprise focus. Steve says it was the second year at the same venue. Asked what they were curating, Dave says they take the code of conduct seriously and had an incident to deal with promptly, and that they look for speaker diversity. Steve is candid: they did an outstanding job on topic diversity and a poor job on representation of groups and demographics, with no excuse, and next year they need to start earlier and go out into the community. Attendees were happy with the range of culture and technical talks, and Steve says they tried to serve the enterprise audience without creating an echo chamber.

Sarah has been to a couple of DevOpsDays in Australia and notes that Toronto has more of a focus on financial tech and banks. Steve adds that Toronto has many attendees from fintech and legacy banks, but the banks don't show up as sponsors, and Steve would like to see them invest at the organization level. Joe says London was very fintech-focused, with Barclays as a sponsor.

Sean's talk was about the team's chatbot, Waterboy, one of two talks Sean submitted, and Sean didn't expect this one to be accepted. Sean's team is very distributed and conversations were happening in private, so moving tools into the chat room made troubleshooting collaborative and left a history. Their tools let you look at a URL and see timing, or take apart a microservice to see how the CDN affects it.

Steve is a fan of ChatOps as an easy path into DevOps patterns: everything happens in the open, is recorded, and is a single interface to many backend operations. Sarah says PagerDuty's operations team moved from deploys to provisioning, decommissioning and Chef converges via ChatOps, so the team no longer provides "keyboard as a service." A plugin that returns context for an IP or hostname means "we stop getting questions about it," a very easy metric. Sarah adds that you should log chat to centralized logging so it's searchable. Dave, a product manager, likes chat in the incident response lifecycle, such as support paging the incident commander from the bot. Steve values not having to interrupt people. Joe says the organizers run their events in Slack, and an audiovisual vendor form posted once gets found by search.

Sean was a first-timer and had been told not to overthink open spaces. Sean enjoyed conversations about metrics, chat and incidents, and liked that the talks were "by people just like us." Dave says "you don't have to be a professional speaker in order to get up and talk at a DevOps Days." Bridget praises the talks about what didn't work, and Steve says that was curated on purpose. If you don't know what to expect, you may think the speakers are beyond your organization, so mixing success stories from companies like Shopify with teams that failed inspires people afraid to start. Dave adds "We all fail. We suck in places," which breaks down boundaries.

Sarah says open space selection was done differently than in Australia, where people pitch for about 20 seconds and others tally marks on cards. Toronto used a Trello board and a show of hands. Bridget likes pitches, since Sean pitching a talk about ChatOps beats the bare word on a board, and Sarah says a pitch framed as a problem gets a bigger response. The Toronto organizers also tried bringing the mic to people so introverts could pitch sitting down.

Bridget says that as an organizer, the approach is to chase the groups you want represented before the CFP opens. In Minneapolis this year Bridget hand-fed the link to a few people, and their talks were upvoted by the rest of the team with names and companies stripped. Bridget thinks a large bank should get an invitation to tell its journey. Steve says the work is continuous, since 2017 starts as soon as 2016 ends. Sarah, who'd usually talk on ChatOps, will get voluntold to give the ChatOps talk next year, and Dave notes they won't invite Sean back: "Sean's done. He had his one chance."

Dave says the different cultural pockets inside large companies are undervalued, and the only thing that works in an enterprise is land and expand, a change agent in one area. Bridget says bimodal IT is bad because it tells coworkers that some get awesome mode and others get sad mode. Sean says "I wish people would stop treating the old stuff as boring. It's the stuff bringing in the money," and that experimentation needs the users and traffic that are on the old stuff. Bridget agrees: "if the old stuff didn't matter, you would just turn it off."

Sean's earlier version of this talk was to federal workers in DC, where the idea of a chatbot doing work seemed foreign and the jokes didn't land. In Toronto people were far more open about their problems, which Bridget says is a hallmark of DevOpsDays, including DevOpsDays DC at the US Patent Office. Joe says single-track events have lulls in vendor booth traffic because everyone is in one room, and Steve says multi-track brings FOMO traffic. Sarah says it's recorded, so you can catch it later.

Looking ahead, Steve wants to get back to speaking, Sean has a ChefConf talk in July, Sarah looks forward to Monitorama, and Joe has t-shirts, AV and menus for Minneapolis. The thing Joe plans to steal from Toronto is a structured tech check for all the day's speakers half an hour before the event, since AV people like early testing more than "I'm speaking in 15 seconds and I haven't tested this." Steve says to take their laptops away afterward so they don't change settings.

devopsdays Toronto

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

For any devopsdays, try the code ADO2016! It should get you 20% off.

Bridget hosts solo, pulled together at the last minute, and talks security with two guests who agree on more than they argue about. Ben Hughes of Etsy last appeared on the show for the security episode about two years earlier, and Jessie Frazelle works on container security at Mesosphere. Ben opens with the line that "YAML is readable by humans if your humans are going through a stroke," a theme that comes back near the end. Ben introduces Jessie as the leading authority on running silly things in containers on a Linux desktop, and the only person who gets audio, networking and everything else working on a bleeding-edge Linux kernel. Jessie, meanwhile, has just gotten back from CraftConf in Budapest.

Bridget asks Ben what security even is. Ben's answer: you have the stuff, and you don't want other people to get it. Computers aren't as deterministic as we'd like, since CPUs and microcode have bugs and Rowhammer showed that physics can corrupt memory, so insecurity goes all the way down. The role of security people is managing that risk, and Ben adds that "security often loses sight of the fact that they are a business function." For most companies the goal is a profit, not being the most secure company in the world.

Jessie's view is measured: "you're better off with containers than without them," as long as you run them correctly. If someone gets into an app inside a container, the world they see is different from what they'd see on the host. It can't save the world. Ben puts containers in the broader category of sandboxing, like Chrome sandboxing Flash, and says the main thing you can do is reduce attack surface, especially by dropping as many permissions as you can. Bridget and Ben both remember jails and chroots.

Jessie explains unprivileged containers, from Jessie's blog post and CraftConf talk. Docker on your host runs as root, and adding a user to the Docker group also gives root, which some people don't realize. With unprivileged containers, the user starting the container is a normal local user with no added capabilities. Jessie also runs Chrome in a container with cgroup limits on RAM and CPU, but had to remove the limits because of a Chrome memory leak, so if Ben and Bridget see Jessie drop out of the Hangout, the out-of-memory killer is probably to blame.

Asked where the sweet spot is between securing everything and nothing, Ben says humans are really bad at risk analysis. Nobody says "safe ride to the airport," they say "safe flight," though Ben has been more terrified by taxis than by pilots. Ben loves a kernel-hardening project that makes whole swathes of kernel exploits stop working, but would rather have people use longer passwords and a password manager. Most compromises are found credentials or very old software, not an amazing new kernel zero-day. The dramatic wins headlines, logos and RSA booth sales, as with the $100,000 appliance that claims to stop all zero-days. Ben uses ImageMagick as an example of an exploit whose barrier is nonexistent: "If you can write a sentence, you can probably exploit it." A logo and a domain got more people to patch than a blog post did.

They wander into online voting, where Ben says the paper system is reasonably trusted and the government contracts go to companies like Diebold. On chip-and-signature cards, Ben has heard the banks didn't want to change two things at once. Jessie says the line is Linux in cars, though open source, Jessie admits, is better than every company writing its own firmware.

Ben's recent blog post, written on a flight back from Berlin, is about impostor syndrome in security, a field with a lot of posturing and an attack-defense mindset. "We should try and be nicer to each other." The field has a huge skill shortage, but it's an unwelcoming place if you have to be popping shells on day one. It also tends to want only breakers, and a team of breakers doesn't build secure software, it finds bugs in all the software you have. Ben's example is Wireshark, which parses loads of wire formats in a big C program: "Wireshark is just a CVE-generating machine." Jessie has a container for it, though it would need a custom seccomp profile, and Ben recommends capturing with tcpdump and loading into Wireshark in a VM or container.

Jessie chose container security because the problem of real multi-tenancy appeals, and says that nobody has 10 years of Docker experience, so you hire people who are good at what they do and can jump in. What Etsy actually finds more useful, Ben says, is people who can talk to developers and explain an attack, which won't get a conference talk but helps the business more. Jessie sums it up as "enabling people to do the right thing versus telling them that they were wrong in the first place," and Bridget adds that at Etsy corporate security enables people to do their job, which differs from how security usually interacts with the business.

Ben spoke in Berlin on the topic, whose name Ben blames on Gareth Rushgrove, to an audience mostly of executives. It included Pete Cheslock's image of the DevOps unicorn emitting rainbows while security shovels them out. Etsy doesn't use the term DevOps much, but the point is to get security involved early, embed security people on other teams and stop shouting at people for writing code with bugs. Ben also mentions an article on detecting curl piped to bash through server-side timing, since the shell buffers differently than a plain curl, which lets a server send different output.

Jessie says the talk was for anyone running containers, and surprisingly popular with the academic physics community, whose servers don't allow running as root. Real sandboxing needs custom seccomp and AppArmor profiles, which will land on the security team. Better tooling would help, since no one likes writing SELinux policy. Jessie's proof of concept for AppArmor uses TOML, which Jessie thinks should be JSON. Ben objects that "JSON isn't a config format," since you can't put comments in it, and that YAML's significant whitespace is "not acceptable in a config format."

Jessie predicts containers will keep getting more secure. Ben's bold prediction: "I predict in the next 12 to 18 months, there will be another OpenSSL vulnerability," and more ImageMagick bugs, since image and config parsing is a minefield. Ben thinks "the container security story is great in the kernel but is terrible on the actual things in the container," with hundreds of things running out-of-date code where there used to be one monolith. Bridget says that if you can't build images repeatably and roll them at a moment's notice, "you probably have no business using containers in production." Ben says the speed everyone was sold on disappears once you build a repeatable, testable build system, and Bridget answers that you can still push fast through CI with tagged images.

For wishes, Ben wants security to stop blaming everyone. You tell people not to click links in emails while employing a recruiting team to click on PDFs from the internet, so "the tools have failed. So make better tools. Stop blaming users." Jessie wants a desktop OS made of containers, like Subgraph, and people to stop making gigantic images. Ben adds: "People should stop using curl in Dockerfiles," and stop using HTTP there too. Bridget adds pinning versions, and Ben's advice is to find an ops person and work it out.

For any devopsdays, try the code ADO2016! It should get you 20% off.

We have t-shirts now! And mugs! They are available at store.arresteddevops.com! Only unisex for now, but more styles coming! Buy one today. Or not. We’re not the boss of you.

Ben:

Jessie:

Bridget:

Bridget hosts solo, with Matty and Trevor out this time, and talks about speaking at conferences with returning guest Ryn Daniels of Etsy. Ryn last appeared on the show for the episode on starting a new DevOps job, and is now finishing a book, Effective DevOps, with Jennifer Davis of Chef, which is known as the Yak Book because O'Reilly's animal for it is the unshaven yak. It is due in late May or early June. Ryn has also been building infrastructure provisioning tooling at Etsy, and just got back from the Codemania conference in New Zealand, talking about applying software development practices to operational tooling. The side project that blew up on Twitter was Necro Atsume, a heavy metal cover of Take on Me from Etsy's talent show, with a logo of a cat in corpse paint that became a Teespring shirt.

Ryn started speaking in 2013, when Jason Dixon, who organizes Monitorama, asked for a talk at Monitorama EU in Berlin. Ryn's first reaction was that there was nothing to say, and Jason's answer was "I follow you on Twitter because you have things to say." Ryn had just changed jobs and had opinions about monitoring changes they hadn't been empowered to make at the previous one, so the talk was about that, and it was well received. Ryn's reasons for speaking are to share stories as a way of helping others learn, and it helps a career to be an established speaker. Conferences also led to Etsy, and to meeting Bridget at DevOpsDays New York, where they were both giving lightning talks.

Ryn is involved with DevOpsDays New York, which anonymizes submissions to reduce unconscious bias, then rates them and discusses as a group, leaning toward newer speakers. "I don't want the DevOps community to become an echo chamber where we just hear from only the same people over and over again." Bridget says that is why Bridget spoke at fewer DevOpsDays in 2015 than in 2014. Ryn says some conferences are invite-only, which can produce a diverse lineup or a group of friends who look just like the organizer. Bridget adds that organizers have to encourage participation from communities who haven't attended. Ryn wants DevOpsDays New York to include more of the organization beyond dev and ops, and Bridget mentions a marketing speaker from Atlassian at Minneapolis.

Ryn is slightly less nervous each time, and thinks a little nervousness shows you care. Preparation starts about three weeks out with a written outline, sometimes as long as a blog post. The Codemania talk drew on a Code as Craft post. Slides come a couple of weeks before, starting with five or six header slides for the problem, the solution and what was learned. Ryn's message to organizers: "I'm not going to give you my slides 2 weeks in advance. I'm a professional." They're still iterating the night before. Ryn rehearses three to five times, since more makes the talk boring, and an Ignite, which has 20 slides that auto-advance, gets much more rehearsal. Full-day trainings are the opposite, because so much depends on the students. Co-speaking works best when it feels like a conversation, and Ryn will present on Nagios at Velocity with a coworker.

Bridget shares that the shortest gap between giving the same talk twice was before and after lunch at a local conference. Bridget used the same deck and it turned into two different talks.

Ryn's first step is to work out what you want to say, since enthusiasm makes for a better talk than something to check off a list. On slides: "If you have enough words on your slides that it takes people more than like 2 seconds to read them, you have too many words on your slides." Bridget adds that corporate decks are different, and Ryn says a deck should still make sense after the fact, unlike some of their own decks that are "just cat pictures with no context."

Next is finding a conference that fits. Look at past lineups. A tutorial submitted to a conference with no tutorial track won't be accepted. Ryn recommends the Callback Women Twitter account, which tweets CFPs from a wide range of tech conferences, and the Technically Speaking newsletter, which also notes whether travel or lodging is covered. Bridget recommends trying a talk at a local meetup first, especially a lightning talk.

As a reviewer, Ryn wants to know who the audience is and what they'll get, since the abstract often becomes the schedule listing. At a multi-track conference you have to say why someone should choose your talk, and at a single-track conference you should still think about who that conference's audience is and check past agendas for how-to versus advanced. Ryn loves 101 talks. "Talking isn't about you as a speaker. It's about the audience." Use the form fields, since they're there for a reason. Bridget, who reviews for Velocity, says vagueness hurts, and "don't leave this a mystery." Ryn suggests two to three paragraphs and a few bullets, since one isn't enough and more than three burdens reviewers, and to put detail in the field for organizers and leave a little mystery for the schedule.

For a speaking portfolio, Ryn keeps videos when a conference makes them available, and has a page on their website listing past events with links to slides on SpeakerDeck and embedded video. Bridget adds a high-resolution headshot, and a short bio saying who you are and what your background is.

Ryn has strong feelings on this. Some conferences don't cover admission for speakers, which Bridget calls complete nonsense, and Ryn points out a talk can take 30 to 40 hours of prep. Conferences that say exposure is payment don't understand that exposure won't cover the rent. Ryn gives leeway to new or small conferences, but a five-year-old conference with too many sponsors to fit on a page that doesn't cover travel and lodging is "taking advantage of them." Bridget prefers to take speakers at their word if they say their company won't pay. Both like a low-key speaker event beforehand, and organizers sending information about where to stay and what to do, since radio silence leaves speakers overwhelmed. Ryn says "there's nothing more stressful to me as a speaker than having no idea what I'm walking into," and praises Bridget's blog post about what organizers should tell speakers.

Ryn prefers to speak early in the conference so they can relax, though Bridget notes speaking later lets you refer to earlier talks. Bridget recommends going to the other talks, so you know what has been said before you add your voice, and can reinforce or push back on other speakers. Bridget's summary is to "go to the other talks, kids."

Afterward, people come up to you, and Ryn's advice to those people is not to be condescending, since the speaker has considered the basics. For Q&A, Ryn sometimes times a talk so there's no time for questions and invites people to find them afterwards, since fewer people make statements when they don't have a captive room. Bridget will sometimes ask a "strategic softball" that lets the speaker elaborate, and as an organizer likes it when the crowd moves outside the room. Bridget also suggests a buddy who holds your things on stage, and says a partner who is an AV professional plugs in and packs up Bridget's laptop. Ryn likes a friend in the front row.

They both recommend putting your Twitter handle on every slide so people can live tweet you, and Bridget embeds the tweets on a page for each talk and has shared them up the management chain at work. Ryn admits to being too polite and Midwestern to retweet strategically, and Bridget says conference organizers love it when speakers publicize their talks. Slides go on SpeakerDeck, and usually as PDF without speaker notes.

Asked what they'd tell themselves in 2013, Ryn says "it's hard to A/B test your own life," but would spread the talks out, since five in two months is exhausting, and would be less nervous about giving technical talks. They gave mostly cultural talks partly from imposter syndrome. Bridget says you don't have to be pigeonholed, and that Bridget's 2015 was all Docker until a job change sparked a wish to talk about how organizations learn. Ryn says branching out gets you to other conferences, like Codemania, a development conference that was out of their comfort zone.

They finish with ways in that aren't conferences: internal lunch and learns, Etsy's weekly Ops School series, local meetups. Ryn's last word: "I used to be super intimidated by all the cool Etsy people who were up on stage talking at Velocity, and now I'm one of them. You can do it too." Ryn has committed publicly to only four conference talks this year, including a keynote at Continuous Lifecycle London in May.

For any devopsdays, try the code ADO2016! It should get you 20% off.

We have t-shirts now! And mugs! They are available at store.arresteddevops.com! Only unisex for now, but more styles coming! Buy one today. Or not. We’re not the boss of you.