Arrested DevOps

Matty and Trevor record in front of a room on day 2 of the first DevOpsDays Chicago, the second in the Midwest after Minneapolis, and Trevor's first DevOpsDays. Joining them are J. Paul Reed, who is "defecting from The Ship Show" for one episode, Michael Ducy, who is named the show's "special field correspondent" for events the hosts can't attend (Michael's reaction: "I'm the Walter Cronkite of DevOps now"), Jason Hand of VictorOps, and Mark Cornick, an Ignite speaker who has been on a self-described DevOpsDays tour this year.

Mark's summary for anyone who hasn't been: it's highly focused on learning, full of ways to participate, and inexpensive, since Mark hasn't paid more than $100 for a ticket. It's usually two days, with invited talks in the morning and an open space in the afternoon, and the sponsors are "just here to be part of it on the same terms as everyone else."

Michael says the Midwest tends to lag in technology adoption, going by data from RedMonk, with New York a little faster and San Francisco "so far in the future." Over the last two years, though, DevOps meetups have taken off, with Minneapolis one of the largest in the country, and Michael thinks the Midwest is doing a lot of cool things without getting together to talk about it. Paul says that on the coasts people are building "Airbnb for your cat," while these cities host industries that are the backbone of the country, like Target in Minneapolis and the futures exchange in Chicago. Michael lists the enterprises that turned up this week, including Allstate, Kroger, McDonald's, Kohl's, and Crate and Barrel.

Paul tells of talking to a Crate and Barrel attendee after having ordered wine glasses two weeks earlier. When Paul mentioned the checkout hadn't gone smoothly, the attendee said the feedback was useful, since they'd changed how they handle cookies, and handed over a test credit card number to try again. Jason adds that at both Minneapolis and Chicago, a large majority of the room raised a hand as first-timers.

Matty, one of the organizers, admits to having sold Chicago short during planning. Matty proposed a blameless postmortem open space, thinking nobody knew about them, and heard that it "got really advanced really fast": "nope, dummy, we know that." They sold out at over 300 people and turned some away, and day 2 turnout was nearly the same as day 1. At the evening event, at least half a dozen people told Matty they'd thought open spaces sounded dumb and changed their minds.

Jason was skeptical too, and now open spaces are Jason's favorite part. Mark, a self-described introvert, says one of Mark's first open spaces was Tom Duffield on enjoying tech conferences as an introvert, and that "as a result, here I am": an Ignite talk followed, and now a spot on this podcast. Justin, a developer from Chicago at a first DevOpsDays, expected someone talking at the audience and then drinks, and instead found topics too small for a full talk but worth discussing. Matty notes that badges say participant, not attendee, because the conference is meant to be interactive.

The venue was on the 37th floor of the Sears Tower, which Justin still calls Sears and Paul calls "the name that shall not be spoken," and Paul says it's the highest DevOpsDays anyone's been to. The hashtag was #DeepDishDevOps, and the evening event was an arcade with Dig Dug, Joust, and Teenage Mutant Ninja Turtles, which Paul says is a lot harder when you're not 10 years old and crowding around the machine.

Matty says the email to Michael went out in spring of the year before, asking why Chicago didn't have one. Michael says the Chicago meetup was one of the first in the country but "a little flaky at times," a remark an organizer has ribbed Michael about, and that in Minneapolis, at a social mixer with four people standing around, Bridget Kromhout put a hand up and said "I'll organize it." Nobody in Chicago had been passionate enough to take it on. Matty asked Patrick Debois for the names of people who'd shown interest in 2013, held a kickoff meeting and sent a SurveyMonkey survey, and ended up with a group in maroon shirts who mostly didn't know each other. It was, Matty says, very flat.

Matty thanks Minneapolis for its 18-page postmortem document, and says you have to make it your own, though it's nice "when someone else has made a couple mud pies." Early on there's little to do and at the end there's a lot, so around six weeks out Matty asked for a reaffirmation of a loyalty oath they had never actually taken. Shannon, another co-organizer, compares it to planning Shannon's own wedding, and says what stands out is that everyone is a volunteer with no ulterior motive. Aaron says organizing gave more than attending would have. Organizers weren't allowed to give talks, first because of conflict of interest and then because they'd have no time, and they ran an on-call rotation on an email address, an idea borrowed from Minneapolis. Michael adds that Amsterdam went all out with headset walkie-talkies.

Michael says the talk to look up is the compliance one, since there's "a lot of fear and uncertainty when you talk about moving faster" and the community rarely discusses audits. Paul liked that it defined the auditors' vocabulary, given that operations people speak their own jargon at everyone else: an auditor asks whether you have a control for something, and ops people have no idea what that means.

Steve Pereira, a consultant visiting from Toronto, says Paul's talk "DevOps == 42" was a highlight, because Steve had expected a talk claiming DevOps was the answer and got one about not knowing the right question. Paul says Matty came up with the title, that the schedule link went to no abstract, and that it seemed so appropriate that the abstract stayed empty.

Michael learned about the twelve-factor app methodology in an open space on hybrid cloud, where most of the developers hadn't heard of it. Paul recommends the #DeepDishDevOps hashtag and the Giordano's pizza served at lunch. Trevor recommends a terminal library that plugs into the Jimmy John's API so you can literally "sudo make me a sandwich," and the local Daisy Cutter beer from Half Acre. Steve mentions rollout.io, which is about pushing hotfixes to a mobile app without waiting on App Store approval. Matty's is DevOps Against Humanity, a Cards Against Humanity-style deck of DevOps jokes started by Bridget Kromhout, played at the after-party where a few people won Raspberry Pis.

Jason Dixon of Librato, who started the Monitorama conference, Bridget Kromhout, an operations engineer at DramaFever who helps organize DevOpsDays Minneapolis, and Pete Cheslock of ThreatStack discuss why conferences are worth attending. Their first ones vary widely. Bridget's was LISA '95 as an undergraduate. Matty's was COMDEX in about 1996, which was really a trade show, and then there was no other until TechEd 2008. Pete's was Surge in 2012, because earlier employers didn't see the value of sending people. Jason's was probably LinuxWorld in New York in the early 2000s.

What you get that you can't get another way, Pete says, is the hallway track, the people chatting between sessions, which at many conferences "is actually the most interesting one." Jason isn't there to learn from the talks, since there's plenty of content online. Jason goes to meet the developers behind open source projects and have the conversation comments can't give you. Bridget's example is standing in a hallway at Velocity Santa Clara in 2013 and asking an AWS employee about Hadoop and MongoDB problems. The employee turned out to have written the MongoDB on AWS white paper. Bridget went home and re-architected everything, moved to SSDs and removed unnecessary sharding, and it cut enough off the Amazon bill that Bridget is "pretty sure that paid for the conference plane ticket."

Engineers hear networking and, "if it's not dealing with routes and subnets," tune out, Pete says. But Pete can reach out to old contacts from years ago and ask if they've done something, or know someone who has. Pete pitches conferences to managers on recruiting: tech recruiters often receive 20 to 30% of a first-year salary, so instead of paying a recruiter $10,000 or $20,000, send engineers to conferences. Pete met Jason at one, they stayed in touch, and when Jason was looking, "you kind of jump the line." Trevor says meetups do the same job, and that's how Trevor and Matty met, at an Azure meetup.

For people who don't know anyone, Jason's answer is admittedly useless: start your own conference, which is what Jason did to get the speakers and interactions Jason wanted. The market, Jason thinks, is moving toward small, community-focused conferences. Matty likes that as making the thing you want. Bridget went to Velocity Santa Clara knowing only one person, and "sat at a different table at lunch every day," looking for people with blue and pink hair, most of whom turned out to work for Etsy. Pete notes everyone at lunch is there to meet everyone else.

Matty, who describes being very introverted, tweets to the conference hashtag a week or two before and follows it, so the first introductions have already happened virtually. Bridget says if you're not on Twitter, stop listening and make an account, since badges often show handles. Jason put avatars on Monitorama badges because people know the avatar and not the face, and admits to feeling like an outsider at a front-end conference where everyone knew each other. Trevor mentions imposter syndrome, and Bridget's tip is that "pretty much everyone loves to talk about themselves," so ask polite questions and listen.

Bridget, Jason and Pete all find events through Twitter, and Pete worries that it's an echo chamber. At DevOpsDays Boston about 10% of the room had been to one before, and Bridget says over 80% in Minneapolis were first-timers, with a later poll at 75%. Pete found a language-specific event, Mountain West Ruby Conference, was a change of scene. Pete also spoke at an Agile conference and polled the room on company size, finding about 90% worked at companies of 10,000 or more, "a room full of Scrum Masters," attacking the same problems at a scale none of the panel can imagine. Bridget points to the Enterprise Scale DevOps conference for that audience.

Pete has too many conference bags now that messenger bags are too much for Pete's back, and had to work hard to get a Monitorama hoodie from Jason. Jason isn't anti-swag but likes practical stuff, and as an organizer wants people to think about the landfill footprint: Monitorama's badges and lanyards are biodegradable, and tries to talk sponsors out of excess swag. Bridget says DevOpsDays Minneapolis gave speakers small battery packs, and that DevOpsDays New York skipped t-shirts and used the money to build a well in Cambodia. Pete's suggestion is to offer donating instead of a free t-shirt at checkout, because "who needs more stuff really?"

Bridget describes a conference as a space where you're at work but not really at work, with bad Wi-Fi, alcohol, and people you don't know well, so professional boundaries get fuzzy. Bridget's rule: have fun, but ask "would I like these people to try to recruit me for their next startup?" Bridget also says being welcoming means not making assumptions about people, like assuming a woman must be a recruiter or an older person is trying to sell you something, and instead asking open-ended questions. Trevor has seen people ask a third party whether someone is a recruiter or an engineer, when the answer is "go talk to them."

Bridget got Patrick Debois to fly from Belgium for the closing keynote of DevOpsDays Minneapolis, and Andrew Clay Shafer, who couldn't attend, introduced Patrick over a Google Hangout, so a "giant floating head" appeared on screen as a surprise. Matty spent an hour talking with Jeffrey Snover at ChefConf and offered Jeffrey the opinion that PowerShell was as readable as Perl.

Jason tells of an OSCON booth demo of a failover firewall on OpenBSD, where Jason gave the whole spiel to someone who, as Jason tells it, turned out to be the creator of FreeBSD, which went unnoticed until the people behind started laughing. Jason also says Monitorama's diversity effort included inviting a speaker on the subject, and that the audience's reaction moved Jason. Pete's story is a sushi dinner in San Jose after Velocity where the person across the table turned out to have written a project Pete depends on, and Trevor's is being asked "are you that guy from the podcast?"

Ben Hughes of Etsy is one of the senior network engineers there, though Ben's team looks after infrastructure, about a thousand machines and a hundred laptops, and leaves networking to a NetOps team. The guest fell into security after dropping out of school on discovering 2600 and Linux, and at one point had around 1,500 accounts on the Unix system at Ben's high school. Ben came to Etsy from Puppet Labs, learning about DevOps Days there.

Ben's current work includes Python on Midas, a host-based intrusion detection system for the Mac that Etsy released with Facebook, and Go to trick Logstash into using different TLS ciphers. Next week Ben is at an incident response conference, because for breaches "the days of it being an if are long gone."

Matty recalls Ben's point about phishing: the question is what you do once someone gets phished. Ben says Etsy focuses on people first. You can't shout at a recruiter for opening a PDF, since that's their job, and "blameless postmortems work for blameless phishing campaigns." What matters is that people tell the security team when something screwy happens, and "the more you chastise people, the less they will tell you." So when someone reports a phish, they say thank you and give them an Etsy gift card. Being able to respond matters more than believing you have full coverage: "Nothing's 100% in security, despite what some people will try and sell you."

Matty raises the belief that you can secure your own stuff better than a cloud provider. Ben says that if you bring the "armadillo" security model of the last 20 or 30 years into the cloud, "you're going to have a bad time," because unless you build a private network, there's no trusted internal network at all. Matty passes on a Microsoft data center manager's remark that after the US government Microsoft is the most attacked entity on the internet, so who better to learn from.

Ben's caution is about outsourcing security to someone else. Without a complete overview of your organization and your threat model, "you're going to be hit by a surprise," and security vendors sell silver bullets using fear. Etsy is big enough to build its own tools, as Netflix and Square do for their particular problems, and those tools "you can't buy commercially."

Ben says throwing something over the wall to security at the last minute is as bad as throwing it over to operations, and everyone who's had a bad time says they should have talked to security sooner. Given budget to allocate, Ben would spend it on a security team early, not on products or pen testing, because until you map what you're defending and from whom, you can't build defenses. Etsy's first security person was one of its toolsmiths who took on security on the side, and Ben says you can find people interested in security in operations, development or QA.

For developers, Ben's number one request is "please stop turning off TLS verification," and the age-old rule of never trusting user input: "never trust users," in the nicest possible way.

Matty rants about transparent HTTPS proxies that decrypt traffic, which break a gem install because the certificate doesn't verify, so people turn verification off. Ben gives the benefit of the doubt, since there are good reasons to inspect web traffic, and suggests trusting the proxy's CA cert in the gem config. The guest says people will route around anything that stops them doing their job, with SSH tunnels, OpenVPN or tunneling IP over DNS: "However you need to get out of a network, you will." So "we can either work with these people or kind of be avoided by these people."

Trust famously doesn't scale, Ben says: at 50 people you know everyone's name, and at 10,000 you don't trust other buildings. Etsy defaults to trusting everyone, monitors everything, and is transparent about what the security team is doing, which is how you get trust back. The guest sums up DevOps as "why don't we work together?"

For a listener's question on secrets, Ben says Chef encrypted data bags "turn your encryption problem into a key management problem," and points to Nordstrom's Chef Vault, which Matty says ships with ChefDK. The guest would also like Square to release its resident-only in-memory file systems built on FUSE. Ben's main advice is to get to where you can change a secret and nothing breaks: once sharing works, randomly change some passwords, "it'll break a ton of stuff, and you'll have a terrible day," but you'll know next time credentials leak, which they do. "Pastebin is a treasure trove of other people's mistakes, and GitHub Gists is just a shopping cart full of logins."

Personally, Ben uses a password manager with around 600 credentials and two-factor on everything. Matty asks front-end developers to name password fields password so managers can find them, and mentions a six-word Diceware passphrase, learned by making 1Password prompt for it every time for a couple of days.

Ben's first misconception is that HTTPS is too slow, and the rebuttal is istlsfastyet.com. Ben's second is the media's fixation on zero days, which "99.99999% of organizations do not need to worry about," when the unpatched Linux kernel and users' four-character passwords are bigger problems.

Trevor asks how to start security where there's no team. Ben says find the person who stays up late on IRC, give them time, and get leadership to treat security like backups. Some security is better than none, and "smaller incremental gains is always the way to do it." To sell it, Ben says you can do it "not out of fear, but out of this is the responsible thing to do," and customers pick the vendor that's more secure. Matty realizes there's no padlock in a mobile app to tell you whether traffic is encrypted, and Ben says the ideal is for platforms to show one only when certificates are validated. The guest tells of a game site that showed a picture of a padlock next to an HTTP URL.

Matty asks whether InfoSec is still the right name, given the band Information Society, and Ben says it's still used, that Ben's own LinkedIn said Security Monkey for a long time, and that hacker versus cracker is settled. On The Phoenix Project's security manager, who's cast as a villain, Ben says many see security as Big Brother, and that Etsy's team held a hackers party watching the film Hackers, taught people to pick locks, and ran a security hack week. If you're the villain, Ben says, "start being nicer to people because you work with them."

Ben also thinks security needs to buck up its own ideas: the military jargon, like the phrase threat intelligence, which "brings me out in a rage," makes the field seem aggressive. The guest wants less preaching that it's 100% or nothing, noting that even CPUs ship with errata lists, so "computers are broken from the ground upwards" and it doesn't all have to be doom and gloom.

Aaron Blythe of Cerner, and John Smyth and Nate Burleson, both senior consultants at 10th Magnitude, all started as developers. Aaron's first taste of ops was deploying a Rails e-commerce application solo: Aaron learned Bash on the fly, decided there had to be a better way, and went from Capistrano to Chef. John says the move into pure operations came because an application was performing badly on SQL Server, and that the split between the two "wasn't so clear-cut." Matty says John has talked about doing this for years, but "we used to just call it working."

Nate says the Unix sysadmins from early in Nate's career taught Nate Perl tricks, and were good programmers who knew the command line and automated their jobs. In Nate's telling the division between dev and ops is modern and artificial, though sysadmins did tend to keep programmers away from the systems, "for fear that we didn't know what we were doing, which was true." Trevor says the move begins with necessity: something doesn't work, you find it isn't your software, and if nobody else can change it and you can, "you become the vector for that change." Trevor's own start was manually changing permissions on the wwwroot folder and blowing up IIS.

John says everything is layers of abstraction, and the deeper you know them the better you can troubleshoot. John's development background started with learning assembler and the hardware by programming it. Aaron says the exchange goes both ways: the development team teaches ops code reuse, and ops has a vast knowledge Aaron never had when the development environment was set up for Aaron, who couldn't have built it from scratch.

Matty quotes Adam Jacob on cultural change: see what happens to sysadmins when you tell them they have to do a code review before an infrastructure change. Aaron says the development teams Aaron works with spend about half their time reviewing or testing someone else's code, which ops colleagues find hard to believe, and that with Lean, "you got to slow down to be able to speed up."

Matty admits not really knowing how to do a code review, and the guests explain. John says there's no right way but there's probably a wrong one: reviewing a sample of someone's code for style, or reviewing commits before they're committed. Trevor says review is subjective without agreed standards, like clean code conventions, and that it's mostly accountability. Trevor describes a project with 100 design patterns "because the person who wrote the code had no one to be accountable to." John says "there's no better way to make sure the code is readable than to make sure somebody reads it other than you," and that with Chef they check attributes are used consistently and things are decomposed. Aaron says tools now let you comment inline on a single line, like a stray rm -rf, and that review is a learning experience for new hires, who are told to grab a line and ask what it does.

Trevor likes pair programming for the same reason, and Aaron has paired with an ops person, staying late and going from whiteboard to keyboard to system, and says "when you hit that magic, it's pretty sweet."

Nate was surprised by the immaturity of tooling in the Windows ops space, including source control and code review, since Nate had found the most technically adept people tended to be in ops. Trevor's surprise was how far over the line toward ops Trevor had already gone: building and configuring machines, adding users and firewall rules, because nobody else was in that role.

A Twitter question asks about incentives for bringing dev and ops together. Aaron says there aren't concrete ones, and that the two groups sit on campuses 15 miles apart in Kansas City, so Aaron's team goes to the ops campus one day a week and ops comes to theirs two days a week, though no manager said they had to. They do it because they "want to be physically in the same location and humanize each other."

Matty says metricizing collaboration is tricky, and John says formal structure doesn't dictate relationships, since teams that are separate but whose people work together have accomplished the goal, and a joint team "doesn't prevent somebody from being a dick." Matty adds that collaboration doesn't require friendship. Matty and John worked together for years at a bank without socializing outside work and collaborated fine.

Trevor's advice for a developer who wants to understand ops is to talk to an ops person, and Matty calls that not really advice, asking how. Trevor: "Like a human being?" John says the opportunities to collaborate are situational, and that when you have a problem you should ask questions and not pretend to know: "Nothing makes me crazier than when somebody feels the need to answer a question despite the fact that they don't know what the answer is." Aaron asks what the most frustrating part of someone's job is, and is blown away by the answers, since Aaron didn't know anyone had to worry about those things. Trevor adds not to ask "to ask the question," but "because you care." Matty asks how John would react if someone interrupted during a fire to say it looks stressful, and John says the reaction would probably be that of a sarcastic ass.

Aaron deployed an application on a Friday at 5 o'clock, and during dinner with Aaron's wife, Aaron's boss's boss said the CSS wasn't loading. It worked on one of the two nodes and not the other, so Aaron had to leave dinner and fix it. John's was a single storage array that periodically ground to a halt and took down a fully virtualized data center. Nate released a new version of an error-logging SOAP service the day before Nate's honeymoon, and it caused an infinite loop of logging its own errors: "the plane lands in Argentina, and my BlackBerry starts lighting up." Nate's lesson was to never release on a weekend or before leaving town.

Matty tells one about a coworker who was handed a decommissioned database server, and panicked when the storage array wasn't visible. Matty checked the SCSI cables and found that, in a hurry, the coworker had plugged the two RAID cards into each other and the two storage arrays into each other, and for at least a year Matty called the coworker Loopback.

You Suck at Technical Interviews

John Vincent - DevOps The Title Match

Learn Linux

ConEmu (Console Emulator):

Nueske’s slab bacon

Jez Humble’s ChefConf talk

TeamCity

Encourage extension Visual Studio

Lumosity Mobile

What exactly does it mean to be a sysadmin? How do you define “sysadmin”?

What is your favorite thing about devops? How does it change your life as a sysadmin?

Matt has a belief that sysadmins are inherently cynical. Is he an idiot?

What do you miss about “the good old days?”

Alex Howells asks: “How often do you think 'Fuck it all, I'm going to be a plumber or electrician?'"

Video of guys changing tires while car is on two wheels http://www.youtube.com/watch?v=5WLwRg3erm4

Amazon Certified Sysops Admin Associate http://aws.amazon.com/certification/certification-levels/certified-sysops-admin-associate

Sasha Rosenbaum, the show's first returning guest, and Dave Gerding, an associate professor at Columbia College Chicago who also does management consulting, take up how to ask for help, with Matty's voice shot after the Agile 2014 conference. Dave teaches the McCarthy protocols, a management doctrine published under an open source license, whose cornerstone skill is asking for help, "which is kind of a funny idea since everyone thinks that they do it and almost nobody does." In the protocol you say "will you help me?", which separates asking from what he jokingly calls announcing a problem. Walking into a room and saying "boy, my shoes are really dirty" is announcing a problem, since you haven't asked anyone for anything.

Announcing a problem, Dave says, is a cop-out that lets you avoid being vulnerable, and it isn't common in tech to lead with vulnerability. Sasha adds that it's a two-way street: it takes confidence both to ask and to respond, and even if you can't help you shouldn't shut people down so that they don't want to ask again.

Matty points out that with open source you don't call a support hotline, you go to IRC or a mailing list, and the community has to welcome you (The Ship Show did an episode on this, linked below). Sasha says she gets few responses when she posts on social networks or Stack Overflow, and goes directly to people she knows. Matty says crowdsourcing on Twitter only works if the people who know the answer follow you, and now that people who know Chef follow him, he needs that help less, which is a vicious cycle. Trevor's approach is to find out who the right person is and ask, and Matty says that if you say Nathen Harvey three more times "he will show up in the podcast like Beetlejuice."

Dave's tip for small open source projects is to kick $10 into their PayPal donate button before asking anything, which in his anecdotal experience gets a better response. He says not asking is "leaving a huge pile of free value on the table," and points out how cheap transacting is. Sasha adds that people you tag or message are usually happy to help, because it makes them feel good and gives them a chance to share their view of their product.

Sasha says the person you ask can go wrong in two ways: by leading you down the wrong path instead of admitting they don't know, and by being condescending, like sending you a "let me Google this for you" link. Trevor adds there's a difference between that and asking "did you Google the issue first?", which Dave used to ask him at work, and Dave apologizes to Trevor and the entire internet.

Matty describes the help vampire, from a Stack Exchange article: someone who asks what everyone has asked, seems unwilling to Google, and expects you to do their thinking. Because experts get deluged, he suggests showing you've done due diligence, and vouching by retweeting someone's question to people you know. Sasha distinguishes strangers, where she does her research first, from a coworker, where she asks first ("what are the best recommended steps?") and then Googles the hell out of it.

Dave says asking a person is different emotional work from posting to a group, and that he's often had the problem reveal itself as soon as someone leans into his frame. Sasha recommends the Ask the Duck article on rubber duck debugging: you don't need the other party to speak, just to phrase your problem as a question, which lets another part of your brain look at it. Dave says he'll go get a rubber duck tomorrow, and Matty says he had a Dalek on his desk but "you would just get exterminated." Dave: "It has one answer for all problems."

Dave says the question format softens ego, because we're often "solving the solution instead of solving the problem," and asking makes us drop the concept we thought was right. Asking a human, he adds, also gets you socialization and sometimes something better than you'd have come up with.

Sasha says the ego problem hits people at the beginning and the end of a career. Beginners feel they must solve everything themselves and that asking makes them look weak. Senior people feel trapped by the need to keep their authority, while "best senior people that I know are absolutely not afraid of admitting that they don't know." Consulting taught her that "you don't know anything," so there's no shame in asking. She notes the waste when you spend two days on a problem that someone at the next desk solved yesterday. Dave calls it sad when executives won't be vulnerable, since they seal their own fate, cut themselves off from new ideas and stop anyone sharing them. Trevor says practicing this teaches you who your go-to people are.

Trevor raises the excuse that you'll be bothering someone. Matty says the answer is boundaries, and that a tweet is low-cost, while cornering a speaker at a conference isn't. He cites Seth Vargo's Twitter bio saying he doesn't reply to technical support questions, so tweeting one at him would violate his boundary. In an office you can just ask: "how do I ask you for help in a way that is effective?" Dave: "will you help me ask you for help?" Sasha prefers asynchronous chat over "do you have a second?" because it lets her finish her own task first.

For people who freeze after one no, Matty says offer options instead of saying no: help in an hour, or write up your question and he'll come find you. Dave thinks that when people say they're afraid of hurting someone's feelings, they're often afraid of their own being hurt by hearing no. His advice for someone hearing no all the time is "you have to ask for help about asking for help. You just got to pop a level and go meta for a minute." Sasha says she finds it hard to ask in her personal life but easy at work, because she focuses on results, and suggests others do the same. Dave says gender is clearly a big parameter and that, stereotypically, men are more likely to fall into the trap of not being vulnerable. His last word is "do it more."

Matty records from the first DevOpsDays Minneapolis, the first one in the Midwest, and his first DevOpsDays and first Ignite talk. With Trevor away, Chef's Julian Dunn co-hosts. Julian says this is about his fourth DevOpsDays and that what's interesting is the local flavor. In an open space the day before on how not to be a pointy-haired boss, someone said direct feedback is harder in Minnesota because of a "Minnesota nice" complex, where giving direct feedback is considered impolite.

Ryn Daniels, the one-person ops team at GameChanger, gave a talk called "DevOps is Dead" that uses a simplified Gartner hype cycle. It argues the term has been used and misused until it's a buzzword, especially for recruiters, and tries to return to first principles. At her company the developers are all on call, which she says aligns incentives, because they're the first line when something breaks and care more about the quality of their own product. She calls it the best DevOpsDays she's attended, with more culture than tools and plenty of open spaces.

JP Morgenthal, who leads cloud computing and DevOps practice at Perficient, is also at his first one. He says people at other conferences won't open up to a stranger about a broken job or a bad boss, while in an open space they share quickly, and the discussions are "far better than any cloud or DevOps conference that I've been to so far." He notes a small team like Ryn's can make headway that a Fortune 2000 company with embedded fiefdoms and legacy systems struggles with, and asks what a company with 40,000 people in IT can learn from a one-person ops team's revenue per employee.

Matty brings in Jez Humble's point from the previous episode about maneuver warfare, and adds that enterprises used to keep quiet about their DevOps work, so people heard only about Facebook and Netflix, and he talks of "no-logo customers" who do the work but won't go on record. His view is that big companies are made of small teams, and small teams are made of people.

Patrick Debois asks JP whether he'd follow command and control if he trusted the command. JP's answer is that in the military the alternative is death, and that the people below "look up and go, what an idiot. I could do it so much better." Matty thinks the answer is that leaders earn trust by showing trust: he'll still coach and sometimes can't share information, but in a culture of trust people assume there's a reason. He says he said trust "like 35 times, and I think there's a reason." Ryn says visibility and transparency are what built her trust in managers, and Matty's rule is that "transparency is the default, and you override it as needed and as rarely as possible."

An audience member, a group technology director, says he oscillates between high-trust and command-and-control and describes directional shepherding, where strong individual contributors form a mesh toward a shared goal. Pure command and control fails because "no one is omnipotent."

Matty wonders whether an organization's culture drives its software architecture, since in places without trust service-oriented architecture never occurs to anyone, because you can't rely on a contract with people you don't trust. JP and Ryn both name Conway's Law. Julian asks whether it's like Moore's Law, something to respect, or something to explode. JP reports an open space the day before on "anti-Conway's Law," concluding that it's an observation, and asking whether organizing yourself on purpose would shape the product. Some examples suggested it did. An audience member adds that language matters, since calling work a product instead of a project changes how a team feels about it.

An audience member from SPS Commerce says enterprises re-evaluate trust with statistics, since they need business numbers, and that incents teams to make the graph look okay. "Are you gonna come to work and do a good job, or do you wanna look like you're doing a good job?"

Matty recalls a Scott Adams story about a company that paid testers a bounty for every bug found and developers a bounty for every bug fixed, and overnight a black market in bugs appeared, until they'd paid out around $6,000 in the first week. The audience member suggests getting the Freakonomics authors to study DevOps culture, and JP recommends the book Business at the Speed of Trust.

Matty asks Patrick and Bridget Kromhout how everyone ended up here. Bridget's version is that she showed up at a local meetup where people were hoping to start a DevOpsDays Minneapolis, was asked to help, and said yes, and that with a great team such things are possible. She told Patrick it's a franchise: "We're not gonna always do things your way." He's run about 30.

Patrick tells the origin story. After 15 years working in government, he was moving a data center and looking for process ideas. The team tried Scrum, which was too slow for day-to-day work, then Kanban, and he wrote a paper for the Agile conference in Toronto in 2008. At the same conference Andrew Clay Shafer had scheduled a birds-of-a-feather session on Agile infrastructure and gave up when nobody came, since Patrick arrived after he'd left. They connected on Twitter, and when Patrick said someone should hold a conference, someone tweeted back "why don't you run your own conference?" He took the open space format from Agile events in Belgium and added morning talks to inspire the afternoon discussions.

Bridget says the hardest part was keeping the schedule on time for the livestream audience, since people on Twitter were asking if it was down. Patrick offers as evidence of how she runs things that when a vegan speaker couldn't have the hotel's cookie, she went to a food co-op to buy a vegan one. Bridget's reasoning: "the chocolate chip cookie is not accessible to you, you're not having a good experience."

Several audience members are at their first DevOpsDays, and praise the quality and depth of the conversations. One asked what the unicorn and goat things were about, and says the strong sense of community is what people are looking for. Matty says his own Ignite talk started as a joke ("All good talks start as a joke, that's my new theory"), and that the room is full of people who want you to win, so it's a good place to speak for the first time. Patrick says he writes presentations to sort out his own ideas, and the audience listening is a bonus.

Bridget takes up the in-jokes: she went to her first DevOpsDays the previous October, so it's not a closed community, and the team that called itself the Sparkly Post-DevOps Princesses got the worst score in the O'Reilly animals trivia, while a local team of people who didn't consider themselves DevOps walked away with the prizes.

An audience member suggests the community define its own identity instead of leaving it to recruiters. Matty relays an open space idea from a Chicago attendee, "what do you want marketing to say about DevOps?" which he loves because it means "stop complaining and help us do it right."

Jez Humble, co-author of the Continuous Delivery book and the forthcoming Lean Enterprise, contrasts continuous delivery with the phase-gate approach, where planning, development, integration and testing come in sequence and the integration and testing phases "telescope and be very unpredictable." Instead, he says, have something very small deployable from day one, even if feature number 1 is just a status page, and keep the software releasable at all times, prioritizing that "over doing new work." He points to the HP LaserJet firmware team, who don't update printers ten times a day but found that keeping software releasable "changes the economics of software development." Matty says he first heard the story on DevOps Cafe, yelling at his car that it wouldn't work for his company, until the firmware example made him say "oh, okay."

It works, Jez says, because it forces you to deal with scalability, availability, logging and monitoring early. Everyone lists those as requirements, but "knowing that you've got to do something is very different from verifying that you actually did it," and the idea that you can fix performance or operability later "is just false."

Starting is harder with a brownfield system, he says, but that's no reason not to. Continuous delivery isn't a project you plan and finish, it's "just continuous improvement": make the release process boring, and start where the biggest payoff is, which you can find by mapping the value stream from check-in to release. If you fix a part that isn't the constraint, he says, you won't change end-to-end cycle time. Step 0 is version control. Step 1 is automated builds with fast feedback, plus the discipline to fix a red build immediately.

Matty asks whether it's still continuous delivery if each product has its own pipeline. Jez describes two patterns. The Amazon and Netflix way is many small, loosely coupled services, deployed independently and versioned side by side, which needs strong monitoring because a request may pass through 100 services and you need to trace where the latency is. Facebook and Google build a huge binary, and Jez says Google compensates with very rigorous code-level CI, running the tests of every downstream dependency to give quick feedback. If you go monolithic, he says, "you have to make sure your CI is really, really, really good."

Matty raises data, since you can't roll back a schema change. Jez points to the expand-contract pattern from Michael Nygard's Release It: never change existing objects, add new ones. To split an address field into two lines, add the new columns beside the old one, have the app read from the new columns and fall back to the old, and write to both so the data migrates lazily. Then you can roll back the app freely and later batch-migrate and drop the old column. Jez says he's heard Facebook makes no guarantees about what database version is in production, so developers code defensively. The cost, he says, is an added layer of indirection and complexity in the app, and what you get is deployment flexibility: "So again, trade-off."

Continuous delivery, Jez says, changes how developers think: breaking large changes into small increments that keep trunk releasable, not going off on a feature branch for days. He cites the Puppet State of DevOps survey for data that working in small incremental steps increases IT performance. To developers who say some things can't be broken up: "There aren't." The question is what you optimize for, and "we don't actually want to optimize for how fast I can say I'm done on my feature branch." What continues to astonish him is that developers who love new languages resist changing their practices, and he asks "why did you go into the technology industry if you don't want to change the way you think about things?"

Trevor asks what versioning means here. Jez's answer is small identifiable changes so you can reproduce any state for debugging, and so that when a soak test finds a performance regression across 20 check-ins, you can binary search for the one that caused it.

On what has changed since the book came out in 2010, Jez says mobile, the cloud and the Internet of Things have grown and a lot of tools have arrived, but practices and process haven't. He says the industry has "a terrible grasp of our history," is bad at communicating and experimenting with process ideas, and is still cliquey. He thinks it will be "5, 10 years, at least" before continuous delivery is standard, and that "this is an echo chamber that we're in right now."

Matty and Jez both misspell continuous, so Jez looks it up on Google, which shows the word's origin: con plus tenere makes continere, "hang together." His theory is "we use continuous because it's easier to spell than uninterrupted." He then suggests searching for recursion, and Google asks whether you meant recursion.

Trevor passes on a listener's question from someone at a company of 35,000 people with 9,000 in IT, where infrastructure staff are 15 miles away. Jez says ThoughtWorks has seen these problems close up, and that a lot of it "is just excuses." He suggests assuming it can work and asking what little thing you could do. Large companies, he says, have friction, and what works at scale is what he found in both the Toyota Production System and maneuver warfare: everyone knows the mission, and you tell people the outcome, not how to get there.

As examples, he says Google has over 10,000 developers who can all check into trunk and revert each other's changes, except for locked-down crown jewels, and that Amazon spent 2001 to 2005 re-architecting a monolith into services, partly to decentralize authority, with the architecture mirroring the organization, which he calls Conway's Law. The obstacle is leadership that clings to command and control, which he says "hasn't been fashionable in military circles since Napoleon destroyed everyone else in Europe in 1806." If leadership is political, people on the ground have to work under the radar until they meet someone incentivized to block them.

Matty remembers a Jez talk in Chicago where he figured most of 100 people were afraid of being fired for the wrong move, and says he told his sysadmin team "let's just pretend it will work and see what happens." Jez says Toyota sets outrageous goals and tells people to work out how, but that it needs trust. He tells of the NUMMI plant, where Toyota decided hiring should be central, not by your own boss, because "your loyalty is to Toyota, not to your boss," which lets people say no or even automate themselves out of a job. The alternative is a stretch goal in an atmosphere of fear, where someone says "we'll ship it in a month" and you spend your evenings eating pizza. It's fine to commit to a month, he says, if you decide what's in the release.

Asked about anti-patterns, Jez lists developers not changing how they think, ignoring a red build, always deprioritizing test and deployment automation for features because management tracks utilization, and automating "horrible, broken manual processes" step for step. You will screw it up, he says, so don't blame people, and make failures safe. He borrows "safe-to-fail experiments" from the Cynefin framework, and applies it to a top-down mandate to automate all tests in QTP: automate five tests in JUnit and put them in the pipeline, since five tests that mean something when they fail beat a comprehensive suite "that everyone ignores because it's just red all the time."

Can you do continuous delivery without infrastructure as code? Jez says probably, if your system is small, but manual point-and-click configuration makes each release error-prone and disaster recovery unpredictable. His acceptance criterion is "can I recreate my production system's state purely from information stored in version control?" with production data as the only exception. He mentions Martin Fowler's thought experiment of blowing up a data center with a flamethrower and timing recovery, and Google's disaster recovery exercises, including disconnecting the campus from the internet. If enterprises were truly risk-averse, he says, they'd worry about disaster recovery, and "the question is, what risks are you actually averse to?"

Asked about the next big thing after continuous delivery becomes standard, Jez says the world has larger problems than automation, and that in software each new technology means relearning things learned 15 or 20 years ago in another domain, like test and deployment automation for mobile. What he'd actually like is for the industry to become less "pseudo-meritocratic" and white-male-dominated, with more women and people of color, and he takes the defensiveness he sees as a sign that things are slowly changing.

Pete Cheslock, Nathen Harvey and Randi Harper each open with what DevOps means to them. Pete takes the historical view, the "cultural and professional movement," a little tools and a little culture. Nathen says it's everyone working toward a common goal, typically a delightful customer experience, across development, operations, business, marketing, sales and finance. Randi came back to engineering after four years away, found a title called DevOps, and concluded it was "sysadmins who actually know what the hell they're doing," the people who aren't afraid of strace and developer tools and who work with developers.

The first misconception, Nathen says, is taking a sysadmin role and stamping DevOps engineer on it. Pete has stopped fighting it, since it works as a qualifier for a senior operator who writes code, and Nathen has capitulated too: "fine, internet, you win." Randi likes the title, and says it's a bridge, because developers see it and don't just see an ops person they throw things over the wall to. Matty's version is that in Chicago there are no sysadmin jobs anymore, only DevOps engineer listings that say install patches on Windows servers and perform backups. Candidates with DevOps on their resumes, he says, answer his question about continuous delivery with "but I'm good at patching, and our group was called DevOps."

Pete says he meets companies that all say the same things about continuous delivery and business value, and some who ask him to "bring me the DevOps." When he lists what would have to change, they say "we don't actually want to change anything. We just want the DevOps." Nathen adds that hiring a DevOps expert and expecting to have DevOps now is "a really good way to fail," and Trevor compares it to bringing in an Agile coach to Agile you up. Pete: "We're going to Scrum like crazy."

Matty also has a preference for how to say it. People hear DevOps as an abbreviation of development operations, he says, when it's a portmanteau, dev plus ops, and that's where it slides into meaning release engineering. Pete says the enterprise teams being called DevOps look a lot like release engineering, "but you know what? That's okay."

Matty borrows code smell and asks what DevOps smell would be. Nathen's first is developers saying "that's DevOps' problem." Matty's is a DevOps project about implementing tools, which brings back Nathen's line that the only DevOps tool is someone with the title Director of DevOps, to Pete's objection. Pete says you can sell DevOps but not buy it, and that a tool on a broken culture gives you "2 problems, essentially: a broken culture and a broken tool," while Nathen says that if implementing the tool is the end goal, "you've clearly missed the point."

Pete's smell is DevOps talk that involves only developers and operations, and leaves out product, QA, security, and even sales, which sells features that don't exist. He has seen initiatives succeed because executives gave real trust, and fail because they wanted the DevOps without trusting anyone to change things. Trevor's contribution is that "the usage of the term DevOps is inversely proportional to a company's understanding of DevOps," and Pete proposes calling it Trevor's Law.

Randi sees it as culture: the same work she did before, with a culture that lets her tell developers about a memory leak instead of adding more memory to the server. Pete says start with culture, but by 2014, if you're not using config management, "you're probably doing something wrong." Nathen says he'd like to call bullshit: culture and tools are too intertwined to separate. His example is version control. Copying a file to .bak isn't version control, Subversion is a central authority, and with Git "you're trusting everyone on your team to have a copy of the repository." You can't claim to trust your team and also insist on a centralized system.

Randi points out there's a difference between having Chef or Puppet installed and using it. Matty says the consultants on the call mostly hear tools conversations, and that Jez Humble told a Chicago audience the bar in the industry is low. Matty adds that culture change "has to happen from the top. It can't come grassroots." Trevor's summary is that doing just culture or just tools is another way to screw it up. Pete agrees: you have to do both with purpose. At Dyn he picked Chef over CFEngine because operations and developers had each pushed a different one and weren't talking, and the right choice, he says, isn't Chef or Puppet but "the thing that delivers value to your customers." Nathen adds, "change for change's sake is stupid."

A listener question asks how to convince coworkers that it's Dev+Ops and not DevOps. Pete says you can't change culture, and that trying is "a fool's errand," but you can build something that shows value, as he did at Dyn with a Chef workflow integrated into a CI pipeline that CFEngine didn't have. Matty adds that you should show and invite, so others help make it better. Randi notices that the best DevOps people she knows are public speakers or podcasters, because this is a communication problem.

Nathen describes two approaches: put developers and operations near each other and have them eat lunch and get beers together, or lock a cross-functional team in a room with a project, since "sometimes the right way to break down silos is to build another silo." He adds not to make the bastard operator from hell the first member. Pete says he took passionate, curious engineers, trained them on something like configuration management, and embedded them in teams as "patient zero," so the excitement spread. Matty's condition is that these teams be temporary, "bridges," so they don't become the team everything gets pushed onto.

Asked for the one way to screw up DevOps, Randi says not to use one tool for everything. Specifically: don't use a Jenkins build job to execute scripts on servers or restart services. Pete's version: "Jenkins is not orchestration." Matty knows an organization that uses Jenkins as the job scheduler for its product's batch jobs, "because hey, Jenkins does that stuff." Randi: "Just because it can do it doesn't mean it should do it."

Matty floats an idea from a night of drinks: whether a tool's own opinions can drive culture, the way an opinionated framework like Rails pushes you in a direction, or the way you train a bonsai tree. Pete first hears it as tool bias and talks about people looking down on others for their language or tool. Trevor takes to the actual idea: using Chef forced him and Matty to answer each other's questions and reach a shared understanding.

Nathen tells how his own team learned. He would change the code, test on a new machine, and then, nervous about running the client in production, make the same change by hand on every production machine. He calls it his fault, not the tool's, and says that once the team trusted the tool, they let the client enforce policy continuously, which changed how they worked. Pete says developers who distrust a new tool will say "Chef changed this file, and it broke everything," when it changed the file because they wrote code to do it. Nathen: every Java developer who moves to Rails writes Rails that looks like Java, and sysadmins write procedural Chef.

Matty is helping plan the first DevOps Days Chicago and asks how to reach the people who haven't gotten religion. Pete calls DevOps Days "a massive echo chamber" and is looking forward to his Agile 2014 talk for a different audience, and thinks enterprises are where the next big shift is. Nathen suggests going to new cities, bringing a developer along to every event since these gatherings lean toward operations, and reading books and listening to podcasts about lean manufacturing and change outside of technology.

Remap your CAPS LOCK key as Ctrl key. Also new Supermarket site for Chef - supermarket.getchef.com

Anything BUT Jenkins.

Ranjib Dey, a system administrator at PagerDuty with earlier stints at ThoughtWorks and Google, joins Matty and Trevor. His definition of software delivery runs from gathering a requirement from an end user to the finished feature reaching customers and drawing feedback. His three consistent themes are incremental changes, regular changes, and a whole process that is automated in a reliable way.

Continuous delivery, for him, is "the attitude towards delivering your software in a well-tested and incremental fashion," usually an extension of CI. What matters is that every merge passes through review and testing, whether that happens up front, as in XP, or after the commit, as with pull requests on GitHub. From there you can release a feature to a subset of users, by geography or age group, for example. In the process, he says, you also "learn the risk-taking abilities," and the organization goes through cultural change, because that wasn't common at the start.

A small commit, to Ranjib, embodies a feature that is independently verifiable: "If you cannot think of a feature and we cannot associate that with our commits, then it's probably not a full commit." Infrastructure work counts, so moving from canary deployments to dark launching is a feature with its own tests, and known errors get a regression test so they don't come back.

More tests is better, Ranjib says, but you don't always get the time. For a greenfield project he'd start with functional tests, since the priority is whether a feature works, and add unit tests as design issues and tech debt show up. Good tests also work as documentation, which helps a new hire, and let you change code you no longer fully understand and know from the failing tests what you broke.

Integration tests are the black-box ones that involve external services, such as a load balancer with five unicorns and two databases, and checking what happens when one goes down. That takes a full simulated environment and automated failure injection, which he says you add as you grow. Matty prefers that meaning of the term, notes the Continuous Delivery book calls these non-functional tests, and points to episode 2 on testing. He also says that a cookbook is code, and Trevor tells him not to knock it now that he's tried it.

Ranjib adds that these terms mean different things depending on the concern, and gives a packaging example. Testing the libcurl code is unit testing, building the package and making a curl call is functional testing, and checking that Fedora works with that version of libcurl, the kernel and glibc is integration testing, because the user wants libcurl working together with the whole platform.

Matty asks whether PagerDuty has one pipeline for all its products. Ranjib says the setup shows their history. There was no CI when he joined a team of 14, so the first step was a Jenkins server with feature branching and per-project builds. Pull requests then piled up in the queue, so they moved that to Travis, which reports red, green or yellow on GitHub and posts to HipChat. Jenkins now takes master after a merge and deploys it to downstream environments. Ranjib is particular that a true pipeline has fan-in and fan-out stages, which Travis doesn't do. Matty's version is that you can have a virtual pipeline, where everything must go through the same steps, without a single physical one.

Ranjib describes a canary deployment as a router, such as nginx, sending a particular URL to a subset of backend servers, which sandboxes a change to those servers. Blue-green deployment and dark launching are similar approaches. Matty asks about rollbacks and mentions Mark Burgess's paper arguing against them. Ranjib says the ability is nice but often hard or impossible, and his organization tends to roll forward, because its heavy automation and testing make a small fix safe to ship. For database migrations, he says, split the release in two, so that the first stops using the column and the second drops it, which preserves the rollback state. Containers let you keep a couple of old versions on the host, and even when you don't intend to roll back, you might do it for a short time to buy time for a hotfix. His rule: "you should always optimize for rolling forward. You should never optimize for rolling back." Matty repeats it "so I remember it."

Trevor is still nervous about a CI server deploying straight to production. Ranjib says there's no best way. His current favorite is a HipChat bot that deploys, with GitHub webhooks showing merges and reviews in the same room, though he stresses the integration is thin and the orchestration behind it matters more. Some people want a button, and you sometimes have to give them one in the CI server. The deciding factor is trust: all the tools "reflect your company's culture and trust at the end."

Matty puts on what he calls the psychiatrist hat, and asks why a person is nervous, because you don't want to steamroll someone's concerns. Ranjib says Trevor's concern is legitimate, since in ten years he's seen 80 to 90% of outages come from deployments. Automation doesn't avoid failure, it makes it faster, so the answer is to make failure "extremely cheap, extremely affordable," with multiple environments to test workflows in. Some things, like migrations on very large databases, will need different tools than a standard CI system. Trevor's summary is to fail quickly and fail cheaply.

The biggest anti-pattern Ranjib sees is people becoming hardliners about the nuance of a buzzword instead of its meaning. Arguing over what counts as integration testing is "a bike shedding discussion," and deploying 20 times a day doesn't make sense for a JVM application that has to restart each time. His second is accumulating tech debt, especially skipping monitoring: a service you can't monitor is "not gonna fly," and isolating services early is much cheaper than later. He'd also rather have small groups with independence, including over tools.

Asked for one piece of advice, he says there's no magic bullet: "Nobody's giving you the button that you click and it will solve your problem."

What is software delivery? There are a lot of approaches to this subject- what does "software delivery" mean at PagerDuty?

What is your idea of "best" way to deliver software, or line of best fit?

What gets in the way of companies or individuals delivering software?

What are some patterns and anti-patterns for consistent software delivery?