Arrested DevOps

Matty says up front that he's on the ops side of DevOps and doesn't know a lot about CI, so he asks the panel to explain it like he's five. Joe Hirn's answer is that CI is "essentially just a build server at its core," and that the problem it solves is the works-on-my-machine dilemma. He thinks of it as "a dial on the responsibility of a team": curing works-on-my-machine is one setting, and continuous delivery is turning the dial up on how much you trust the team to cover everything.

Mathias Meyer, who handles infrastructure at Travis CI, finds CI more interesting as culture than as tooling. It's about integrating changes into master often and iterating quickly, with the CI server as "the unbiased judge" of whether something works beyond one developer's machine. Joe agrees, and adds that having to stare at each other and ask why the build is broken, who broke it and who's going to fix it "sets the tone for a different culture on a team."

Matty's clients say they'll do CI, but they want their feature branches too. Mathias's response is a question: why can't you develop this feature on master, and what would it take to let you? GitHub, he notes, uses plenty of feature branches, but they ship them to a small share of production servers, so they still see whether the change works. If people insist, the answer might be a feature flip or better isolation, and he credits Jez Humble for pushing him to think about it.

Joe is blunter: "commits don't happen if they're not on master." He'll accept a private branch for saving your work overnight, and he points people to Paul Hammant's writing on trunk-based development. A feature branch, he says, is "almost like a little mini coup," and the longer it stays out of master, the longer before teammates can give feedback or use the helper functions you wrote. He has "taken down many Git flow posters off people's walls." Both allow that open source is different: when you don't know the contributor, the pull request model fits.

Trevor describes the branch-per-story workflow his team runs, where you keep merging development into your branch through the day so nothing that reaches UAT or QA gets broken. Matty's reaction: "Boy, that sounds like a lot of work." Trevor's: "It is. It really is."

Asked where an unconvinced team begins, Joe says "By doing." Point Travis at the repo, or download Jenkins and run it locally, and show teammates the build that's been broken for a week. Mathias says the real prerequisite is an automated build, which used to be a barrier for big Java and C++ projects (his first automated builds were Ant and Make) and mostly isn't anymore, since Django and Rails come with build tooling.

Matty's summary is to forget unit tests and coverage and just ask whether the build worked. Joe says that's the right first step, and that the CI server does the nudging from there: once it's set up, "you should feel this internal shame that there's not a single command that you can run to compile your project." Then the empty test phase suggests a passing test or two, and the deployment checkbox suggests the next step.

Matty asks whether using a CI tool for CI differs from using it to orchestrate workflow automation. Mathias says at its core it runs commands, so orchestrating a pipeline of unit tests, integration tests, QA sign-off and deploy is a natural evolution, and removing friction from shipping is good. Trevor's team runs everything through it: moving databases and code, and running unit and UI tests from development through QA and UAT to production. Matty runs Chef cookbooks through CI and spins up Vagrant VMs to check they compile, which a few years ago would have drawn a "you're doing what with the what now?"

He also tells a story about presenting configuration management to a client. An ops executive who had said nothing through the whole session asked, "You're telling me that I can have the developers do the work, but I still get to push the button that says it's okay because I know it's okay?" Matty said yes. The executive said okay, sold, and walked out of the room. Mathias adds that the word he keeps coming back to is confidence, and that automation only works if everyone keeps caring that the build is green and fast.

Joe says the Rails community's testing culture is such that a gem without a how-to-test note in its README isn't going to be widely used. His team tests first, to drive the design "as God intended, not in its diluted form," and the hardest part is choosing the isolation level for each test, such as whether to use Capybara or hit the database. Working in Clojure, where the ecosystem is less baked, has meant leaving "the padded, cozy, warm fireplace, bear-rug testing environment of Rails," and building things like the test database setup by hand.

For hosting, they use Travis CI for open source, and he likes seeing the Travis flag on a gem before he uses it. On client projects they've used CodeShip, and he praises its support, including help with firing up a Capybara server and its dependencies like PhantomJS.

Matty describes preflighting as committing to a staging branch or repo that the CI tool builds, with only passing changes promoted to trunk. Joe runs most of his tests in a local pre-commit hook and defers the slow ones, like multi-browser Selenium runs, to CI, and he calls a preflight gate "sort of a smell." His reasoning is that "there's probably a deeper problem if you can't trust your developers to commit code to master." Mathias agrees: "It's a barrier, and the question is, why do you put it up?" Trevor sums it up as "trust versus control."

Mathias points out that at Google, tens of thousands of developers commit to a single branch every day, and asks why your company can't. Joe's version is that a team with a Git flow poster on the wall lets everybody stand around it and figure out how they're supposed to develop software. Joe grants that forks and pull requests are a good model for open source, but for a team in the same room, he says, "it's really just an inconvenience." Mathias adds that "you're all on the same team," and that private forks show little trust in the people building the product.

For common mistakes, Joe starts with not having a visible status indicator in the room. His line is "you don't want to look at the status of the build. You want the status of the build looking at you." Otherwise people filter the CI emails, and someone eventually notices the build has been broken for a week. Mathias says visibility is also the argument against preflight checks: if you're worried about people breaking master, worry about fixing it fast.

Joe has seen teams gamify it, including a Hudson plugin that tracked who broke the most builds, and the traditional build gnome for the last person to break it. He cautions that it turns into punishment for people who are sensitive about it. What he prefers is a build master of the day who owns getting it running regardless of whose commit broke it, and the principle behind it: who broke the build doesn't really matter, and "is the build fixed matters."

Matt hates Subversion.

Trever attended a Chef training class and is super excited about it. Even though he only learned how to make it configure Linux machines.

This is a new section of the podcast where we introduce a new topic in just a couple sentences. This episode's "requirement" is Configuration Management.

Want to learn more about Configuration Management? Check out The Food Fight Show podcast!

Len Lagestee, an Agile coach who has been teaching or practicing Agile since 2004, and Patrick O'Brien, a lifelong consultant and project manager who spent years fighting Agile "tooth and nail" before converting, join Matty and Trevor. Trevor opens with a question Matty has long held an opinion on: should anyone outside the delivery team care how the sausage gets made, or is the team a black box that takes in a feature request and outputs a feature?

Patrick objects to the word should in that question. It depends on the culture and maturity of the organization, he says, because an unready organization is "either blinded by the transparency or intimidated by it" and starts grabbing at details. Len coaches transparency from the start, with an open invitation for any stakeholder to stop by a review, stand-up or planning session. The retrospective is the exception. He keeps it "a place for the family to talk about family business," since teams shut down and hold back bad news when managers with direct reports sit in. The review, where the product owner or a tester reads out each story and its acceptance criteria while the team demos it, is a separate ceremony, after which everyone else is excused. Trevor's team folds the review into iteration planning and rolls straight into the retro, where "anybody who's not a core member of the team is out of there."

Should work be split into dev tasks, QA tasks and UX tasks, or should the team just collaborate? Patrick's answer is that it works best with specific tasks going to specific groups, because that at least gives the illusion of ownership, and throwing everything out there is "like a steak to a pack of dogs." He is firmest about testing: don't let testers be the developers, especially at UAT, because "you're literally letting the fox into the henhouse," however tempting it is to reuse dev people for capacity.

Trevor's counterexample comes from that week. He'd set up the automation server to run click tests, and since he as a developer didn't know the tool, it made more sense to pair with the QA person.

Len says DevOps-flavored work is not a user story, since no end user benefits directly, so his teams write architecture or technical debt stories and size them in sprint planning alongside features. Sometimes a team will decide to spend "a quarter of our time" on technical stories and 75% on feature work. Patrick asks whether these get wrapped in an epic. Only if they're big, Len says, and what he'd really like is for them to line up with an architectural roadmap that gets merged into the product owner's roadmap. Business folks historically give a deer in the headlights stare and say "I just want you guys to build features," and a single roadmap is his fix. He adds that writing a story as "as a developer I need this" is a first sign of a bad user story, but the work still has to live somewhere, and once product owners see what it takes to deliver, some start asking the architects what the team needs.

Trevor's team files that work as chores, and Trevor points out that "it's hard to size a chore." Len adds that it's hard to test one too.

The recording dropped out here, and Matty came back from a staff meeting to a reminder that "this is Arrested DevOps, not Abandoned DevOps." The question he'd missed was what a Scrum Master does. Len's answer is to take a neutral stance on process and watch for team dysfunction, and above all to remove impediments, by shepherding the removal, not doing it personally. The role also ends up being "a bit of a psychologist," and Len's own exit strategy from a client is getting the Scrum Masters to make the same observations he does.

Patrick describes it from the trenches: enforcing whatever process the team agreed to, running the daily round of questions, and letting team members move the cards on the board themselves because moving something from dev done to QA ready is cathartic. He wants a coach, not a manager. Matty had first thought of the Scrum Master as the Agile cop before deciding that was a non-Agile thing to say, and notes that people often want the Scrum Master to be a project manager who writes reports. Patrick will come after you if you're late, "but I'll be nice about it." Len prefers a stealthier version: if a two-hour task has sat in progress for three days, whisper to a teammate to ask about it, because "the best Scrum Masters are the ones that have to say the fewest words."

Patrick says Agile begins as public humiliation for teams coming from waterfall or project management by heroics, and that he doesn't deny it to them. Matty asks that the humiliation stay inside the delivery team, not at the demo where a missed feature gets pinned on Joe, and Patrick agrees: "public humiliation within a microcosm." Len finds the phrase too strong and would sooner have the teammate ask "do you need any help with that?" because the developer is probably new, stuck, or afraid to raise an impediment.

Matty connects it to blameless culture. The internal accountability cuts two ways, he says: pick your teammate up, and also don't make the rest of the team fail the review. Len's version has the last word: "what can I do to help you get out of this foxhole?"

Matty asks whether a Scrum Master is a natural fit to coach DevOps-style collaboration. Len says yes for getting the right people talking, with an architect for the technical side. Leadership can't just tell teams to be more DevOpsy, he says. He'd ask about their pain points instead, and bring those to communities of practice across teams, since "if it's a mandate from leadership, I very rarely see that actually work." He also mentions clients who say they're going Agile and then mention "it takes us 4 weeks to get something out into production," through CAB boards and approvals, and he tells them they won't be very agile until that's addressed. Len ventures, with a caveat that he's throwing the number out, that maybe 10 to 15% of organizations do Agile well.

Patrick's reaction to "we're going Agile" is "what do you mean by that?" Matty says the same happens with DevOps, with people asking to "download the DevOps, hire me some DevOps." Patrick thinks people treat it as a tool they can install, and if you're going to do Scrum, do all of it: "That's not a Scrum, that's a status meeting" if it isn't daily. He also describes a middle ground between waterfall and Agile that he's implemented, and Trevor supplies the name: "We called it fragile."

Matty asks why an operations person isn't part of delivery all the time. The delivery triangle has a developer and a tester, "and I don't remember who the third one is. I know it sure as hell wasn't ops." His previous organization invented a system engineer role inside the team, without doing an awesome job of it. Len would like anyone with a vested interest in shipping to be on the team, but with 30 teams that's expensive, so a lead engineer or architect who understands DevOps stands in, if you can even find one. Matty, a 20-year sysadmin, argues you should have far more developers than sysadmins, which is why the answer has to be practices, not headcount. He returns to the challenge from the top of the episode, to pull a sticky off the board that isn't yours, which drew no reports back ("either nobody did it because you're a bunch of chickens, or nobody told us how it went"). He can't take the task that says build a server in production, but "why can't it sometimes be test the software that Trevor wrote?"

Len's remedy for teams that throw work over the wall is to have them support the application for a while, interrupting or even aborting the sprint when something breaks, so they feel the pain of release. Matty calls it the classic Amazon example of developers carrying a pager. Patrick describes trying to get his team to own a story all the way to deployment, and Trevor notes that after he picked up more ops tasks, more than one person asked to pair with him. Len's summary is that any place you'd have had a handoff is now a co-creation point, which Matty repeats back at the end of the hour to a "well played, sir" from Len.

Things get off to a great start during the retro, where Trevor complains about destroying his USB 3.0 drivers due to a Win 8.1 upgrade and Matt turns into a Cylon.

Test Kitchen is now officially 1.0, but does’t really support Windows, but that doesn’t stop Matt from wanting to hack it to make it work anyway.

“Testing is any action taken to give you information about the actual state of your software, vs your assumptions” – Lanette

“A lot of developers look at testing like insurance – it’s not going to prevent a disaster, but it’s going to help you mitigate those problems” – John

Spirited discussion about the value of code coverage as a metric, and our panelists mostly violenly agree that it is not a valuable number in a vacuum. We also discuss that it is possible to approach all of life like a QA tester.