Arrested DevOps

Bridget, Matty and Trevor bring in two longtime ops people to talk about dependency management and points of failure, after the npm left-pad episode rekindled the perennial argument. Pete Cheslock runs operations and support at Threat Stack, and Charity Majors just co-founded Hound after being the first infrastructure hire at Parse, which Facebook acquired. Bridget invited Charity for some ranting, and the conversation delivers.

Bridget points to whoownsmyavailability.com, which keeps telling you that you do. Charity loves it as a gut check, but says it is also not only you. It's your team, your processes and the teams around you. Operations people tend to have a "hero-martyr complex," and Charity wants to say both that "You can't pass the buck" on a vendor or platform and that it isn't up to you to be a hero. Pete says blaming is comical, since the one guarantee is that "if it's online and on the internet, it's going to go down."

Matty notes podcasters have their own version, own your own RSS feed, since everyone loved FeedBurner, and it's a reminder that the service you depend on is somebody else's business. They pour one out for Parse, though Charity says it didn't fail for technical reasons and won't say more. Bridget was impressed that the Parse community pulled together on open source options. Charity says the same holds for Parse, Heroku and AWS: if an availability zone goes down and you chose to be single-homed, that may have been the right choice, but you can't say it's their fault.

Trevor reads from the npm blog: a package many projects depend on, directly or indirectly, was unpublished by its author in a dispute over a package name. Bridget's summary is that the namespace is global, and lots of sites were importing the package live, so when it disappeared they couldn't build. Go, PyPI and Chef Supermarket users have all had a version of this problem, which brings up Pete's rage-tweeted advice to vendor your dependencies.

Pete says Threat Stack has a lot of Node.js, and it wasn't affected because they use Artifactory. Vendoring means taking a copy of a package, bringing it internal and serving it yourself. Every dependency adds risk. Pete vendors all Chef dependencies and doesn't talk to Supermarket, and does the same for full Debian packages such as Cassandra's third-party packages. Charity says Cassandra stopped publishing an older package they relied on, and they couldn't bring up new instances without upgrading the cluster. Pete says it was the exact scenario they hit: only the newest version was kept.

Charity adds a caveat about company lifecycle. At a three-month startup vendoring every package would be a waste of time, since availability requirements and time are both in different places. But there's a stage where these failures affect your ability to push code and roll back, and at that point you should have a local cache of every apt package, gem and npm package. Bridget recalls a broken Docker registry release pushed without changing the version number, which they worked around by copying the official registry container into their own account. Trevor says Jenkins only hosts the latest Debian package, and a release that broke Trevor's Groovy scripts forced learning to build a package feed.

Matty points out that the enterprises everyone teases for banning internet access from build systems didn't get burned, though not for availability reasons. Matty adds that the depth of dependencies is scary, since people use systems that depend on left-pad without knowing they touch Node. Charity's example is a first boot script that turns out to call half a dozen apt repos and gem sources, when the goal was just to bring up a host. When trying to fix a scaling problem, the last thing Charity wants is to "detangle somebody else's outage."

Pete is one of a few people who sign the packages Threat Stack ships to customers, and says there is "a high pucker factor" when thousands of systems can grab what you push. Charity's advice on resilience is to avoid reinstalling every package every time a node boots. Build a base image with Packer, bake in everything except the package that changes, such as Cassandra, and cache that in an apt repo you control, so if it fails you can blame yourself and fix it quickly. As Charity puts it, "you should have as few things that can fail while you're doing critical things as possible." Bridget's team at Drama Fever built AMIs with Packer from a Chef definition using Jenkins jobs, and a failed image build is annoying but doesn't affect production.

Pete says Threat Stack started with a ten-minute Chef run on every node, and only packerized the base when uptime and scale demanded it, so nodes come up in a minute. Charity's message on best practice is "Dude, it's contextual," since over-engineering early can kill a company as surely as neglecting things later. Pete adds that a launch date moved up to re:Invent forced choices, and the important part is going back to pay down the tech debt. Matty says you make a decision about acceptable risk with your eyes open, and revisit it as the business changes.

Asked what reduces operational risk, Charity says decentralizing accountability. In Charity's words, "your operations engineers honestly are not responsible for your reliability." Software engineers should own their services end to end, and ops should be in the first design meeting asking how it scales and how it's instrumented. The toss-it-over-the-wall model fails because "the feedback isn't effective if it isn't immediate and if it isn't somewhat painful." The goal is not never going down but limping along in a degraded state when a component fails, which Bridget calls circuit breakers and continuous partial failure.

Pete adds a security angle. Threat Stack monitors what systems do, and connections to Russia or China turned out to be apt repositories served over anycast. Vendoring cuts those random connections so an odd connection stands out. Matty warns you can overcorrect: if doing the right thing is hard, people will route around it. Matty quotes Sascha Bates that "if you treat your employees like children, they're going to behave like children." Pete says people will route everything over port 80 if it's the only port open, and Bridget recalls finding Comcast IPs in a production MongoDB security group that someone had added from home.

Pete's starting point is an object store like S3. The Ruby gem deb-s3 pushes Debian packages to S3, which is basically apt repos on the cheap. Charity suggests wrapping package installs in your image build so they stash a copy in S3, and Bridget says to make it a step in the CI job, not a 42-item checklist. Pete points to paid options like PackageCloud, Artifactory and Nexus, and says "open source is only free if your time is worth nothing." Bridget suggests not depending only on public Docker Hub, having been paged more by Docker going down than by Bridget's own stuff. Charity says that if it's just caching package files, "that's seriously one of the easiest problems in computer science."

Pete raises verifying that the packages you get are the ones you expect and treads lightly around GPG. Deb-s3, Artifactory and PackageCloud all make signing easy. Charity has built Debian packages at every job, and says the packages that have to be built become the seed of the local repo. Matty says the tension is trust: you don't want to build everything, but vendors mean trusting someone else.

Charity says an agent that isn't open source is a non-starter if it runs on Charity's own hosts. When you get a stack trace, you need to read the code that generated it. Trevor likes walking traditional closed-source shops through Chef cookbooks and answering "what does this action do" with "you tell me." Matty adds that if a vendor goes out of business, understanding how it worked matters. Pete's example is an engineer who wanted to gem install Sensu plugins, when Pete preferred to bring them into a cookbook. A week later a chunk of the code turned out to be dead code that would have confused them if they had just installed it. Bridget says you want a good picture of what normal looks like before 3 a.m., and Pete says every dependency adds risk that some organizations can absorb.

Charity offers a cautionary note from repeatedly moving the source of truth from an untrusted repo to GitHub, "and then you have 2 problems."

Matty asks about bosses who want someone to blame or sue. Pete says you sometimes have to play the game and call it a security issue, recruiting the biggest curmudgeon to hammer on it. Matty says SLAs are a lawyer's favorite thing to say, since they're made up and never apply. Charity says it's about audiences. Developers trust you when you give the dirty details and take ownership, while executives report to people who don't want to understand software. Translate for them: paying for something and getting an SLA won't reduce outages, and probably means more of them because "we won't be able to debug them." Charity says to translate that into corporate-ese, but it will build a better product.

Pete's closing point is that everyone's stage, risk tolerance and budget differ, and "you don't have to listen to what talking heads on the internet say" about vendoring every dependency. Use each crack in the mortar as a chance to make things a little more durable. Charity says the answer is "it's you, but not you personally": a successful culture is one where every person feels they own it, and it's not some other team's, a DBA's or a vendor's problem. Bridget notes the subtlety in English: "is you singular or you plural?" And the answer is yes.

Matty and Trevor talk personal brand with Andrea Javor, Senior Director of Global Digital and Media at Beam Suntory, and Michael Hedgpeth, a senior software architect at NCR in the hospitality division, whom Matty worked with to bring Chef into NCR. The episode grew out of a talk idea Matty floated, where Michael's feedback was enough that Matty said Michael had basically written the talk. The pitch is that people problems are the hard ones in any DevOps change, and a personal brand is how you get people to listen without resorting to self-aggrandizing boasts or claims of thought leadership.

Michael says a personal brand is "what people say about you when they go to lunch and you're not there." You don't control it, and you can't compile it or run tests on it, but you can influence it. Andrea puts it as what colleagues would tell someone about how to get something done through you, whether you are collaborative or prefer to work alone. Asked to distinguish it from reputation, Michael says they're very similar, and Andrea adds that reputation is built from past experience and shapes how others play back your brand. Michael's point to technical people is that "everybody gets a brand," and it isn't only for sales and marketing.

Matty says it makes you a more effective change agent, and asks whether it's unfair that Matt gets listened to more because people think Matt is an expert. Michael's answer is a story about a colleague who dictated a policy that Michael complied with outwardly but didn't follow. If people feel you're a jerk or that they aren't heard, the parts of the organization most resistant to change win, because they can say you live in an ivory tower. Matty ties it to compliance versus commitment, and Michael compares it to going upstream or downstream in a river.

Andrea says how you get things done is as important as what gets done, and you have to manage up, down and laterally. Andrea stresses congruence with the company's culture, and describes Beam Suntory as fast-moving, which suits Andrea as a fixer. Michael says technical people start out with no social expectations, get rewarded for being loud and right, and are often surrounded by managers who cover for their awkwardness, so "behind every technical person who thinks that they're immune from those things is a manager getting a big, fat bonus."

Andrea describes feeling the opposite burden, leading a social media team, and tells of a Predictive Index personality test in which Andrea was the second most extroverted person at the company, behind the woman who runs the distillery in Clermont, Kentucky. Andrea suggests IT people might learn from marketing counterparts how much the how matters.

Trevor asks about the person whose brand is the one you throw every problem at. Michael says they secretly like it, and "you cannot get out of your real brand without looking fake," so if that is who you are, embrace it, maybe ask for a raise, and if you don't want it, make changes at the action level. Andrea describes a colleague who broadened the role by swinging by Andrea's desk and asking what was important to Andrea, until the colleague was working across marketing, sales and legal.

Matty notes that in consulting and customer-facing work, Matty and Trevor think about external brand more than internal, whereas a big Twitter following retweeting Michael's Ansible opinions wouldn't help Michael inside NCR. Where your brand is consumed matters, though Matty doesn't think it is either/or.

Trevor asks how you work out your brand. Andrea says you need data: self-awareness, feedback from people you trust, and tests like the Predictive Index or the diagnostic from Sally Hogshead's How the World Sees You. Michael brings up Branding Pays by Karen Kang, where a brand is cake and icing. The cake is your rational value, like being a Chef expert who is good at sales. The icing is how people feel when you're around, and people go for both. Andrea sums it up as "the what and the how." Trevor points out you can be great at .NET and still be someone nobody wants to work with.

Michael's caution is that if you're the jerk, nobody tells you, so self-awareness has to be iterative. You find one person who will tell you and show them you'll change. Matty links to a blog post called Empathy Is CI for the Soul and tells of a colleague who asked whether Matty wanted direct feedback after a proof of concept. It was hard to read, and part of it was that Matty talks too much, which the other three on the call agreed with.

Matty raises imposter syndrome and the worry of seeming to brag. Michael thinks the branding metaphor is unfortunate, because people picture someone full of themselves. What Michael wants people to say is the truth, and "the truth only comes about by results." Michael has been working on going from the person who is always right and always talking to one who listens, lets others have the idea and facilitates. Andrea adds that being authentic means you needn't fear being called braggy, and describes doing a guest lecture at a university about once a quarter and not keeping separate work and personal social accounts.

Michael tells three project stories. The first involves TeamCity, which Michael introduced years ago, feeling a duty to help anyone get on it, even meeting people at 2 a.m., and it became what hospitality at NCR uses for continuous integration. On another, a multi-product initiative, Michael ignored a lack of buy-in and pushed through on technical prowess, leaving half the people unhappy, and it felt like swimming upstream. For the configuration tool search, Michael wanted something people would adopt on their own merit, and when Michael admitted to Matty not knowing how they'd decide, Matty said it would be 45 days. Michael hung up thinking it was impossible. Chef has been like TeamCity, with people coming to Michael out of the blue with Test Kitchen questions.

Trevor says the terrifying part is looking vulnerable in community Slacks and GitHub, where Trevor is a Chef expert for a consultancy. Even at 90% certain, Trevor fears someone will call the answer wrong and cost credibility. Matty has struggled with it too. Joshua Timberman told Matty that Chef is too big for anyone to know everything, and that Adam gets things wrong too. If you answer only when you're 100 billion percent sure, you damage your brand because you stop talking. Andrea says being an expert means being endlessly hungry to learn, and Michael tells Trevor the cake isn't the whole of Trevor's value.

Matty says as your knowledge grows you raise the bar for what counts as table stakes, and tells of being corrected in a Slack channel, recoiling, and then realizing it was an opportunity. There was some "writing to the lurkers" happening, and beginners won't have to say the stupid stuff later because an advanced person said it for them. Trevor says answers come freely in a room but the public setting raises the discomfort. Michael says people "really hate a know-it-all," they feel belittled, and Michael has been trying to clamp down on the urge to answer every question.

Matty's first tip is to share what you know and bring others along instead of reciting what you've read, so you're cemented as the person who knows about that thing. Andrea suggests guest lecturing, industry events where the real value is networking, and being the same person on and off the clock. Matty says if you don't like public speaking, that's fine, and if you do, anything you have to say is valuable, and that meetups and DevOpsDays hallway tracks can connect you with people who have done the migration you're considering. Andrea says it's okay to be deliberate about networking, for example setting a goal of three unprompted hallway conversations a week, as long as you're genuinely interested in the other person. For the virtual version, Matty suggests asking on Stack Exchange or the HangOps Slack before calling the vendor.

Matty admits the title may sound like snake oil, and says you shouldn't decide to become the foremost authority on testing Go code so you can keynote GopherCon. You accentuate something you already believe. Michael says people know who the fakes are, and imagines the business dinner where their name comes up. Andrea says "real thought leaders don't consistently call themselves thought leaders." They are "humble enough to know that they need other thoughts to lead any thought of their own." When people call Andrea a guru, the reply is that Andrea is passionate and excited to learn, along with a suggestion to look at the data together. Matty's version: "I would say I'm an opinion-haver." Trevor closes on the difference between wanting to be a master at something and wanting to talk about being one.

Empathy is CI For The Soul by David Shackelford

Matty talks with Job van der Voort, GitLab's VP of Product, about how the project started and what it's like to run a company and a large open source project in the open. Job is responsible for what goes into each monthly release, did some engineering before that, and has a degree in cognitive neuroscience. Matty jokes that Job should have been on the cognitive neuroscience episode, since Job says the ADO archive is being worked through backwards.

In 2011, Job says, a PHP developer in Ukraine named Dmitry wanted the company to move to Git, but it wouldn't allow code on GitHub off-premises and there was no good alternative. So Dmitry built one in Ruby on Rails, working through the night after the day job. Job likes to mention that Dmitry's house had no running water, so whenever water was needed, Dmitry walked about 100 meters to a well with a bucket, and kept developing GitLab in between. Matty wonders what water-bucket-driven development did to the early commit patterns. Dmitry put the project on GitHub, where the programmers were, and it gained traction.

Around 2013, Sid, a Dutch guy whose real name is Sietse, thought it would make a good SaaS and told Dmitry that gitlab.com would launch without Dmitry's involvement. Dmitry said go for it. A few months later Dmitry tweeted about wanting to work on GitLab full time, Sid saw it, and they started a company together.

Matty's first encounter with GitLab was for a customer who couldn't put code on GitHub.com and was stuck on an ancient version of AccuRev, and the Omnibus install, the Chef packaging, won Matty over. Job says GitLab hadn't always used Omnibus. Before that, installing a Rails app meant a manual with at least 12 involved steps, and switching to a one-command install made downloads go up "like 1,000%." Job credits a good part of the project's success to it. There is now a package repository so you can just apt-get install gitlab. Matty calls it time to first delight, and Job says for a developer-oriented product, "it has to be extremely easy to use," installation included.

Job says GitLab does in the open everything it reasonably can where the community benefits. All development is public, from an idea or bug report through code review, merge and release, on the public gitlab.com instance. Then they opened the company handbook, a website of markdown files at about.gitlab.com/handbook where every page links to its file in the repository, and later support and operations. Product direction is open too. It lives on a direction page, not called a roadmap because they don't want to commit to dates, and in the public issue tracker where all of Job's work happens and feedback from customers and the community arrives.

Matty argues that the list of things you can't share is shorter than you think, and asks how to move an organization that way. Job says there was a lot of internal resistance, including Job's own. Customers rarely want to be named, so a support ticket becomes a public issue with the name sanitized and a link to the private ticket, labeled as a big or medium customer. It adds some process, "but the positives massively outweigh" the negatives. Being open also means saying in public that you don't think a feature is a good idea and being open to hearing why you're wrong: "We don't want to be right. We don't want to be authoritative. We want to build a really good product." Matty says the worst feeling is being ignored, and it's better to hear no in a discussion. GitLab doesn't share revenue or salaries, which Job says wouldn't be immediately valuable to the community, unlike the things they invite people to contribute to.

GitLab.com is free so they have a good place to load test, but it runs as production, with the team's own work on it. In 2014 it had a brownout of many hours, when there were about six people, all engineers in Europe, and part of it happened while they slept. They opened a Google Doc to discuss it and then decided to share it, put a link on Hacker News, and got a positive response. People said they had more faith in them after seeing the team work hard and disclose what was happening. It wasn't the moment they opened everything, but it led up to it. The site later carried a message saying it was slow and had downtime but the data was safe.

Job loves the GitLab status Twitter account, run by the DevOps engineers, who share their emotions, and says "I think it humanizes it." Matty read one saying that because GitLab.com has three single points of failure, they expect it to be unavailable at some point, and Job says they are now down to one. Matty connects this to HugOps and to blameless culture, recalling Charity Majors telling an engineer who had gone eight months without breaking production to step it up. Job says being visible online has never been a regret: "There has not been a single situation that we regretted being very present online."

Job joined at the beginning of 2014 with six people, four of them engineers. A year later there were nine and they had been through Y Combinator, and now there are about 50, about half engineers. Dmitry released the first version on the 22nd of the month and GitLab still ships on the 22nd every month. Job calls it the release train and announces in Slack, "choo choo, the release train is going." They've done it 51 times and are going for 52 without fault.

GitLab tries to use GitLab for everything, and when it hits a wall it builds the feature. Job's example is folding an external feedback tracker into the internal issue tracker, which gained a voting feature but then had thousands of issues instead of hundreds, so they look for ways to improve the product for that. Matty asks when a vendor should stop and integrate with something else. Job's example of a firm no is permission management on specific directories, which SVN migrants ask for. Because a clone contains the whole history, "we're not even going to try this because this is simply not the way Git works." Matty says that's cruel empathy: if your old way worked, you wouldn't be shopping.

Job says there are about 1,000 contributors, from a single commit to hundreds. Some of the best features came from the community, like the button to merge when the build succeeds, contributed by someone who was then offered an internship and is now a developer, and the fuzzy file finder. Customers contribute too. CERN, a customer, built several Enterprise Edition authentication features and worked in the open on issues and merge requests.

This year GitLab has at least one developer whose full-time job is to coach incoming community merge requests, finishing abandoned ones if needed, and an "up for grabs" label for small, quick issues that a first-time contributor with a little Ruby or JavaScript can tackle. Matty notes the parallel with the Phil Dibowitz episode about starting in open source.

Matty asks JJ Asghar to explain what's going on in the OpenStack world. JJ is the self-described OpenStack Chef guy, representing OpenStack in the Chef community and Chef in the OpenStack community, and owns the Knife plugin, the Test Kitchen plugin and the Chef Provisioning Fog integration. JJ got involved back around the Diablo release, the fourth one, so about four years ago, and OpenStack is about five years old. Matty opens by saying the story sounds like Chicago politics, which makes Matty feel right at home, and comes back to it later.

JJ's 50,000-foot view is that OpenStack is a private cloud you build yourself, from database as a service and compute to imaging, object storage and even container clusters. It started between Rackspace and NASA, with NASA's front end to Nova for spinning up VMs and Rackspace's Swift for object storage. It has since grown to about 19 projects on a six-month release cadence, which JJ says is a challenge for operators, and you can pick and choose which ones you use. The core ones are Keystone for identity, Glance for images and Nova, with Neutron the subject of debate about whether it counts as core. JJ says OpenStack recently got nonprofit status in the US under the same tax code as the NFL and churches, so contributing can be described to your employer as charitable work, which Matty calls joining the Church of OpenStack.

JJ says if you care about owning your data in your data center, OpenStack is an open source way to build a cloud instead of paying a VMware tax. At scale, VMware "gets prohibitively expensive around the 10,000 hypervisor mark," and JJ would rather spend that on hiring an engineer you can train than on a license and support contract. It is also good for dev and QA, where a machine sitting idle on EC2 for a year costs $1,200, and reclaimed hardware can run the same APIs as EC2 locally and get more life out of its depreciation.

According to JJ, "9 out of 10 times, it's teaching a company to go to the cloud." At a previous employer, they spent money and effort on an OpenStack infrastructure, but the idea that a misbehaving machine gets rebuilt from scratch never took hold, since the company couldn't accept ephemeral machines. JJ's warning is "OpenStack is not free VMware." Handing someone who has spent their career on vSphere an OpenStack cloud is like putting a lifelong Windows user in front of a FreeBSD box: they'll get by, but with a long learning curve. Matty compares it to Windows being API-based and Unix file-based.

The Microsoft footprint is surprisingly high, JJ says. The main hypervisors are KVM and QEMU, there are Hyper-V ports, JJ knows of production Windows 2012 R2 boxes, and one company, cloudbase.it, built its business on Windows images for OpenStack clouds.

JJ says OpenStack has three camps: developers who commit code to OpenStack, users JJ prefers to call consumers who just want an API endpoint for a Test Kitchen instance or a Redis server, and operators who run the clouds. Operators have been underrepresented for a long time. The Foundation responded with mid-cycle operator meetups, the next in Manchester, UK, the first outside the US. Operators also usually can't get an ATC, the Active Technical Contributor status that earns a free summit ticket worth $600 or $700 for anyone who has contributed in the last 12 to 18 months, because their bash, Python and Ruby scripts don't go into official projects. OSOps is a place for operators to share those tools, with the goal of getting them ATC someday. JJ says the Foundation is happy with how it has taken off, and is considering making it a core project, since you need people to run these clouds.

JJ says to run away from DevStack, which "has grown into a monstrosity" and won't teach you what you need. First figure out what you want to build, because saying you want an OpenStack cloud gets you choice paralysis. Projects like OpenStack Chef and OpenStack Model T, along with some Puppet and Ansible builds, let you build small clouds to learn. Commercial options include Mirantis and Blue Box, now IBM, which are useful, but you are handing off understanding of the cloud to a third party, no different from asking an MSP. JJ comes from the camp that if you want full control, you should understand how the whole thing works, though the engineering time may be cost prohibitive, JJ admits.

OpenStack fits with other technologies: apps.openstack.org has a one-button push to pull a Glance image and build a Kubernetes cluster, and a project called Magnum uses Heat to spin up CoreOS machines with Fleet and etcd and points your API endpoint at the cluster so you can use Docker as usual.

JJ is running for the OpenStack Board of Directors, and Matty says JJ would get a vote from Matty if Matty had one. JJ's argument is that board members have been removed from day-to-day OpenStack. JJ uses a story about pioneers, town planners and city builders. CIO magazines and Foundation blog posts make OpenStack sound like it is in the city-building phase, but JJ believes "we are just getting out of the pioneering phase and just getting into the town planning phase." JJ sees apps.openstack.org as the general store, a milestone, but says features get pushed into releases before operators have adopted the previous ones. JJ's example is that people are still arguing about basic layer 2 and layer 3 networking while vendors fund edge routers inside SDNs. JJ wants a seat to ask why, and to see "how the sausage is made." Who has the right to vote isn't clear to JJ either. JJ had it last year but not the year before, which is how Matty got to Chicago politics.

JJ is proud of OpenStack Model T, an opinionated build of OpenStack in one Chef cookbook, which automates the long OpenStack install guide, with RabbitMQ as the queue and Ubuntu as the base OS. Run one recipe on a controller and one on each compute node you add, and you have a horizontally scalable cloud on reclaimed hardware. JJ hopes to give a ChefConf 2016 talk on a reference architecture bootstrapped with a cookbook called Pixie Dust and tested with Test Kitchen on bare metal.

The OpenStack Chef project has cookbooks for all the major projects, and a small team that needs more help. It is in the middle of a refactor because the last two cycles added everything including the kitchen sink, which JJ thinks scared people off. With a 16-gig MacBook Pro you can build a multi-node OpenStack cluster with Chef Provisioning and Vagrant, and JJ used the same approach on a donated 96-gig quad-core Xeon to build a production-ready all-in-one cloud in about 45 minutes, which is still running. JJ's parting message is that operators are trying to come together, and if you have a tool, drop it into OSOps and, once ATC is possible, you'll get it.

This is a joint episode with The Goat Farm, the enterprise DevOps podcast, with all three Arrested DevOps hosts plus Michael Ducy from The Goat Farm. The conversation started when Matty and Michael, both at Chef, were in Seattle training the sales engineers and got talking about being a partner to customers rather than someone handing over a bill of sale. Bridget is at Pivotal and Trevor is at 10th Magnitude, so everyone has been on both sides of the vendor table. Trevor opens with a customer who recognized Trevor as the guy from the podcast and wants to work together because sometimes Trevor sounds smart.

Michael says the Seattle training was built around value stream mapping and Kanban, so the sales engineers would have some DevOps fundamentals as tools for conversations with customers. Trevor asks what value stream mapping is, and Michael explains that you draw a current state map of how a piece of work flows through a process and where time is spent, then build a future state map that removes the waste. It comes from lean manufacturing, and Gene Kim is a proponent of it in The Phoenix Project. Michael has done enterprise software sales for eight years, mostly on the presales side at BMC, Instratius (which Dell acquired) and Chef, and now manages Chef's East presales team. Michael wants to be less of a demo jockey and bring more value to the conversation.

Bridget says the golf-and-wine stereotype doesn't match the field people encountered at Pivotal. Matty introduces The Challenger Sale, which describes five profiles: the hard worker, who makes the calls and follows the process, the lone wolf, who ignores the process and closes deals with a Rolodex, the relationship builder, the problem solver and the challenger. The problem solver says "tell me where it hurts and I will sell you the Band-Aid," while the challenger teaches you something about your business you didn't know and acts as a trusted advisor.

Bridget pushes back that you need a relationship before you can challenge anyone. Matty's answer is that the relationship builder means you buy from me because you like me, and Michael adds that many CIOs buy from the same salesperson they have bought from for 20 years. Trevor, as a principal consultant, lands in the problem solver role, and says Matty is probably half problem solver and half challenger. Michael says you have to wear different hats depending on the customer, since some know their problem and some have read about DevOps and called the DevOps company to buy it.

That leads to a recurring question: "do you want to sell them 8 units of DevOps today, or do you want to sell them 4 units of DevOps over the next 15 years?" Bridget says you sometimes have to give people what they need, not what they want. Matty explains that with a subscription model, selling a customer something they aren't ready for creates a churn customer who will cancel in a year.

Michael contrasts the proprietary days. The customer knew little about the software, the documentation was behind a login, and a $2 million deal meant convincing everyone through executive relationships. In the open source world, the customer is often smarter about the software than Michael is and asks why they should pay Michael. The sale runs through the individual contributors who sell it upward to the economic buyer. Bridget adds that if those people can't look at your code on GitHub they will veto it, and Michael says you're also selling the community, which gives customers other people to share the pain with, where in the proprietary world you're "always angry at the vendor for not giving you what you need." Bridget adds that customers can choose among vendors, since other companies also help with Chef and commercial Cloud Foundry.

Matty describes r/sysadmin complaints about vendors' cold calls, with SolarWinds getting the worst reputation, and the flip side: when you do your own discovery, there's a lot you don't know to look for, which is the advantage of a trusted advisor. Michael says one reason for leaving BMC was the constant wondering "what are you trying to hide" when documentation was behind a paywall, and recalls an executive saying the customer shouldn't be able to download the software because it's enterprise software and complicated. Bridget says that at Pivotal, missing public information is usually because nobody has had time, not malice, and describes a ThoughtWorks person who found an error in Pivotal's docs and was pointed to the GitHub repo to submit a PR.

Bridget says that as a Chef customer at Drama Fever, Bridget happened to run into Julian Dunn, who was doing product management for the thing Bridget was unhappy about, and who told Bridget where it was on the roadmap, with no secret handshake. Matty's example is a rep from Serena Software on a small deal of about $50K, who talked about the roadmap and told Matty that Matty wasn't ready for something yet, so they'd get Matty there. It paid off: they later partnered in a big way, and Matty says a hard sell on the problem as described would have gotten them barely anything.

Michael says Chef used to run proofs of concept the traditional way, with experts setting up the tool, writing the customer's use case and demoing that the software solves the problem. They realized that was broken, and now "it's more about can the customer solve their use cases with the software." Matty recalls the Fast search POC at Apartments.com, six weeks with the vendor's engineer building everything. If Matty came in and wrote Chef code to install WebLogic, "it proves that Matt can write Chef code," which doesn't help you because Matty doesn't work for you. Matty calls it a proof of experience, a time machine to see what it will be like, and says it is physically and emotionally exhausting and Matty's favorite part of the job. The sign that it is working: "when the stickers go on the laptop, you know the deal's on its way."

Michael says the point is to challenge expectations in enterprises that say they could never work that way, by having people do it. Bridget adds that if only one part of the organization is in the room, the parts that are always at war aren't, and you sell them the units and they don't use them, so you have to get everyone in the room, since you can't just buy a tool. Michael quotes Adam Jacob: the tool reinforces the culture and the culture reinforces the tool.

Matty quotes Nathan Harvey: "everyone's in sales in your organization." Bridget found a 2014 tweet of Bridget's own for a presentation to a sales organization at a quarterly business review, "DevOps is culture, princess. Anyone who tells you differently is trying to sell something," and Andrew Clay Shafer's reply was "everyone is selling." Michael brings up Daniel Pink's To Sell Is Human, which says one in nine Americans work in sales and the other eight are selling something too. Michael says the presales team teaches value stream mapping, Kanban and pair programming during the POC, and then at the end tells customers that what they just did is foundational DevOps, and that they had no idea the presales team had done "the DevOps on them." Bridget adds that they did the DevOps for themselves.

Closing thoughts: Matty, who never expected to be good at sales, doesn't think wanting to change the world and being a good salesperson are mutually exclusive, and asks listeners to skip the assumption that everyone is selling snake oil: "Assume positive intent." Bridget, who as a sysadmin was lied to about IOPS, is humbled by how much customers trust them, and says sometimes the right answer is to stop hating each other and talk, which is what Bridget calls "DevOps therapy."

This was a special co-production with The Goat Farm. Don't forget to subscribe to their Enterprise DevOps podcast on iTunes or Stitcher!

Trevor said a few things about Chocolatey on the switching operating systems episode that caught the attention of Rob Reynolds, its creator, so Rob comes on to set the record straight and walk through where the project is headed. Rob lives in Topeka, Kansas, is a senior software engineer on the Windows team at Puppet Labs, and created Chocolatey a little over four years ago. Trevor opens with an apology for spreading misinformation, and Rob's answer is that nothing said was really incorrect.

Rob was moving from company to company and repeating the same silent-installer work each time, and wanted to make the concept more global so the wheel wouldn't need restarting, including when working outside the company firewall. The goal was to be able to sit down to pair with someone who didn't have Notepad++ and fix it quickly instead of doing "the nice long yak shave." At the time Rob was on the NuGet team, which had decided not to approach machine package management. They joked that if they ever built one they'd call it Chocolatey NuGet, since it wasn't vanilla NuGet packages, and the name stuck.

Rob's one clarification is that Chocolatey is a decentralized package management tool, and in production "you definitely don't want to be depending on the community feed." There is little control and very low trust there. Trust is something they are building, and package signing is the next big step, giving traceability from who you think created a package to who actually did. Companies can and do use Chocolatey by creating their own packages, hosting them on an internal repository, and never touching the internet. Community feed packages typically point to a binary at an official distribution point, with a checksum (many packages have one, some still don't) and the silent arguments to install or upgrade it.

Underneath, Chocolatey works through the NuGet packaging framework plus automation scripts, which are currently PowerShell, with Script.cs support to come. Script.cs is a C# scripting tool with a REPL that runs on Mono, which Trevor finds exciting.

Chocolatey started as a command line app written entirely in PowerShell, which meant starting PowerShell on every command and paying a one to two second startup penalty. It had about 20-some thousand lines of code and became interesting to maintain. For maintainability, speed and the possibility of going cross-platform, Rob decided to take the pain of a rewrite in C#. It was in beta for a while and came out that March. Rob says the goal is to make Chocolatey look like what you would expect from a Linux package manager, and that there is more work to do.

Trevor asks how a company mirrors community packages when they point at external download sites, as when Trevor's Notepad++ install broke. Rob says you currently have to edit each package, a process called repackaging. The business version coming next year will make that a single command that fetches the package and its downloads, puts them internally, and rewrites the package to point there. The paid versions will also add virus checking and, for pro users still on the community feed, an alternate private CDN so a vendor's 404 doesn't break the install.

Rob says a free service still costs money, and after three or four years of running the site, it came time to decide whether to keep pouring money into it or find a business model. Rob ran a Kickstarter last year, which succeeded. Moderation was introduced in October 2014, and before that a package went live the moment it was uploaded. It was popular but they probably turned it on too early, before the infrastructure automation was ready. Some maintainers push up to 200 packages at a time, so automation on one side and human review on the other doesn't work very well.

A couple of weeks earlier they released the verifier, which checks existing packages every two weeks to see if they still install, and emails the maintainer if not. On new submissions it runs headlessly through Vagrant, installs Chocolatey, sets up a sandbox, installs the package, and checks the uninstall too. Chocolatey takes a registry snapshot on install so it can uninstall automatically, which means about 80% of packages need no uninstall script. The validator checks quality and consistency against about 35 rules, such as naming guidelines, where an icon is hosted, and use of Start-Process, and posts notes on the package page. Only when a package installs cleanly and passes validation does a human reviewer step in, mostly to read the install script and check that the download comes from the official distribution point. Rob says the messaging is not being reported back to users yet, because they want it short and rock solid.

Trevor asks about malicious packages. Rob says they haven't found one, but "it only takes one." They have seen packages that use SourceForge, which pulls down malware. To see more of what an install touches, they plan to add Turbo, a tool formerly called Spoon that can diff a container to show every registry key and file touched.

Rob says Chocolatey lacks virtual packages, so a package can't depend on "a PDF reader" and have Adobe or Sumatra satisfy it, and it lacks concepts like replaces or conflicts. It also has no package indexes, so it hits remote systems whenever you query. Checking for upgrades took about four minutes in the old PowerShell version and about 40 seconds in the C# version, which is still too long, and with indexes it should take under a second. It has a CLI and a GUI, called Chocolatey GUI, which Trevor suggests could be renamed.

Microsoft's OneGet is a package manager aggregator that talks to other package managers, and the Chocolatey provider for it is unfinished, since the few volunteers working on it are stretched thin. Rob wants help from people who know PowerShell and C#, and would like it done in the first half of the year. Rob is also dealing with growth: downloads went from about 5 million total before moderation to, by Rob's account, another 20 to 25 million in the following year, and the site handles about a terabyte a day and 4 to 7 million requests. Approvals have slowed, and "we've sort of failed, that we didn't get the automation in place as quickly as we needed to." Rob works on stability problems on personal time, sometimes taking PTO, because as the guy behind Chocolatey, Rob hears about outages.

To get involved, Rob points to the mailing list and Gitter, taking over a package whose maintainer has disappeared, or the Up for Grabs items, which are often small and documentation-related. Rob says a longtime user once said "once you go chocolatey, you don't go back." Puppet has a Chocolatey provider and Chef has a cookbook, and there is more coming for offline use. Rob also likes the open source business model, since the paid version keeps driving features into the free one. Rob's closing plea is for better documentation, ending with: "Chocolatey, it's awesome. Use it."

Rob's pick is Turbo, and another pick slipped Rob's mind, so Trevor jokes it will be posted as Rob's latte thoughts. Trevor recommends playing with DSC and Windows Management Framework 5, Agents of S.H.I.E.L.D., and David Bowie's new video Blackstar. Rob adds that both Puppet and Chef have DSC providers, and that it is a sign of Microsoft doing things differently and more openly.

Matty corners Phil Dibowitz, a production engineer at Facebook, at the Chef Community Summit in October 2015 and talks open source: how it has changed over Phil's career, why companies do it, and how someone who benefits from it can start giving back. Phil hasn't been on the show before, and, as Matty tells it, didn't run away fast enough.

Phil has spent a couple of years building a pattern at Facebook for teams to own their own tiers of machines. Phil's team wrote low-level core cookbooks that expose an API on attributes, so setting a sysctl or adding a cron job is a variable assignment in your own cookbook, with no need to understand every kernel tunable or every other cron job on the system. They deliberately used templates and notifications so that deleting those lines from a cookbook cleans everything up. It took a couple of years to get everyone migrated. Along the way Phil got deeply involved in the Chef community as a maintainer, most recently adding multi-package support, and mentions internal Facebook tools people have found useful, like Taste Tester and Grocery Delivery.

Phil will hit five years at Facebook in November. Before that came Google, working on Gmail infrastructure, and before that at Ticketmaster, where Puppet and Chef didn't exist yet, so a group of them wrote their own configuration management system called Spine. It was very cool for its day, Phil says, but it never had a community and wasn't maintained, and the industry had moved on by the time of Phil's return to it.

Matty describes the odd alternate universe at ChefConf, with Mark Russinovich and Jeffrey Snover on stage talking about running Linux on Azure, and says it feels like Michael Corleone: every time the Windows chapter looks closed, they pull Matty back in. Matty asks how open source has evolved over Phil's career.

Phil got into the industry young by hacking on open source as a kid, and when interviewing at Facebook wanted to be able to open source what they built. Phil sees a shift since the late '90s, as Linux, Apache and the browsers showed open source would not only work but be better. What changed is that companies no longer do it for moral reasons: "the best way to move fast is to let everyone else help you move fast." Phil's example is the HipHop PHP to C compiler, which Facebook released and tried to build a real community around, including engaging the Zend community, so a lot of the work now gets done for them. It is also why nobody at Facebook wanted to write a new config management system when good ones were being maintained.

On Microsoft, Phil says it was lip service for a long time, but thinks large parts of the company have bought in and finds it exciting. Like Chrome and Firefox, each side makes the other better, and Phil doesn't expect Linux and Windows to merge. Matty adds the open source that doesn't accept PRs as the lip-service version.

Matty says enterprises used to keep quiet about how they worked, so the only stories were from Facebook, Netflix and Etsy, which is where the unicorn idea came from, and clients would say Matty was the fifth person that day to tell them they should be like Netflix. Matty credits the DevOps Enterprise Summit for getting companies like Target, Nordstrom and GE to talk. Most of what gets shared is plumbing, not competitive advantage. Netflix doesn't open source its video encoding, and if someone is going to disrupt Facebook it won't be by being better at building 300,000 servers fast. Phil calls it infrastructure scaffolding.

Phil wants to comment on the history. As a junior admin in the '90s, Phil asked a mentor what made a senior sysadmin, and the answer was being able to read the code of the applications you deploy and to bridge the gap with developers and DBAs. "That's what DevOps is," Phil says, and "it drives me nuts when people talk about, oh, look at this new idea that we just came up with, and it's never been new." Matty quotes a friend who has been doing DevOps for a long time, "back when I just called it work," and suggests the current wave corrects a pendulum swing toward admins who only care about the box. Phil's take is "those people are just bad at their job." Matty says the problem is that there are a lot of them.

Getting a company to release code is the harder part, Phil says. Inside a company you have to persuade a legal team to give a thing away, and Phil has been lucky to work with lawyers who weigh the risk against the value and decide. Lawyers weren't trained on releasing internal intellectual property as open source until recently. Now there is a decent chance someone in marketing, business and legal will get it, and you can find a path through the gauntlet even at an older company.

Matty says that's the realization that every company is a software company, and the rest of the business has to catch up. Matty adds that lawyers worry about setting a precedent for the IP they are protective of, and Phil says that is where written policies come in, guidelines on what you'd release and what you wouldn't. Phil's sign that open source has reached critical mass is that a stranger on a plane, even a flight attendant, has heard of it.

Phil says two things have changed in communities. The first is tooling. Running a project once meant juggling patches that no longer applied and email threads you forgot to reply to, and contributing meant knowing a project's patch policy. With GitHub you send a pull request and don't even need to join the mailing list.

The second is inclusivity, which Phil says the community is struggling with. Phil describes being a "loud, obnoxious, aggressive guy," and says someone who isn't may feel steamrolled and go away. The project needs both: people who feel able to bring ideas, and the culture of ripping apart a patch so the contributor can make a better one, without hard code review becoming an excuse to attack someone. Phil says ChefConf strikes a good balance, with people around to help with code of conduct issues, and that "the tooling kind of was stage 1, and this is stage 2."

Matty asks how someone who gets a lot of value from Chef but isn't going to commit to core as a first step should begin. Phil started around 12 or 13, when a little Perl was readable but bug fixes were out of reach, by writing docs. Phil maintained the IPFilter FAQ, watched the mailing list, and put questions that came up three or four times on a webpage. Phil's view is "there's no meaningless contribution," and adds "Any contribution is a contribution, we're a community." Matty adds meetups, facilitating a talk, and even whitespace fixes. Phil mentions people on projects who aren't coders but triage bug reports, asking for debug logs and version numbers so developers don't go back and forth six times.

Phil also wants people to remember there are no gatekeepers. Phil cites a DEF CON 101 talk that tells newcomers that no speaker is above them, and says the same goes at Chef: go talk to Adam, who loves talking to people, and "there is no one above you or better than you." If someone comes off as a jerk, tell an organizer, because everyone does by accident sometimes.

Asked for something surprising, Phil says internally the goal is to do the best thing for the user, with constant discussions about privacy, and that user trust is your bread and butter for any company where people share things. Phil points to a line in the S-1: "we make money to build products, we don't build products to make money." Phil has never stayed at a company longer than two and a half years except here, and says the technology was the draw and the company is why Phil stayed.

Bridget hosts solo and talks platforms with Kelsey Hightower and Andrew Clay Shafer, a colleague of Bridget's at Pivotal. The show opens and closes with Andrew's New Year's goal of being more like Kelsey Hightower, to which Kelsey replies that it is Kelsey's goal too and Andrew wishes Kelsey luck. It is a wandering conversation, and Bridget lets it wander, but the two of them agree on more than a cage match would need.

Andrew says it wasn't a moment when things switched. Andrew saw the same patterns of success across projects of many scales, and realized the architecture you are automating matters as much as the fact that you automate it, and that automation can entrench bad practices. Kelsey says when good patterns are found they become foundational, like moving something into a language's standard library. The trap is people wanting the platform to do everything, and if you're truly a unique snowflake you'll need third-party packages outside the standard.

Andrew separates standards from patterns, since standards aren't always good patterns, and says what we are participating in is a trend for complexity to move up the stack. As practitioners recognize patterns and abstract them, we can stop reimplementing them and solve some other problem that creates value for the business.

Kelsey says people rarely account for situation. Kelsey's analogy is house hunting: the same style of house is built differently in the hills, the city or the suburbs. If you get one user per hour you can use almost anything for your platform, and if you get a billion requests per second some things that don't matter to the rest of the world matter to you. Engineers rarely get enough time to evaluate the full situation before someone says they need something by Friday.

Andrew adds that experience changes what you can see. Without scars, two solutions look equivalent when one is far better, though Andrew notes some organizations have frozen their tech stack in amber for a decade, so years of experience isn't the only factor. Andrew thinks microservices, platform as a service, DevOps and continuous delivery are all aspects of one phenomenon, the patterns from high-performing organizations that deliver services that are highly available and rapidly changing. If you want all those qualities, "you're going to converge on something that starts to look suspiciously like a platform as a service."

Kelsey says when it works, it works, and Andrew counters that everyone has a different threshold of pain and what they call working may not be. Andrew brings in Tolstoy: happy families are alike and unhappy ones are unhappy in different ways, and the high-functioning organizations look similar.

Bridget asks what to pay attention to among containers, schedulers and orchestration. Kelsey says if managing infrastructure isn't your job, you pick a platform like Heroku and write the app. Otherwise you'll have to compromise: adopt something that exists, or use a fraction of it and build on top. Kelsey thinks the debate is fueled by emotional attachment to tools, and Andrew adds that "everything in tech, even though it looks like it's about logic and reason, is really about fashion and tribalism." Kelsey's version is that if you're great at Bash you attack every problem with Bash, like Batman's utility belt, and "You can write bad code in any language." Kelsey also thinks people play a zero-sum game, and abandon a platform like Heroku over one missing feature and reinvent the whole wheel.

Andrew describes the arc of the thinking as it played out for Andrew. You feel powerful the first time you configure a server with Puppet, and then 100 servers, and then you have Amazon, and it turns into the Fantasia broom story: the brooms become their own problem and you need another layer to orchestrate them. Bridget asks if it's abstractions all the way down, and Andrew says of course, the complexity is moving up the stack.

Kelsey says there is a real fear of yielding control to a service that just works. If we do this long enough, will we never be able to buy a server and configure it, and end up renting forever? Kelsey half sympathizes, because some people want to look under the hood, though Kelsey says it slows platforms down: Cloud Foundry needs BOSH partly because it has to support so many install targets, and the number cited is 80,000. Andrew says some of the fear is people attaching their tribal identity to their tasks, so that the ability to do that thing is what defines them. Kelsey jokes about people who think they can hit eight nines in a single data center in their backyard, and Andrew calls it Dunning-Kruger.

Andrew points out that this replays every time: moving from assembler to C brought the same worry about losing control. "You don't fight the future. You just have to figure out where you want to live inside of it." Kelsey notes stuff from the '70s is still running, and Andrew describes sedimentary layers that never go away, pointing out that the mainframe business's top line grew every year for the last decade.

Kelsey says the true benefit of containers, stripped of hype, is that they let you swap out what's underneath more easily, and that the new legacy we create will be easier to deal with in 30 years. Andrew argues that much of the benefit comes from Linux having won as a standard: the syscall interface is what got standardized. Bridget notes cgroups and namespaces predate Docker, which made them accessible. Andrew agrees, but says what became accessible was using them, not the mental model of how they work, so what's being iterated on is the abstraction.

Kelsey says "containers mean nothing without the API," and when people say containers they mean Docker's API. Unikernels would only swap one containment technology for another, and you can't introduce a new containment boundary without an API. Andrew agrees they aren't usable until they're in fabrics with tooling like Docker's, and recommends an interview that argues unikernels are exokernels, which Andrew doesn't buy.

Kelsey also says "You should not be using Kubernetes directly for all of your needs." Kelsey works lower in the stack and Andrew higher, and if you're building a Cloud Foundry-like thing on top of Kubernetes, you're probably wasting someone's time. Kelsey's endgame is that you have an app in a bundle, and you don't care whether it runs on unikernels, VMs or containers.

Andrew introduces 12-factor ops, the other side of the 12-factor contract. Each factor implies that something exists to make the app work, so if the app is configured through environment variables, something had better inject them. When people say they don't need a platform because they have Docker, Andrew says "that's adorable," and asks how they deployed things before platform as a service. Your platform might be Heroku, or a configuration management tool, or Julie running a shell script in a loop. Kelsey adds that we've neglected app behavior because humans will look at the logs and do the needful.

Asked to be prescriptive, Kelsey has people whiteboard how their organization works before they look at tools, since no tool does everything, and then asks whether they are willing to own the missing piece. Andrew says to work out what you're trying to accomplish and back into a solution, and to think about how automatable your architecture is. Andrew gives an example: if your service needs an order-dependent stateful install process, that process is the lower bound on your recovery time, and in practice worse, because you have to figure out what state you're in and unwind first. Separating state from statelessness gives you more power regardless of automation choices.

Kelsey says people need to get over the hump of change, and some code changes need to be on the table. Andrew points to a sentence in the Borg paper that Andrew thinks most people miss, that most of what ran on Borg had an embedded web server broadcasting metrics about the application's health. Kelsey says we used to probe apps from outside with Nagios, and they should just report their health, as New Relic did by importing one package: "We've learned too much in 30 years. No more excuses." Andrew adds that Spring Boot ships with embedded metrics, and the Netflix open source circuit breakers come with those patterns.

Andrew expects mass enterprise adoption of cloud and many failures, because adoption isn't always enlightened. Andrew tells of being asked to help a company that wants the DevOps and then, after the analysis, asking whether Andrew could help them do it without changing anything. At that point, Andrew says, you should just use Puppet, because they have political issues to work through first.

Kelsey predicts people will be forced to outsource compute by things outside their control. Machine learning services, petabytes of data per day and IoT data volumes go beyond what a team can build, and Kelsey thinks there will be six or seven platforms that mature and work. Andrew adds that high performers didn't set out to do DevOps, they were pushed by Darwinian force, and the organizations that seem healthy haven't met theirs yet.

Kelsey's last word, on the Kubernetes workshop at OSCON that Kelsey now chairs, is a conclusion: "Most people do not want to run it themselves. It's a fallacy." So the material will be about what to do next.

Matty, Trevor and Bridget record the second year-end episode together, with Trevor and Bridget in the same room in Minneapolis, and go through conferences, favorite episodes, site stats and what they saw in the wider DevOps world. Trevor opens by noting they've made it through two years, slightly over. The cold open is Bridget: "Pink-haired thought leadership as a service is valuable enough for Pivotal to pay me to do it."

Matty spoke at ChefConf, the first time all three hosts were in the same place and all spoke, with a talk on automating animated GIFs in HipChat. Bridget's case for it is that people dread swapping a heavyweight process for a new terrible one, and animated GIFs in a chat app make them realize "I could actually have fun with this DevOps thing." Matty ties it to Adam Jacob's ChefConf keynote on bringing delight: the one feature Adam required in Chef Delivery was support for animated GIFs, and the system profiler being called Ohai adds no technical value but makes people grin. Matty shows embedded YouTube videos and GIFs in Chef Delivery demos, and even the stodgiest enterprise loves them.

Matty went to DevOpsDays Rockies, which was held in a data center, and the talk of tours prompts Bridget and Matty to trade data center stories: Bridget recalls a designed-as versus built-as floor plan mismatch that meant moving two racks days before a $5 million supercomputer delivery, and Matty still likes pretty cable management. Matty also spoke at ALM Forum, enjoying non-DevOps audiences because "you don't have to be super insightful," and gave The 5 Love Languages of DevOps, which works well at agile and app dev events. Bridget's counter is that many people attend Velocity for the first time and gravitate to the culture track, with the CFP deadline on January 11th.

The best story is DevOpsDays Minneapolis. Bridget pushed Matty to submit an Ignite talk about Pete Chessbot, a Markov bot that mangles Pete Cheslock's tweets, and Matty confirms, on the record, to be the bot's owner. To keep the talk funny, Matty asked Pete to fill the slides with bot tweets and send them to Bridget so Matty wouldn't see them. The first slide appeared with no warning, so the first words of the talk were "fucking Cheslock." Pete insisted no instructions had arrived, and Matty pulled up the sent email from the front row. The email said: "dude, how do you DevOps while you're illiterate?" Bridget thought the real surprise made it better.

Matty also praises That Conference in the Wisconsin Dells, "summer camp for geeks," where 60 to 70 people came to a talk about Chef at a .NET-focused audience and Channel 9 interviewed Matty. Trevor made a speaking debut at ChefConf with the Fresh Prince of Azure rap, which now serves as an icebreaker with every client, then spoke at DevOpsDays Chicago on contributing to open source and at Days of .NET about Chef, twice, after a coworker took a job at Microsoft. Bridget spoke at about 23 events, including OSCON, Velocity New York and Amsterdam, ChefConf and KubeCon, and co-presented with former and current coworkers, preferring the conversational format, which "feels more like a podcast."

A running joke is that being on the show gets people new jobs or promotions, with Trevor, Bridget, Katherine Daniels and Jill Jablinski cited, and a listener tweets that getting mentioned got them a new job too. Matty's proposed fee for a promotion is a high five, a tweet or an iTunes review. Bridget notes that being a regular host helped with moving to Pivotal, and that Kyle Kingsbury opened a consultancy around Jepsen after the distributed systems episode.

The hosts call out the Microsoft episode with Jeffrey Snover and Jessica DeVita, which became the most viewed YouTube video, the Docker episode with James Turnbull, which was the most listened to, finally landing Andrew Clay Shafer during a day and a half in Minneapolis, and an episode with John Willis that hadn't been released yet. Bridget's highlight is the cognitive neuroscience episode with Courtney Nash and Lindsay Holmwood, and Matty discovered that episodes without Matty were fine, after resisting them at first. Matty says "I don't think we had any turkeys this year." Bridget and Trevor also credit Matty for the editing, the site overhaul and the feed wrangling.

The site is now statically generated from a GitHub repository, so pull requests with fixes are welcome, and Matty fixed a title typo pointed out weeks late. Visits went from 14,000 to 24,000, listens from 76,000 to 204,000, and per-episode averages doubled to between 5,000 and 7,000.

Bridget now visits customers, including government types working hard to change, and some who want DevOps "without changing anything." Matty says it's the year the C-levels get it, as part of strategic planning, but organizations balk at "the right hard thing." Trevor sees Windows shops adopting test-driven infrastructure. Matty fields the question "what's your container story?" from people with no plan, comparing it to asking "what's your computer strategy?" A manager at Bridget's Agile Day open space said a VP had decided the company was doing microservices and asked about the downsides.

Matty says enterprise DevOps as a separate category faded in 2015, credit to DevOps Enterprise Summit, and reminds everyone that under 15 percent of organizations do configuration management, so the echo chamber is "1% of 1% of 1% of IT in the whole world." Bridget adds that "there is no such thing as greenfield," since what ships yesterday is legacy today, and that a vendor's answer should be "I don't care which one you use, just use something." Matty says the real question is how to do the thing and not which tool, and that the best thing about the year is companies like Target talking about what's behind their firewall. Bridget describes Target's dojo, where groups get a 30-day challenge and come back to spread the change: "It's practice."

Software Defined Talk (they need a more memorable URL)

Matty sits down with Chef's Dr. Nicole Forsgren, recorded at the Chef Community Summit, to cover the two parts of CAMS the show hasn't gotten to: measurement and sharing. Matty apologizes at the end for taking so long to edit and publish it. The conversation is mostly practical: what to measure first, how to get the numbers in front of people who don't speak your language, and how to measure culture without giving up and calling it squishy.

Matty says Adam Jacob likes to say Nicole has a doctorate in charts and graphs. Nicole started on an AS/400 doing programming and system administration, in green screen, "laying cable in 4-inch heels," which Nicole says is true. Nicole then got a PhD in management information systems, wanting to look at how technology use plays into business outcomes, and how organizational and cultural factors affect it, which is really CAMS, Nicole says. Nicole's focus since the dissertation has been tech professionals, and a story follows about the dissertation advisor calling excitedly after the announcement that Nicole was joining Chef to say "you were right" that sysadmins were a population worth studying.

Matty says measurement isn't just monitoring, like Nagios paging you when a disk hits 80%. It's how you tell that the needle is moving as you change things, and shorten the feedback loop. Nicole agrees that monitoring is good, since you want to know when everything is on fire, but says measurement at the start of a transformation matters because you have to understand where you've been and where you're going, and communicate that to your team and beyond. Vocabulary is part of the problem. Lead time can mean code commit to deploy, ideation to deploy, or deploy to delivery, and dev, ops and QA may each use the same term differently.

Nicole's advice is to measure it, write down what it is and how you capture it, and name it something meaningful. If Nicole says lead time and you say cycle time and the numbers don't line up, you can compare notes on what each of you is measuring and why they differ, and "it's no longer a fight." If you start with a baseline, even an ugly one, you can tell whether changes to tooling, practice, process, culture or team structure are making a difference. Nicole describes the path as starting with a handful of metrics in a spreadsheet, moving to exploratory correlations and visualizations, and eventually predictive analyses of what happens if you turn a knob, which some of the most innovative companies are doing.

Matty says the ultimate driver is selling shoes, so the job of the sysadmin is "really not uptime," it is facilitating the business selling shoes. Nicole's suggestion is to treat your work as a product for a customer who may be elsewhere in the company, and to create your own little marketing deck of basic metrics that show quarter over quarter what value you provide. Nicole thinks Amazon is an example of internal infrastructure that became a product. To find where to start, ask what your company does, how it makes money, what your executives care about, and how you directly support that. If you sell shoes and you're on the infrastructure team, you limit downtime because you provide the infrastructure for shoes to be sold, even if that means EDI and the FedEx order making it through.

Nicole was on the metrics group of Gene Kim's DevOps Enterprise Forum, which set out to name the ten most essential metrics for everyone and, by Nicole's recollection, started with 155. They settled on three areas: internal facing, external facing, and culture, since culture is a big indicator of when things are going well or about to break. The white paper was released at the DevOps Enterprise Summit and is linked below. Nicole's practical advice for a team is "Start with 3. Honestly, start with 3," and treat it like an MVP you can add to. Nicole also warns that measurements drive behavior, and that showing improvement with the resources you've been given may earn you more resources, or a seat at the table when those decisions are made.

Matty gives a snarky summary of Gartner's bimodal IT, where one half of IT does the cool DevOps and Docker work and the other keeps the mail servers and desktops running. Matty's view is that the traditional half should be measuring how it drives the business as well, and that it often measures things that just make its own job easier, and brings up the sysadmin subreddit complaining about the head of marketing wanting a MacBook.

Nicole says bimodal IT is a thing right now, since companies can't transform all at once, but Nicole doesn't think it is sustainable. As lead on the 2014 State of DevOps report, Nicole has data from 20,000 respondents, and the trade-offs between throughput and stability that ITIL suggests, which would justify a bimodal approach, "never show up anywhere in the data." Instead, "you're either doing it or you're not," all fast, all slow or all in the middle. Nicole is watching stock trading and air travel, with systems like Sabre, where everyone is terrified to touch legacy systems. Matty tells a story from Matty's time at a big bank, where a computer operator in the basement pointed at a machine nobody had seen before and said nobody knew what it did, but they weren't going to turn it off.

For traditional IT that has a solid reason to run as it does, Nicole says that is an even stronger reason to collect metrics and tell the business, because otherwise the business asks, as Matty puts it, "why aren't we just using Gmail?" Or, Nicole adds, why aren't you doing the DevOps. Nicole's advice is to be the one architecting the new service and not to have the rug pulled out from under you. Nicole and Carolyn Rowland have given a half-day workshop on communicating value and being a strategic partner to the business.

Matty's talk The 5 Love Languages of DevOps comes up, and the point that what matters to you may not matter to your CFO, so it just sounds like complaining. Matty's story is from Apartments.com. A web service was throwing something like 14,000 fake errors a minute because of a bug, flooding the logs. The sysadmin on Matty's team kept trying to get it into a sprint, and Matty at the time figured product just didn't understand. It wasn't communicated in a way product cared about, since product's question was whether there had been an outage. What actually got it fixed was Splunk dashboards on the wall, one with a dial in the red at 14,000 errors a minute, which the GM walked past and asked what the hell it was. Matty admits that isn't a good way to sell. Matty quotes Bill Joyce from the DevOps culture change episode: "if you ever find yourself saying, this is crystal clear to me, why aren't they seeing it? Then it's more about you than it's about them."

Nicole loves the dashboard. Three measures is a nice number because you can hold it in your head, and it fits on a reasonably priced monitor on a wall. Across three periods, even if the first baseline was horrendous, the GM or CFO who walks by sees progress. Nicole suggests big letters and few words, so it's readable across a room. Matty remembers a website performance report sent to the board with so much data nobody cared about most of it, and Nicole's answer is that "your team is gonna want 37 metrics," but executives should get three or four, the ones that show improvement plus maybe one that doesn't. That one is your business case: we've optimized as much as we can with tooling, and I need the headcount, because I refuse to work my team insane hours, and here's the data to show it.

Matty says people insist you can't measure culture. Matty recalls Bernard Golden's line that "culture is for yogurt." Nicole says the squishy culture girl label gets applied to Nicole, but culture matters, and it tends to be a leading indicator: if it tanks for no expected reason, go check on your team, since it is often a sign your tooling is about to fall apart. If you want to avoid the squishiness entirely, you can proxy. You can't take the temperature of culture, but you can look at HR data like attrition rates, at access to natural light, which Nicole jokes is bad news for teams in the basement, and at work hours.

Better, in the 2014 State of DevOps report they used a six-item measure based on research by Ron Westrum, also used in 2015 and included in the white paper. It's a survey on a 1 to 7 scale from strongly disagree to strongly agree, and Nicole, who has a background in psychometrics, says it has been shown to be statistically valid and reliable across samples of 9,000 and 5,000. The six statements are "On my team, information is actively sought," "On my team, failures are learning opportunities, and messengers of them are not punished," "On my team, responsibilities are shared," "On my team, cross-functional collaboration is encouraged and rewarded," "On my team, failure causes inquiry," and "On my team, new ideas are welcomed." Matty recognizes it from taking it at Chef, and Nicole says Jez was hired shortly after and used it.

Because it is a latent construct, you can average the answers into one number per person and combine them for a team, and look at any single item that comes out low. If everyone scores low on failure causing inquiry, look at your practices: are you doing blameless postmortems, and are your backup systems so bad that failure is scary? Nicole suggests a hack day around making things fail, with silly prizes, and then remeasuring. That item in particular is a strong predictor of both IT performance, in throughput and reliability, and of organizational performance. It's all open. Nicole's last word is not to give up when a measure stops working, because "it's always just an improvement kata," and it has probably stopped working because your processes have improved.