Arrested DevOps

Open source projects have a lot of benefits, as we all know. But sometimes it can be a real challenge to take tools and projects developed internally and open-source them, especially from a traditional enterprise. Guest Aaron Rinehart shares his journey and story with open sourcing his internal security chaos engineering tool, Chaoslingr.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Discount codes: ADO2018 for 20% off lots of devopsdays, 10% off ChefConf

Matt refers to an episode of “Coupling” on the BBC - the episode was Series 1, Episode 2, “Size Matters” - here’s the quote:

Patrick: Oh, don’t be so piecey.

Image credit

Bridget sits down for a remote chat with Alice Goldfuss, a site reliability engineer at GitHub who loves systems, lower-level problems and kernel panics, and also tea, cats, chocolate and box forts. Alice joins from the rainy Northwest with a desk lamp aimed at the face. The cold open is Alice's line, "Ladies is gender-neutral now," which turns out to be the name of a t-shirt. At the end, Alice says that if it had been announced as a fireside chat, Alice would have set something on fire in the background.

Alice was talks co-chair for LISA, the Large Installation System Administration Conference, which USENIX runs and which has been around for over 30 years, back when a large installation meant 10 workstations. USENIX handles logistics and brings in people from the community each year to select talks, workshops, tutorials and keynotes. Alice likes that it is non-vendor-specific and for practitioners, and describes three days of training and workshops followed by three days of a more traditional conference. The 2017 event was in San Francisco, the next in Tennessee and the following one in Portland.

In choosing talks, Alice wants the how: "there's a difference between how to use Docker and how we used Docker to change up our infrastructure." Alice also says "I also really love a good outage story," and wanted Niantic to talk about the Pokémon GO launch. Bridget asks how to get stories with failure in them past a company's lawyers. Alice says it's hard, since companies fear customers questioning an SLA, but an engineer hearing honest failure stories is more likely to buy, and talks from Google have gotten through when the incidents were old or vague enough. LISA, Alice says, sits at the end of the year when people are tired of flashy conferences and more willing to talk.

Alice ran devopsdays Portland's vendors and MC duties, and did not choose talks that year, but encouraged people to submit who otherwise wouldn't have. A more diverse selection committee used its networks, telling people it would be a safe space. Bridget says that when you post a CFP and do nothing else, you get the people whose job it is to submit, so a broader range means reaching out. Alice's first conference talk, at LISA in 2015, happened because someone who was a chair that year reached out, and "someone reached out to me and said, I think you have something to add here." Alice adds that looking at past talks by people with PhDs would have scared Alice off.

Alice says it matters who is reaching out and why. An invitation to a Python conference said it was looking for more women, for a framework Alice hadn't worked with in years, and Alice's reaction was "you just want a woman, that's all this is." Bridget says no one wants to be a decorative prop. Reaching out works when it names something specific, which Bridget calls "conference invitation Mad Libs," and when it says why the conference is good for the speaker, since Alice has to justify conference travel to an employer.

At GitHub, Alice started on the Edge team, which handled the network tier and a custom load balancer, and now works on the Kubernetes platform that all of GitHub.com runs on, with plans for kernel-level work. Alice grew up in a very small upstate New York town with a Windows 95 computer and no internet until 16, so learning happened alone, with Compton's Encyclopedia for AP Calc. Alice has a film degree, started in tech support, tried Ruby and disliked it, then liked Python, and took an early Coursera programming class that built games in eight weeks. Alice says "I've had a career of doing one job during the day and learning to do another job at night," and prefers "self-guided" to self-taught, since other people wrote the materials. The method is the Socratic why, which keeps leading lower in the stack.

Alice's laptop was resting on Understanding the Linux Kernel, the Practical Linux Security Cookbook and Docker Up and Running, and since the kernel book's examples are in C, Alice learned C for fun. Alice brought the kernel book to jury duty. For concepts like Kubernetes, YouTube videos with diagrams help, and Stack Overflow answers supply jargon to search the man pages. Alice carries a chip on the shoulder about lacking a CS degree, but says "I've made it this far without having to know really any computer science," and that knowing arrays and hashes gets you far. Bridget notes that even with a CS degree you still Google constantly.

Alice made a t-shirt with a Panic! at the Colonel design after wanting shirts that weren't unisex tech swag, and when it sold, began donating the money. The first went to Women in Linux. Then came the "ladies is gender-neutral" shirt, made to answer "guys is gender neutral," offered only in fitted sizes and sold with every line conference organizers use on unisex shirts. It raised over $5,000 for Outreachy. A Manic Pixie Dream Girl shirt with PXE as the pixie benefits Free Geek, a Portland organization that refurbishes donated equipment. Alice moved from Teespring to Threadless for more cuts and colors, and includes sizes up to 4X. Fitted shirts, Alice says, should be table stakes for vendors, though they don't fit all women. Alice adds that scarves made things much easier for devopsdays Portland's swag, since no one needs a size.

Bridget asks what Alice wants people to take from posting things that poke the bear. Alice says humor works: "comedy allows you to point out things that otherwise would be considered offensive or people would instantly shut down to," like the fool in a Shakespearean play, and people remember it better than a blog post with citations. Jokes about sexism may also embolden people to talk about it at work. Alice says the loudness comes from being vocal about things that get in the way, like sexism and racism: "I talk about them a lot because I want them to go away." Alice also talks about technical things and baking.

Alice had been at GitHub over seven months before ordering an office chair, a Steelcase Gesture with narrow armrests that suit a smaller frame, describing the research habit as being "like the missing stair enabler of office equipment." Alice runs the PDX DevOps Meetup on the last Wednesday of the month, and encourages speakers to come, since the bar is lower than at a conference, with no recording, and a repeated talk is welcome. Bridget's tip is to give a conference talk at a local meetup the month before, pointing to a local speaker who changed about half the slides after the first run. Alice gives at least two practice talks, and plans to use the PDX meetup for the next new one. Alice likes devopsdays for the local angle, since it's harder to claim something can't be done locally when an organization three blocks away has done it.

Alice plans the year during the holidays, because of a need to manage emotional highs and introvertness, and wants to keep 2018 domestic with not too many new talks. A planned January keynote was postponed indefinitely. Alice would like to give a technical keynote on three or four years of running containers in production: at New Relic, Docker went into prod at version 0.6, and Alice has run Dockerized databases and now runs Kubernetes at GitHub. Alice calls it "the mechanic's view of containers," leaning on the podium with a rusty wrench saying it's going to break, and jokes "Your problem is all of the whales that you put all your apps in." Bridget counts 29 public talks in 2016 and calls that too many.

Asked how people grow up to be Alice, the answer mixes self-guided learning with a network outside your employer. Alice says soft skills are a real deficit in tech, and that giving talks and keeping a social media presence is "kind of like a bridge between all of your jobs" and a safety net if a job ends, since you don't want your only network inside one company. Befriending women in tech has made life easier, Alice says, and the network doesn't have to be women. "Don't let your job be your whole life," because failing at work shouldn't take everything else down with it. Bridget adds that networking means connecting over common interests, and that following someone on Twitter doesn't make them your BFF. Alice adds that you can't control a promotion, but you can control learning a new language.

For listeners trying to DevOps at their own companies, Alice says things are easier with buy-in from the top, since a grassroots effort without a VP's backing is hard, and convincing a manager's manager helps. If you keep hitting obstacles and aren't enjoying work, Alice says it may be time to look elsewhere, since "your job shouldn't be your whole life."

Bridget sat down for a fireside chat with Alice Goldfuss (GitHub). No actual fires were harmed in the making of this episode.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Use code "ADO2017" for a discount on many devopsdays.

Devopsdays Madison 2017 was the second year for this event. With great speakers, workshops, open spaces, and sponsors, this event brought the community together to learn and share.

Bridget and Matt sat down with a speaker (Emily Freeman) and a couple of organizers (Joshua Zimmerman & Christian Herro) to talk about the wider themes of organizational change when you may not call all the shots in your org.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Use code "ADO2017" for a discount on many devopsdays.

Ines Sombra (Fastly) and James Turnbull (Empatico) are chairs of the Velocity conference series, which is celebrating its 10th year in 2017. They joined Bridget to talk about the events next month in New York and London, and share tips for making the most of your conference as well as submitting talks to future conferences.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Use code "ADO2017" for 20% off at Velocity New York Oct 1-4.

Use code "ADO2017" for a discount on many devopsdays.

Food Fight Show and Arrested DevOps joined forces to host our first live devops call in show! We featured Dr. Nicole Forsgren to answer your DevOps questions about measuring effectiveness, ROI of DevOps initiatives, and more!

Recorded May 8, 2017.

The unofficial title of this episode: "Ya sure, Nicole?"

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Use code "ADO2017" for 20% off at Velocity New York Oct 1-4.

Use code "ADO2017" for a discount on many devopsdays.

Bridget talks with Jez Humble, who was last on the show in episode 15, about three years earlier, on continuous delivery. Since then Jez spent a year at 18F, a team inside the US federal government, working on infrastructure and on cloud.gov, a platform as a service built with Amazon and open source Pivotal Cloud Foundry to show agencies that continuous practices work in government. Now Jez works at DORA, DevOps Research and Assessment, with Nicole Forsgren, Sue Choi and Gene Kim, the team that produced the State of DevOps report with Puppet Labs. The cold open is a story about a Zen temple that returns at the end of the episode.

DORA's assessment surveys an organization's people against 20 capabilities the research has linked to IT performance, from culture to practices like automated deployment and managing work in process. It compares the organization to the industry, and with a group of 50 or more people it can say which investments give the most return. Bridget raises the customer who wants a RACI chart for who should do each thing. Jez's answer is that an organization should have a few strategic priorities and each team should work out how to collaborate, agreeing on a measurable goal for the next month. There's no way of knowing in advance who will have to change, "but, you know, my money is on everyone." Jez says DORA is "all about data," including measurable culture, and that picking a fight with Nicole over stats is a losing one.

Jez defines continuous integration as building and testing every change, working off a shared trunk and not long-lived feature branches, with a quick, comprehensive automated test suite, so that the default state of the system is a working one. Jez tests people with three questions: does everyone check into a shared trunk at least once a day, do tests run on every check-in, and when the build breaks, is it typically fixed within 10 minutes? Most people can't answer yes to all three. Running Jenkins against feature branches and ignoring it when it turns red isn't continuous integration: "It's a practice and mindset, not a tool." Bridget notes the CI/CD theater, and Jez says DORA uses yes/no and agree/disagree questions, and that "asking the questions is, in many ways, the hard part," and "People love redefining terms so they can say they're already doing it."

Continuous deployment means a passing build goes to production automatically, with high performers deploying hundreds or thousands of times a day. Continuous delivery is behaving as if you'd do that without deploying every build, which suits firmware or mobile apps, and still improves quality, cost, time to market and feedback loops. Bridget plays devil's advocate with Schrödinger's deployment, where a build could go out but a schema migration wasn't accounted for. Jez says to "do continuous deployment wherever you can," and that Jez wouldn't build non-web software unless forced to, noting people at Facebook who deploy mobile apps every couple of weeks aren't happy about it. Jez points to a recent Charity Majors blog post about testing in production, and agrees that production needs good monitoring, logging and tracing, and cheap, low-risk deploys through blue-green deployments, canaries, feature flags and circuit breakers. Jez says to accept that "failure is inevitable," and recalls that John Allspaw's focus on mean time to restore over mean time between failures was an aha moment. Bridget adds Andrew Clay Shafer's state of continuous partial failure.

Asked about the common thread among organizations on the path, Jez's answer is that "The best organizations are always trying to get better," and are never satisfied. Even Amazon and Facebook invested huge effort, and Jez notes Amazon was a monolith that spent four years re-architecting after making it a priority at all levels, consistently and with money. Jez adds that Amazon has miserable, dysfunctional pockets too, and Bridget asks whether that means software is made of humans.

Bridget asks for the Cliff Notes of the end of Jez's Agile 2017 keynote. Jez explains that an ex-Google employee, James Damore, had written a 10-page document about Google and the causes of low representation of women and people of color in the industry, and that Jez was angry we're still having the discussion in 2017, like being angry about talking about continuous integration after 15 years. Jez holds up Douglas McGregor's The Human Side of Enterprise, from 1960, on motivating teams. Jez says there are biological differences, but real outcomes come from a complex interaction of genetics, epigenetics and environment, and "you can't go from those biological differences straight to differences in ability." Jez recalls telling the audience that math and science can't explain the lack of diversity, but sharing a work environment with people like Damore can, and some people walked out. Jez argues that unequal access to highly paid, high-status tech jobs is inequitable, produces worse products and affects the economy, and that people who care about the minutiae of tech but not how teams are composed will do worse, since these are systemic problems and the system is what dominates.

Bridget asks about siloes. Jez says it's highly variable and depends on who leads each team. A high point of Jez's career was the federal government, where Jez met mission-driven, brilliant, hardworking people, including in InfoSec, and learned from Mark Schwartz's talk that bureaucracy can be helpful because it encodes what people think is the right way to do things. Jez's "number one DevOps hack" is to find the person everyone slags off, often InfoSec in the dev world, take them to lunch, and spend an hour actively listening to what's in their way. Bridget asks for a British-to-American translation of "slag off," and Jez says it means trash-talking someone.

Bridget asks what individual contributors can do. Jez says it's making friends and understanding different perspectives, as in learning why a rule exists from the person who had the problem. Jez argues that "Most of the problems we face are due to a lack of empathy," pushing back on the idea that the obstacle is a failure to systemize. DORA measures collaboration between Dev and Ops, asking whether the outcome is win-win, and measures culture with the Westrum model, a construct from sociologist Ron Westrum, which John Allspaw also pointed Jez to, studying safety outcomes in healthcare and aviation along six axes. Jez says information flow is critical for resilient organizations, and "You can build the best systems in the world and use the best tools in the world, but if your cultural organizational structure is wrong, it just won't work."

Jez says large agile frameworks get adopted without changing procurement, contracting, leadership or organizational structure, and then "you're just shuffling around the chairs on the Titanic." The point of practices is to change the organization's behavior, and Jez objects to cargo culting. Jez says knowing where you are and where you're going in measurable terms comes first, and when people copied Toyota, "You're copying the practices, you're not copying the mindset and the culture of continuous improvement." Bridget gives the example of an andon cord in a place where pulling it gets you fired. Jez points to the NUMMI story on This American Life, where other GM factories copied the cord, but managers were rewarded by how many cars came off the line whether or not they worked, so pulling it got you fired.

Jez cites John Kotter's Leading Change, whose first step is that most employees, about 75% of management and virtually all top executives need to believe that considerable change is essential. That shows up as ruthless prioritization, and it's "much easier to change an organization where there's a burning platform than it is to change an organization where everyone thinks that everything's just fine." Some leaders won't state priorities in measurable terms because it takes away the ability to change their minds. Jez says individuals can still effect change, though the impact will be limited and making it stick is the hard part, and organizations have backslid after people left. The original Continuous Delivery book came from a team of eight and colleagues working on miserable projects with Bash and CVS, transforming how an application was deployed, which shows "you can achieve great things with small teams." Jez adds "There is no linear path from A to B."

Jez closes with the Zen temple story from the opening, told by the teacher at an introduction to meditation: sometimes you come to the temple feeling down, sometimes feeling great, "But don't worry, because that feeling will pass too." Jez thinks that sums up DevOps and the human condition. Jez also encourages listeners to speak at conferences, since you're probably doing things other people would be excited to hear about.

Bridget chats with Jez Humble (DORA) about continuous... everything!

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Use code "ADO2017" for 20% off at Velocity New York Oct 1-4.

Bridget will be at Uptime in Pittsburgh this week.

Use code "ADO2017" for a discount on many devopsdays.

Bridget and Matty record live at devopsdays Minneapolis 2017, putting two keynote speakers in one conversation. Bryan Liles of Capital One, who opened the conference, took what Bryan calls a total non-tech approach: people matter more than anything, plus experiences of DevOps in the enterprise and how it could be better. Jessie Frazelle of Google, who closed, talked about security in a containerized world, making security on by default so that 99% of users benefit. Bridget calls the pairing "fanfic of the conference talks." Midway through, Andrew Clay Shafer wanders in from a game of werewolf in another room and describes Andrew as "A villager, not a werewolf." The cold open is Bryan explaining the vein of a banana.

Bridget asks how to incentivize people in tech. Bryan says "the only incentive that I have is a smile," and describes spheres of influence: tensions occur where bubbles touch, so Bryan tries to make a bubble more permeable and bigger. Jessie says incentives are hard in open source because you can't pay people, so you use their interests, and don't push them into work they don't want to do, since it's free labor. Matty notes that Jessie's slides paired each deep technical point with a person problem, and that making security hard is itself an incentive, since people turn it off. Bryan adds that incentives are harder at big companies because "you can really blend in" and do the bare minimum for years.

Bryan took a note from Jessie's talk: "We should always allow our users to fail well." People will do the worst thing possible, so the default should leave no room for it: "You can't add privileges to Docker. You get what you get and you can only take away." Bridget says defaults apply to organizations too, and asks what a team does if no one takes any action. Bryan says people do the easiest thing, like turning off SELinux or booting cloud instances without thinking about security groups, and wants all that to be on by default, like a banana you have to peel.

Matty tells of a sysadmin who wanted a release manager reporting to the CTO with hiring and firing power over developers so they'd follow a new release process. Matty's answer was to "make the right way the easy way," as in the book Switch, whose factory machine was redesigned to need both hands on switches away from the blade. The happy path should be a glide path, where "the default should be the happy path," and doing it wrong means fighting the current.

Bridget points to a GitHub thread in Jessie's slides where users wanted a use case that would harm the general one. Jessie says the maintainer takes the heat for saying no, and the aim is to leave the mass of users unaffected, while "someone's always going to be pissed off at the end of the day." Bryan recalls one key moment in that thread, when someone came 100 posts down and admitted reading none of it. Jessie remembers the reply, "maybe if you find the time to read the rest of the issue, you will know what is happening right now," and says friends in Slack absorb the heat not sent back on GitHub. Matty calls it a troll flame radiator.

Bridget asks Bryan how to handle people fighting for their local optimizations. Bryan says a job isn't a place where you get your way all the time, and "at the end of the day, it's org first," but the approach is to set expectations early and offer tradeoffs, inside what Bryan calls a "circle of respect." Jessie says in open source there is mutual respect, and multiple maintainers will take the wheel when someone gets frustrated.

Matty, who works for a vendor, explains that Matty and Bridget are advocates inside their companies, and a vendor doesn't have infinite resources. When a large customer demands a feature, a company may redo everything, run a 90-hour week, and compromise 10 to 20 other customers, without anyone asking five whys. Bryan's rule is "It's don't be a jerk," and every decision asks whether it's being a jerk, since there's a scale of jerkiness. Bryan wouldn't demand a feature now, but would point out that a vendor hinted at three months, and that was four months ago. Matty says roadmaps are not promises of dates.

Bridget wonders whether unpaid users are more demanding. Jessie says "Everybody wants their thing, their Turing-complete thing that will make their life so easy," and that some users eventually say thanks after a maintainer adds the feature out of spite. Jessie notices when an issue comes from someone at a company using the tool, and those people are mostly willing to contribute the fix. Bridget notes Allstate contributed authentication and authorization work to open source Cloud Foundry, which Pivotal sells in a commercial distribution.

Bryan describes intersourcing, open source to a company and no one else, and says it exists for good reasons in a specialized vertical like a bank. Inside Capital One, Bryan uses GitHub and gets pull requests all the time, and some are closed because they ignore the roadmap. Bryan says when people push hard internally it's because someone else is prodding them with a fork, and "we're all archaeologists." Matty calls it closed-door open source, and says open source has already solved collaborative coding, so vendors should stop trying to invent it. Matty adds that asking why uncovers the driver three levels up, and Bryan would call the person and say both of you are trying to get things done.

Matty asks about incentives for smaller maintainers. Jessie says "giving people responsibility actually goes a long way," meaning commit rights or code review, and it becomes gamification of reaching maintainer, though some people just want a thank you.

Andrew says open source isn't always shiny, happy people. You can create a lot of value and capture little, and people assume they're entitled to enterprise-level support for weekend efforts. Bridget asks Bryan the right way for an enterprise to consume and contribute. Bryan says it's complicated: licenses and patents, for example, and without patent indemnity, using someone's open source could lead to being sued for IP, so "there's levels to this." Jessie says Jessie stays far from Google's secret sauce to avoid messing up.

Andrew says some people want to give money to projects and there's no good way to, and that adoption sometimes stalls for lack of governance or indemnification. Others go YOLO, and Andrew says "I would be shocked if most enterprises realize what code's actually in production." Matty knows a CTO who requires code review of every open source project, and someone submits 10,000 lines of code that goes straight to prod. Bryan has been at Capital One since the previous October, after doing open source full time, including contributing to Terraform and writing Go for DigitalOcean. Bryan says the bank tries to know everything that goes into production, given the stakes. Bridget adds that during a startup acquisition Bridget had to find every license and rip out one library whose terms would have given the acquirer's IP away.

Asked how to get people to do the tech work and the people work, Bryan says to be the example, with empathy. Jessie says being kind makes people kind back. Matty, who coaches customers and isn't a line manager, says to be genuine, because fake coaches get smelled out. Andrew reinforces that, and says the two can't be solved separately: "there's not really a tech and a people thing. That it's one system."

Bridget and Matt chat about enterprise transformation and open source with guests Bryan Liles, Jessie Frazelle, and Andrew Clay Shafer, in front of a live studio audience at devopsdays Minneapolis 2017.

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf

Use code "ADO2017" for a discount on many devopsdays.

Bridget and Matty record the last of six live episodes at GOTO Chicago, a panel on microservices with Kenny Bastani and Daphne Chong. Kenny is a Spring developer advocate at Pivotal, on Bridget's team. Daphne is a software engineer at Amazon who has lived in the UK, the US and Australia, and gave a talk on video transcoding at the ABC, which Daphne clarifies is the Australian Broadcasting Corporation. The episode opens with Bridget asking what advice Daphne would give someone who thinks they need some microservices, and the answer is "Don't."

Daphne's reason for microservices at the ABC was scale. One part of the system, transcoding, has to scale far more than everything else, and the microservices part "tended by accident really," once that piece was separated. Transcoding means converting a video or audio file from one format to another, and the ABC needed to do it over its whole catalog, with requirements that commercial transcoders and Elastic Transcoder didn't meet. The transcoding service takes a JSON packet saying which file to transcode and where to save the output. It can run on a larger AWS instance type suited to the job, and Daphne says the cost of that instance goes 100% to transcoding. Matty calls that an interesting angle on showback and chargeback.

Bridget points out that spreading work across services adds complexity, and Daphne answers "I think you just move the complexity. Because there's always complexity everywhere." Bridget says that is Tim Gross's "conservation of complexity." Daphne says independently deployed services tend to be stable, and adding a captioning extraction service would sit apart from the others, leaving the main complexity in coordinating which service is called when.

Matty asks how this differs from service-oriented architecture, which Matty worked with at a dot-com years earlier, with a mail service and a lead conversion service behind versioned APIs. Kenny says recent research into continuous delivery changed the definition Kenny started with. A shared delivery pipeline means you're "technically taking public transportation to production," batching up changes from, say, 500 engineers on one monolith. Splitting the pipelines lets people commit and deploy independently. Kenny admits "We're really bad at naming things," then lists small teams organized around business capabilities, independent deployability and a decomposition strategy, and calls it "a better SOA." A rule of thumb: if a new engineer takes longer than a day to ramp up on a service, it should probably be two services. Bridget, speaking as someone on the receiving end of the pager, adds that if a health check can only say some of it is working, too much is crammed into the service.

Kenny's talk covered event-driven microservices, using events to maintain the integrity of foreign key relationships when a large shared database is torn apart. Somebody had found the Cloud Foundry management endpoint on a Spring Boot app in one of Kenny's demos, and crashed the application of ten microservices from an iPhone. Matty guesses that someone thought they were at DEF CON.

Kenny passes along Matt Stein's line that microservices aren't the solution: you already have a problem, and want to go faster or scale. Matty says every VP wants a fill-in-the-blank this quarter, and Bridget recalls an open space at an Agile conference where someone said their VP wanted microservices that quarter. Kenny's advice is to get data first, for instance "How much unchanged code are you deploying per deployment?" Kenny notes that even a monolith on continuous delivery could be deploying every 11.6 seconds. Matty adds that the question is whether you need to go faster at all. Daphne says unchanged code is risk, since every deployment might go wrong, and Matty says small changes minimize the blast radius, which is a reason even if speed isn't.

Kenny says shared resources are a good reason to consider microservices or serverless, and expects a healthy microservice architecture to reduce unchanged code per deploy. Kenny would like to test that by mining GitHub, though no enterprise is going to put its code there. Kenny and Josh Long spent two years writing a book, Cloud Native Java, on cloud-native applications with Spring Boot, Spring Cloud and Cloud Foundry, which had gone to press, with a copy expected in early June.

Daphne's own route to microservices wasn't a formal journey. A team averaging about three people, each knowing a different part, built small pieces, and the scale question drove it. The project "kind of secretly started a bit off-piste" as a proof of concept that would save money.

Bridget says teams sometimes want independently deployable services to avoid interacting with other teams, and asks about weaponizing microservices. Kenny: "Oh, they're already weapons." Kenny says teams need empathy for the services they consume and produce, and describes consumer-driven contract testing, where you publish a contract and other services test against a mock, so you can't reach production without passing consumer tests. That pushes you to go to other teams instead of them coming to you, "a good way to prevent evil." Matty adds that Conway's Law also works in reverse: at places with little trust, a change from hunter green to forest green on the front end meant testing the whole website down to the data warehouse, because nobody trusted the contracts. "No matter what, people are terrible," Matty says, and Kenny says to make the less terrible thing easy.

Bridget asks how serverless fits. Daphne says you still need to deploy and debug the serverless pieces, and the smaller the piece the easier it is: "it's a tool that you should wield in particular circumstances, and otherwise you're just gonna be shooting yourself in the foot." Kenny is researching how Lambda binds you to event sources that lock you into AWS, and suggests a Spring Boot microservice as an event source with Lambda functions as event handlers, though Kenny doesn't know of anyone doing it in production.

Matty says enterprises move slowly, and most enterprise customers on AWS or Azure use only compute, not things like RDS, because they want to hold on to configuration, and a DBA wants to look at a slow query log. Matty's private serverless GIF is an empty data center. The appeal of tools like Habitat is getting some of the value without rewriting an app into a 12-factor app, and Matty can't make a legacy .NET app on Windows 2008 serverless. Kenny defines replatforming as modifying applications to suit a platform that runs them differently, such as a cloud-native one, and says it makes sense where competition is rough. The IRS, Kenny guesses, wants to move faster but faces higher risk and little competition.

Matty brings in Marty Cagan's kill, maintain and innovate for products. A product on the kill list shouldn't be rewritten, one in maintenance doesn't need faster delivery, and only the innovate ones do. Matty says that's where bimodal IT goes wrong, though the Gartner idea properly read is that "different teams move at different speeds," like "one transmission for all of our teams, they're just in different gears." Bridget calls bimodal IT horseshit, and Matty's point is that declaring "microservice all the shit" skips thinking about each product's lifecycle.

Asked for the most ridiculously stupid thing to do, Kenny says creating microservices for things the business doesn't drive, or special snowflake services like image filtering for one consumer. Daphne's answer is languages: the ABC system used different ones for different services, which is fine as long as you don't end up with 20, so "Don't write them all in 10 different things." Matty adds "Don't write one in Ada." Kenny adds shared libraries, since upgrading one across 500 microservices is trouble. Bridget warns about the hidden distributed monolith, where everything still talks to the same database, and Kenny's test is that if one change means deploying all ten microservices, "you've got a distributed monolith."

Bridget and Matt chat with Daphne Chong (Amazon) and Kenny Bastani (Pivotal).

If you have an upcoming conference you would like to see promoted on ADO, you can fill out the handy form at arresteddevops.com/conf