Pulling the Strings

Sign up for our podcast on iTunes or subscribe to our feed with your favorite player.

Welcome to the final episode of our three-part infrastructure security series. Puppet senior software engineer Gareth Rushgrove chats about the future of infrastructure security, and how breaking down organizational silos can help devs, sysadmins and security experts work better together.

Gareth discusses how other DevOps practices like test-driven development and continuous delivery can help you create more secure software and infrastructure.

If you've had the chance to hear Gareth at security conference talks, you know how good he is on this topic. If you haven't heard him, here's a great opportunity to get the benefit of his experience in infrastructure security.

Beth Cornils is senior product manager at Puppet.

Beth Cornils, senior product manager: Hi, everybody. I am Beth Cornils. And we are here to kick off the first of our Security Podcasts. To do that, we have the delightful Gareth Rushgrove. Thank you so much for joining us.

Gareth Rushgrove, senior software engineer: No problem.

Beth Cornils: Gareth has an extensive work experience within security in a variety of capacities. And we'll probably give him an opportunity to talk about that shortly. He is actually one of my go-to people here at Puppet when I have questions or as I'm going through what our roadmap should be and say, "Oh my God, does this make sense or not?" So I'm super happy to have you here, and thanks so much because I know the time zone difference is a little bit of a bugger.

So let's start out with the general niceties. I've said a little bit about me. Gareth, why don't you tell us a little bit about you?

Gareth Rushgrove: Yeah, no problem. So hi, everyone. I'm Gareth Rushgrove. Pretty much garethr everywhere on the Internet. I'm a senior software developer here at Puppet, mainly working on a whole bunch of things around containers, new operating systems, integration.

As an aside, I also curate the DevOps Weekly email newsletter, which lots of people know and love.

Beth Cornils: Yes, I know. I signed up for that once I realized that that was actually a thing. So that's been pretty fun and good. I really appreciate it. Okay. So let's kick off with a couple of questions. So this one's going to sound like, why are we talking about this? We're talking about DevOps and not security. But trust me, we'll get there.

So Gareth, you've spent some time thinking about the DevOps world and how it's evolved. So let's take some time to talk a little bit about that, kind of like from your perspective.

Gareth Rushgrove: Yes. So I'm — I guess I've been lucky enough to be involved in two different ways: One, to be have been part of this DevOps movement from pretty early. I wasn't at the first DevOps days event where like the word DevOps came from. But I think I was at the second or third one, the second one in Europe anyway, in Hamburg. And I've been around the rest of the community. I helped out with some of the DevOps Days events. I've spoken at a bunch of DevOps events and been along and recently spoke at DevOps Enterprise Summit. And I — in parallel to that, I've been in worlds where I've been a software developer.

I've been in worlds where I've been much more on the operations side, operations management side. And I did a lot of crossover bits and pieces. And I think that's how I ended up in that community, as part of that community, interested in the things that this whole DevOps community's been interested in. And one of those things that's become much more of a theme, I would say, maybe two, three years ago now, was security.

And to start with it was very much just a conversation between the typical sysadmin and the typical developer taking a outside-in look at things. But more and more — it was to start with all about everyone, and everyone has just become much more about , I guess, this being testing and quality folks, and performance, and increasingly security.

But one of the things to realize and sort like — in all this everyone knows the word DevOps. Everyone is increasingly familiar with some of the practices. If you jump back five or 10 years, things looked very different. Certainly before we had the word DevOps. But also before a lot of the practices were — and anything even talked about.

Agile systems administration was the precursor. And again, like 10 years ago that was not the way anyone really worked. Alternation was really not the way anyone worked, apart from like on the edges and behind the scenes in a few very specific places. And operations at that point was definitely like a silo. It was definitely something that was not well regarded like from outside its own sphere, that had arguments to relate to developers.

And it was procedures of bottleneck. There was like this stereotype of operations and the stereotype of developers. Like 10 years ago it was — the stereotype was much closer to the reality in most places. And I think it's easy to forget that in where we are now where there's a DevOps event even somewhere around the world every week or two.

Where there're newsletters, where there are companies, where there are tools, where there's a constant conversation about how we can like better operations as a discipline. And there are tools that you can buy, tools that you can download from the Internet free that can help you, and people to talk to and places to go. And it feels like operations has come a long way in the last like five and 10 years.

Beth Cornils: Yeah, I can see that. And I've been — as the product owner for a security, I've spent an awful lot of time talking to security people, and not necessarily Puppet users who are in security, but other people. And one of the common themes I've been hearing is we have all these security conferences that security go and talk to each other about security, about all the things that they already know because they're all concerned about the same things.

And they're starting to actually go to DevOps conferences and talk about security at DevOps conferences to raise that visibility and see how we can all kind of work together. And they're trying to get developers to come to the security conferences as well. So it does seem like there's a shift that is starting to happen. So yeah, I would definitely agree with that. I've been hearing that and it seems like it's a good thing, because any time you keep any particular group in a silo, nothing good can come from that, in my opinion.

Gareth Rushgrove: Yeah. I'd say those people are pretty progressive on the security side. I think there's a growing interest and there's some awareness on the security side. But it definitely feels that it's only been in the last year or two. So that's starting to happen.

Beth Cornils: Yes. Yeah. No, I would agree with that.

And that's one of the things that I've been sending my developers to security conferences, whether they like it or not. And having some of the security people who have a security background here are going to DevOps conferences. So that's one of the things that, from a Puppet perspective, has been really important to me.

As you mentioned, it's only in the last couple of years, as I've heard like a couple of people say, "Yeah, every time I go to a security conference, it's only security people, and we keep complaining about the same things. Or maybe we're evolving a little bit, but how do we get the rest of the industry to follow us along."

Gareth Rushgrove: And I think that's the area between where we are today with security and maybe where we were with our operations systems administration five, 10 years ago. But it looks the same from my viewpoint.

You've got the early adopters. You've got those people who are enthusiastic. They are saying, "Well, actually, I'll step over the line." And whether that was the operators going to developer conferences, developers going to IT prototype conferences, or today security professionals going to DevOps events. It always takes those few people to start with to step over some imaginary nonexistent really, but quite firm line.

I think as well, there's an one to one event in particular in London, and it's on again this year, was the first event I went to where I think they did an amazing job. The audience really felt like it was in thirds. Software development as a third. I saw people from more of an IT operations background, and a third from a security background.

And that was security people from a policy side as well as security people from the more hands-on, security engineering side. And it was operations people from a automation, like software systems engineering point of view, but also operations people from a service management point of view. They did a really good job of bringing together a really wide range of people.

And it felt like one of the first events I've been to where it really was everyone in the room as opposed to being a developer event, which has some security people who have come along to help like developers learn more or operators learn more or whatever. And so I think we'll see more of that as well with like really good crossover events.

Oh, I think that's exciting. I hope we do because it's definitely exciting when you think about it, security has been in a silo for so long.

And when I think back — so I've been in the industry for about 15 years. We had devs who were like, "Oh, I'm not going to go talk to ops." And ops who were like, "Yeah, devs are always going to like screw me over. So whatever." And now they're working together. Before they worked so hard to keep in their own silo. And I see security people in a lot of ways doing the same thing. And as you said, there are some of the cutting edge people who are willing to go over the imaginary line that are starting to like try to integrate and bring everybody together.

So I think it's really exciting. It's kind of an exciting time to be in security, as far as I'm concerned.

Gareth Rushgrove: Oh definitely.

Beth Cornils: So let's see. So how do you see the world changing in security? I find it interesting that there are more people who are willing to reinvent instead of using open source.

There seems to be almost like this “keep everything close to yourself” when you're in security instead of sharing and being a little more open about how you're handling your security, how you're handling security within your industry. Do you think that will start to change? Do you think open source is going to be a thing that will be more relevant and useful in the security world? What are your thought on that?

Gareth Rushgrove: I guess, broadly, yes. But also not just open source but openness in general. I did a recent talk at Information Security Europe in London about the benefits and challenges of openness is security, because I think one of the things that's happened over the last 10-year period or made-up timescale and of operations and developers working more closely together, infrastructure as a service, and like to the DevOps movement.

There's been an explosion in software. But in reality, a lot of that type of software existed, people had their scripts. People had their ways of doing those things. But by collaborating together, they were able to come up with actually high-level tools or more polished tools.

And there's been a lot more effort spent on a small number of tools rather than everyone have their own scripts. And Puppet it's a great example of that. The number of people from such a large number of organizations that have come a bit over the last 10 or so years has provides that network effect. And everyone contributing, everyone using, benefits from everyone else contributing and using.

And I don't think the same things have happened yet in the security environment. There are definitely individual products used for security purposes that are open source. And many of them are low level, and many of them will then the low levels of development tools, the low levels of inspection tools. But you don't really have the higher level and frameworks for the most part.

And I think some of that collaboration around software comes hand in hand with conversations and sharing and being more open in general. And I don't think Puppet would be where it is today as a tool or as a company if there hadn't been all those people talking about it.

Luke did travel quite a lot and started talking about Puppet in the early days. But way more people outside Puppet have given talks about Puppet and showing their colleagues or showing their friends or showing people at user groups Puppet. That happened inside a company, and that effect, that large community of people has I don't see anything quite like that within security yet.

And I think that's interesting because the effect open source and the open source culture and sharing and the network effect of these sorts of platforms in infrastructure, platforms, in operations have had on not just how operators do their work but also the industry, like the companies and players in it. I think the same thing could completely play out in the security environment.

There's a lot of really good tools and products. But in a similar way to maybe things on the infrastructure side, they can be perceived as expensive and for the very high end. And I think in infrastructure we've seen the the high-end tools become as readily available for your three-person startup or your like education establishment, just with different bells and whistles and different ways of working.

I can definitely see the same thing playing out in security.

Beth Cornils: Yeah. I think it's interesting because when I've talked to, again, people in the industry — that's my job as a product manager, which is why I keep bringing it up — it seems like not a lot has changed within the tools that are being used. And so it remains very much the same.

And security can actually be a fairly lucrative kind of environment. And it can, from both the companies who create the tools as well as the people who do things like countries who are willing to steal personal information. You have like Mafia is actually into this.

You have holding companies ransom with DDOS ransomware. So you see a lot of that happening. And so I see the connection between security companies or security teams keeping things closed and more siloed, and the fact that that allows security software companies to not evolve.

They don't have to evolve if the security teams aren't evolving. So do you see that security companies will change as we start to make this change and as people are starting to cross over that line, do you think that that will happen? I mean, I feel like it kind of — it has to happen, right, because otherwise, the security vendors are not keeping up with what people are doing, like the Mafia and different companies stealing PII data or whatever data, that thing.

Like it has to change. So do you feel like this is the time that it's going to happen, or do you have any like ideas of how you think that will happen?

Gareth Rushgrove: I think it is happening already. I think in hindsight, it's easy to look at what happened with operations tools as happening overnight. In reality, it played out over 10, 15 years from the point where certain organizations had taken a different approach and said, "We're not going to use the same tools everyone else is using.

"We want different tools. They don't exist, so we're going to create them for ourselves." And if you talk to very large but not always very large companies, like technology centric companies, not necessarily technology companies as well, where they have real security problems and they have an awareness of the security problems.

And they're often building that tool, really good tools, themselves. And I think you don't just have to do it by piecing very complete solutions together. You can often start seeing it coming out with organizations that piece things together from small components and they invest their money then in integration and gluing things together. And that approach will work for all organizations.

But I think what it will demonstrate is that there are other approaches to securing data, securing infrastructure. A good example of that, I guess, is the — that much more historical model of securing the perimeter, investing everything in a single device or single devices that secure your perimeter and ultimately, say, not having a big wall.

And over time that space become clear that like actually, yes, progress reports are important, but unless like you're actually defending inside your network and you've probably got more problems than you realize. And those problems are completely bypassing your often quite expensive perimeter security, you've put all your eggs in one basket.

And I think like that doesn't have to be the approach, but when you're in a world where you can only buy off the shelf, you often then don't have Internet tradeoffs like that, because you can maybe only invest that. It's that much money and you have to pick one or the other. And when you're doing a lot more bits yourself, you can be a lot more nuanced. But you need the skills and the ability and the trust and the environment to do that.

And I think like with operations, we've shown that organizations can move towards like doing some more of those things. But you will find extreme organizations that do it in an extreme way and pay costs for that and pay an ongoing cost for that, but probably move first. But a lot of those things can then be more industrialized and then later productized and commercialized and play out everywhere else. I think we're already seeing it.

It's just that it's unevenly distributed. And in many cases, you can't buy it today, in the same way as you couldn't buy really good infrastructure automation tools maybe five, six, seven, eight years ago. It's not that they didn't exist. It's how they were preserved by the early adopters. And now, obviously, there's a lot more like modern infrastructure tooling. So I think we're seeing it.

It's just that change takes a lot longer than people think.

Beth Cornils: Yes, it does. Yeah, no that's true. And it's one of those things that I've thought about that what drives some of that change. And it seems like we hear a lot more about breaches. We hear a lot more about security issues. We had very high visibility with the FBI and Apple, that I wonder if part of that will start to help us evolve more quickly within like, oh, it seems like we have to kind of like change more quickly to keep up with the Joneses, so to speak.

In terms of some of our security software and what we're doing in our approach and how do we take a look at it, and do we need to reframe how we're handling security, because it seems like, for a long time, there was a very stagnant way of looking at security - do this, you have these rules and regs that you follow, you follow this procedure, you check off this box.

And those things are good. Those are not bad and we should not throw them out. But there's also the ability to be — and I hate to use the word — but a little more agile in how we address some of our security stuff and how we can take a look, and can we learn from that — if we're less closed off, we have the ability to be more open and make everything better for everyone.

And then it's more of a united front against some of the attackers as opposed to this one company over here found this amazing way to identify anomalies, and they're working on making it open source. And so maybe we can get that in the next year-type thing. Is that what you're talking about? Or did I kind of take us off track there a little bit?

Gareth Rushgrove: No, one example. I think security's one of those areas, and the security experience tends to be quite good as a somewhat close community of sharing within itself, because there's generally a sense that like you don't — you don't win at security by someone else being like compromised, if you like.

There's like — it's not the same. The — as a — like working for an organization that isn't a software vendor, that's not selling some things. Someone else not being compromised is good for you as well. I think that the issue there is that real change tends to come about from this, I guess, existential crisis.

So what's been happening, and Apple being a good example, Apple really saw reinvesting and investing in infrastructure, and using that and pushing that out as part of their marketing, even. And so it's not just technology; it's policy as well. But they had a real existential crisis there. And everyone else can look at that. But most organizations will think, "Well, glad it wasn't me," and move on.

I fear that most organizations change because they have an existential crisis, not because someone else did. And there is that risk that as an entire group, we do know how to do things better, [but] actually like we did individual organizations, it's difficult to find how you apply the pressure within your own organization to do things better, to do things differently.

I think it's going to be one-faceted. And security's becoming more visible. It's becoming more obviously problematic. I think one of the things that's happening there is going to be insurance and how the insurance industry, interestingly, like becomes involved in applying pressure for people to be more secure.

And I think that's — that's going to be one of the things that increasingly plays out. I think in lots of cases already, security is being to a degree driven by compliance made in modern security. And there's a link there that being compliant with something like PCIs, like they are trying to drive standards across an industry and ensure things are more secure.

But that large brush strokes and they're across everyone. And so there's a baseline there. And rather than it being much more contextual, much more about your organization and your own threats, and something that maybe less specific — that is more specific to your organization, is more specific to you as a company. And driving security would probably be good.

And the best driver is always that existential crisis. But that tends to mean something's gone very wrong beforehand, and there's a question of can we, as an industry and as individual companies, do things without that. You hope so. But how that plays out is difficult to see in some cases. I think that the only way that really happens across lots of organizations is with sharing.

It is with people being — people finding it easier to know what works and what doesn't work, and easier to adopt new things quickly. You mentioned this. A lot of that in lots of cases is being able to change our action, be able to not always just move faster but ultimately adapt faster.

And that's something that in lots of cases, security — often security driven by compliance finds it very difficult to do, and is adopted on a like — on a regular cadence, on a week by week, on a day by day basis, rather than seeing through the eyes of a compliance regime which might be much more six month, or yearly or biyearly basis. And it's centered around those audits rather than centered around the day to day security risks, which may have actually little to do with compliance.

Beth Cornils: Right. No, absolutely. And it seems like if I kind of tie loosely with the DevOps world and just getting the security and getting security people out of their silo, having other people within the organization aware of the security risk that they could be helping to prevent.

Or why when a security person is coming to them with their hair on fire with a the bottle of gin in their hand going, "Oh my God, the world is on fire and I need this gin because it's going to make me feel better." They understand the reason why they're doing that. And they take it seriously instead of like, "Oh God, here's someone who's going to tell me no, or who's just going to like take me off track for everything." So it's that visibility and bringing people together.

Gareth Rushgrove: Well, visibility and context.

But is it’s an example that, I guess, that some of the practices that are — actually have really good support in the data about being good, and are being adopted by larger and larger organizations. And so, for example, around continuous deployment, continuous delivery, releasing in small batches and much, much more regularly, with literally the opposite, if you like, of the best practice 10 years ago.

And what's interesting there is that we have data to back that up. So to start with, people were doing it and everyone was saying they're crazy. And to a degree, there're still people saying that today. But it's much easier to have a grownup conversation because we have really good data from across industries, and, for example, in the DevOps report, that shows this works. And I think having the — that only happens when there's — like there's an ability to share, there's an ability to question existing best practices.

And I think the same thing will happen in security. And it's that questioning that has happened in some organizations, and they have new ways of working and they think are better. The main thing is, if they are better, can we prove it? And if we can prove it, can we get the word out?

And again the operations, they're like — the parallel, to me, is someone who's being on lots of different sides of this, really strong and the answer feels like yes. I don't know what the answers are. I don't know what all the new practices are or what the new tools might look like. But they definitely exist. There are definitely better ways of doing like what we're doing today.

Beth Cornils: Right. Yeah. Well, in a way, we kind of have to, right? We can't say where we are.

Okay. So one of the — well, I have two final questions for you. One is open-ended. You tell me if I've missed anything. But what are some of the interesting changes that you've seen recently in the security area, or maybe adjacent to security, that you think is like, wow, this is amazing. This could be — this can really change how the world thinks about things, or not the world but this part of the world.

Gareth Rushgrove: So I think there's, first off, several approaches. And as someone who's been a guest and both a software engineer, operator and has interest in security, and I'm drawn to a number of the tools that are around applying software, applying code to security problems.

I think that's to an extent a product of like my background and my interests. But it was the same thing with infrastructure as code and with Puppet as well. And that worked out. Like we demonstrated that there's value in taking things that were maybe previously in people's heads, differently in different people's heads, written down in human language, maybe across language barriers, maybe not, but just in a lofty way.

This , oh, how do I make this change? Oh, there's a book somewhere on the Wiki. When was it last updated? It was last updated when someone started and did that task because the previous one was out of date. That's like massive information, turning that into well-structured code that we can reuse as being a theme that DevOps may have been.

And there's a number of tools popping in security that — they're all starting to do the same things. They maybe lack the sharing platform at the moment, but the low-level tools are there. So this was actually one of the things that was mentioned around testing in the recent cyber reliability engineering book from Google.

And it's interesting to see that it was something that they'd been doing in parallel to everyone else and doing as well. So tools like Service Pack can be used here like writing unit tests based around security policy for your code. They inspect framework as well from the folks at T-Mobile Labs. There's also BDD security is an interesting one, applying human language code using the cucumber gherkin syntax.

But the tool itself comes with lots of built in security. Has scanning features, if you like. So it's not just that. It moves you away from the idea that an expert runs the scan on an ad hoc basis and interprets the tea leaves, interprets this all this data that's difficult to analyze and gives you a result at the other end. Rather than having — like having code actually scan — and interpret — and you making specific assertions against that.

So you can be much more fine-grained. Also you can push that — those fine-grained checks out into individual application teams or individual service teams or infrastructure teams. But you can all see all of the new answers. So you still need that expertise to know what those things should be in the first place.

But you can much more easily share the information about what's working and what's not, what needs to be fixed, and what's bad, like what's been broken for a while. So I think that's a really interesting area. — you can nearly run with the idea. It's not like — there's not like this equivalent to infrastructure as code. What would we mean by security as code? It's all a good — like a good hypothetical question. Like do some of the same approaches work?

Do some of the same lessons that we've learned through applying software engineering practices to infrastructure? Do they work in security? And I don't think we — I think the answer is probably yes in some way. And we got like anecdotes about that working in some way. But I think that will be a good thing to run with over the next few years.

Beth Cornils: Yeah. That's interesting. I like that. Okay. So then last question: Is there anything that I missed that maybe you wanted to talk about in this? Because sometimes I miss things, it turns out.

Gareth Rushgrove: I don't know. I think that's a pretty good run through a series of topics.

Beth Cornils: I thought you kind of rocked it.

Gareth Rushgrove: I think it's just — it's interesting observing the , I guess, the tectonic plates of all these different IT disciplines moving. And I think if you talked to some people at some organizations, they really don't see themselves as separate. But actually, when — if you're forced to think and look at how you work and how you've always worked and how your department is structured, and how people are paid and how people are incentivized and who reports to whom.

And really take a step back, you can start — you start seeing that actually, yeah, we often have structured IT organizations to have these characteristics of siloing themselves off. And I think that maybe the only thing we haven't touched on is when you realize that from a security point of view, how do you change your organization structurally to best take advantage of maybe this serendipity like of security people talking to developers, talking to operators.

Beth Cornils: Right.

Gareth Rushgrove: And I think that's something else that we've seen on the operations side is that lots of organizations are changing how they structure themselves, that they're undergoing , in some cases, incredibly large reorganizations to take advantage of operating developers working more closely together and ultimately pushing towards being able to move faster.

And I think that's a pressure on security organizations, because if the rest of the organization changes shape and does so in pursuit of moving faster, if a security organization could just keep up before, it's definitely not going to be able to do it in the future if he retains the same shape.

And in lots of those cases, they're probably also not going to — if it's — if they're — if applications are being deployed 10 times faster, let's say, after early, and the security organization's probably not going to get 10 times as many people or 10 times the budget, or 10 times anything else. So it's going to have to change in order to adapt to that new normal.

And I think we'll hear a lot more about — over the next few years how security organizations are reorganizing - what sorts of skills are required, what people are required in them to keep up with that rate of change.

Beth Cornils: Yeah. No, that makes sense, because, well, that's the nature of software. That happens in every group, and I can see that happening in security especially. All right. Excellent. Thank you so much, Gareth, for your time. I really appreciate it.

This was a super fun kickoff to our security podcast series that we're going to be doing. So I hope everyone looks forward to the next one that we have coming up.

Thanks so much, Gareth. I appreciate your time.

Gareth Rushgrove: No. No problems. Always good to chat.

Sign up for our podcast on iTunes or subscribe to our feed with your favorite player.

In this podcast about IT security, hear Puppet founder Luke Kanies talk about how he built security into Puppet from the beginning; how his thinking on security has evolved; how locking down systems too tightly can actually make your IT infrastructure less secure; the security challenges of containers and the Internet of Things; and more.

Puppet engineer Chris Barker brings his experience to the table, talking about how customers actually do security, and why the data security model you use for machines isn't appropriate for humans.

It’s a great listen as Luke and Chris go back and forth on security and Puppet, and the evolution of Puppet's handling of security. Go on, enjoy it now! (You can also read the transcript below.)

Beth Cornils is a senior product manager at Puppet.

Beth Cornils, Puppet senior product manager : Today I'm super excited for our security podcast. I have the honor of having Luke Kanies and Chris Barker here. So thank you both very much for taking the time to talk about security and Puppet. Let's kick it off by Luke, tell us a little bit about yourself, a little ditty.

Luke Kanies, Puppet founder: I'm the founder and CEO of Puppet, and I started Puppet 11 and a half years ago. Prior to that, I was a sysadmin by trade.

Beth: Chris?

Chris Barker: I've been at Puppet for a little over four years now. Before that, I was kind of an IT consultant for hire everything from small businesses to Fortune 500. So I spent a lot of time either doing small business therapy or technical work.

Beth: All right. Excellent. The reason why I have both of you here is because this is a Thunderdome situation. You didn't know that when you walked into the room, and so good luck. No, it's really not. The idea is

Luke Kanies: Well, there's three of us here, so that would be complicated.

Beth: It would be complicated

Luke Kanies: Wait, is that

Beth: but I'm on the other side of the desk.

Luke Kanies: Does that make you Tina Turner?

Beth: It does. It does. [Laughs]

Luke Kanies: Since I get to be Master in this situation, then I get to ride on Chris' shoulders.

Beth: The idea is just to talk about, Luke, from your perspective, why you built in security, how you built it in, and then, Chris, sort of the realistic when you're going in and you're talking to customers. That is why I really have you. There's no fighting. I mean, you can fight, but that's up to you.

We're going to kick it off with you, Luke. When you first started Puppet, there wasn't the same level of concern as there is today from a security perspective. It was a known thing, but people kind of pushed it off to the sides. What did you do to include security in Puppet?

Luke Kanies: There's two ways to think about the security question, and one is "How secure is Puppet as a system? How secure is the platform itself?" And the second one is "To what extent can Puppet be used to make your infrastructure overall more secure?"

On the first side, everybody in security now knows that there's usually a tradeoff between how big of a pain you make it for the user and how secure you end up. You can make your users change passwords every week and have them be 400 characters long and all special characters, but you're going to end up with a less secure system, not a more secure system, if you do that.

So one of my early priorities in Puppet was "How do we end up with a hopefully highly secure system while asking that user to make the smallest tradeoff possible?" One example is, when I started, I wanted to use a security system that everybody already knew, so people pretty well understood X.509 certificates.

In particular, I was moving from CFEngine, where we only had symmetric pairs they used SSH style certificates for both authentication and encryption. The problem with SSH is it's all mutual trust, so if you want to have somebody talk to somebody else, they have to have preshared their keys, and that key presharing was a huge pain in the CFEngine world, and that pain meant people did weird things to avoid it.

One of my goals was "How do I avoid that pain?" Well, X.509 fixes that, because you only have to trust one certificate authority, and then everybody else, from there, you're fine.

There are still some other rules you have to figure out, because, in the world of SSH, presence of a key is kind of equivalent to "I trust them to talk to me," and in this world, you have to build other authorization layers on top of it.

But once I figured out, "Okay, this is my basic security. My authentication model is going to be these certificates," now I have to make the certificates themselves incredibly easy. At the time, we only had one certificate authority that pretty much the whole world used, which was CA.pl.

And even, I think, today, if you want to go use that, it requires you to fill out all this information that, in general, doesn't matter at all to anybody, and you still have to fill that all in, and it doesn't make any sense. So I said, "Why don't I just build a CA that doesn't ask any questions at all?"

That was one of my goals, was get to the point where you can go through the process of building an entirely secure, entirely trustable system without asking the user a single question about security. In fact, most people, unless they have a problem with their certificates, shouldn't be even exposed to the fact that they are using certificates or there is a CA in there.

You end up learning it almost by osmosis, because you have to go through this "Hey, a machine would like to get into the network. Are you willing to sign its cert?" You learn that from there. That was a huge part of "How do I reduce the tradeoff? How do I make it so you're going to end up with a secure system, but don't have to make this big sacrifice to get there?" which again drives people to make insecure decisions, if you ask them to make a big sacrifice.

The second was my experience at Puppet was heavily influenced by tools like Titan, and there's another one.

I think it was Titan and Satan were the two tools. These were tools that were, I think, originally developed for the Solaris world. I know Titan was a Solaris tool. It was essentially a huge collection of Shell scripts that would entirely lock your machine down. I remember having in 2001 or so, I had a Solaris box that Nmap was like, "This might be a Linux machine," because I had done so much to the core configuration to make it more secure

That this idea of building on shared security practices that were essentially removing as much as possible from the system, and then whatever was left, locking it down as much as possible, was a really good idea.

One of the things I'd say I tried to do, but I don't think I did as well as I did the first one, was design in the ability to use Puppet to secure your system really well. I think we've got the platform does really well. What we haven't built is that shared practice.

It's not like you can download the hey, if you're running this version, here are the 10, 20,000 checks to do or switches to toggle to make your system significantly more secure.

Beth: Right.

Luke Kanies: Those are the two areas that I thought a lot about in the buildup, and I think some of that, we still have a lot more work to do, and some of it, I think we've done pretty well, actually.

Beth: Okay. Excellent. Chris, can you talk a little bit about some of the things that you've seen in the field regarding security. I know, just based on some of the stuff that Luke just mentioned, I think it's interesting that we've already kind of thought about baking things in.

I think that helps with some of the disconnect between DevOps and security. But what are some of the things that you're seeing in real life?

Chris Barker, Puppet principal solutions engineer: The biggest areas I've seen are either kind of people using tools such as Puppet, I'd say, as a post-audit failure remediation task, where they've been given a list of "These are all the things I have to change to get back into a compliant state to pass that" and then kind of leaving it in that state, going forward, and Puppet just becomes a monitoring tool in some fashion.

Usually, it's kind of the first step into using Puppet. The other area I've seen, though, is, going back to Luke's point of if you make it harder for users to be secure, they're going to just be less secure. They're going to take the path of least resistance. The password rotation process pretty much means that everyone just has a password with a number in it that's incremented, and that password is then on a PostIt note underneath your keyboard, and that's pretty much how everybody does it.

So if you want to root a system, you just find the EA's keyboard and flip it upside-down, and then that person has because they're the EA, they have the president's email account address and all those other wonderful things.

The thing is getting it easier to do security, which really, as DevOps started taking off, meant that, pretty much, there was still the security was the last thing people ever looked at, because it was so hard to do security.

Trying to build a process and pipeline and tooling to make it easier for people to do security has always been something I've been watching, really to not much success. I've seen a lot of larger vendors come out there with this idea of "Well, just let your developers go off and build a box, and then what we'll do is, we'll wrap it in a hard candy shell of" it used to be "We'll put it behind a firewall," and now it's a

Luke Kanies: Firewalls are magic.

Chris Barker: Yeah. Now, with software defined networking, we'll put the network driver on the machine, but we'll still just encapsulate all the traffic inside the box.

Luke Kanies: There was that period of time, too, where everyone got their own firewall, and it was like, "Oh, a firewall isn't the answer. Many, many firewalls is the answer."

Chris Barker: Well, that's pretty much what SDN is now, is like, "Every box gets its own firewall that talks to all the other boxes," but it's still in this like we don't actually know what the hell people are doing with the box or what the system state is or what the configuration is. We just know that these two boxes have to talk to each other and no one else, and we were going to try really hard to do that.

But that doesn't mean that they're actually going to monitor the traffic going into the box, which means, "Great. That web server can only talk to the database server." Someone just owned that web server, and you're still going to let it talk to that database server and retrieve all of the credit card numbers.

Beth: That seems fine.

Chris Barker: Oh, yeah. It's totally okay.

Beth: Yeah.

Chris Barker: So I think that we've seen a lot more people moving towards having tools like Puppet deploying the workload, so they know what it's doing, and then using Puppet as part of the security implementation of it.

There's still some struggles with it, but that still is providing a lot more transparency than before. In some ways, the easiest adoption has been the security team and the other people working together at least have a language to talk about all these things, whereas before it was security guy had a Perl script that he ran on the box to audit it, and then

Luke Kanies: If you were lucky.

Chris Barker: Yeah.

Luke Kanies: It was often a Word doc.

Chris Barker: Yeah. Well, no. He had a Perl script he ran and then took the output and put that in a Word doc and then sent it to your manager, which you then got two weeks later and then you had to go fix by hand.

And then a month later he'd come back and run the Perl script again.

Luke Kanies: Where I was, literally, the security guy had a Word doc, and he went around typing the commands from the Word doc into a Shell and then would take the output and turn that into a report.

Chris Barker: That's actually been stuff I've put into demos for people, is taking the command from the Word doc and putting it in that fact to then show that

Beth: Right.

Luke Kanies: It's like magic.

Chris Barker: Yeah.

Beth: Yeah.

Luke Kanies: Yeah. You brought up an area that I didn't bring up when I was talking about the design of Puppet, and that's the principle of least privilege, which is, essentially, you want to, as much as possible, basically give people as little privilege as possible.

Any machine in your network should have the absolute least amount of privilege. One of the big controversy isn't maybe the right way to put it, but disagreements about how CM tools should be built is should you distribute all of your code to the edge nodes and let them do all of the work locally, or should you do like Puppet recommends? Now, many of our users don't do this, but have all your code stored centrally and then do all the work there.

00:11:01 And one of the major benefits of putting the code centrally is that you're able to give much less privilege to the edge nodes, and your edge nodes are unquestionably and by edge in this case, what I mean is your web server is your database server is your mail server; it's all those things they're the least secure systems in your network, because they're the systems that your customers are actually directly talking to, and that means they are exposed to the outside world, which means, if your firewall is working well, those machines are available to the Internet.

00:11:27 So if you're able to give them less privilege, that means, if somebody roots that box, they can't read all the code that exists. They can't read the code for how any other machine in the entire network is configured. But even better, when you think about how many systems your configuration arrangement system has to have access to, to correctly compile a complete configuration, everybody who has access to that code needs access to all those systems.

Now, instead of doing either a single or a small pool of integrations, of connections between your Puppet masters and all of these core systems, whether it's your auth. systems or things like that that's, I think, the most secure model.

In this model, now every edge machine needs to have direct access to any data source, that it needs not to apply its configuration, but to compile its configuration, just to figure out what the valid configuration is. So now you've got essentially end-to-end authentications. Every core service is accessed by every machine in your network, which means every machine has, frankly, a good chunk of privilege.

So anything you can do to avoid that is inherently dramatically more secure. Reducing the ability for one machine to read another machine's configuration, and then also making it so that the machines that have access to core services are isolated from the machines that the users have access to, gives you two additional walls of security, just in the architecture, which I think makes a pretty big difference.

Chris Barker: Yeah, and also, actually, in a situation like that, even distributing to the edge nodes, those services do end up building some sort of RBAC service or some sort of service broker.

But that becomes the idea of well, we're going to move the work out there, but now we have to have a governance system, and now all of a sudden the governance system has to extend out to every system as well. And the problem with that is, if the governance system has been extended out to every system, that is now the biggest, juiciest target in your infrastructure. That is literally the governance system that says, "Okay. This node can only talk to this other node," as long as this governance system says so. Well, that's your prime target now.

Because node A has to talk to it to figure out who it can talk to, well, then if you compromise node A, you now have a launch point and attack surface into the governance system. In some cases, there's the argument of "Where do I want to put my CPUs to work?" for the centralized versus decentralized model, but then the funny thing is the decentralized model still has to have a central authority configuration.

Luke Kanies: Right. And one of the things we've worked hard at over the course of the last decade or so is trusting even less of the client machines.

Part of the core architecture of Puppet is hey, went you send facts up, you're just sending data. You're not sending code. So we're not going to let an edge machine produce code that runs on an individual machine.

In fact, we're not even going to let the central server generally write code that runs locally. As part of your configuration package, when you get your catalog from the server, there's very little in there that we trust to run, and if there's an exec in there, we work pretty hard to make sure that's relatively secure, relatively straightforward. You're not going to hide information on an exec not to say you can't, but we work hard to make it complicated to.

Well, over the years, we've tried to have even less trust of those facts along the server side. We want to make sure that you can't inject code into the server by sending data up from the client not to say that it's impossible. Again, there's always some edge in there somewhere, but it's an area where I think we've invested a good chunk, and I think we're close than we've ever been on that front.

Beth: Right.

I'm relatively new in the security space, and as the product manager, I'm learning a whole bunch about different security products. One of the things that it seems like is security is bolted on. That's one of the things that I really like and appreciate about PE, is that you baked it in. I mean, we've talked about some of the benefits. Are there other benefits?

But then what are some of the things that you might like to see change, or not that you would like to see change, but that you see the entire industry is changing and how Puppet is following that along, or Puppet may already be there? What would be some of the things that you experience, either you, Luke and then you, Chris?

Luke Kanies: There's probably three or four areas I can think of off the top of my head where, I think, if you look up in two or three years, we'll be doing things differently, or I hope we'll be doing things differently.

One is we tend to encrypt in transport, but not on destinations, so the connections between two machines are entirely encrypted, but once you're on a machine, things tend not to be stored in an encrypted way. I think we can do better there. Because we've got these public and private keys everywhere, any package that a client sends to the server, they should be able to encrypt that package so that it's not just that you can't listen in on the connection, but that, actually, if you were to go on that client machine and find that file, it would actually still be encrypted there.

Now, you're still not perfect, because that machine has to have the ability to decrypt the file; otherwise, there's no point. But it does make it a little more it's a bit of a security by obscurity, because, if you know where the private key is and you know how public key cryptography works, then you can decrypt it, but it is actually more secure, and you don't get things like leakage by somebody accidentally CATing a file to the screen or copying the wrong files around. Another aspect of the encrypt by default is internally, there are a number of, essentially, secrets that the team's working pretty hard on "How do we encrypt those secrets?"

Right now we try to hash them when we print them, but they aren't actually printed. They aren't encrypted in the back end. How do we encrypt, essentially, in the short term, key secrets; in the long term, quite possibly, any value?

Beth: Yes.

Luke Kanies: Then you get to the point where, yes, the edges can all decrypt, but you essentially have to have those key pairs in place. We've already got the CAs. We've already got both key pairs on both ends. Everybody already has access to those keys.

The infrastructure's there. Now you've just got to build the code paths to make it all work. So that's another area where I think we can and we've already been investing in better security. Once you've got that platform in place, where you can essentially give the system "Here's a thing that I'm going to hand you in one part of the system, and I don't want anybody anywhere throughout the rest of the process to get access, other than this one individual machine." There's a ton of power in that. Going back to the credit cards, it means somebody having access to the database it doesn't help them, right?

You can get a full copy of the database, and it's still entirely encrypted, because the only machine in the entire world that can decrypt that field is the individual node that that data was destined for, because it has the private key. So that makes a pretty big difference.

I think the other areas that are kind of foundationally complicated in a way that's only ever going to get better, but never going to become good, are essentially better trust models and authorization models.

When you talk about what has access to what, who has access to what, we've worked hard to build RBAC into the core architecture, but that's an area where we had some aspects of RBAC early on, but we didn't have anywhere near a sufficient system for what our customers need today. We've done a ton of investment over the years. The problem becomes there are so many capabilities and data types that you care about. If you try to matrix that against all the kinds of people and systems that could care about those, it's just foundational and complex, and so that's an area where you need to keep investing.

Again, the challenge is always how do I keep this simple enough that, when I go visit a customer, they haven't just given up, and they've either made it so everyone logs in with the same account, or everyone is essentially all in one big group, and they all have admin access? And that's an area where I think we've got a lot of work to do to really improve that and especially to push the security models down into the data models down into the access models. That, I think, is what makes it really hard.

Beth: I think what I'm hearing is that my roadmap is approved by the CEO, so that's awesome.

Luke Kanies: Well, doing something is approved.

Beth: [Laughs]

Luke Kanies: I don't know about your specific roadmap.

Beth: My roadmap does include some of those things. No, but having worked in a SAS environment where we had the second highest number of Social Security numbers and PII next to the Social Security Administration, I totally understand the importance of keeping that information secure.

Chris Barker: Yeah. I think the last point you touched on is an area that I'd really like us to see improve a lot more on, which is we have access. I mean, really, we have the access controls down, but now it's the actual data model, so there's some areas deep in Puppet internals where it's like, once you get there, you just see everything. PuppetDB is still kind of in the point of like I can pull up the catalog. If I can talk to it and gets facts, I can pull out the catalog. I can pull out the status.

And those, all of a sudden, realize there's a lot of data in there that you don't initially think of as being sensitive until you realize that sensitive data just ends up in there just because of the act of "This is a great tool. We're using it to automate our infrastructure. Let's just throw stuff in there, throw stuff in there, throw stuff in there. Oh, wait. We just put our backup software in there with the credentials we use, the shared password for all of our backups."

And then, all of a sudden, that means that, going back to the "Well, I got the shared password for the backups over from a web node. Now I'm going to do a restore for the database server and go walk away" I think those types of isolations are the things where from a security standpoint, the things I always look at is the lateral movement that people can do inside an organization, where people just don't realize that there's these privilege access.

I remember, at a previous company, I was doing kind of some audit of their active directory trust rules and found out that there was a front desk receptionist who had full domain access privileges for the entire AD environment, because they were trying to troubleshoot a printer problem.

And they ended up just granting that person they just kind of kept going down and put them in on one group and another group, and then all of a sudden, that whole spiraled outwards into that group technically had domain admin access, because another person troubleshooting another issue added that group to that, not realizing the impact of that change.

Beth: Right.

Chris Barker: So this person literally could have done anything they wanted in the organization. Luckily, they were just trying to print. Going back to also

Luke Kanies: [Laughs] It's pretty insecure, actually.

Beth: [Laughs] Yes.

Chris Barker: Printing is always a terrible thing.

Luke Kanies: When your printing description format is Turing complete, you've got some real problems.

Chris Barker: Yeah. But, no, I think that's really the area that, for us, in the product, is finding better ways to isolate that data and finding either other systems because so much of the design of, for example, PuppetDB was optimized for fast catalog compilation and distribution of facts and storage of facts and being able to kind of be that central data warehouse specific for, really, Puppet masters to do their job very quickly.

Then that turned into "We can use that for other information" and then realizing that there maybe other data model services that would be more optimized for the human-facing integration into that. But all of a sudden, that means that, just because machines have to have access to this data doesn't mean that's the same model that's appropriate for humans, too.

Luke Kanies: Yeah. I think one of the things that I think we have a unique ability to do better at is so much of security can be dramatically improved by taking things that the world already knows and making it so the people who are using your tools also know those as a result.

A great example is it's not that hard to get a list of vulnerable software. It's not that hard to matrix that list against the software that's installed in your system, but it turns out there aren't a lot of tools out there that do that really well

Beth: Right.

Luke Kanies: effectively and efficiently. And then it's not that hard to connect that with remediation software that says, "Hey, make sure that's not the case anymore."

Beth: Correct.

Luke Kanies: But closing that complete loop is not something that other people have done well before.

That's something that I think we're somewhat uniquely set up I don't know that you can be "somewhat unique" but we are well set up for, and very few others are. Again, any case where you can add security into the system with the user not having to answer any questions, not noticing it's there until something would have leaked, but didn't, and they went, "Huh, I didn't realize that was encrypted" any time you can do that, you've gotten a huge security win.

So that point about encrypting the data in PuppetDB, not just encrypting the connection, but encrypting the data there, encrypting the data on the client there are all these places where we shouldn't even ask. It should just like, "Duh. Of course it's encrypted. Of course we're going to do that. Of course this is how it's going to work." You should prenegotiate it. The client server should just do all that for you. Of course, the problem with the negotiation is, as we've learned with SSL recently, if you can tell the server, "Oh, I'm sorry. I can't support that new fancy model," then it will just give you the old bad secure model, so you need to figure that out a little bit, too.

But I think there's a ton of opportunity to go through the system. This is where you cross over from Puppet's architectural security to the security of the applications and things that your customers install. If you can tell your customers, "Look, all these machines you have: here are the insecure systems they have. Here are the insecure package versions or system versions they have. Here are the remediation tasks to take." Because, again, the feeds all exist, but most customers don't directly connect those feeds into production.

Then, of course, the second step is, you need to make it ridiculously simple for your customers to test and employ any potential changes, because, ideally, you do a security fix, it doesn't change production behavior. But that's the ideal world, and we're talking about the real world, unfortunately, so you need to make that test cycle incredibly efficient and incredibly cheap.

Once you can connect it all together in a way that the customer just kind of you put a light up that says, "Here are all the things I remediated last night, and I confirmed that they didn't have any behavior or performance problems" or "Hey, here are the things that I'm going to do, as long as you're okay with it. I'll do them tonight on the schedule that you've already set," and it's essentially all to the point where the user, at worst, has to kind of you know how, once in a while, you look at Chrome, and there's the yellow hamburger menu instead of the green one, and you have to say, "Yes, I'd like to restart Chrome right now." That should be the worst that your customer ever has to see in terms of doing all the security remediations.

That requires the whole system to be connected from end to end with backend data sources that tell you about the reality of vulnerabilities today. Again, most changes that happen on systems today are changes that were well-intentioned, even if they were probably stupid. Being able to differentiate really clearly between "Hey, there were 10 billion changes to your systems yesterday to your 50,000 production servers. There were 120 that we weren't expecting." [Laughs]

Beth: Right.

Luke Kanies: Pulling that 120 out of the thousands and thousands of intentional changes is fantastically complicated, but with the right automation in place, with the right systems in place, it becomes dramatically simpler.

Beth: Right.

Chris Barker: Yeah. I think, when you were talking about the [making a quick redaction to] the remediation stage that you can make that change with confidence speaks to the fact that almost all of the large credit card breaches that we've had in large corporations have never been like, "Hey, I just found this zero day and I've just trolled through the infrastructure in like 30 seconds and punched through all the firewalls" I mean, unless you're a government agency who put them there in the first place.

But, I mean, you could see that most of the stuff was they were there for months and they were going through months-old, years-old vulnerabilities and just kind of going through a playlist and just moving through the infrastructure, and even people were seeing it, but they didn't have any execution ability to make those changes. I think the biggest thing that

When I talk to people about what Puppet as a tool brings to them is the ability to make changes quickly and make them with confidence and know what you're actually doing, and whether or not that means fixing a production issue, removing an admin who has been let go, or literally just rolling out a series of patches and being able to say, "We can fix open SSL today for these problems and know that our application is still going to work tomorrow" I think, at the end of the day, if you can make it easier to get those changes out there to the people, so they can roll out those changes because there's always going to be security problems. There's always going to be remediation. There's always going to be changes that have to happen.

Pushing that forward is really it's the question I've always had, which is, "How do you roll back?" And it's like, well, you can never really ever I mean, you can get really existential about this and you can never actually roll back, but the best thing you can do is roll forward as quickly as possible to the next new known good state. The faster we can shorten those cycles for people to get to that point is really the area that I think will drive a lot of this adoption.

Beth: All right. I could probably ask dozens and dozens of more questions to keep you all going, but we're at probably about the 30minute mark, so I just wanted to leave it. I always like to leave the podcast with Luke, do you have anything else or any questions that I didn't ask or anything that you'd like to talk about that I didn't cover already?

Luke Kanies: I'm probably most interested in the collection and distribution of preexisting security solutions, so essentially going back to the role of Titan, where some expert somewhere took the time to collect "Hey, here's a way to secure the heck out of your systems" and nobody else had to learn that expertise as a result. I feel like we're in this weird world where we're foolishly expecting nearly every organization to be good at security.

I'd love to rely more successfully on the ways in which Puppet is good at allowing people to share expertise. "Hey, if you're really good at installing and configuring this service, then I don't need to be good at it. I can rely on the module you built." I think that's probably one of my kind of biggest unsolved opportunities in the Puppet world, where, if we can get your world as good as Titan was in the year 2000 on Solaris, but not just on Solaris on every platform you use, because why wouldn't you, right?

Why wouldn't you just do all the things that security experts say you should start with this? Now, in a lot of cases, you have to back it off, because, if you start with the most secure version, then suddenly, almost everything is impossible, but without some ability to ratchet security down as much as possible, as quickly as possible, without having to think very much, you're always going to have this massive attack surface. One of the best things that tools like Titan can do is help you reduce that attack surface.

Ironically, we're getting to the point where "Oh, wow. We're going to have containers. We're going to have" and everyone talks about it like, frankly, no one talks about the security role in that world yet, because they're not using it in production yet, so they haven't had to worry about the reality of problems like security, but you're talking about a world where you often have the consequence of a vulnerability gets much larger, because, instead of having one operating system instance, like the old physical world, or 10, like the virtual world, you have a hundred or a thousand.

Let's say every container that you have up, which only is up for 30 seconds, is a vulnerable container, and it's doing pernicious work. It's only for 30 seconds. How are you ever even going to figure out that it's doing pernicious work? Where is it logging in a way that you can even figure out what's happening? It's nearly impossible. So you talk about it's already hard to find the traces of malevolent activity, right? Imagine that in a world where everything is temporary. Everything is essentially going to disappear in 30 seconds.

It's just, if you don't think about security in that world, and if you don't take every opportunity to rely on the expertise of others rather than to try to build it internally, then we're going to be super lost.

Obviously, we haven't even talked about things like IOT, which are such a disaster when it comes to security. It's ridiculous. Some of it's the obvious stuff, like all those operating systems are unpatched and completely vulnerable, but of it's less obvious stuff, like almost anybody who is a super-geek on Internet of Things you can walk up their front porch, yell, "Hey, Siri," open the front door, and walk right in.

Of course, those keys they had before the locks they had before weren't hard to pick, necessarily, but it's a little bit different just being able to walk up to somebody's front door and look like you're allowed in. All that stuff if you could connect that into a platform that knew how to secure things, knew what vulnerabilities existed, had expertise that other people had gone to the effort of building, and just connected that directly into production if you can build that connected platform, suddenly, the whole world looks really different.

And that's got to be the world we're driving to.

Chris Barker: Yeah. I'd say, actually, the Internet of Things is kind of the more terrifying space that

Luke Kanies: Oh my God.

Chris Barker: I feel like some of it comes down to

Beth: Yeah. [Laughs]

Chris Barker: as people who have been working kind of in the virtual world for the last six years or so, where building infrastructure, provisioning infrastructure that is painless, and so being able to iterate on that is pretty easy, versus you're now back to "I have to build something that's going to be shipped on a ship, and there are going to be a thousand of them, and how do I patch it after they've been made and printed" and all this other stuff makes it a terrible process to figure out how to get updates and config changes to it.

At Defcon, there was a talk about how a guy got a solar panel control system that was left with an open VPN connection back to the manufacturer that was on the same subnet as all the other solar panel control systems, so he could go from his network to the manufacturer's network

Beth: Oh my God.

Chris Barker: to someone else's

Luke Kanies: Every other customer in the network.

Chris Barker: Every other customer.

Beth: Yes.

Chris Barker: And they were like, "Oops. We're sorry. We shipped you the wrong one. That was the dev unit," and they fixed it and they remediated after he had gone through the whole process. I'm assuming this was a public talk at Defcon, so everyone knows about it now. You can find it.

But again, the thing of we talk about Agile and DevOps and these quick iterative processes, but then you go to the world of physical manufacturing and you realize that, where we got the ideas from, the state of software development in physical manufacturing gave us the ideas of Agile and Kanban and Lean management, but then the flip side is their software policies, which we are now finally getting adopted in the software community and in the IT community, are still way behind what we're doing right now.

Luke Kanies: Yeah. I think, by 2020, the modern home is going to look dangerously and painfully like a data center around 1999.

Lots of physical infrastructure with direct, poorly written software that's poorly secured. It's all oneoff. It's in completely unknown states. You can't update it. No one in that building knows how to upgrade it. You can't do any assessments on it. All this stuff just missing. It's going to be crazy. And then you're going to have the same number of machines, right? An average data center was, back then, a couple hundred machines. You'll have a

Beth: Right.

Luke Kanies: couple hundred IP-connected devices in your home. And when something breaks when it works, it's going to be like magic, and when it doesn't work, you're going to want to burn the damn thing down.

Beth: Yep. All right. Okay. Excellent. Thank you both for your time. I really appreciate it. This was

Luke Kanies: Thank you for having us on the show.

Chris Barker: Thank you.

Beth: This was a really fun talk. Thanks.

Sign up for our podcast on iTunes or subscribe to our feed with your favorite player.

The Puppet Podcast is back with a three-part series on security. “But wait, isn’t Puppet all about DevOps?” you ask. Here’s the thing: People have always used Puppet for security. (Check out Bill Weiss’ talk at PuppetConf to learn more.) In this episode, we chat with Ben Hughes from Etsy about DevOps and security working together. Turns out, security professionals are people too, and having empathy for them can go a long way to improving relations with them.

In the second half, we talk gender diversity. As many people know, InfoSec consists of 10 percent women and underrepresented groups. I think we can do better, and doing so will make the industry, as a whole, better. Diversity in thought, approach, and problem solving is needed everywhere.

We also chat about various industry issues, including industry shifts and where Puppet is growing its security and compliance offering. And there may be a wisecrack or two about fedoras.

A transcript of the podcast is below.

If you like what you’ve heard, check out Ben Hughes’ blog post on Imposter Syndrome. It’s really well done, and I suspect will resonate whether you work in security or not. I also recommend two DevOpsDays PDX talks:

Ben discusses Why won’t my DevOps team talk to my security team.

Tune in this November to get a glimpse into another side of how Puppet has security baked in, with Luke Kanies and Chris Barker.

Beth Cornils is senior product manager at Puppet.

• Check out our white paper on maximizing security with configuration management.

Kara Sowles, Puppet community manager: Hi, everyone. Welcome to the Puppet podcast. I'm [Kara]. I'm your usual host. But today, I'm so excited to be introducing our security podcast miniseries. Beth from Puppet is here and will be leading the series of podcasts on — you guessed it — security.

We've still got plenty of Puppet-filled podcasts ready for you as well. Until then, Beth, do you want to talk a little bit about why we find this topic so important as to dedicate a whole miniseries to it?

Beth Cornils, senior product manager, Puppet: Yes, Kara. I would love to talk about that. In the world today, security is really important. It's important for a variety of reasons. One of the things that I love about the first podcast that we're doing to kickoff is that a lot of people don't understand that every single person within the organization plays a part within security.

It doesn't matter if you're the receptionist or if you're the security architect. Everybody is important. So the goal is to talk about security, to make the security people real people, to help everyone understand why it's important.

Kara: Excellent. That's — actually, I'm hearing a lot of pressure for me even as the community manager it's — I'm part of security too.

Beth: You are turns out —

Kara: Uh-oh.

Beth: — a really important one. [laughter]

Kara: Well, who is our first guest here today?

Beth: So our first guest is Ben Hughes, the delightful Ben Hughes.

Ben Hughes, security monkey at Etsy: Hello.

Beth: So — hello. And while I've never personally met Ben, I have heard great things about him. So I've also been sent to a number of his videos and podcasts. And I would recommend everybody else do the same thing. I find them entertaining and informative.

You can't find that very often in a video, especially about security. I mean, come on. And also, he did this amazing write-up about imposter syndrome in security, which I think is relevant for everyone, not just people in security.

So that was — I would encourage everyone to kind of go down that path and read it if you haven't already. So I kind of feel like I've talked enough at this point. So let's turn it over to Ben. Ben, tell us a little about you.

Ben Hughes, security expert, Etsy: Hi. I'm Ben. I work for Etsy. I'm based in San Francisco [at Etsy, predominantly] based on New York, Brooklyn, but they have offices throughout the world. Before that, I was actually at Puppet Labs, as it was back then.

Kara: I thought I met you somewhere before.

Ben Hughes: [Reductive Labs community leader]. [laughter] I went to the first Puppet camp back in like the 1800s, which was the first-round interview for everyone at Puppet now. So I've been involved in the Puppet world — was involved in the Puppet community and seemingly still am, which is why I'm here. Before that, I've — in a number of successful and failed startups throughout London and other places in the world.

Beth: Excellent. So one of the things that I wanted to talk about is that people often assume that, when they're going to talk to a security person, that the first thing the security person is going to tell them is no. No, you can't do that, whether you're a developer, whether you're ops. It doesn't matter who you are. They're going to go to security.

And it's like you're going to security because you're in trouble. So I wanted to get your perception about, over the years, kind of how you've handled that and if you've seen that that's changed at all and if it depends on the organization or maybe like how you've changed over time.

Ben Hughes: Yeah. I've definitely changed over time. I remember, 15 years ago in one of my first security roles, I was quite young, pretty immature, not always the most thoughtful about how my actions would impact anyone else. I've certainly taken out a lot of Solaris servers by trying to find vulnerabilities in them. But that's kind of your own fault.

I think the industry or at least parts of the industry has matured over the years. Certainly, I'd love to hope I have. But the jury is still out. If you just say no to people, then they'll ignore you and go around you rather than actually doing the right thing.

So to steal the line from the exciting world of improv, saying. "Yes. And what can I do instead?" or, "Can we compromise?" or, "Do you really need this access?" or, "How can we actually redesign this in a sensible way?" and actually thanking them for coming and talking to you rather than punishing them because not that many people like just being punished for asking a fairly innocuous question.

Beth: I would imagine — and you can correct me if I'm wrong — that there's a difference between a developer and working with a developer in terms of from a security perspective and sort of the no aspect and someone working in HR or finance or someone who maybe is not in developing the code that goes into the features that get sent out.

Ben Hughes: Yeah. I think security actually needs to look at what the actual impact is and what people are trying to do. The majority of people in finance or developers or any part of your organization generally don't wake up in the morning and go, "How can I be the most malicious and unhelpful to security I can?"

They, like everyone else, just want to get their job done so that they can go home and [endure] Netflix. So how you like help them do that in a way that doesn't alienate them from you is actually the job of security not to just like block everyone from doing everything.

Because it turns out blocking everyone from doing everything is not a great way to make money. And yeah capitalism.

It's really not it turns out. No. As I've been working on some of the security stuff for Puppet, that's one of the things that I've heard from people all over. It's like the most secure thing is to just unplug everything and don't let anybody do work. It turns out, can't do that.

Ben Hughes: Yeah. It's not — I don't know that many successful businesses that have taken all their computers and filled them with concrete.

Beth: [laughs] I don't either. It's a [very weird thing].

Ben Hughes: But I didn't study economics. So I'm not an MBA. So I could be way off base there.

Beth: [laughs] You could. But I think you're probably right.

Ben Hughes: Yeah. The phrase security nihilism, which Alex Stamos of formerly Yahoo and now Facebook has kind of coined, rubs into that view of it being very much [idio-secure], as in it's smashed into tiny pieces and then launched into the sun, or it's not secure. And it turns out there is a step in between those, in fact numerous steps in terms of how much time, effort and money you're willing to throw at securing things and how important things are.

Beth: So with that, especially from a developer's perspective, how — or ops — how do you get people comfortable with the fact that nothing is ever going to be 100 percent secure? So like you saying, "No, but," or "Hey, have you thought about the consequences of this?"

How do you approach that so that they understand that you understand that nothing is ever going to be 100 percent secure? I'm assuming you understand that. Should I have clarified that first?

Ben Hughes: So I think developers and operations people certainly understand that better than security people. Security people — this trend is beginning to erode. But certainly — and I've been as guilty of this and brilliantly continue to be of like, "We can't do this. There are ways around it. Or there are bypasses. Or there are flaws in it."

And you're like, sure. But like most homes have terrible locks on them. It turns out that's fine because the majority of criminals don't pick locks. That's actually quite rare. They do have things like bricks or wait till people leave their door open.

And like spending like thousands of dollars on an amazing set of locks for like a $10 wooden door frame, also pointless. So they kind of get lost in those weeds.

Whereas, if you ask a developer, "Does your code have any bugs then?" all but the most arrogant will go, "Well, yeah. Of course." And we'll find those. And that will be exciting. And you ask an ops person, "So we need 100 percent up time on this."

And anyone with their beans will go, "You don't pay me enough to make that 100 percent uptime. And we're going to need a much bigger budget to do that," because those are like untenable goals.

But you say to a security person, "We need to make this 100 percent secure," and you'll either have some that are still laughing to this day about that notion or some that go, "Okay. Then, we need to do all these things." And it's not going to happen.

Beth: Right. Okay. So how do you bring awareness to security within an organization without making people completely paranoid? Because I know, once I started investigating and getting into, as a product manager into the world of security, all I could think is, my gawd, the world is burning.

And it seemed a little bit terrifying. So are there ways to make it fun so that people understand that, hey, I do play a part, and it's not the end of the world if something gets screwed up?

Ben Hughes: Yeah. I'd say two parts to that. So with the whole the world is burning, it is. But the world has also been going quite a while. And it hasn't magically ended yet. So as insecure as you were yesterday, just because you have new knowledge about that, that doesn't actually change anything. It just makes you paranoid but not necessarily with too much more real reason.

Beth: Well, not everybody is as Zen as you are or have been doing this as long as you are.

Ben Hughes: I'm very glad people aren't as Zen as me, as there would have to be a lot more sleeping tablets. On the raising awareness, certainly at Etsy, we have a couple of concepts for doing that.

One of them is security candy where we just spend lots of money on candy and have jars of it at our desk, which encourages people to swing by our desks and, inadvertently, sometimes have to talk to us and build up rapport with us and occasionally even talk to us about security things, all while slowly killing themselves through fructose.

And I think we have an unsaid policy of all emails from us by near law must have some amusing GIF on them so that you know it's from the security team because there's — if you're going to give people bad news, at least give them like a dog falling over or something equally wild to lighten the load, which also like occasionally doubles as a, "Was this really sent by you? There was no ridiculous animated animal associated with it."

We're like, "No. That was. We just missed it." So I think those seemingly small things actually do quite a lot. And I know my friend [Astira] at SoundCloud actually prints — you call them buttons. The free world calls them badges with like a button press, badge press. I'm looking at Kara —

Kara: Yeah. It's a button.

Ben Hughes: [Because you most certainly] have like six of these [at your house].

Kara: Yeah. No. I have a whole box of buttons but no button maker. So if you've got a button maker, we should talk later.

Ben Hughes: I know. You can probably buy one off Etsy.com. [laughter] It makes those for when people do good security things. If like you reported some phishing, you get a button.

That actually — their ridiculous gamification of buttons — they're still badges — seems to work quite well, fun things like that. Security doesn't have to be all tears and sadness.

Beth: It doesn't? Gawd, that's good to know because, wow, it's been kind of depressing.

Ben Hughes: It's still depressing. But you're less depressed with a button.

Beth: Okay. Okay. You're right. That can make the whole world better.

Ben Hughes: Yeah.

Beth: The whole world gets better with a button.

Ben Hughes: Certainly better than most things vended than it is of RSA or blackout.

Beth: Yeah. Yeah.

Ben Hughes: So —

Beth: Okay. I'll give you that. Fair enough. And I guess we've kind of already talked about that because, you know, I prepared, and we didn't really go off my script kind of like what you said. That was awesome. Thanks for the heads up.

Beth: [laughs] What? So what do you wish people understood about your area of security and what you do? And do you think that that would help understanding with kind of — so here's what I'm kind of looking for is the humanization —

Ben Hughes: What do you want me to say?

Beth: I want you to answer from your heart. And — [laughs] so let me give you an example. Oftentimes, I go into meetings. And people will say, "Well, product didn't tell us. So we didn't do it."

And it's like, hi. My name is Beth. I am product. I'm right here. So can you say, "Beth, I need some information from you"? Does security have that same issue that it's like you're security; you're not Ben who works in security?

And what would you want people to understand about your day-to-day life that kind of like humanizes you? Because we all have bad days. We all have good days. Sometimes, we like — I can't swear — screw things up. There's all kinds of things that can go wrong in a given day.

Sometimes, you can have an entire bad week. And it doesn't make you a bad — I know. But it doesn't make you a bad person. So how do you kind of like —

Ben Hughes: In security, we dream of only having bad weeks as opposed to like bad epochs. I blame Solaris. Humanizing us — I d — I visit a lot of security people out there who probably aren't that human and who are very good at like smashing out bugs or reversing things or writing exploits.

But those skills are amazing and years beyond me. I'm never going to be that smart. But they're not the area I specialize in. And they're not what I spend my time doing. I have always — that's not true.

Since being in the world of employment, have always been trying to defend things, which is a different skillset in that you have to know what you have. And you have to know what's valuable. And you have to defend it that way.

So I've not suffered too much from just being an amorphous blob of security. I'm sorry, in product, that you are.

Kara: This is a pretty familiar topic, I think, for a lot of — I know we have a lot of listeners that are sys administrators and work in IT. Like this question kind of comes up a lot I think in those contexts as well.

Beth: Right.

Ben Hughes: I think, to associate it back to that, the struggle I've had in days doing systems work is, if — and this isn't to start the sales and engineering war that has been hashed to death in the world.

But if sales has a good week, everyone is like cheers. Yay. You've sold this much stuff. If the website still works, no one goes around high-fiving your sys admins. They're like, yeah. Still works. Lights are still on.

Security is like that. And to have my own bleeding-heart story — but even worse because I'm very hard done by — that like we don't think we got hacked this week because it's really hard to prove that. Yay. It's like an even less empty, hollow victory than the website is still up.

It's like as far as we're aware with the tools we have available, we haven't been owned by a large nation state who are going to do something in like two years' time, which is not the same as dehumanizing in that way. But it's very hard to celebrate those wins.

Beth: Right. Okay. No. I think that's important because I think one of the things that we do within organizations is we spend an awful lot of time dehumanizing the people that are like the sysadmins that go through. And they just make sure that everything works.

And I think security gets that same — they are perceived in a very similar way except in a way maybe almost worse because they're the ones who are going to tell you, "No, you can't do that," as opposed to a sys admin that you go to.

And it's like, "Hey, everything broke. And everything is on fire. Can you fix it?" And they're like, "Okay. Yeah. It's 4:00 in the morning. Sure."

Ben Hughes: Or they like build things and like —

Beth: Toss it over.

Ben Hughes: — provide you with — well, like systems people provide you a platform to run your thing on. Or like they've enabled this growth. Like security hasn't stopped you doing your work in an annoying way.

And you're like, cool. Thanks for not making my life as annoying as you could have. But you've made it this annoying, which I see as more annoying. But the delta of annoying is not as much as it could be. And like no one celebrates that.

Beth: Right.

Ben Hughes: No one is like, "You brought in awesome things like Duo for 2FA rather than SecurID." And you're like, "Now, I have to find my phone," rather than, "Now, I have to find a small token that may run out of batteries in two years and may or may not have been lost somewhere in my house." So that's like actual real-world challenges, the integration of these things.

Beth: Excellent. So have you found any tools that really help make that easier? Or are you just going to drink a lot of gin?

Ben Hughes: Yeah. Just gin is the main tool.

Beth: Okay. Yeah. It's a good tool.

Ben Hughes: Yeah.

Beth: I find that, in product, it works pretty well. Okay. Fair enough. So I only have one more question that I think we didn't really cover of my highly prepared, talking to my cat of all the questions, which is, is there anything that I missed that you'd like to add?

Ben Hughes: Sure.

Beth: That's my favorite question.

Ben Hughes: Yeah. I just got back from sunny Las Vegas for DEF CON and Black Hat, the two biggest security conferences of the year. Well, one of them is a security conference. The other one is DEF CON. Throw some shade already. [laughter] See who I haven't upset in the industry yet. And then —

Beth: There's no fedoras in the room.

Ben Hughes: Okay. That's good. From that, there's certainly a lot of interest and hype in the newest bug or the coolest exploit. And there's not been as much focus on the, "Well, this is the coolest mitigation against this exploit. Or this is how you defend against these things," because part of how humans appear to work in that we quite like competition and pushing each other down and part of how marketing — bless them — works is you have to have the excitement and the hype.

But that means you don't focus on the actual useful things like defending stuff. You focus on the cool media thing like, in the exciting world of soccer — which we have in the rest of the world — no one can name five goalkeepers. But they can probably name five strikers. And you're like, yeah. That seems a fair balance.

So it's that kind of mentality. But I think slowly the industry is realizing that, if we actually want to make things better, that we can't just keep breaking stuff and going, "Found a hole in your stuff. You're an idiot."

You're like, "Cool. But you couldn't make this in the first place. So cool. But thanks for telling me about the vulnerability," and actually work out how to actually make things more secure rather than just — what's a polite way of saying ego contest? Ego contest.

And there are some cool bugs. I don't want to take anything away from security researchers and all those people who do things I could never even begin to do. But not all the time are they actually pushing the agenda of making things more secure. They're just finding cool bugs for fun, which is totally a cool thing to do. But — yeah. It may not be making the world a happier, better place.

Beth: Fair enough. All right. That's a wrap for our very first security podcast. Thank you very much, Ben Hughes. As I mentioned at the beginning, Ben — you can find on YouTube a number of his security videos and look for other podcasts that he's done.

Also, if you find him on the Twittersphere, you can find a link to his imposter syndrome and security document, which is very good. So thanks very much everybody for listening.

And thanks, Ben, for helping to talk about the more human side of the security world. It was much appreciated. And I hope that was useful for all of those listening.

Note: This podcast was originally posted on November 6th, and we're reposting it to our main podcast channel.

Welcome to the Conversations with Puppet Labs podcast. We'll be talking with folks in the industry about a variety of topics that affect IT operations.

Over the next few weeks, we'll focus on what it means to be a change agent in your organization, and how ops can influence positive change. We'll explore the challenges and roadblocks you might come across when trying to implement change, and we'll dive into some of the technical aspects of implementing IT automation and other DevOps practices.

We're kicking things off with Gene Kim, author of The Phoenix Project and founder of the DevOps Enterprise Summit. Listen as Gene describes what it means to be a change agent and some of the challenges you may face, such as:

• Doing automated testing right (or at all)

Have a story about making positive change on your IT operations team? Tell us about it: email [email protected].

Theme music: "Smirking" by BenSoundBeats

In the latest edition of the Puppet Podcast, you’ll find out about Razor, the Puppet provisioning application. The Puppet Labs software engineers who work on Razor, David Lutterkort and Scott McClellan, join Kent Bye and Kara Sowles to discuss the history of Razor and the new capabilities you can look forward to in the upcoming release.

You can also check out our many other recent podcasts by visiting our podcast page or subscribing in your favorite podcast tool.

• How to provision with Razor on Puppet Enterprise.

Jonathan Fletcher of Hiscox Insurance talks about building a reusable change platform with Puppet that includes continuous integration and testing tools. His teams have seen a radical increase in the pace of change, going from a half-dozen deployments in a 10-week change cycle to as many as 50 deployments per week! Deployments used to be big events, and are now routine non-events.

In this interview, Jonathan also describes:

• How the increased pace of change allows Hiscox to maintain a competitive advantage in its market: niche specialist insurance.

Finally, Jonathan talks about the fact that some of Hiscox's most difficult changes have been cultural, underlining the importance of plenty of communication and demonstrating success with a proof of concept. Puppet has brought software development best practices to Hiscox's operations teams, and Jonathan talks about how that's changed behavior and helped to cultivate a collaborative DevOps culture.

• Read Jonathan's blog post about how he implemented DevOps at Hiscox.

Elisabeth Hendrickson is currently the senior director of engineering at Pivotal, and a longtime evangelist for testing. In this week's podcast, Elisabeth talks about the process of cultivating a cohesive engineering culture at Pivotal through the introduction of extreme programming techniques. She discusses a wide variety of topics related to change management, including:

• How change can trigger the grief process as you seek to change established practices and processes.

Elisabeth says if you're considering bringing about change in your organization, the No. 1 thing you can do is start holding regular retrospectives in order to cultivate an open and safe place for authentic communication.

• See video interviews from Getty Images, HP, Stanford University and more.

Linda Rising is the co-author of the book Fearless Change: Patterns for Introducing New Ideas. She's studied change management within organizations for over 20 years, and discovered 48 patterns of behavior that are associated with successful change. In this podcast, Linda covers the importance of:

• Becoming an evangelist to sell your ideas

Linda approaches both the strategic and tactical sides of bringing about change to an organization, and her insights feel particularly germane for building cross-functional teams and work groups.

• Did you catch our last conversation? Derek Austin shares practical advice for launching a continuous delivery initiative.