Sign up for our podcast on iTunes or subscribe to our feed with your favorite player.

Welcome to the final episode of our three-part infrastructure security series. Puppet senior software engineer Gareth Rushgrove chats about the future of infrastructure security, and how breaking down organizational silos can help devs, sysadmins and security experts work better together.

Gareth discusses how other DevOps practices like test-driven development and continuous delivery can help you create more secure software and infrastructure.

If you've had the chance to hear Gareth at security conference talks, you know how good he is on this topic. If you haven't heard him, here's a great opportunity to get the benefit of his experience in infrastructure security.

Beth Cornils is senior product manager at Puppet.

Beth Cornils, senior product manager: Hi, everybody. I am Beth Cornils. And we are here to kick off the first of our Security Podcasts. To do that, we have the delightful Gareth Rushgrove. Thank you so much for joining us.

Gareth Rushgrove, senior software engineer: No problem.

Beth Cornils: Gareth has an extensive work experience within security in a variety of capacities. And we'll probably give him an opportunity to talk about that shortly. He is actually one of my go-to people here at Puppet when I have questions or as I'm going through what our roadmap should be and say, "Oh my God, does this make sense or not?" So I'm super happy to have you here, and thanks so much because I know the time zone difference is a little bit of a bugger.

So let's start out with the general niceties. I've said a little bit about me. Gareth, why don't you tell us a little bit about you?

Gareth Rushgrove: Yeah, no problem. So hi, everyone. I'm Gareth Rushgrove. Pretty much garethr everywhere on the Internet. I'm a senior software developer here at Puppet, mainly working on a whole bunch of things around containers, new operating systems, integration.

As an aside, I also curate the DevOps Weekly email newsletter, which lots of people know and love.

Beth Cornils: Yes, I know. I signed up for that once I realized that that was actually a thing. So that's been pretty fun and good. I really appreciate it. Okay. So let's kick off with a couple of questions. So this one's going to sound like, why are we talking about this? We're talking about DevOps and not security. But trust me, we'll get there.

So Gareth, you've spent some time thinking about the DevOps world and how it's evolved. So let's take some time to talk a little bit about that, kind of like from your perspective.

Gareth Rushgrove: Yes. So I'm — I guess I've been lucky enough to be involved in two different ways: One, to be have been part of this DevOps movement from pretty early. I wasn't at the first DevOps days event where like the word DevOps came from. But I think I was at the second or third one, the second one in Europe anyway, in Hamburg. And I've been around the rest of the community. I helped out with some of the DevOps Days events. I've spoken at a bunch of DevOps events and been along and recently spoke at DevOps Enterprise Summit. And I — in parallel to that, I've been in worlds where I've been a software developer.

I've been in worlds where I've been much more on the operations side, operations management side. And I did a lot of crossover bits and pieces. And I think that's how I ended up in that community, as part of that community, interested in the things that this whole DevOps community's been interested in. And one of those things that's become much more of a theme, I would say, maybe two, three years ago now, was security.

And to start with it was very much just a conversation between the typical sysadmin and the typical developer taking a outside-in look at things. But more and more — it was to start with all about everyone, and everyone has just become much more about , I guess, this being testing and quality folks, and performance, and increasingly security.

But one of the things to realize and sort like — in all this everyone knows the word DevOps. Everyone is increasingly familiar with some of the practices. If you jump back five or 10 years, things looked very different. Certainly before we had the word DevOps. But also before a lot of the practices were — and anything even talked about.

Agile systems administration was the precursor. And again, like 10 years ago that was not the way anyone really worked. Alternation was really not the way anyone worked, apart from like on the edges and behind the scenes in a few very specific places. And operations at that point was definitely like a silo. It was definitely something that was not well regarded like from outside its own sphere, that had arguments to relate to developers.

And it was procedures of bottleneck. There was like this stereotype of operations and the stereotype of developers. Like 10 years ago it was — the stereotype was much closer to the reality in most places. And I think it's easy to forget that in where we are now where there's a DevOps event even somewhere around the world every week or two.

Where there're newsletters, where there are companies, where there are tools, where there's a constant conversation about how we can like better operations as a discipline. And there are tools that you can buy, tools that you can download from the Internet free that can help you, and people to talk to and places to go. And it feels like operations has come a long way in the last like five and 10 years.

Beth Cornils: Yeah, I can see that. And I've been — as the product owner for a security, I've spent an awful lot of time talking to security people, and not necessarily Puppet users who are in security, but other people. And one of the common themes I've been hearing is we have all these security conferences that security go and talk to each other about security, about all the things that they already know because they're all concerned about the same things.

And they're starting to actually go to DevOps conferences and talk about security at DevOps conferences to raise that visibility and see how we can all kind of work together. And they're trying to get developers to come to the security conferences as well. So it does seem like there's a shift that is starting to happen. So yeah, I would definitely agree with that. I've been hearing that and it seems like it's a good thing, because any time you keep any particular group in a silo, nothing good can come from that, in my opinion.

Gareth Rushgrove: Yeah. I'd say those people are pretty progressive on the security side. I think there's a growing interest and there's some awareness on the security side. But it definitely feels that it's only been in the last year or two. So that's starting to happen.

Beth Cornils: Yes. Yeah. No, I would agree with that.

And that's one of the things that I've been sending my developers to security conferences, whether they like it or not. And having some of the security people who have a security background here are going to DevOps conferences. So that's one of the things that, from a Puppet perspective, has been really important to me.

As you mentioned, it's only in the last couple of years, as I've heard like a couple of people say, "Yeah, every time I go to a security conference, it's only security people, and we keep complaining about the same things. Or maybe we're evolving a little bit, but how do we get the rest of the industry to follow us along."

Gareth Rushgrove: And I think that's the area between where we are today with security and maybe where we were with our operations systems administration five, 10 years ago. But it looks the same from my viewpoint.

You've got the early adopters. You've got those people who are enthusiastic. They are saying, "Well, actually, I'll step over the line." And whether that was the operators going to developer conferences, developers going to IT prototype conferences, or today security professionals going to DevOps events. It always takes those few people to start with to step over some imaginary nonexistent really, but quite firm line.

I think as well, there's an one to one event in particular in London, and it's on again this year, was the first event I went to where I think they did an amazing job. The audience really felt like it was in thirds. Software development as a third. I saw people from more of an IT operations background, and a third from a security background.

And that was security people from a policy side as well as security people from the more hands-on, security engineering side. And it was operations people from a automation, like software systems engineering point of view, but also operations people from a service management point of view. They did a really good job of bringing together a really wide range of people.

And it felt like one of the first events I've been to where it really was everyone in the room as opposed to being a developer event, which has some security people who have come along to help like developers learn more or operators learn more or whatever. And so I think we'll see more of that as well with like really good crossover events.

Oh, I think that's exciting. I hope we do because it's definitely exciting when you think about it, security has been in a silo for so long.

And when I think back — so I've been in the industry for about 15 years. We had devs who were like, "Oh, I'm not going to go talk to ops." And ops who were like, "Yeah, devs are always going to like screw me over. So whatever." And now they're working together. Before they worked so hard to keep in their own silo. And I see security people in a lot of ways doing the same thing. And as you said, there are some of the cutting edge people who are willing to go over the imaginary line that are starting to like try to integrate and bring everybody together.

So I think it's really exciting. It's kind of an exciting time to be in security, as far as I'm concerned.

Gareth Rushgrove: Oh definitely.

Beth Cornils: So let's see. So how do you see the world changing in security? I find it interesting that there are more people who are willing to reinvent instead of using open source.

There seems to be almost like this “keep everything close to yourself” when you're in security instead of sharing and being a little more open about how you're handling your security, how you're handling security within your industry. Do you think that will start to change? Do you think open source is going to be a thing that will be more relevant and useful in the security world? What are your thought on that?

Gareth Rushgrove: I guess, broadly, yes. But also not just open source but openness in general. I did a recent talk at Information Security Europe in London about the benefits and challenges of openness is security, because I think one of the things that's happened over the last 10-year period or made-up timescale and of operations and developers working more closely together, infrastructure as a service, and like to the DevOps movement.

There's been an explosion in software. But in reality, a lot of that type of software existed, people had their scripts. People had their ways of doing those things. But by collaborating together, they were able to come up with actually high-level tools or more polished tools.

And there's been a lot more effort spent on a small number of tools rather than everyone have their own scripts. And Puppet it's a great example of that. The number of people from such a large number of organizations that have come a bit over the last 10 or so years has provides that network effect. And everyone contributing, everyone using, benefits from everyone else contributing and using.

And I don't think the same things have happened yet in the security environment. There are definitely individual products used for security purposes that are open source. And many of them are low level, and many of them will then the low levels of development tools, the low levels of inspection tools. But you don't really have the higher level and frameworks for the most part.

And I think some of that collaboration around software comes hand in hand with conversations and sharing and being more open in general. And I don't think Puppet would be where it is today as a tool or as a company if there hadn't been all those people talking about it.

Luke did travel quite a lot and started talking about Puppet in the early days. But way more people outside Puppet have given talks about Puppet and showing their colleagues or showing their friends or showing people at user groups Puppet. That happened inside a company, and that effect, that large community of people has I don't see anything quite like that within security yet.

And I think that's interesting because the effect open source and the open source culture and sharing and the network effect of these sorts of platforms in infrastructure, platforms, in operations have had on not just how operators do their work but also the industry, like the companies and players in it. I think the same thing could completely play out in the security environment.

There's a lot of really good tools and products. But in a similar way to maybe things on the infrastructure side, they can be perceived as expensive and for the very high end. And I think in infrastructure we've seen the the high-end tools become as readily available for your three-person startup or your like education establishment, just with different bells and whistles and different ways of working.

I can definitely see the same thing playing out in security.

Beth Cornils: Yeah. I think it's interesting because when I've talked to, again, people in the industry — that's my job as a product manager, which is why I keep bringing it up — it seems like not a lot has changed within the tools that are being used. And so it remains very much the same.

And security can actually be a fairly lucrative kind of environment. And it can, from both the companies who create the tools as well as the people who do things like countries who are willing to steal personal information. You have like Mafia is actually into this.

You have holding companies ransom with DDOS ransomware. So you see a lot of that happening. And so I see the connection between security companies or security teams keeping things closed and more siloed, and the fact that that allows security software companies to not evolve.

They don't have to evolve if the security teams aren't evolving. So do you see that security companies will change as we start to make this change and as people are starting to cross over that line, do you think that that will happen? I mean, I feel like it kind of — it has to happen, right, because otherwise, the security vendors are not keeping up with what people are doing, like the Mafia and different companies stealing PII data or whatever data, that thing.

Like it has to change. So do you feel like this is the time that it's going to happen, or do you have any like ideas of how you think that will happen?

Gareth Rushgrove: I think it is happening already. I think in hindsight, it's easy to look at what happened with operations tools as happening overnight. In reality, it played out over 10, 15 years from the point where certain organizations had taken a different approach and said, "We're not going to use the same tools everyone else is using.

"We want different tools. They don't exist, so we're going to create them for ourselves." And if you talk to very large but not always very large companies, like technology centric companies, not necessarily technology companies as well, where they have real security problems and they have an awareness of the security problems.

And they're often building that tool, really good tools, themselves. And I think you don't just have to do it by piecing very complete solutions together. You can often start seeing it coming out with organizations that piece things together from small components and they invest their money then in integration and gluing things together. And that approach will work for all organizations.

But I think what it will demonstrate is that there are other approaches to securing data, securing infrastructure. A good example of that, I guess, is the — that much more historical model of securing the perimeter, investing everything in a single device or single devices that secure your perimeter and ultimately, say, not having a big wall.

And over time that space become clear that like actually, yes, progress reports are important, but unless like you're actually defending inside your network and you've probably got more problems than you realize. And those problems are completely bypassing your often quite expensive perimeter security, you've put all your eggs in one basket.

And I think like that doesn't have to be the approach, but when you're in a world where you can only buy off the shelf, you often then don't have Internet tradeoffs like that, because you can maybe only invest that. It's that much money and you have to pick one or the other. And when you're doing a lot more bits yourself, you can be a lot more nuanced. But you need the skills and the ability and the trust and the environment to do that.

And I think like with operations, we've shown that organizations can move towards like doing some more of those things. But you will find extreme organizations that do it in an extreme way and pay costs for that and pay an ongoing cost for that, but probably move first. But a lot of those things can then be more industrialized and then later productized and commercialized and play out everywhere else. I think we're already seeing it.

It's just that it's unevenly distributed. And in many cases, you can't buy it today, in the same way as you couldn't buy really good infrastructure automation tools maybe five, six, seven, eight years ago. It's not that they didn't exist. It's how they were preserved by the early adopters. And now, obviously, there's a lot more like modern infrastructure tooling. So I think we're seeing it.

It's just that change takes a lot longer than people think.

Beth Cornils: Yes, it does. Yeah, no that's true. And it's one of those things that I've thought about that what drives some of that change. And it seems like we hear a lot more about breaches. We hear a lot more about security issues. We had very high visibility with the FBI and Apple, that I wonder if part of that will start to help us evolve more quickly within like, oh, it seems like we have to kind of like change more quickly to keep up with the Joneses, so to speak.

In terms of some of our security software and what we're doing in our approach and how do we take a look at it, and do we need to reframe how we're handling security, because it seems like, for a long time, there was a very stagnant way of looking at security - do this, you have these rules and regs that you follow, you follow this procedure, you check off this box.

And those things are good. Those are not bad and we should not throw them out. But there's also the ability to be — and I hate to use the word — but a little more agile in how we address some of our security stuff and how we can take a look, and can we learn from that — if we're less closed off, we have the ability to be more open and make everything better for everyone.

And then it's more of a united front against some of the attackers as opposed to this one company over here found this amazing way to identify anomalies, and they're working on making it open source. And so maybe we can get that in the next year-type thing. Is that what you're talking about? Or did I kind of take us off track there a little bit?

Gareth Rushgrove: No, one example. I think security's one of those areas, and the security experience tends to be quite good as a somewhat close community of sharing within itself, because there's generally a sense that like you don't — you don't win at security by someone else being like compromised, if you like.

There's like — it's not the same. The — as a — like working for an organization that isn't a software vendor, that's not selling some things. Someone else not being compromised is good for you as well. I think that the issue there is that real change tends to come about from this, I guess, existential crisis.

So what's been happening, and Apple being a good example, Apple really saw reinvesting and investing in infrastructure, and using that and pushing that out as part of their marketing, even. And so it's not just technology; it's policy as well. But they had a real existential crisis there. And everyone else can look at that. But most organizations will think, "Well, glad it wasn't me," and move on.

I fear that most organizations change because they have an existential crisis, not because someone else did. And there is that risk that as an entire group, we do know how to do things better, [but] actually like we did individual organizations, it's difficult to find how you apply the pressure within your own organization to do things better, to do things differently.

I think it's going to be one-faceted. And security's becoming more visible. It's becoming more obviously problematic. I think one of the things that's happening there is going to be insurance and how the insurance industry, interestingly, like becomes involved in applying pressure for people to be more secure.

And I think that's — that's going to be one of the things that increasingly plays out. I think in lots of cases already, security is being to a degree driven by compliance made in modern security. And there's a link there that being compliant with something like PCIs, like they are trying to drive standards across an industry and ensure things are more secure.

But that large brush strokes and they're across everyone. And so there's a baseline there. And rather than it being much more contextual, much more about your organization and your own threats, and something that maybe less specific — that is more specific to your organization, is more specific to you as a company. And driving security would probably be good.

And the best driver is always that existential crisis. But that tends to mean something's gone very wrong beforehand, and there's a question of can we, as an industry and as individual companies, do things without that. You hope so. But how that plays out is difficult to see in some cases. I think that the only way that really happens across lots of organizations is with sharing.

It is with people being — people finding it easier to know what works and what doesn't work, and easier to adopt new things quickly. You mentioned this. A lot of that in lots of cases is being able to change our action, be able to not always just move faster but ultimately adapt faster.

And that's something that in lots of cases, security — often security driven by compliance finds it very difficult to do, and is adopted on a like — on a regular cadence, on a week by week, on a day by day basis, rather than seeing through the eyes of a compliance regime which might be much more six month, or yearly or biyearly basis. And it's centered around those audits rather than centered around the day to day security risks, which may have actually little to do with compliance.

Beth Cornils: Right. No, absolutely. And it seems like if I kind of tie loosely with the DevOps world and just getting the security and getting security people out of their silo, having other people within the organization aware of the security risk that they could be helping to prevent.

Or why when a security person is coming to them with their hair on fire with a the bottle of gin in their hand going, "Oh my God, the world is on fire and I need this gin because it's going to make me feel better." They understand the reason why they're doing that. And they take it seriously instead of like, "Oh God, here's someone who's going to tell me no, or who's just going to like take me off track for everything." So it's that visibility and bringing people together.

Gareth Rushgrove: Well, visibility and context.

But is it’s an example that, I guess, that some of the practices that are — actually have really good support in the data about being good, and are being adopted by larger and larger organizations. And so, for example, around continuous deployment, continuous delivery, releasing in small batches and much, much more regularly, with literally the opposite, if you like, of the best practice 10 years ago.

And what's interesting there is that we have data to back that up. So to start with, people were doing it and everyone was saying they're crazy. And to a degree, there're still people saying that today. But it's much easier to have a grownup conversation because we have really good data from across industries, and, for example, in the DevOps report, that shows this works. And I think having the — that only happens when there's — like there's an ability to share, there's an ability to question existing best practices.

And I think the same thing will happen in security. And it's that questioning that has happened in some organizations, and they have new ways of working and they think are better. The main thing is, if they are better, can we prove it? And if we can prove it, can we get the word out?

And again the operations, they're like — the parallel, to me, is someone who's being on lots of different sides of this, really strong and the answer feels like yes. I don't know what the answers are. I don't know what all the new practices are or what the new tools might look like. But they definitely exist. There are definitely better ways of doing like what we're doing today.

Beth Cornils: Right. Yeah. Well, in a way, we kind of have to, right? We can't say where we are.

Okay. So one of the — well, I have two final questions for you. One is open-ended. You tell me if I've missed anything. But what are some of the interesting changes that you've seen recently in the security area, or maybe adjacent to security, that you think is like, wow, this is amazing. This could be — this can really change how the world thinks about things, or not the world but this part of the world.

Gareth Rushgrove: So I think there's, first off, several approaches. And as someone who's been a guest and both a software engineer, operator and has interest in security, and I'm drawn to a number of the tools that are around applying software, applying code to security problems.

I think that's to an extent a product of like my background and my interests. But it was the same thing with infrastructure as code and with Puppet as well. And that worked out. Like we demonstrated that there's value in taking things that were maybe previously in people's heads, differently in different people's heads, written down in human language, maybe across language barriers, maybe not, but just in a lofty way.

This , oh, how do I make this change? Oh, there's a book somewhere on the Wiki. When was it last updated? It was last updated when someone started and did that task because the previous one was out of date. That's like massive information, turning that into well-structured code that we can reuse as being a theme that DevOps may have been.

And there's a number of tools popping in security that — they're all starting to do the same things. They maybe lack the sharing platform at the moment, but the low-level tools are there. So this was actually one of the things that was mentioned around testing in the recent cyber reliability engineering book from Google.

And it's interesting to see that it was something that they'd been doing in parallel to everyone else and doing as well. So tools like Service Pack can be used here like writing unit tests based around security policy for your code. They inspect framework as well from the folks at T-Mobile Labs. There's also BDD security is an interesting one, applying human language code using the cucumber gherkin syntax.

But the tool itself comes with lots of built in security. Has scanning features, if you like. So it's not just that. It moves you away from the idea that an expert runs the scan on an ad hoc basis and interprets the tea leaves, interprets this all this data that's difficult to analyze and gives you a result at the other end. Rather than having — like having code actually scan — and interpret — and you making specific assertions against that.

So you can be much more fine-grained. Also you can push that — those fine-grained checks out into individual application teams or individual service teams or infrastructure teams. But you can all see all of the new answers. So you still need that expertise to know what those things should be in the first place.

But you can much more easily share the information about what's working and what's not, what needs to be fixed, and what's bad, like what's been broken for a while. So I think that's a really interesting area. — you can nearly run with the idea. It's not like — there's not like this equivalent to infrastructure as code. What would we mean by security as code? It's all a good — like a good hypothetical question. Like do some of the same approaches work?

Do some of the same lessons that we've learned through applying software engineering practices to infrastructure? Do they work in security? And I don't think we — I think the answer is probably yes in some way. And we got like anecdotes about that working in some way. But I think that will be a good thing to run with over the next few years.

Beth Cornils: Yeah. That's interesting. I like that. Okay. So then last question: Is there anything that I missed that maybe you wanted to talk about in this? Because sometimes I miss things, it turns out.

Gareth Rushgrove: I don't know. I think that's a pretty good run through a series of topics.

Beth Cornils: I thought you kind of rocked it.

Gareth Rushgrove: I think it's just — it's interesting observing the , I guess, the tectonic plates of all these different IT disciplines moving. And I think if you talked to some people at some organizations, they really don't see themselves as separate. But actually, when — if you're forced to think and look at how you work and how you've always worked and how your department is structured, and how people are paid and how people are incentivized and who reports to whom.

And really take a step back, you can start — you start seeing that actually, yeah, we often have structured IT organizations to have these characteristics of siloing themselves off. And I think that maybe the only thing we haven't touched on is when you realize that from a security point of view, how do you change your organization structurally to best take advantage of maybe this serendipity like of security people talking to developers, talking to operators.

Beth Cornils: Right.

Gareth Rushgrove: And I think that's something else that we've seen on the operations side is that lots of organizations are changing how they structure themselves, that they're undergoing , in some cases, incredibly large reorganizations to take advantage of operating developers working more closely together and ultimately pushing towards being able to move faster.

And I think that's a pressure on security organizations, because if the rest of the organization changes shape and does so in pursuit of moving faster, if a security organization could just keep up before, it's definitely not going to be able to do it in the future if he retains the same shape.

And in lots of those cases, they're probably also not going to — if it's — if they're — if applications are being deployed 10 times faster, let's say, after early, and the security organization's probably not going to get 10 times as many people or 10 times the budget, or 10 times anything else. So it's going to have to change in order to adapt to that new normal.

And I think we'll hear a lot more about — over the next few years how security organizations are reorganizing - what sorts of skills are required, what people are required in them to keep up with that rate of change.

Beth Cornils: Yeah. No, that makes sense, because, well, that's the nature of software. That happens in every group, and I can see that happening in security especially. All right. Excellent. Thank you so much, Gareth, for your time. I really appreciate it.

This was a super fun kickoff to our security podcast series that we're going to be doing. So I hope everyone looks forward to the next one that we have coming up.

Thanks so much, Gareth. I appreciate your time.

Gareth Rushgrove: No. No problems. Always good to chat.