Pulling the Strings

Vox Pupuli has long been the backbone of the open source Puppet community. But as often happens when a group is so capable and ubiquitous, sometimes the work done by individuals in the group goes unsung. In this episode, Eric and Hunter tell the story of their own involvement in the group and share some fun historical stories.

Today's musical intro was recorded by Eric Putnam.

*Ben Ford is a developer advocate at Puppet.*

One of Puppet’s big strengths is declarative automation – you model the desired end state of your infrastructure and Puppet does the hard work of enforcing it. This concept is quite powerful and has become the industry standard for configuration management. But sometimes your workflow doesn't quite fit into that pattern. Sometimes you really do need to deploy point-in-time changes, or mix imperative and declarative approaches together in a single workflow.

With Bolt, we've learned from the lessons of Puppet and used those to design the next generation of infrastructure automation. Listen as Eric Sorenson and Yasmin Rajabi explain the tools of the future.

The music in this episode is part of a live performance Eric Sorenson, Puppet Director of Product, prepared for Volt Divers. The full track is available here

*Ben Ford is a developer advocate at Puppet.*

One of Puppet’s big strengths is declarative automation – you model the desired end state of your infrastructure and Puppet does the hard work of enforcing it. This concept is quite powerful and has become the industry standard for configuration management. But sometimes your workflow doesn't quite fit into that pattern. Sometimes you really do need to deploy point-in-time changes, or mix imperative and declarative approaches together in a single workflow.

With Bolt, we've learned from the lessons of Puppet and used those to design the next generation of infrastructure automation. Listen as Eric Sorenson and Yasmin Rajabi explain the tools of the future.

The music in this episode is part of a live performance Eric Sorenson, Puppet Director of Product, prepared for Volt Divers. The full track is available here.

Ben Ford is a developer advocate at Puppet.

Ben: [00:00:06] Hi, my name is Ben. I am our developer advocate here on Puppet's community team. One of my jobs is to sort of connect the dots between people using our tools and the people building our tools. And part of that is bringing this line of technical podcasts to you. Here with me today is Eric who has been our product manager and sort of look like idea guy around the Puppet platform for many, many years. And Yasmin who is our product manager for Bolt. They're going to be talking about maybe you could call it the next generation of what configuration management might be.

Eric: [00:00:47] As you mentioned manager Eric Sorenson and for most of the time I've been here I've been working on Puppet and we introduced Bolt back in 2017 and the project has really taken off since its initial introduction.

Yasmin: [00:01:04] I'm also super excited to be here like Ben mentioned. My name's Yasmin and I'm the product manager and I work on Bolt. I've been at Puppet almost two years now so it's been a lot of fun and it's super cool to be here with Eric because when I told someone that I got a job at Puppet their first response was "Oh, you get to work with Eric Sorenson?" And so I do actually get to work with Eric Sorenson so that's cool.

Eric: [00:01:26] That's terrifying actually, but it really gratifying as well. That's awesome. Bolt's been a really interesting project. We started off with an idea that we should have something that would work alongside of Puppet that would complement the things that Puppet does really well by filling in some of the things that it doesn't do super well or that people have to hack around in order to make it work. Largely that's around tasks. It's really hard to make Puppet the tool do something once and never do it again. It's really hard to make Puppet change the state of say a service twice in one catalog run. It's just kind of like fighting against the way the tool was designed which you know as I think most people know is intended to enforce one particular state and to keep that state in compliance with what your manifest says all the time.

Yasmin: [00:02:22] The way I kind of look at it is Puppet's really good at telling systems to be a certain way. What our community asks for are tools that allow them to do things a certain way. So puppet's great at making your systems be a certain way and Bolt's great at making your systems do a certain thing.

Eric: [00:02:40] Yeah, I think that's a great way to look at it. When I thought back over the history of configuration management - and I've been doing this kind of a terrifyingly long time - I gave a talk at Configuration Management Camp in Belgium in early February and called it fourth generation configuration management because there really have been a number of iterations and evolutions of how we think about and perform configuration management in our systems. The fourth generation is... maybe it's a little arbitrary. I had to come up with something that fit the title of the talk and I don't know whether it's completely accurate but the way I thought about it was first we had shell scripts before we had any kind of configuration management people were doing config management they just didn't call it that. You wrote shell scripts, and if you got a new machine which was not a very frequent occurrence you might get a new like a Sun Server or something like that once a year and so you didn't really have to spend a lot of time building repeatable configuration for the systems because it wasn't just something that happened all the time. But when you did, you wanted to pull something out of your home directory or someone else's home directory and run a shell script to do kind of baseline configuration on it, and that sort of set the stage for the first generation of actual tools that were purpose built for doing configuration management. I think a lot of people are familiar with CFEngine and its author Mark Burgess who did a lot of really important early work in both the theory of systems administration and the theory of configuration management as well as writing and developing CFEngine, which was a pretty groundbreaking tool at the time. It did things that no other tool in the open source ecosystem could actually do. And it was pretty complicated to use. It was tough, but once you got your CFEngine stuff up and running it was like "Oh my gosh, I have superpowers now. I can do things that used to take me weeks and weeks and I can get them done in just a few hours." And that was kind of the first iteration of it. Have either of you poked at CFEngine, have you ever used it?

Ben: [00:04:57] Only in anger.

Yasmin: [00:04:59] I think that was pre-my time in working with computers. Recently [Michael] Stahnke sent me an article about his thoughts on config management from like 2011 or something and it's just really cool to like get the history of that because I was way just before my time and it reminds me of like when we were starting on Bolt and tasks and I was doing product manager-y stuff, whatever reading docs, and catching up. I read at least three or four docs from 2019, 2012, like early early years that read like they were true. The same problem was there, it's just a more evolved way of solving it, which I just think is really cool as we look at the history that the problems are essentially the same, it's just how we solve them have evolved.

Eric: [00:05:42] Yep. Luke Kanies, who wrote Puppet originally and started the company obviously, and I were both CFEngine community people originally - in fact that's where I met him back in 2002 or 2003 or something like that. Funny story: When Puppet was very first getting started one of the earliest customers of Luke's consulting business was Stanford University in Palo Alto. They were really early adopters of Puppet and they in fact just paid for him as a you know Reductive Labs consultant, which was basically just him and was his company but it was just him to come out to Stanford for a couple of weeks and work on Puppet for them and to implement some of the features that they needed, some of which are still in the code base which also is sort of terrifying. But when he came out I knew from talking to him on IRC and from CFEngine and stuff that he was into bikes, so I loaned him my bike so he could get around the Stanford campus and so he rode around on a fixed gear bike at Stanford in 2006 and built some of the early Puppet features there.

Ben: [00:06:53] That sounds very Luke.

Eric: [00:06:54] It was a fun time and that kind of leads into the idea that like that CFEngine being the first generation - it did some stuff fairly well, but it was ,as I mentioned, hard too hard to use and a lot of that dissatisfaction with how user unfriendly parts of CFEngine were design goals that Luke wanted to fix with Puppet. And that's where we got things like the resource abstraction layer, the fact that before things got really complicated in the Puppet language initially it was more like a config file like the syntax and the way a Puppet manifest looks was modeled after the way a Nagios configuration file would look, rather than having like a lot of complicated logic that you would have to follow through and try to understand. Another related project that I think also sort of went into that second generation is Chef. Similar to how Puppet came out of the CFEngine community, Chef actually came out of the Puppet community. Adam Jacob, who was the initial author of Chef and their CTO up until very recently, was a Puppet consultant and had a consulting company where they just went around and implemented Puppet in a lot of places and similar to Luke's unhappiness with CFEngine, it was Adam's unhappiness with Puppet that led him to create Chef in the first place. And they're really pretty similar tools, but there are a couple of points of I guess philosophical difference that encapsulate like why people would pick one or the other or why they would tend towards using Chef instead of Puppet. One of them is that Chef cookbooks are written in Ruby effectively - there is a DSL over them, but you can just start writing Ruby in the center of the cookbook, which is really powerful and it's great if you are a Ruby developer but it does sort of hurt that idea of the simple configuration file style syntax - the idea that instead of having to run the Puppet manifest in order to see what happened you can reason about it you can run it in noop mode and you can check whether or not it's going to do what you expect before you actually make any changes on the system. That's really really hard to do in Chef.

Eric: [00:09:14] Moving on to the next kind of iteration I would say that Ansible and Salt are kind of the epitome of the third generation. These are both tools that came out kind of in the early 2000s - 2014 for Ansible and about 2012 for Salt and they are pretty orchestration driven. Yeah most of the configuration that you do with them is written in yaml and they're very much like you were saying earlier go out and get stuff done make changes your system. There is idempotence in Ansible playbooks if you if you choose to make it so, but it doesn't enforce it in the same way that Puppet manifests too. And the tools are great for a certain set of problems and in fact it was you know we took a lot of inspiration I would say and ideas about how people wanted to solve problems from the success and the rise of Ansible which kind of came onto the scene and really got a lot of mindshare and got a lot of users in a very short amount of time in around 2015 which sort of leads us to this fourth generation now, which I think is kind of like the blending of all of those approaches and bringing all of those things together in a consistent way and allowing you to do both model based and state driven config management and imperative tasks and ad hoc commands and you know to just do whatever you need to do in order to get the work done that you're trying to accomplish with the tool. And that's kind of where we're at with Bolt.

Ben: [00:10:46] One thing that I find kind of interesting about that - I think I'd probably also add another layer there is like the zero-th level was sort of like cut and pasting from a wiki or something like that. And I admit that that's kind of where I went back to after running my head into CFEngine a couple of times, but every iteration that we come through it seems like the tools are simplifying a little bit getting a little bit simpler a little bit more streamlined and making it easier to get to your desired end result.

Yasmin: [00:11:14] I think that's one of the main goals that we like - our charter, you could say, for Bolt is if we're building something does it make someone's life simpler? Is it easier for them to do the thing they're trying to do? Because at the end of the day the software we build is meant to make your life easier. It's meant to make the thing you're doing, not only just faster but like more enjoyable so that you go to work and you're like "Oh okay, I have to do this task and I'm going to use some tools to do that task." Hopefully that is an enjoyable experience vs. like "Oh no I have to use that tool to get my job done" and kind of that was one of the main things with Bolt that we continue to try and embody is is it making someone's life simpler. And it's almost like expected from people nowadays of it used to take maybe an hour to get something done. Now it's minutes then it'll be seconds and it's kind of like there's a level of patience people have and simplicity they expect that we kind of need to operate in, and that's kind of one of the main goals with Bolt.

Ben: [00:12:13] But at the same time it also kind of gives you those steps towards the more powerful like model language that we still employ with Puppet.

Eric: [00:12:21] Yeah that's that's been a really awesome feature to see develop. So initially, and just to walk very quickly through the initial sort of phases of Bolt, we started actually writing code in June of 2017 if I recall correctly. And the name initially came from one of the developers Josh Cooper, who's also been around Puppet for a long time and has been one of the core code developers, he was working on this little project and we needed to call it something. We had a few different goofy code names, none of which were ever destined for the outside world. But he said, "Well, can we can we call it Bolt?" And there was no reason not to. So that's what we picked and it stuck. But initially it was a pretty simple tool and we wanted to do this progression of being able to take stuff that you already have, whether that's just a shell script like you're saying something you command that you have in your batch history or something that you ran one time and it worked for you, and be able to use the tool in order to get that done. Like Yasmin was saying quicker and simpler and maybe more enjoyably than having to run it by hand, then progressed from that into what we call a task and a task with a capital T is a shell command or a program that you've written that expects certain input and produces output in a predictable way. And ideally the task is accompanied with a little bit of metadata inside a JSON file that lives alongside of the task and is named the same thing as the task with the JSON extension that tells Bolt what that task does and lets you make human readable descriptions of it. It uses the Puppet type system which we introduced in Puppet 4 in order to describe what the parameters of the task are using a pretty rich expressive set of types. So you can say like it's an Enum, so one of a few defined values that way the tool can do input validation before we ever even try to run the command. We can say well you supplied a value that's not one of the possible acceptable values of this Enum. So we're just going to bomb out early and tell you that it needs to be one of these predefined values. We can make sure that if you need an array that you get an array and those kinds of things. So that type system is really really helpful. And because it's reused from Puppet a lot of people were already used to it from writing modules and it was great to be able to leverage that capability that we already had in the different part of that ecosystem.

[00:14:53] Then the next step once you have tasks is to be able to connect those tasks together in what's called a task plan that really is like a workflow. It's written in the Puppet language. It lets you use those parts of the language which make sense for a plan - like for instance you can't directly define a resource, you can't do like exported resources or anything like that, but you can use all the logic and you can run functions and string things together in a way that allows you to catch errors, to respond intelligently to failures of parts of one of the steps, really to link all of the work that you've built or downloaded from the Forge together in a consistent workflow. And plans are kind of like the end stage I think for all of that for Bolt and for tasks overall because there's so much stuff that you can do inside of them and one of the most recent additions to the plan language and to the stuff that you can do with them is actually apply Puppet code, apply resources onto individual nodes by including an apply statement inside of your plan.

Yasmin: [00:16:01] I think this is one of my favorite features of Bolt. This came along because we were having a user research interview with one of our users Nick Maludy and he's like "I love Bolt. I can get a lot of things done with it and I like Puppet, but I use another tool to kind of do a lot of my more ad hoc automation." And he's like "I would love to take the investment I already have with Puppet and use that and apply it as more of a one off." And that's kind of where the apply functionality started. We knew that there's so much value in the Puppet language, the content that we have, the community that we've built, and we want to be able to reuse that with Bolt. Bolt is meant to help you get started with Puppet faster and expand the kind of capabilities you have today that you're using Puppet for with Bolt so you can use it as a standalone as you're getting started or you can use it to expand your current kind of automation workflows. And so because of that we wanted to make sure that we're reusing our existing investments and that's where Bolt apply came. As soon as we shipped it, it became the second most used function within a plan which was super cool and that kind of drove us to build a command for it so you can actually just do Bolt apply run give your Puppet code and then apply it to the unknown the catalog gets compiled locally and then connects out to that node and collects the facts and does all the things and you get some Puppet code applied, which is really cool. And we've seen that popping up in different demos where people are literally just copy pasting from the Forge and they've built the thing they want to do. They don't have to write code for it and they can reuse investment that they already have.

Ben: [00:17:39] So do you mean that people are using modules from the Forge by just copy and pasting the include name of the module whatever and it just works?

Yasmin: [00:17:46] Yep. And it literally just works. We're working with the Forge team as well to kind of have some highlights on there so you can see how you use this content with Bolt. But it literally just works.

Eric: [00:17:57] It's pretty cool because you can take those apply blocks and sprinkle them throughout your plans and that way you can intermingle, say a build step with a Puppet code step configures the application server with another imperative step that say restarts the server once it's done, and each of those steps can take a set of targets that Bolt will know to go out and execute the stuff on a bunch of those targets and then move on to the next step. So it's really powerful and allows you to string together what used to be maybe a bunch of unconnected steps into one cohesive place that is shareable and reusable plans just like tasks themselves live inside of a module so they can be uploaded to the Forge, they can be tested. The PDK [Puppet Development Kit] knows how to create tasks. I'm not sure it has plan support yet, but it's probably close at hand and the whole ecosystem has kind of added on this capability of being able to do task-based work alongside of state-based work and that's why it kind of leads back to that fourth generation thing. You want to be able to have it both ways, to have your cake and eat it too.

Ben: [00:19:09] And it sounds like you're talking about taking the state definition away from just one single node and saying you can define state across an infrastructure of a handful of nodes so you can look at like an application where part of it runs on one node and part of it runs on another node and kind of orchestrate application of state across those nodes.

Eric: [00:19:27] Yeah, exactly and each one of those can be again some mixture of existing Puppet code, it can be tasks that you've got off the Forge or you've written yourself, or it can be individual commands - just whatever whatever is appropriate for the job.

Yasmin: [00:19:39] One of the things that's kind of cool about the combination of the Puppet code with Bolt is that you can use Bolt for a very long time and not realize you're actually using Puppet. So without typing the word Puppet you're actually using you're using the RAL, you're using the content, you may have even used Bolt to install the agents, configure all the settings you want, and you've gotten so far in actually being able to accomplish your automation goals and setting up kind of your tool and what the tool itself can do. With Bolt it lets you get really far into actually accomplishing your automation goal and under the hood you have gotten so much farther with Puppet, where if you want to make that switch too I'm doing something I'm applying it as a one-off and now I want to actually enforce it, it becomes really easy because you've gotten used to the language, the content, the RAL, and you may even used the tool to install all the bits and configure them as well - without having to actually think about using Puppet.

Eric: [00:20:37] Yeah there's an interesting side effect of adding all this stuff into Bolt, which is that for a long time Puppet supported this master list style operation where you could have Puppet code distributed out to your end nodes and they would run Puppet apply locally and not have a server setup. Bolt with apply mode is sort of like a supercharged master list setup. You don't need to have a PuppetDB necessarily, you don't need a Puppet server, but you get to make use of that content. Bolt knows about Hiera and it can pull Hiera data. So if you're used to using yaml configuration for configuring your modules, Bolt can make use of that and will collect facts and be able to interpolate variables and Hiera correctly for for each of the nodes that it's compiling a little apply block for and like you said you can get really far without really having to do a bunch of setting up a bunch of infrastructure in order to get the work done. It can all kind of go from your laptop until you're ready to move it onto an actual server.

Ben: [00:21:41] And there's no need to setup anything on the target nodes. There's no need to like rsync out code or install any agents or anything, all you need is SSH. And then Bolt will handle all the setting up of the tool, right.

Eric: [00:21:55] I have one quibble with what you just said. You do need to have the Puppet agent installed on the end nodes in order for the apply to work. You just have to have the package installed. You don't need to have an agent running necessarily, but the way it works is that it will ship over a compiled catalog and feed that into the agent. It will feed that into Puppet as its input, rather than have the agent request a catalog from the master, but you do need to have the Puppet software there again, but it doesn't need to have SSL certificates, it doesn't need to be running permanently on the nodes, those kinds of things. So the setup cost is much less.

Ben: [00:22:28] And Bolt'll automate that for you with the apply prep.

Yasmin: [00:22:31] Yep. Apply prep will gather all the facts it needs, make sure everything that needs to be installed is there, and then allow you to apply that compiled Puppet code.

Eric: [00:22:40] People have really taken Bolt to heart and it's really kind of heartwarming I would say to see the amount of things that people have been able to accomplish with it, without even really you know we haven't we've just sort of just started ramping up on talking about it a lot. Getting a lot of talks at DevOps Days and other kinds of events that are pretty Bolt-focused but people have really latched onto the tool and done some really amazing things with it. I mean, I at least didn't expect it to get as much popularity and get as awesome as it's gotten.

Yasmin: [00:23:13] Yeah it's definitely so exciting to see all the usage. I track the Slack channel and every day I see new faces in there and it's just like super heartwarming and it's just exciting to see that people have it resonates, you know. When we're building software we want to make sure we're building software that actually solves problems, not that we think could be cool. And so seeing people either discuss how to use it or share how they're using it, it really it's validating kind of to know that you're solving a problem and you're not kind of making assumptions on the things you're building. One of the things we've seen with traditional Puppet users getting started with Bolt is actually using the Bolt in the Bolt apply mode to test out Puppet code. So rather than kind of commit changes, test it out, deploy them, see if it works. They can just take a chunk of Puppet code and keep iterating on it right from their laptop. And once they feel comfortable with it, it works on whatever the targets they're addressing, it's just a copy paste into your control repo and deploying it out to your traditional Puppet deployment. So that's kind of one of the things we've seen with our core Puppet users.

Yasmin: [00:24:27] We also have a couple cool stories. The one I really like is there's a team out there I won't name them, but they had a vendor come in and apply a change to over a thousand of their nodes and it was approved or whatever, the change went out and they realized like it just took everything down and we need to make a fix real quick and with Bolt they were able to do this within minutes and deploy out a fix get their systems back into the state they need. And they had mentioned that if they didn't have Bolt this would take them days to go through and check which servers have that issue, which ones need to be updated, and actually go make the update. Bolt was really crucial in that moment to help them kind of solve that.

Ben: [00:25:11] I have a story that I can tell about an interaction that I had on IRC earlier today. A user had certificates that were ready to expire and he was kind of worried about like how he's going to navigate that - go from old certificates, new certificates re-sign all the agents certificates and what happens if he makes a mistake along the way, and now the agents are no longer talking like authorized to talk to the master and everything. I suggested that he set up autosign and then use Bolt to just delete the certificate and rerun Puppet and that gives him like a kind of a second way in, so that even if he made a mistake he still could control the nodes with Bolt to re-request certificates. And he says that sounds like a great idea. And no kidding, like eight minutes later he came back and said that totally worked. And that went from like zero like he had just heard about Bolt.

Yasmin: [00:26:06] That's super awesome. It's amazing just watching people install and get started with the software, how simple it is. I mean I don't believe it usually, but when we actually have it in the workshops that we're doing people install Bolt and actually try and solve a problem with it - seeing that, just seeing how easy it is - it's still surprising. I know it shouldn't be surprising, but it's still surprising how easy it is to get started and actually accomplish something before going through all the steps of setup and everything.

Eric: [00:26:36] Pretty cool. Let's talk a little bit about what's what's coming up, what's next for Bolt, where it's headed. There's some there's some work in flight. I know around continuing to bridge that gap between agent-based management and state management. And can you talk a little bit about the network and network device features?

Yasmin: [00:26:59] Yeah. One of the most common things people ask about are supporting network devices because we've expanded the scope of what you can automate through more agentless technologies. And because of that a commonly requested thing is "I want to automate some network devices" and we have a team at Puppet that's been focused on this area for a while now and they've been building really good content, they've been building tools to kind of help people push forward in this area. And what we've been working on with them is combining efforts and allowing Bolt to be that interface into any type of automation. So reusing again our existing investments in the network automation space, the content that we have, making sure its agentless first, using a more remote transports to target those network devices using Bolt. So whether you want to apply changes on a device right on the device, you want to apply those changes, have them run locally and remotely applied to that device, or maybe you want to send that to a proxy node where the proxy node can actually target those devices and run those changes. And so we've been doing work with our network automation team internally at Puppet to help drive that and let Bolt be the tool and help empower automation workflows using Bolt.

Eric: [00:28:21] That way you can really bring those network devices into that same workflow. Maybe you have a task plan that involves interacting with a load balancer or a switch or something like that, you can bring that into the same place where you would do a code deployment or apply Puppet code all inside the same plan.

Yasmin: [00:28:39] Right. Exactly. And the goal is to be seamless so you as a user don't have to think about what type of target that is. But rest assured that bolt will connect out using the right credentials, the right transports, and running the right code against them.

Eric: [00:28:53] Are there particular device types or manufacturers that we're targeting first?

Yasmin: [00:28:59] We're first targeting Cisco and Palo Alto firewalls, so you'll see that shortly. All the code's open anyway so you probably are seeing the changes come out either in this week or next week's release of Bolt should have support for the new remote transport. We try and ship Bolt weekly so that people can get those updates quickly. Sometimes we can't for whatever reason, but we try and stick to a weekly schedule.

Eric: [00:29:25] It seems to like the contributor cycle for Bolt has really gotten a lot of improvements too so that if people want to get started not just using it but also working on the tool and maybe contributing improvements or even filing bugs and helping out in that regard that the team's been in my experience super responsive with everything from GitHub issues to pull requests, right.

Yasmin: [00:29:53] Yeah, that's something super important to the team. If you're ever on the Bolt Slack channel, you'll see all of the engineers are out there answering questions, they do office hours, and merging pull requests or issues on GitHub is super - like it's something we value and we've been trying to have more of our discussions on Slack so that people can provide input and even one of the reasons why we ship weekly is so that if somebody merges a change they can actually see their change quickly and that's - I mean it's a lot more gratifying to make a fix and see that than wait maybe six months until you see that fix in the software. We definitely appreciate any pull requests or issues and if there is a way that we can do this better, please tell us on Slack. One of the things that surprised me a lot is we kind of ran a fun Twitter campaign. We wanted to hand out some stickers and honestly I had told someone "Oh, it's fine. We'll get like a couple responses. I'll write the envelopes. It's not a big deal." Turns out we got a lot of responses, but that's a good problem to have. And just what was so cool is that people were publicly sharing how they're using the software and there are themes, but everyone had a pretty unique way in how they were using Bolt to solve their problems whether it was for their job or for home automation - that was cool. And then it gave us ideas internally that we started thinking like "Oh, I want to automate a Nest with Bolt" or like cool fun after work things and what I really liked about that was one of our engineers was like I like Bolt because it's fun, not just because I actually have to solve a problem, but because I have fun using the tool and that that was just really cool to hear and see like all over Twitter.

Eric: [00:31:34] So if people want to participate in that and get a sticker pack what should they do.

Yasmin: [00:31:37] Just tweet how you're using bolt and add hashtag #PuppetBolt and we'll send you some stickers.

Ben: [00:31:44] Awesome. I can definitely attest to the fun part of that. I actually just kind of for fun threw Bolt into a presentation that I did that kind of like automated a couple of different parts of demonstrations on some machines and it was really fun to do.

Ben: [00:32:00] So we've talked about Slack a couple of times here. If you don't have an account you would want to go to slack.puppet.com and you can request an account there and then just stop by. There's people in both the #puppet channel and the #bolt channel that are always happy to help. Always happy to talk about problems, usually have some kind of answers or some advice. And every Tuesday right now we're doing office hours on Bolt and somebody will be there to help you get started if you need it. Later on this summer, if you happen to be in Europe we are running our annual Contributor Summit event - that's June 4th and 5th in Budapest and we're going to talk about a lot of Bolt things this year, so I would love it if I saw you there. We will have announcements coming out real soon about that. In the meantime I would ask for you to let us know about things that community members are doing that you would love to see them get recognized for at Contributor Summit and you would do that at pup.pt/mvp. Just let us know what what they did and if you have contact information for them, that would be great.

Yasmin: [00:33:05] One other thing that we're doing this year and we're starting in the EMEA region but we want to expand out to the US are just Bolt workshops. They're like a half day session of starting with Bolt, using it to solve problems, learning it, and then ideally you leave the workshop ready to go tackle any automation problem that you have. So if this is interesting to you, reach out to us, let us know where you'd like to see it. If people are interested, then we can go from there.

Ben: [00:33:32] That sounds great and they should probably just ping somebody in like the #bolt channel on Slack right.

Yasmin: [00:33:36] Yeah, that would be great.

Ben: [00:33:38] Thanks for showing up and hope to see you here next time.

Eric: [00:33:41] Thanks Ben.

Yasmin: [00:33:42] Thank you.

It seems like every day is another paradigm shift anymore. SaaS, PaaS, configuration management, cloud native, immutable infrastructure, serverless, and so on. But one has to ask, what do we lose by always chasing the bleeding edge? Information hiding, API contracts, and other ways of abstracting away implementation details are all part of solid system architecture. Are there patterns from yesterday that we should continue applying towards the new ways of working today and tomorrow?

George Pandzik (@DevOpsFables) thinks so, and he's here to talk about patterns that he doesn't think should be lost to the sands of time.

*Ben Ford is a developer advocate at Puppet.*

It seems like every day is another paradigm shift anymore. SaaS, PaaS, configuration management, cloud native, immutable infrastructure, serverless, and so on. But one has to ask, what do we lose by always chasing the bleeding edge? Information hiding, API contracts, and other ways of abstracting away implementation details are all part of solid system architecture. Are there patterns from yesterday that we should continue applying towards the new ways of working today and tomorrow?

George Pandzik (@DevOpsFables) thinks so, and he's here to talk about patterns that he doesn't think should be lost to the sands of time.

Ben Ford is a developer advocate at Puppet.

Ben: [00:00:08] Hi everybody. My name is Ben Ford. I'm the developer advocate here at puppet. I'm here with George Pandzik. He's in the envious position of having formal computer science training and also many years of kind of boots on the ground sort of ops experience. And he's here to talk to us about how those two things relate to one another. Could you introduce yourself and tell us a little bit about what you will be talking about?

George: [00:00:33] Sure. My name is George Pandzik. I have a Masters of Science in Computer Science with a focus in software engineering and also over a decade's experience in everything from performance analysis to handing out flashlights instead of fixing light bulbs. I've worked on z/OS mainframes, Windows systems, Linux systems and automating the same so I've seen a lot of things. Kind of what we're going to talk about today is a topic that's near and dear to my heart is I had a lot of good mentors when I was new who had been around since the '70s back in the days when they were still using punch cards. So I learned that there's a lot that we can pick up from people who've been there and done that before us. That's a big thing. We should ask questions like what worked for them and why did that work. Because if we get to the why we can look for patterns that we can incorporate today.

Ben: [00:01:38] Right. If we look at how past people solved problems when we see problems that are similar and maybe not the specifics of the same thing that they're working on but the same kind of patterns then we can learn from how they solved those problems and we can move forward with our own solutions that you know take from that. So how how do you think that that's something that will affect the way that people write their Puppet code today?

George: [00:02:02] Well so it's really easy to naively just dump thousands and thousands of lines into a site.pp file. That's really hard to maintain because you can make one change anywhere you might have to make a change everywhere. So Dr. Parnas back in 1972 wrote a paper on data encapsulation and information hiding that I think we can learn from and incorporate into our Puppet code.

Ben: [00:02:33] So this is something that's been around and it's been evolving for years and years and years. Could you tell us a little bit about that story.

George: [00:02:40] Sure. A big thing to remember is none of this is new. I mean Parnas wrote his paper 46 years ago. He used an example of what he called the quick algorithm for a keyword in context. Just at a high level. If you want the gory details you can go read the paper from the ACM. But it would take a sentence, print out the sentence in full, and then it would rotate the first word to the end, shift everything and then print out that next sentence. Shift print shift print shift print.

Ben: [00:03:18] Pretty simple algorithm. I'm sure it was a little bit more complex without all of the libraries we have today.

George: [00:03:23] Well there were no libraries back in the day he wrote pseudocode which smacks very strongly of like PL1 or assembler so he actually shows three different implementations of that program including a naive like top down writing nested loops to do the parsing of each sentence, figuring out what is space, what's a comma.

Ben: [00:03:48] So like the the spaghetti code in site.pp sort of example.

George: [00:03:51] Exactly. And he showed why that really doesn't work. He showed kind of intermediate solution but the final solution he presented he called encapsulation where you basically if you think about this program there's really three parts of it. There's the parser, there's the rotator, and there's the printer so in the naive solution those were all just kind of jumbled together and you're doing printing as you go. So any one part of the program could wind up messing with the output stream, which could wind up breaking the whole thing. And remember this was '72, so we're talking about entire address spaces. So what he showed was encapsulating the parser, which would only have access to the input stream, the rotator that would just take a intermediate sentence, some string of bytes, figure out what a word boundary was and manipulate the string. And then you had an output printer. That was the one thing that could that could touch output.

Ben: [00:05:06] So these are things that you might call like functions today or even maybe component modules.

George: [00:05:12] Right. And that's what's really beautiful about this concept is this is what gave rise to objects and methods. The interface concept which became contracts. These are all very similar ideas which come from pretty much the same place. The good interfaces make good code. The idea is you take all the parts that could change. So in "Quick's" example if we wanted to change what a word boundary was if you go with a naive solution and just go by spaces, commas, periods are themselves words and so you could wind up with a period beginning of a sentence. Well that's not useful.

Ben: [00:06:01] Right and so you'd want to be able to to iterate and fix what these things are without breaking other parts of the code.

George: [00:06:07] Right. And the output function if you will shouldn't have to worry about that. So what is important to get with encapsulation is it allows you to just focus on the couple of things that you need to worry about. I call it ignoring things for fun and profit. It lets you be good at one thing.

Ben: [00:06:33] Or like they say you know that good fences make good neighbors.

George: [00:06:36] Yeah exactly. And this is why I like metaphors because it's the same concept but verbal. We reuse stuff. So the other thing that encapsulation does for you is you stop thinking about the implementation. How does this thing that I'm trying to get done work and you just focus on what effect do we want to have. So just as with the function, it's an interface. If I feed it a certain set of parameters I should expect certain behaviors. And it's a black box. I don't know what kind of internal variables it uses. I don't know what syscalls it makes and I don't have to care. It can change them at whim.

Ben: [00:07:24] Coincidentally that's a lot about how like Puppet itself works. You declare a resource you don't have to care about File.open, File.write and all of that. It does it for you.

George: [00:07:33] Exactly. The way it works with Puppet is what Adrien Thebo called in SysAdvent "Configuration is Legos", which kind of led into Roles and Profiles. The idea is that if I'm defining a server as a role they should never have anything but include statements period. If you're doing that you're getting too deep in the weeds. Roles include profiles. Profiles should just define the module resources that they need from a purist standpoint. If you have a file or an exec in a profile you're doing it wrong.

Ben: [00:08:22] Definitely.

George: [00:08:23] I mean that's academically pure so we'll have deadlines I understand. But the modules then do all the nitty gritty of defining users, defining files, exec-ing different install commands, dealing with yum repos and all that stuff. The beautiful thing about that model is if - let's say I'm the manager and for whatever silly reason I am dealing with roles. Managers shouldn't always have their hands in code. But for those of us who do.

Ben: [00:08:55] It happens.

George: [00:08:56] Yeah. They should be able to find out, based on what profiles are included, this is what this role server will do.

Ben: [00:09:05] It's almost self documenting in a sense.

George: [00:09:07] Right and you don't have to discern that from all of the files and exec statements and all that stuff that gets hidden by the profiles and by the modules. It puts it right there in front of you. It's as you said, self documenting.

Ben: [00:09:27] Right, so that kind of gets people away from you know bike shedding about the implementation details.

George: [00:09:32] Exactly. And that really lets you free up your mind and your concentration to do the one thing you're supposed to do well. Like a good Unix programmer. So since you can not have to worry about the implementation you can focus on the effect of what you're trying to do. And focus on the desired outcome that lets you actually write tests and mocks. Because if I don't care what the result of your function is, I can pretend it can be whatever I want to see. This lets you write good tests. It requires that you put trust into other parts of the program. If you don't trust, you might as well write it yourself. And We're all people, we all need to trust each other, whether we're managers or the professionals.

Ben: [00:10:20] Have you read Martin Fowler?

George: [00:10:22] Yes, I will say I've been influenced quite heavily by some of his refactoring books. I highly recommend them in whatever language you want to go look them up. He gives examples of how to, for instance, extract method which is it takes a common chunk of code, you wrap some tests around it and then you actually pull it out and put it into a separate definition. It's a really good practice for simplifying and refactoring your code.

Ben: [00:10:52] What do you think you would say is kind of like the moral of the story here.

George: [00:10:58] So, I guess my DevOps fables moral for the day is don't ask questions you're not prepared to have answered and trust those who are doing work that you don't want to do yourself. We have all these complex systems that when you build up all this trust, none of us have time to re-implement all of standard C every time we read a program.

Ben: [00:11:23] Or even just stdlib.

George: [00:11:24] Right. Or rewrite our operating system to schedule processors. We have to trust that those things are done.

Ben: [00:11:34] For sure. You'd totally want to group everything together, group the modules you write into the component modules and write profiles of group those together. Nobody wants to write spaghetti code - that's nothing to be proud of. So what would you do - like what advice would you give Puppet module authors? How would you apply this directly to how people are working?

George: [00:11:54] So this is where we get the benefits of Adrian Thebo's Lego concept and Roles and Profiles is really harden those boundaries between things. If you are writing a role, only write the role. If you're writing a profile, only write the profile.

Ben: [00:12:12] Like would you would you suggest like totally separating yourselves away from that? So you you're not even involved with the implementation of that?

George: [00:12:19] In an ideal sense, yeah. I mean from if I were just spit balling an org chart the role authors would be more of your business analysts who are talking the business and figuring out what do we need to meet the need - your solutions architects, that nice high level where nobody actually does any work. I saw that as a solutions architect. Your profiles would be kind of your mid-level folks who are actually taking - OK, you want me to set up Apache or nginx or SQL Server. And here are the things I need to do to make those things functional. Finally you would have your most senior people, your real more locked down in the depths, writing the guts the modules to actually do the thing.

Ben: [00:13:13] But now in a kind of a more real world implementation where you don't have that luxury of being separated, where are the people writing the profiles often are also writing component modules. How do you keep yourself like in that mindset of right now I'm writing a profile, tomorrow I'll write a component module or the other way around?

George: [00:13:34] That's a really good question. It takes some self-discipline and a whole lot of self-deception. And when I figured it out I'll let you know.

Ben: [00:13:48] We'll have you back to to talk about that next time then.

George: [00:13:50] Sure. Kind of a real way to implement some of this is like I said harden the boundary around what a thing does. Define your interfaces and document. These are the methods that I expose. These are the options that I can accept, and if anybody tries to get cute with that throw an error. And it goes the other way too because if you have behavior that is either undocumented or the documentation says if you call me you call it this way it'll do this and it doesn't, or does something else entirely. That's a bug. I take a little broader definition of bug. I know it's still controversial, but any undocumented behavior or behavior that is different from the documentation is a bug.

Ben: [00:14:44] Do you use Puppet strings to make this documentation or are you talking about like writing manual user docs for it?

George: [00:14:50] That is really up to however you want to implement it. I'm going to use the old academic defense and say this is just the idea. Implementation is something else. Another really good way of kind of carving these things up is - I'm going to take a page from 1980s Lean here - is map your value flow. Identify each section of your process and group the things that affect or change the same things. Those become a capsule, a module. Now it's going to require you to be able to understand what a business process is and how to express that in code. None of this is new. This is 1980s Lean on top of 1980s ITIL.

Ben: [00:15:44] It's a new way of looking at the same old lessons that we've been learning.

George: [00:15:48] Exactly. And we're keeping the stuff that works. Adjusting it where it doesn't quite fit and we're throwing away the things that don't.

Ben: [00:15:55] Right on. Do you have like any closing thoughts or anything here?

George: [00:15:58] Yes and this is a big one. I encourage everybody to learn from the past. Nothing is ever new. Go find the gray hair in your office who has been doing this for 40 years. Talk to him about the good old days and try to understand why what they did back then worked, and what you can learn to do today.

Ben: [00:16:22] Someday we're all gonna be the greybeards.

George: [00:16:25] It happens. I'm going there myself.

Ben: [00:16:28] That sounds like a very good lesson to take from this. Thanks for sharing George.

George: [00:16:31] Yep and always remember if it doesn't work right, fix it.

We're all familiar with Puppet's main forte; that of managing configuration for computers, servers, cloud instances, etc. But as with any tool, community members often… color outside the lines.

Carrying on with our community spotlight series, this episode highlights one of those unusual usages. Corey Osman started this project by using Puppet to manage tiny IoT devices but ended up building a whole management framework out of Puppet and Bolt technologies. Let's get started and hear his story.

Today's musical intro was recorded by Manny, a software engineer on our hosted services team.

*Ben Ford is a developer advocate at Puppet.*

We're kicking off 2019 with a bang and a series of spotlights on notable community members and their achievements. We've got some great shows lined up for you, from useful development tools, to community experiences, to lessons we could learn from the history of computing. Let's get started today with Corey Osman of Portland, Oregon who's built a command line REPL. 

This quick screencast shows some of the capabilities of the tool:


A REPL, or command-line shell, allows you to interactively work within Puppet and see what effect each line of code you write will have. Rather than trying to understand an entire system at once, you can drop into the shell and try out a language construct or see how a variable will be interpolated. For even greater understanding, you can add breakpoints to your Puppet code so you can stop compilation at a known spot and inspect your variables or see what classes have been declared.

Today's musical intro was recorded by Michelle on our SRE team and her band, Great Niece.

Learn more

• https://www.puppet-debugger.com

• https://github.com/nwops/puppet-debugger

• http://logicminds.github.io/categories/debugger

*Ben Ford is a developer advocate at Puppet.*

We're kicking off 2019 with a bang and a series of spotlights on notable community members and their achievements. We've got some great shows lined up for you, from useful development tools, to community experiences, to lessons we could learn from the history of computing. Let's get started today with Corey Osman of Portland, Oregon who's built a command line REPL debugging tool for Puppet.

This quick screencast shows some of the capabilities of the tool:

A REPL, or command-line shell, allows you to interactively work within Puppet and see what effect each line of code you write will have. Rather than trying to understand an entire system at once, you can drop into the shell and try out a language construct or see how a variable will be interpolated. For even greater understanding, you can add breakpoints to your Puppet code so you can stop compilation at a known spot and inspect your variables or see what classes have been declared.

Today's musical intro was recorded by Michelle on our SRE team and her band, Great Niece.

• https://www.puppet-debugger.com

Ben Ford is a developer advocate at Puppet.

Ben: [00:00:10] Hi everybody. My name is Ben Ford. I'm the developer advocate here at Puppet. I'm here today with Corey Osman. Could you tell us a little bit about yourself?

Corey: [00:00:18] Hi. So I am a DevOps consultant. I've been working in the Puppet space for, man, just over I would say nine years now and I write tools for the DevOps community.

Ben: [00:00:35] That's pretty awesome. That's a very long time. We're here to talk about Corey's Puppet Debugger project. Could you tell us a little bit about what that means?

Corey: [00:00:41] So the Puppet Debugger project, what it initially started out was just a way to run Puppet code without having to write a manifest and it's a Ruby-based tool that you install as a gem and it is designed to help the user learn and understand the Puppet language.

Ben: [00:01:04] And how does it do that? Do you, can you work with the language interactively or...?

Corey: [00:01:10] You know first back up a second and we can talk about what a REPL is because underneath the covers the Puppet Debugger is a REPL in every language that I can think of except for maybe one or two has a REPL and REPL stands for read evaluate print loop. And if you just break those down first it's going to read code, then it's going to evaluate the code, then it's going to print the output, and then it's going to loop back around and do the same thing over and over again. So if we take those foundations and we bring that into a tool, you have a way into the Puppet compiler so that you can write code and then it evaluates it and prints it back out and shows you wat would become about if it was actually applied to a node. The debugger portion of that you know the debugger in general is just a tool that helps you find the problems in the code base itself, so in a lot of this is you know logic errors and if you're writing unit tests with rspec-puppet then that would also help. But there's quite a bit of setup involved in writing tests so this is a tool that will at least help you understand your code and help you find problems much quicker than than doing all of the unit tests set up.

Ben: [00:02:43] So a REPL is kind of a, I don't know, maybe an esoteric name for something that's really really common. I think the one thing that we all know that we've all used is just running bash interactively. Like when you open up a command prompt and you start typing commands into bash that's a REPL right there. So what you're what you're saying what you've built for us is basically the way to interact with Puppet code more or less the same way you would interact with a with a bash shell.

Corey: [00:03:10] Correct, yeah. So you know in bash you're typing something in and if you mistype it tells you that you mistyped it. And that's exactly what the debugger does is if you tried to declare a variable twice, well you can't do that in Puppet and it's going to tell you what you did wrong. But the cool thing is is that it doesn't take 10 or 20 seconds to compile that code and spit out the problem. It's immediate.

Ben: [00:03:39] Could you tell us just like a little bit of like how it works internally? Like how were you able to compile code using Puppet?

Corey: [00:03:46] Underneath the covers it takes your input and we'll just say from a while loop what it's doing is it's setting up the environment. It knows where your Puppet modules are. It grabs the facts from the node that you're running Puppet apply on and it brings all of that together for the parser to eval it. So one of the differentiating factors between say puppet apply and running the puppet debugger is that I just provide like a common set of facts for you so that you don't have to pull them. But I also give you the option to get real facts from your system as well.

Ben: [00:04:30] Could you tell us a little bit about like how this came to be? Like is there like an inception story of where this tool came from, why you built it?

Corey: [00:04:40] Once I understood what was going on in the covers it just sort of came to me and I was actually eating breakfast at a place right up the street from me. And as I was like oh my god like we have not had a REPL in the public community for, well, ever. And I think I just discovered a way to how to do this. And so I think that day I had like a small proof of concept. I mean it wasn't great but it actually read the code that I entered in and spit out the same thing that puppet apply would do, except it did it immediately.

Ben: [00:05:22] So you're saying you can stop in the middle of a catalog compilation and you can actually print out the value of a variable.

Corey: [00:05:30] Yeah that's been something that has been on my want to do list for some time. I think a lot of problems with unit testing is we all want to try to test the intermediate state and to this day you can only test the the end state like what what resources have have been compiled in the catalog and whether or not those things are in the catalog. With the debugger you get the end state but you get everything in between all the intermediate states all of the things inside ephemeral loops. So if you want to know the value of the sixth iteration of an array you can do that. And when I finally discovered why I need a puppet functions I was like oh because I need to manipulate this incoming data. But now it's like OK well there's over 100 functions available to me. What do they do? And there isn't really a good way to do it. So the Puppet Debugger it allows you to use the function and if it's using facts or you have to supply data to it, it allows you to run those functions over and over again and really understand what they're doing.

Ben: [00:06:59] Just kind of interactively play with them a little bit rather than having to write code and run the code. You can just try it and see what happens.

Corey: [00:07:06] Facter DB is another tool that is essentially a database of a pre-canned facts.

Ben: [00:07:14] So what you're saying is you can compile and poke at this code as if it were a CentOS system or as if it were an Ubuntu system or anything like that. Even different versions like CentOS 6 or CentOS 7.

Corey: [00:07:26] Even Windows.

Ben: [00:07:27] Even Windows. So can you do things like run hybrid data lookups or testing functions or maybe like iteration or anything can you look like validate this code you're writing while you're writing it.

Corey: [00:07:41] Absolutely. You know I used to think Hiera was just magic and then I realized it was just a core function that was included with Puppet and I'm like "He,y I can just call the Hiera function and pass in the values and I can have it look up values within my Hiera database. So if you understand that then now you can set up a whole Hiera lookup all the data files and now you can start to test out your Hiera restructure to see if that works correctly and you can swap Facter sets in and out to to validate that your Hiera setup is now so.

Ben: [00:08:27] So can you change facts while you're running? So you can you can test a look up against Red Hat and test that same look up immediately against Debian?

Corey: [00:08:35] Yes you can. There's a reset command that you can run that basically wipes out the environment and you can reset the the Facter query that selects effects.

Ben: [00:08:48] That's even faster. Wow, that's cool. One thing that I know that some people have struggled with quite a bit is understanding Puppet's data type system it can be it's relatively straightforward but it can be pretty complex if you have like different validations or if you have complex types. Would this be useful in trying to validate making sure that you are using the data type properly?

Corey: [00:09:14] Yeah, that is one of my use cases as well. And I have a whole, I think I wrote a whole blog article on validating your data types.

Corey: [00:09:24] There is a couple of things that you can do and you might never discover these if you if you're not using a debugger tool where you can interactively play with the code, but there's a function I think it's called type of - basically there's an operator that I want to say the bacon canon. But that's not the operator name. Yeah it's the equal tilde tilde equal. And that operator will tell you true or false if the data type matches on both sides.

Ben: [00:10:00] OK. So what's your favorite feature that this thing offers?

Corey: [00:10:03] By far my favorite feature in the Puppet Debugger is the ability to set a breakpoint anywhere in your code. This includes Puppet functions. This includes anywhere in the manifests and you can set a breakpoint and the breakpoint is really just another Puppet function and it's just debug colon colon break and when Puppet compiles that catalog with whatever means that it normally does through puppet apply or maybe you're running a unit test or maybe you're just feeding the Puppet Debugger a file through standard in it's going to run that brake function and just stop right there, right in the middle of the compiler compiling the code, and that allows you to see everything that the compiler can see so that allows you to inspect the ephemeral loops the values that are around you at the time the top scope variables the facts that are fed in the output of functions that will run either before or after that break command.

Ben: [00:11:21] So at that point all you really had to do is put in a breakpoint right after the look up call and then see that the variable that should have been set by the lookup didn't have anything set and then the rest was just a little bit of playing with the values and the variables and running the look up command before you noticed that what you were passing in was the wrong case.

Corey: [00:11:43] Yeah correct. So it made it just really simple to determine what my fault was. And after that I just remove the break command and off I went. I finished the code I was working on. So that that is my favorite feature and I'm going to say I totally ripped it off Pry. Pry is an amazing tool for Ruby debugging and I just wanted that same feature in Puppet code because it is such a valuable thing to have to be able to stop inside the code. And I did whatever I could to make it work. And I believe that I did I even inspired heavily from Pry. So if you've never used Pry for really debugging, go check it out because it's awesome.

Ben: [00:12:38] Now I think maybe my last question for you is, it would be really really useful if I were able to impersonate a specific node while it was validating code so can I like load up a set of facts for that specific machine in my environment?

Corey: [00:12:55] So there's two things you can do. What I've been meaning to write this Facter DB article for a couple of weeks now but if I wanted to spoof a system I could capture the Facter set of a real system and use that as a external Facter set for Facter DB. And because the Puppet Debugger uses Facter DB I can now load that externalize Facter set of some of that real node in the real data that it has inside the Puppet Debugger and get to the bottom of why my code does not work the way that I think it should.

Ben: [00:13:40] So how would somebody get started using your project? What's the first steps that they ought to do in order to try it out?

Corey: [00:13:47] So if you just want to try it out, I have a website: www.puppet-debugger.com. There is a demo there and essentially it's a live application where you can just you know test the waters out and see see if you want to download it and try it on your system. But it's an interactive it's real it's running on Heroku it opens up a shell to a remote system using web sockets and it looks like a real system and it loads up the debugger.

Ben: [00:14:25] And then installing it is just as easy as gem install and running Puppet Debugger?

Corey: [00:14:30] Right. So to to install it, it's just gem install puppet dash debugger. It does require you to have Facter and Puppet but I think Puppet is a dependency so she'll come along for the ride. If you're using PDK I think that you should I think it I'm not sure if it ships with it. I think at some point I was trying to get it in there but it may already come with it, it may not. But if you know how to use the gem command you could install it in the PDK environment and then you could install it on a remote node if you want to. I've used it that way before you can install it on your laptop. Yeah. So there's there's multiple ways to use it.

Ben: [00:15:25] Right on. Well, thanks for talking with us Corey and for sharing this project with the community. I think it can really help people out when they're trying to learn how Puppet parses their code base and figures out what to do. Thanks for your time today.

The Puppet Development Kit makes it easier than ever to develop and test Puppet modules by providing a simple, unified interface to a set of helpful tools for anyone who writes or consumes Puppet code. Leveraging the Puppet Development Kit, it’s now possible to:

• Quickly get started developing modules using best practices and new tools that enable you to develop, test and publish high-quality Puppet modules with confidence.

Learn more about the PDK: Listen to the podcast now!

Anna Velasco is an associate community manager at Puppet.

• Download the Puppet Development Kit

Editor's note: The Lumogon open source project is now unmaintained. The technology is part of Puppet Discovery, which lets you discover what's running in your hybrid infrastructure — from servers to VMs, cloud instances and containers.

On May 11th, we announced Lumogon™, a new technology for inspecting, reporting on, and analyzing your container applications. In our podcast about Lumogon, we hear from Kenaz Kwa, Lumogon’s product manager, and Tyler Pace, Lumogon’s UX architect, on the challenges of operating containers at scale while providing visibility into what makes up your containers.

Containers are quickly shifting how applications are delivered. Their black-box nature is great when it comes to running them, but not everyone’s life is being made easier by containers. If, for example, you need to know whether software in a container needs a security update, the black-box nature of containers makes your life a lot harder.

We hope you enjoy learning more about Lumogon straight from the mouths of the project leads. Get started with Lumogon at https://lumogon.com/. You can also chat with the Lumogon team and other Lumogon users in the #lumogon Puppet Slack channel.

Find the rest of the Puppet podcasts here. And if you would like to read the transcript of this one, it's below the Learn more section.

Carl Caum is a technical product marketing manager at Puppet.

• Get going with Lumogon.

Carl Caum: Hello. Welcome to the Puppet Podcast. My name is Carl Caum. Today we're going to be talking about Lumogon, a project for inspecting your container applications and understanding what makes up those container applications in production and its scale.

Today, [we're] talking with us — we're going to have Kenaz Kwa, the product manager of Lumogon. And we're also going to have Tyler Pace, a UX engineer. And I'll let them introduce themselves. Kenaz?

Kenaz Kwa: Hi, my name is Kenaz.

Tyler Pace: Hello. This is Tyler.

Carl Caum: All right. So let's just jump right in. Kenaz, what is Lumogon?

Kenaz Kwa: So, Lumogon is a tool that we're releasing for inspecting and reporting on, and really analyzing the contents of a container. This is everything from the actual packages and the actual bits that make up that container, as well as some of the metadata that you might find interesting.

We're observing that many people, as they adopt containers, are taking the existing applications that live in their enterprise infrastructures and packaging them as containers, and sort of throwing them into production. And that gets a lot of benefits, from taking that packaging format and re-using it for some of their existing applications.

But what they find is now they have a giant box of applications that are running in these containers, and they don't really know what's going on inside of them. So what Lumogon is, the tool, is supposed to be used for is to understand: "What are the contents of that container? And, how do I get more information about things actually running in production?"

The tools — there's two components to it; one is an open-source CLI tool that you can download from Docker Hub, and you just docker run a container in your environment. And then, there's also a web application reporting tool that gives you more information or gives you a nice report of everything that's going on inside that container.

Awesome. So essentially, it's about solving the problem of containers just being a black box and understanding what is exactly in those black boxes at any time.

Tyler Pace: Mm-hmm. Yep.

Carl Caum: Awesome. So why is that important?

Tyler Pace: Well, from the scheduler perspective, a lot of best practice for running large amounts of containers on container hosts is using some type of scheduler that basically treats all of your compute resources as a single contiguous blob of compute resources.

And so containers today are treated by those schedulers — like Cooper Netties and Docker Swarm and DCOS — as a black box. Because all it really cares about is the amount of CPU that's being used, the amount of memory that's being used, the amount of disk that's being used.

But what it doesn't really take into account today is everything that's actually going on inside that container. And we're seeing that that's becoming more and more important. Because there could be lots of things that you don't necessarily know about that's actually running in production; that could be security vulnerabilities; that could be compliance concerns; the same sorts of problems that we typically see in configuration management: around visibility, around audit control, status accounting.

Those are all problems that still remain present when you move all of your workloads to containers.

Carl Caum: Yeah. I know that when I work with containers — especially working with someone like Cooper Netties — if I ever have a question of, "What was there a week ago?" that's exceptionally difficult to answer. Is that something that Lumogon is going to be able to help with?

Tyler Pace: That's sort of the direction that we're going. At the beginning, it's obviously a very simplistic tool. It's really just to show you what -- on a single point in time and for a single report or for a collection of containers -- what's actually going on inside of them.

But where we see Lumogon going is to be able to answer those kinds of questions. Uou know, in: "I noticed that there was a vulnerability in this application at 3 p.m. last week. Can you tell me what was going on five minutes before that? What was everything in my production infrastructure up until that point? And what was in my infrastructure 10 minutes after that?" — so that you can really drill down, and one: understand, "Were you safe or not?" and two: be able to debug and diagnose what the issues are.

Carl Caum: Awesome. So as we're talking through this, a number of questions come into my head that, I think, that a lot of different personas would ask. There's not just the operator of the containers being able to say like, "What's in this thing?" But also, InfoSec asking about compliance, auditors asking about what the past state was. Tyler, can you talk a little bit about everyone and the different personas that would benefit from this?

Tyler Pace: Yeah. Absolutely. We envision a lot of different people involved in container workflows benefiting from a tool like Lumogon. I mean, first in that list are actual application developers who are starting to use containers as part of their application development and deployment workflows.

And we know that increasingly, application developers are taking on more responsibilities, not just the development of an application but it's ongoing maintenance, updates, rollouts, taking some ownership in the uptime of the application, and sort of just being more involved in the end-to-end work that's required to have a sort of fully, powered up application and production.

And so having a tool, like Lumogon, to be able to inspect not just the images you're building — and being able to get more insight into the packages that your application is depending on, the packages that are in those images — to be able to also look at containers that are running, so you [would] actually look at the live-running application; be able to get information on that similar metadata that you might be interested in.

So you can compare: "What does my development state look like versus the production state? This job that's in Jenkins right now, what image is it playing with? And what's the state of that image? And how does that differ from the other environments that I might be interested in?"

So it allows you to start asking questions about your application, sort of different parts in its timeline. You can see how that would be, in particular, helpful in CI/CD environments where you can start to gather reports on: What your application looked like? Did it look like what you thought it was going to look like? Was the right third-party library in there? Is that missing library a reason why you failed your Jenkins build? Now, you can start to get more insight into those practices.

We also see InfoSec people being interested in a tool like this where you can start to gather the data that you need for audits. We know that containers turn over quite quickly. There's a recent report from Datadog where containers are turning over about two days on average.

And so, you know, that's a very short window to be able to put something into production, scan it, and make sure it's the thing that you thought you were going to put into production. And then a month from now, when you've had 15 different versions of that container spin up and spin down, how do you make sure that some OpenSSL vulnerability that existed, very briefly, for three days, three weeks ago -- that you weren't somehow impacted by that?

And tools like Lumogon help you be able to answer questions like that by giving you the data you need, to know what was running and when.

And then, of course, also the more traditional operations teams that are trying to do things like understand: "Why are our applications going down? Are we dealing with some buggy third-party libraries? -- you know, we all write bugs so buggy first-party libraries -- What might be the root cause of our applications going down?

And, of course, a tool like Lumogon that gives you sort of extracted data in a JSON format, makes it really easy to feed into your monitoring systems. It can go into that ELK stack or Splunk tool that you're using to conglomerate all of your data to give you all of your data to give you that birds-eye view of your infrastructure.

And this is another tool that can help you take containers and make them a first-class member of that reporting dual-chain that you've built for your virtual machine infrastructure.

Carl Caum: Cool. That makes a lot of sense. There's a lot of things that jump out to me there. You know, one: you were talking about containers living for about two days.

I recently attended a talk by Diogo Monica from Docker, and he was talking about some stuff that Docker's working on where they expect a container to live for about 60 seconds; it's just a constant state of refresh, even if there's no update that needs to be refreshed. It's more around security of: "If a container gets compromised at any time, well, you want to kill that as soon as possible. So that if you do get compromised, the threat level is removed very, very quickly."

And he was talking about this, and of course, my head is exploding because it was just amazing. But then that question of: "Well, wait a minute. If we're just killing things every 60 seconds, how do we ever have a sense of what's actually going on? How do we actually know what things are running when and where?" So I'm pretty excited about Lumogon being able to start answering some of those questions.

Another area that really jumps out at me is CI/CD, particularly around the peer-review process. So right now when somebody -- actually even this morning, I was looking at a container that somebody was trying to push through a pipeline -- that somebody may have been me.

But when I went to look into it — like: "I'm having a problem with this thing. I need to know what operating system it is. All right. So I open up the Docker file, and it has a from clause. Okay. Well, let's go find that image. Well, I don't actually know where that GitHub project is so let me go Google for that. So I find that and find its Docker file. Oh, wait. It has another from image."

And so it's just this rabbit hole of stuff just to answer a very question of: What the heck -- what is the OS in this thing? So having this data available as a peer-review process, so we could look at those and say: "Hey, you might want to update this library. Hey, you might want to add this Docker label." You know, having one single-source of truth through the pipeline, makes a lot of sense.

Kenaz Kwa: Yeah. And just to add to that -- because the data is being represented in a structured format, then you can inject that as a sort statement into your CI process. So that as operations people are participating in deploying containers, they're able to prescribe the policies that they need in order to be able to make those kinds of guarantees to their stakeholders that: "Yeah. You know, that we are running in a safe, secure sort of way."

Carl Caum: Yeah. That makes a lot of sense. So to that point, Tyler, would you say this is a security tool?

Yes and no. We're not designing explicitly for security in mind, but good security starts with an awareness of what's running in your infrastructure.

You have to be able to understand that landscape of: "What does your surface area look like for your applications? What are all the different dependencies that you're relying on, both the ones that you are developing yourself, and the ones that you're inheriting from different vendors and other third-party providers?"

And that, one of the -- sort of a long-term goal that we see with a lot of people stepping into containers is they understand that in the beginning, they -- from the perspective of containers – kind of have a big surface area.

They're importing a full-fledged operating system like Ubuntu or CentOS into their container, and the hundreds of default packages that you get in an operating system like that -- some of which are probably required for your application, many of which aren't.

And you don't, without an awareness of what your starting point is -- it's hard for you to be able to say, "Okay. This is my big starting point surface area. This was a good place to be for my first application, my first Brownfield application, that I moved to a container. But now I need to start reducing that surface area."

And the only way to get there is to understand what you're dealing with, and just so you can start making educated guesses about: "What can I take away? What can I remove from this image?" And then test and remove, and test and remove, so that you can [winnow] that down to a small surface area.

And that reduced surface area, that's a win for everyone. It's easier to understand an application with the minimum number of dependencies. They're easier to run. They're easier to develop, and they're easier to harden and secure.

Carl Caum: Awesome. So how do people get it?

Tyler Pace: Well, you can get Lumogon from lumogon.com. It's also available on GitHub and Docker Hub.

Carl Caum: Great. So I'm assuming people, as they try this thing out, they're going to want to get in touch with the team and learn what they can do -- some of the hidden stuff that might be in there. How do people get feedback?

Tyler Pace: The Lumogon team, we're hanging out all the time in our Slack channel. And so we encourage anyone to come out and say, "Hello," and tell us, "What they're doing? And what they like? And what they'd like to see in the future?"

And we'd love to get in touch with you. And you can also reach out at [email protected].

Carl Caum: Great. Well, Kenaz, I'm curious -- that it's a really interesting name. Does it mean anything?

Kenaz Kwa: So it turns out, it's shockingly difficult to name things, especially when everyone is taking the four-letter, five-letter words and domain names.

So, it doesn't mean anything. It's completely made up. But the idea of lum- having some implications of light and some notions of shining a light into your containers, and -gon sort of the suffix for a polygon -- the idea of like: "When you shine that light and you see what sort of shape it takes, that you have a better idea of what actually you're looking at." And so that seems workable. So we're going to go with that.

Carl Caum: Awesome. So you talked a little bit earlier about where we're going with it and being able to understand past states of your infrastructure. Is there any more to it? Like, where is this thing going to go in the future?

Kenaz Kwa: Yeah. For sure. Right now we started with a fairly basic set of metadata that we're collecting from the containers. Over time, we expect that collection of information to grow, both from our conversations with customers and with users as well as potentially community contributions.

The entire CLI tool that is collecting the information is all open-source. And we'll help to define what some of those -- that contribution model -- what [it] might look like.

And the other thing is we're looking at expanding the types of targets that you can run this tool on. Right now, it's basically running it on a single container or a container host. But we know specifically, with things like schedulers -- like Cooper Netties -- people are no longer interacting directly with the container host, and instead, all they see are these higher level interfaces like a Cooper Netties replication set or a Docker Swarm service.

So we hope to be able to then instead, point the same tool, just at a higher level of abstraction, that produces the collection of information underneath. So you can say, "Okay. In this service, what are actually all the packages that are in there? What are all the labels that have been applied to these containers?"

Carl Caum: Awesome. So that does beg one question that we haven't asked yet? Usually, when you try to understand what's in a container, you look at its image. And that requires a change to your build process. So you have to build something into the container image, and then when you spin it up, it's running in that environment and can then ship data back. Is that how Lumogon is working?

Kenaz Kwa: No, so it actually -- the whole goal is that you don't have to change anything about your infrastructure. That, we know that you're going to do your infrastructure the way you're going to do it. And we want to get out of the way, but provide you as much value as possible by being able to see inside your containers without having to change anything about the build process.

So to that end, all you really have to do is run another container inside, on your container host, and point that container at the running containers, which will harvest and collect information and report it out to you.

Carl Caum: Awesome. Sounds easy.

Kenaz Kwa: It is.

Carl Caum: All right. Well, that's all the questions I have. Anything else we need to -- think we should cover?

Kenaz Kwa: No.

Tyler Pace: Try it out and let us know what you think.

Carl Caum: All right. Sounds good. Thanks, guys.

Kenaz Kwa: Thank you.