Pulling the Strings

Sometimes big challenges need to be tested and solved for at a much smaller scale. In this case, it’s the 1:87 model train scale with an interesting experiment coming out of Puppet.  Learn more about a new way to bring people together and automate with Bolt.

On this episode, Engineers Cas, Ken, Jacob, and Managing Editor Andrew chat about “bolting” remote task orchestration and automation onto a HO-scale model train set. The vision: sending  Bolt plans to operate a model train at Puppet’s Portland headquarters.

Hear how this cross-functional team solves for the automation of remote devices, navigating network security requirements, possibly provisioning containers, and (most importantly) building a fun project that brings people together.

We’re covering a few DevOps parallels, data-driven diaper changing, and, apparently, Baby Shark.

Taking care of a newborn is a gargantuan task for everyone involved. We invite one of our technologists to tell us more about how he and his partner are automating the rough manual tasks away to make more time for rest and peace of mind.

Automating chunks of the newborn/baby experience is beyond software. It’s about taking out manual tasks to make more time for important things… like sleep.

Andrew Nhem is the managing editor at Puppet.

• Check out Dad Verb's YouTube Channel

Andrew [00:00:00] Hey everyone, welcome to today's episode of pulling the strings. Today we're going to be talking about DevOps, Automation, and guess what - parenting. Bet you didn't see that one coming. This is a special episode as a lot of us at Puppet, yeah, we work day to day on helping folks make their infrastructure more compliant, secure and what-not; automating ad hoc tasks, orchestrating them into complex plans. But sometimes you need to apply those skills back at home, you know, improve the mental wellbeing, get some more sleep. One of our product managers was telling us all about this really cool automation setup that he has for his newborn. So we have this special episode today with Bill Tang, product manager. Introduce yourself real quick.

Bill [00:00:58] Hey everyone, my name is Bill Tang, and I'm one of the product managers here at Puppet. And as Andrew mentioned I'm a new parent to a 2 month old.

Andrew [00:01:06] And joining us remotely which is a super special treat is Alexa Sevilla, one of our product marketers, who is also a parent of young children. Alexa please introduce yourself.

Alexa [00:01:14] Thanks Andrew. So my name is Alexa Sevilla and I have a daughter that is almost eight heading into second grade and a son that is five and a half heading into kindergarten. So I've got some experience on this podcast.

Andrew [00:01:34] Right on. Myself, Andrew Nhem, managing editor of puppet.com. Actually have a little young one, almost 4 years old as of this summer, and summer is actually a really great time to start automating away things that we probably don't want to do. So we can spend more time with the little ones doing things like you know for us, campfires are huge, bike rides, going to the park, grilling. And speaking of grilling I think we could probably get ready to ask Bill a bunch of tough questions about how in the world you are automating your newborn experience. This is amazing, Bill.

Bill [00:02:05] Yeah. So my wife and I first had the baby. It's very difficult as all new parents know of like getting to bed, getting the baby soothed and quiet and actually getting enough sleep. I mean that's where our biggest challenge was - getting enough sleep.

Alexa [00:02:20] So one of the things that we used to do was we would just take babies on a car ride and that was like our way of like soothing, where we reached that ultimate point of like, I can't rock you anymore. I can't cradle you like, you want to be carried you want to be held. I'm just wiped out. I'm going to put you in the car and we're gonna go for a drive.

Andrew [00:02:43] Yeah I recall those as well.

Alexa [00:02:45] They're usually successful until the one time you come home baby sleeps and then the minute you take them out it's like it starts all over again.

Andrew [00:02:54] But Alexa now that Bill has eliminated the car white noise, put the baby to sleep tactic. How is he going to get his fast food fix because that is the great enabler of the late night snack.

Alexa [00:03:07] Yes. I don't think I ever went through the drive through with the baby that late at night. I'l have to think about that that that's what would have been an excellent idea. I was probably so exhausted that I don't even know what street I was driving on to be honest.

Bill [00:03:23] You know one of the key things that we've done in the past two months or so is invest in something called a snoo. If you're not familiar with this new is it's a smart bassinet. It basically automates the bedtime routine of putting your baby to bed, swaddling the baby, getting the baby shushed following Dr. Karp's five S's. You know shushing, swaddling, side the side, sucking and sound. I believe, I may have that mixed up but well.

Andrew [00:03:52] We'll add it to the show notes so people can check out that link if you want to learn more.

Bill [00:03:55] Yeah. So so basically it allowed us to do is basically cut down our routine of after we feed the baby trying to put the baby to bed. That was the biggest challenge of getting her quiet and soothed, and this thing works like magic. You put it down it takes away that manual step of like swaddling the baby and actually rocking the baby to bed instead of taking half an hour put a baby to bed. Now it takes all of two minutes which is amazing.

Andrew [00:04:19] That is amazing. And it helps you and your partner spend a little more time figuring out you know maybe the next day. I mean it's it's it's hard when the kiddos first there and your schedule's completely on its head. You know it's funny because when we were planning this episode out, we were drawing all these connections between DevOps and parenting, you know people probably don't put those two things together, but we did come across some pretty interesting DevOps Days presentations, right? Was it in Seattle or San Francisco?

Bill [00:04:49] Yeah it was in DevOps Days there's talks about parenting and the three major parallels. Like for me the parallels are slightly different. I think it's being a brand new parent, you're thrown in this role right and it's chaos. You have no idea where to start. Very similar to your DevOps journey. You have no idea where to start but you're forced together to work as a team. And I think that's one of the big parallels that I have that I've taken away from being a new parent and DevOps is that it's a cultural shift or a mind shift on the way you think about things. You're no longer thinking about two separate tracks of work for lack of a better word. Now you're working as a team with a single outcome of like putting the baby to bed.

Andrew [00:05:34] There are a lot of connections around celebrating both little and big wins you know. Yeah, we're not shipping tickets but we're changing diapers and changing bottles or cleaning bottles, right. And like Bill said it's preparing for the unexpected. Being able to have a system in place between you or your partner or whatever your system of caregivers are to handle the unexpected and most importantly working as a team.

Bill [00:05:55] Yep and Andrew just to touch upon that I think another key step that kind of parallels parenting or newborn parenting to DevOps is having the right tools and having the right workflow for that. And one of the things that I've learned very early on was having a separate station for each one, it's is almost like a pipeline for feeding the baby, changing the baby, you know putting the baby to bed and just having the right tools that makes those each one of those steps a lot easier.

Andrew [00:06:24] So you have a continuous deployment methodology for dealing with, not dealing with, that's a hard word, but for raising your child.

Bill [00:06:31] Absolutely. So like feeding a baby at 2:00 in the morning or trying to get the baby changed and fed. And what I've learned is like if you minimize the steps as much as possible and simplify so it's repeatable and leveraging the right workflow and leveraging the right tools, like for example, when we first had our baby we had no idea if she was eating enough. So one of the things we've we ended up doing was getting a changing table that was also a scale.

Andrew [00:06:58] Are you kidding me?

Bill [00:07:00] So as soon as you're changing the baby's diaper, you know exactly, we change it before she starts feeding and then it automatically captures your weight. And after she finishes feeding we put her back on. We know exactly how much she's eaten and if we actually have to supplement or not.

Andrew [00:07:14] Wow. So you're telling me you have data driven like diaper changing.

Bill [00:07:19] Absolutely, and I think that's one of the key things as well is the monitoring or the data tracking and I think that's another parallel between DevOps, right? Knowing how you're progressing knowing if you're actually making a difference and if you're successful or not.

Andrew [00:07:32] Wow you're blowing my mind, Bill. Alexa and I we we talk about our kids a lot. We work on a lot of projects together so we often will chill out, talk about the fam. Alexa, you and I both have young kiddos past the toddler stage. I don't recall, you know I wasn't allowed to bring my iPhone into the room. What about you? Did you all have a bunch of gadgets and whatnot in the newborn room?

Alexa [00:07:55] No. And while I'm, you know, I am happy to say we're good with the two kids, Bill's, the gadgets and stuff that he has, it almost makes me want to have another just so I can do all this stuff. Your comment about the changing table with the scale is such an awesome find and I totally wish we had something like that with my daughter because she also was weight challenged. But yeah no it's just fascinating and I feel like you know every year there's so many great things that come out that make it easier whether it be strollers or changing tables or you know there's just so many fascinating things. It's really incredible. I do wish I had some of these things.

Andrew [00:08:40] So what kind of value do you get, Bill, out of the data that you're collecting? What do you do with it? Do you and your wife sit around at dinner saying like oh it's like kiddo's at a great weight, she's she's pooping normal, you know what are these conversations like?

Bill [00:08:54] Well I think a lot of it helps like soothe or is more of peace of mind for our insecurities. I mean being brand new parents, you have no idea what you're getting into. Not knowing if the baby's healthy, not knowing if the baby's eating enough, what not. Like if the baby's sick or not, having the right tools and having the right data, we can track her weight, we can track her heart rate as well and her blood oxygen levels to ensure that - in middle of night, my worst fear is SIDS, right? I thnk it's many parents. And being to look over and seeing that, you know, she's still breathing, seeing her blood oxygen level is where it needs to be, gives another sense of comfort. I mean, obviously, you shouldn't rely on those exclusively. But it gives that extra layer of comfort, of seeing the progression of how the baby has grown from you know, seven pounds, now she's almost 11 pounds. And then seeing the progression of the amount of food she's eating. We have concerns that she's has a fever or not, we have automated ways of taking her temperature as well. That just kind of helps with everything. You know it's not a crutch but it's is a way for us, you know we're both very scientifically driven. So having that data approach really helps us. So it may not work for every parent but at least it works for us.

Andrew [00:10:14] And that is a really good point, Bill, that you make. You know it's not for everyone. You know it's an interesting topic for us that there on the podcast here because you know we work for a company that helps folks with automation and what not. But you're right. You know to each their own, everyone has their own specific needs. It's interesting to me that you did not have to spend as much time as say, like, I know I sure did and I'd love to hear from Alexa, too, you know you get to sneak into the room at night sometimes, you know make sure they're not in a funky position in the crib. I'm sure every parent knows that test where you're like wow they're sleeping way soundly. I should put my hand in front of their face just to make sure I feel a breath, like to be able to ignore, or not ignore, but just to have that peace of mind. I'm sure that's awesome. And you know not roll the dice of like hey if I open this door will this kid wake up? And then you know you gotta put them back to bed and all that other stuff.

Alexa [00:11:04] Right. I would say I would agree with that, like the tiptoeing in and that watching, like is the child rising? Are they, you know, and then creeping back out staring at the monitor. Yeah.

Andrew [00:11:17] It must be really cool at some point to be able to map out that data over the course of you know infancy. Or I guess it goes newborn to infancy to toddler hood. That's got to be really neat to see your kid grow in that way. You're already seeing them you know crawl, flip over or roll over, you know all the other cute baby things. Walk eventually. But then you're going to have this whole other set of data you know along with watching their skills grow. That that sounds really amazing, Bill.

Bill [00:11:44] Yeah I mean, I am a gadget guy as well and I think some of my friends tease me about like if the things, have you ever seen Black Mirror?

Andrew [00:11:53] I have not.

Alexa [00:11:54] I haven't.

Bill [00:11:55] There's an episode about like raising a kid and over monitoring. And I think that's one of the things that I need to be more cognizant about is not over monitoring a kid.

Andrew [00:12:04] Oh yeah Black Mirror is like a millennial Twilight Zone.

Bill [00:12:07] Yeah but yeah I think just having that data and seeing that her progression, it's more about ensuring that she's healthy versus like being a helicopter parent.

Andrew [00:12:16] Yeah. You're getting the right data so that you don't have to do that, right. You just know you have that peace of mind you can focus more on just being a parent. Being there, too. You know I think there are, I mean pick your favorite blogg or article. There's so much about like you know, are you there and are you like actually present with your children? And I like the promise that, you know, if we can find legit ways to incorporate, integrate automation into everyday life that you can then spend less time being stressed out, burned out by having to wrangle the giant you know intense workload that is raising someone, and instead have more mental capacity, emotional capacity to be there and really enjoy it.

Bill [00:12:59] The one thing I've really relied on is like this baby bottle warmer, right where it takes away the errors that I would introduce into the process.

Andrew [00:13:09] Tell us more.

Bill [00:13:10] So when the baby first came out we had the supplement the feeding right where you had the heat the food or the formula or the milk to a certain temperature. I'm not that good at that and I would always constantly have it too hot or too cold. Now there's, we have this baby bottle warmer you turn it to a certain time it guarantees that the milk where the food is the right temperature so I'm not having to like monitor and watch over and turn on the stove at 2:00 a.m. in the morning.

Andrew [00:13:40] I remember having problems getting the food right at like 3:00 p.m. in the afternoon. When you're up in the middle of the night. It's way harder to do that stuff. So to be able to rely on a gadget to ensure that the milk or the food is spot on, yes. Absolutely yes.

Bill [00:13:56] It's taken away that fear for me if like burning the kid with my incompetency at 2:00 a.m. in the morning. So it's been really helpful. I remember the early days of feeding cycle we were doing hour and a half on average of getting. Right when we fed her to the time we got her back to bed and through all this different stuff that we've tried, different automation stuff we've been able to get down to about under half an hour from start to finish from feeding the baby, changing the baby and putting her back to bed.

Andrew [00:14:28] That's pretty good. I mean I mean everyone loves a little return on investment for their work but that streamline-ness is great in that you get that process down. You know I remeber at least with my kiddo, repetition, structure, they're all about it, right. So to be able to make that super efficient just helps you all put the kiddo to bed, so you all could go to bed.

Bill [00:14:50] Yeah I mean I think getting more sleep has really allowed us to enjoy those precious moments when we're awake and when she's awake full, dry and happy. You can actually take advantage of that.

Andrew [00:15:02] So have you figured out a way whether it's via automation or a data driven approach to eliminate blowouts?

Bill [00:15:09] That is a great question and I think maybe would be based on data because at this stage of her life she doesn't poo for three to five days so knock that cadence out, we can potentially narrow down when we need to put an extra diaper in.

Andrew [00:15:26] Wow you heard that here, everyone. We're getting closer to data driven insights that will help us avoid the time honor tradition of dealing with the newborn/toddler blowout, a time honored trial by fire for all parents new and old. So you've covered weight, changing diapers, bottles. What about the bath? Anything for the bath yet?

Bill [00:15:50] We haven't given her a full bath, yet. Well actually we have. Not like every day. It's more like we hold her over the shower, we spray her.

Andrew [00:15:59] Are you telling me you've invested so much into the automation of the other processes that you don't even have like the little cute bath insert?

Bill [00:16:07] One of the things we've overdone is like we bought too much in raising the kid and preparing for the kid.

Andrew [00:16:14] Which is also a DevOps problem, you know, too many tools in the tool chain, right? And all the sudden you're stuck with like how do we fully utilize this? Do we need to utilize this? Do we need this at all? Are we still focusing on the right problems at hand? Got to know, are you using Bolt to orchestrate any kind of task?

Bill [00:16:32] Not in the baby automation, but overall my household. Sort of like doing routine maintenance of resetting or installing a patch on my home server. I'm absolutely using Bolt for that but not in the baby automation park.

Andrew [00:16:48] Do you think you'll have like a dashboard on home that shows like baby stats?

Bill [00:16:52] Potentially.

Andrew [00:16:53] Wow I was half kidding when I asked that question.

Bill [00:16:55] I will have to invest or investigate the server's API, so pull that data from the services that I'm using.

Andrew [00:17:01] Then put it up on the family TV. Picture in a picture. Have like a Daniel striped Tiger up on the big picture and then baby stats on the corner.

Bill [00:17:10] Yeah.

Andrew [00:17:12] Yeah, there you go. That's some advanced stuff.

Andrew [00:17:13] So Bill, I have the pleasure of working with you day to day and I know you're big on optimizing the data you receive on your phone. Are you piping any kind of these baby stats to your phone then getting the ultimate - I feel like you have the ultimate dad view now. You get pictures of your kid, you get texts about your kid, and you have data about your kid. Like you just get this whole view of baby.

Bill [00:17:37] Yeah I have. If you look at my phone I have how much she slept, how much she ate, how much she weighs and if my wife inserts in or records how many diapers she's changed during the day.

Andrew [00:17:50] Wow. Bill, I kid you not. My partner was out of town for a bit. I was solo parenting for about a week or so. I'm pretty sure I skipped the teeth brushing. Not on purpose but because you know maybe I need to invest in a bit of this automation and what not to clear my mind a bit, open up more time to actually really make sure I hit the schedule for my my soon to be 4 year old. I'm impressed. So if we have other parents out there who are wanting to get started, what are the baby steps?

Bill [00:18:21] That's a great question. For me, there's this great youtube channel called, I believe is his channel is called, Dad Verbs.Really highlights all the cool gadgets you can put in to your nursery. So that's where I got started but I kind of went full tilt on that.

Andrew [00:18:40] Wow. I'll have to check that out. We'll throw that link into the show notes as well for folks who want to check that out.

Alexa [00:18:47] Bill, do you know baby shark?

Bill [00:18:49] I don't think I do.

Andrew [00:18:52] That's good.

Alexa [00:18:54] Give it a couple years.

Andrew [00:18:55] Yeah. I'm sure it'll be different by then. So, I'm sure baby shark is like mainstream meme enough, if you don't know what it is it's this, I think it's from Hong Kong. Is it a Hong Kong or Taiwanese kids show.

Alexa [00:19:09] Yeah, I think so, my son loves it.

Andrew [00:19:10] There's this segment where like this kid - it's just a catchy song about baby shark, Grandma shark, momma shark, dada shark. And it just became this viral hit. I'm sure if you listen to it it would just destroy all sense of automation and data collection that you have going on with your kid. It's just one of those things, such as, it's a brain breaker but kids love it.

Alexa [00:19:33] They do. I saw a meme not too long ago that was like you know our kids are gonna be dancing in the club to baby shark like it's all of a sudden like some remix is gonna come on and all the young kids who are our age kids that are now club age. Yeah Baby Shark!

Andrew [00:19:49] Yeah you're probably right. Like some remix, the 20th year anniversary of Baby shark or something like that. I kid you not my my daughter jams out to this random Abba Mamma Mia workout mix. There are a couple layers here that we need to unpack. The first layer is that anything by Abba is more or less pretty darn catchy. OK. That's Level 1. You add in Mamma Mia which is you know love or hate that play. It's a play, right? I've only ever known it as a movie. There's an actual play. Which one came first? The movie or the play?

Alexa [00:20:22] I think the play came first.

Andrew [00:20:25] OK, and then they made the movie, the musical. Then another layer of just like aweome musicality via Mamma Mia the play. And you add this third layer where some geinuses out there turned it into like a 80s 90s style workout mix. 20 tracks of workout Mamma Mia. This is in the family car all the time. I know it. Track by track as is my daughter. It's incredible. She's jamming out to that. So I don't mind because it's I think it's better than baby shark, but hey, to each their own opinion.

Alexa [00:20:56] Yeah. I'd much prefer listening to I think Mamma Mia in the car than baby shark on repeat but occasionally you like the hand motions are kind of fun.

Andrew [00:21:06] They are. It's like that a Dada, finger Mama finger, where are you song. It's like the next version of that.

Alexa [00:21:17] Totally.

Bill [00:21:17] Sounds like I have a lot to learn. Or a lot to look forward to.

Andrew [00:21:18] Yeah I think when you're on the right track. Did you go too hard on automation front, you need to catch up culturally now.

Bill [00:21:24] I think so. Like after the baby we have not watched any TV shows. We're not even caught up with game of thrones yet either.

Andrew [00:21:32] Oh, no spoilers.

Bill [00:21:33] Yeah, no spoilers please.

Alexa [00:21:33] So I will share though my catching up on TV that doesn't get better, like my husband and I just, it took us three days to watch A Star is Born. We just finished it last night. So just a level of don't get your hopes up too much with catching up on shows and being on top of everything from that regard because that's still hard.

Andrew [00:21:59] So what you're saying is you've adopted an iterative approach to watching movies and shows post having children.

Alexa [00:22:06] Yes.

Andrew [00:22:06] Got it. Got it. See. Full circle. DevOps, everyone.

Bill [00:22:12] Andrew, I wanted to mention one of the other things I've automated in our nursery. My my daughter has sensitive skin so the humidity level has to be just right. I actually have a humidity sensor in the nursery or in our room that detects when it's either too dry or too moist and it kicks on and off the humidifier.

Andrew [00:22:33] Wow can you add essential oils too?

Bill [00:22:37] Maybe I'm not sure. She's allergic to any of the smells yet. That's something we have to check.

Andrew [00:22:40] I guess I was thinking of the full adult version like I would love that personally. Hey maybe that's our next startup idea, Bill. Keep that in mind. Right on so we've covered sleep, we've covered changing, we've covered feeding, we've covered heart rate, blood oxygen monitoring, we've covered sensitive skin and how to deal with humidity in the room. Bill Tang, what haven't you automated yet?

Bill [00:23:01] The actual food delivery to the baby. That's one thing I would like to do. Oh yeah.

Andrew [00:23:07] Like in terms of like weird possibly illegal means of doing it or it more of like something cute that you'd see out of like a Pixar movie?

Bill [00:23:17] No idea.

Andrew [00:23:18] Right On. Well maybe that's an episode for the future. All right. Well we've covered the entirety of how Bill Tang is automating the newborn experience. We'd love to hear more from listeners, feel free to hit us up on slack or on Twitter and if you've got any ideas or comments about that. If you'd like to check out Bolt, our open source task orchestrator, go on ahead to puppet.com/bolt. You can check out the rest of puppet at puppet.com. Otherwise thank you all so much for joining today, Bill and Alexa.

Alexa [00:23:50] Thank you for having me. It was really awesome to chat and listen to how Bill has everything automated.

Bill [00:23:56] Thanks Andrew for having me.

Andrew [00:23:57] And another big thank you to all our listeners out there. We totally appreciate it. And yeah we're going to try and add more segments on just the other ways folks are automating their day to day lives. A big part of Puppet is to eliminate soul crushing tasks and what not. And you know they're not always at work, right? So let's all find a better way to make our lives better. Be better to one another. And, yeah, take it easy everyone.

Andrew [00:24:21] When Bill isn't doing his product management duties and his parental duties he's probably working on his awesome talk for the upcoming PuppetizePDX. So for those of you who are interested, PuppetizePDX is our user conference on October 9th through 10th. You'll learn a lot more about it at puppet.com/puppetize. We hope to see you there.

You've probably heard that we've got a conference coming up soon. But have you also heard about the exciting additional events going on around Puppetize PDX? We're coming back home to Portland, and are super excited to show you some of our favorite parts of our city.

We've added a few extra events into the Puppetize PDX agenda and in this podcast, we'll talk with a small handful of Puppet employees who're helping facilitate some of those experiences for you. They range from integral parts of the conference, like the Birds of a Feather discussion sessions that Gene is facilitating, to the technical extras, like the Bolt workshop and the Hackday, all the way to the unrelated-but-still-fun Puppet-sponsored Portland Marathon team and the informal pinball tournament at Ground Kontrol.

Come along; we hope you enjoy the listen!

*Ben Ford is a software engineer and developer advocate at Puppet.*

Learn more

• See the Ground Kontrol facilities.

• See the Puppetize PDX agenda

• Read about the Birds of a Feather format

• Play with the self-paced version of the Bolt workshop

You've probably heard that we've got a conference coming up soon. But have you also heard about the exciting additional events going on around Puppetize PDX? We're coming back home to Portland, and are super excited to show you some of our favorite parts of our city.

We've added a few extra events into the Puppetize PDX agenda and in this podcast, we'll talk with a small handful of Puppet employees who're helping facilitate some of those experiences for you. They range from integral parts of the conference, like the Birds of a Feather discussion sessions that Gene is facilitating, to the technical extras, like the Bolt workshop and the Hackday, all the way to the unrelated-but-still-fun Puppet-sponsored Portland Marathon team and the informal pinball tournament at Ground Kontrol.

Come along; we hope you enjoy the listen!

Ben Ford is a software engineer and developer advocate at Puppet.

• See the Ground Kontrol facilities.

Ben [00:00:10] Hi everybody. Welcome back to pulling the strings here at puppet. My name is Ben Ford, I am our developer advocate, and today we're going to be sharing some of the ancillary events that we're doing alongside Puppetize. We've got Gene Liverman, Lucy Wyman, Nick Lewis and Ken Johnson. And I'll be asking them to tell me a little bit about some of the different things that we've got going on. I hope you enjoy hearing about them.

Ben [00:00:40] We've got Gene, who's going to be talking about our birds of a feather session. What is that anyways? Could you tell us a little bit about yourself.

Gene [00:00:48] Yeah sure. So I'm an SRE here at puppet. I work on the team that helps manage the infrastructure the engineers need to get their job done. So I work with the physical servers with our cloud environments, help manage vmware, starting to help manage Kubernetes and multitude of other smaller applications.

Ben [00:01:08] Right on, how long you've been doing that?

Gene [00:01:10] I've been doing it here with puppet for a little over two years and I've been working in systems administration kind of work for well over a decade now. Both on the Windows and Linux side. The place I came from a university and it was about 50/50 Linux and 50 Windows. So kind of doing a little bit of all that for quite a long time now and also working with some of the more esoteric things like Solaris and stuff like that.

Ben [00:01:35] That's pretty cool it sounds like you've got a lot of experience for like a lot of very disparate topics which is kind of what we're here to talk about anyways. Do you want to tell us a little bit about a birds of a feather session is.

Gene [00:01:47] Yeah sure. So I think of birds of a feather session to be kind of like a little roundtable discussion with some loose coordination added on to it. So the conferences I've been to that have had them before, there's a set of topics that have been decided on ahead of time and maybe one or two ad hoc topics based on presentations that were done at the conference that seemed to generate a whole lot of interest and in each topic somebody has agreed to kind of lead maybe in a very loose sense moderated a little bit.

Ben [00:02:18] Kind of facilitate -

Gene [00:02:20] Facilitate is a really good way of putting it. And that person will usually kick off the discussion with something relevant to the topic and then encourage other people to chime in either with answering a question that they posed or talking about similar things or just generally discussing it. Kind of an example as I went to a DevOps Days in Portland a couple of years ago. One of the Birds Of A Feather Sessions was owned using Power Shell as a DevOps practitioner. And so anybody who was interested in that general topic could come and sit in and we sat around and talked about some of the different ways people were currently doing things and the tools that they had found to use for it. And you know what went well and what didn't and had more than enough to chat about for the time period.

Ben [00:03:02] It seems like a really really good way of like sort of sharing ideas with each other and even generating new ideas. It seems like a like a what would you call it a target rich environment for ideas.

Gene [00:03:14] Absolutely. By nature of being at a given conference you've already got some level of mindshare or similar challenges that you're trying to solve or similar tooling that you're using you know all depending on what the conference is sometimes a little of all of that. And so people are kind of in the same world but they've also got lots of different experiences. And this is a way for those people who have gathered with those different experiences to be able to chat with each other for a little bit without the kind of one way nature of most of the presentations at a conference, it's really a dialogue.

Ben [00:03:46] I really like that a lot that resonates a lot with me it's like a like a peer to peer dialogue not necessarily like somebody telling you how to go do your work or what not. It's like the two of you coming up or multiple of you coming up with collaboratively better practices.

Gene [00:04:03] It's kind of like the preplanned version of the hallway conversations that you end up having with people at a conference.

Ben [00:04:08] That's cool. And you you mentioned topics a little bit and you said something about like ad hoc topics and something about like planned ahead topics what like how are you you coming up with this list of topics.

Gene [00:04:20] So for Puppetize PDX what we're looking to do is gather from the community some ideas for topics and have them vote on the ideas that are proposed and then whichever one seem to be the most popular talked about the top five and we'll point those out to have those as preplanned topics and then probably also pointing to have space set up so that if there is one or two additional topics that become evident that people are really interested in from say overflowing attendance at a session or people saying "hey I'd really like to learn more about this" we'll have space where we could set up one or two additional conversations also.

Ben [00:04:59] So it sounds like you're asking people almost to submit ideas.

Gene [00:05:03] Yes. That's exactly what it is. So we'll just ask people, say "are there things you would like to chat with other practitioners in the field about?" The Power Shell example is a really good one. But say if kind of playing off of that one maybe people who work with Puppet on Windows or just do Windows Administration in general, wanted to get together and talk with some of their peers because they've got challenges that they're hoping maybe somebody else has a good answer for. You know this would be a great time to come in. Maybe they propose to talk to say, "hey I'm trying to, I'm trying to figure out how to manage this set of applications." And that's their topic idea. And other people are like "hey you know, we run that. That's a big challenge for us too. I wonder if anyone elise is doing that." So they decide they want to vote on it. And a whole bunch of people get together and chat about it. Or a few people get together and chat about it.

Ben [00:05:54] That sounds great. So Puppetize PDX is October 9th and 10th. You know when you'll be have like putting these topics out for people to start suggesting or voting or whatnot on?

Gene [00:06:07] So we're looking to send out a survey to the community and posted on our slack and probably post it on social media sometime early in August. Probably going to use Google Forms. We looked at a couple of different tools and there's some other interesting tools in the space but either they cost a lot more money than the benefit you get out of them or they only kind of touched on the topic or touched on the problem space that we're trying to get information from. And turned out that something just a simple google form is going to be the easiest way to gather the info so gather that during August and then probably late in August, we'll send out a second survey say "hey these are the ones people seem to like the most or these are the topics that people submitted. What are your top one or two topics that you would like to hear?"

Ben [00:06:55] So voting like you adding your vote to.

Gene [00:06:58] So do voting kind of as a separate round since Google Forms doesn't allow you to see what other people have done and instantly vote on it. So gather in one round and then vote in a second round.

Ben [00:07:07] I like that a lot. So it seems very kind of grassroots it seems like exactly the sort of thing that an SRE would put together right?

Gene [00:07:14] Well that's the idea. That's that's what we're hoping for. And community involvement is key in something like this. Cause I mean the whole point of it is for people in our community to sit down and chat with each other. This isn't about you know, a puppet employee coming and saying this is what we're going to talk about. It's Joe Practioner or Jane Practioner or saying, "hey I'd like to talk to somebody about this" and then we talked about it.

Ben [00:07:35] Well then here is maybe a challenge like like you're you're a Puppet employee you're you're putting this together and whatnot but you're also like a practitioner you're you're somebody in the field doing this. So what would be like your topic if you had one topic to pick out and put on this list and vote up what do you think you would want to talk about and hear ideas from?

Gene [00:07:53] I think I would be most interested right now in talking with people that aren't working in a major corporation, how they're managing Windows stuff. I don't do much in the way of the Windows management since joining puppet but because of my previous role I know there is a tremendous amount of room for improving the management of Windows servers and for doing real configuration management on Windows. And I know a lot of it can be done but the documentation around it and the knowledge share around it is lacking compared to say doing the same thing on a Linux system. So learning from our community about what they're doing and then maybe helping spread that around a little bit.

Ben [00:08:33] That sounds like a great topic. I know several people who would love to be in that conversation with you, some with some really really good ideas.

Gene [00:08:40] And don't get me wrong I'm not discouraging the people from the big corporations from participating in that conversation. It's just interesting. A lot of times they've got some really great ideas but not everything not everything works the same when you've got you know 300 people focused on a problem is when you've got three people focused on a problem. Your solutions have to stay with the size of your staff that are managing it sometimes.

Ben [00:09:02] That's really true. And the solutions that are put together by somebody who's writing the documentation for a thing are not always a solution that a person in the field going "Oh wow, how does this thing work again?" will come up with.

Gene [00:09:13] Yeah absolutely.

Ben [00:09:16] Anything else you'd like to say about the birds of a feather session?

Gene [00:09:20] Just encourage people to pitch an idea even if you don't think anybody else would be interested in it. Feel free to propose it. You never know who's going to be interested in something. It's you can't ever really predict what application somebody is running in their shop, it may not even seem like something that's relevant but they may have some some interesting need that causes them to be doing the same kind of things you're doing and if you're trying to solve the same problems you can learn from each other.

Ben [00:09:44] That's true. There is no bad idea. And if your topic doesn't get voted up it just doesn't get voted up.

Gene [00:09:50] Yeah there's no there's no shame.

Ben [00:09:52] There's no way to get wrong.

Gene [00:09:53] There's no shame in that I mean, it doesn't hurt to put the idea out there and if nothing else maybe, as a byproduct of it, you might be able to find a - find out that there's other people in the community that are interested in it, maybe you can connect with them on the community slack as part of it.

Ben [00:10:07] That sounds great. Well thanks for chatting with us, Gene.

Gene [00:10:09] Thanks for having me.

Ben [00:10:12] I am looking forward to the birds of a feather session and I hope everybody listening is, too.

Gene [00:10:15] Cool, thank you.

Ben [00:10:16] All right and next up we've got Lucy Wyman. She's an engineer on the Bolt team. Could you tell us a little bit about yourself, Lucy?

Luca [00:10:22] Yeah hi everyone. My name is Lucy Wyman and like Ben said, I am an engineer on the Bolt team.

Ben [00:10:31] We'll be talking to her about the Bolt workshop that we're running the day before Puppetize. So what is it about this Bolt workshop that makes it valuable for people to come? What do you think?

Lucy [00:10:39] Oh my gosh. I think there's almost too much to say, like the Bolt workshop is a very comprehensive overview of everything you could possibly want to know about Bolt to get started with it. So it's about four hours, but there's a decent amount of breaks, you know a lot of interactivity. It's entirely hands-on. So it's composed of labs that you do to learn basic Bolt concepts. And by basic I mean all Bolt concepts like this workshop really covers the gamut from really early just running commands on machines all the way to applying puppet manifests without an agent, and doing really advanced Bolt work. So by the end of it you'll have a really good idea of like all of the capabilities of Bolt and how to do them and you should be pretty comfortable doing them yourself because you'll be going through all of the exercises and there'll be hands on support there from the people leading the workshop. So it's it's a really great way to evaluate if Bolt is a good option for you and everything that it can do.

Ben [00:11:43] Almost sounds a little intimidating. What sort of things do people need to know coming into it?

Lucy [00:11:47] I can see how it would be intimidating because it's so long and it's so much of Bolt but luckily Bolt is a really simple tool. So all you really need to understand going in is the concept of like remoting into another machine. So being able to SSH each into another machine and run a command. If you can do that then you can definitely understand Bolt and learn more about what it does.

Ben [00:12:10] Or WinRm. So you can use this of the same on on Windows machines. Yes absolutely. So what's an example of one of the first exercises that people might be running when they get started here?

Lucy [00:12:20] Yeah. One of the first exercises we have people do is literally just running ping on handful of Linux nodes.

Ben [00:12:28] That seems really really easy to do. Yeah.

Lucy [00:12:31] We also have one for on Windows nodes as well. So both platforms. Just making sure that the servers that we expect to be there are in fact there.

Ben [00:12:40] And you get that output back from the output of the Bolt run shows all the machines that are up and running and responding to your queries.

Lucy [00:12:47] Yeah, so bolt aggregates the output from all of the nodes and then outputs those from when you run a command. And so you can see each node and then what its ping status or ping output is.

Ben [00:13:02] So what do people need to do to register for this?

Lucy [00:13:04] When you're registering for Puppetize, you want to make sure that you click, there will be like a list of events that are happening right before Puppetize on that Monday. And one of them is the Bolt workshop. So you just want to make sure that you say that you're going to go.

Ben [00:13:20] That also seems very easy. I'm seeing a trend here.

Lucy [00:13:23] Yeah. I think if you're having trouble like automating across large infrastructure or even if you're just looking to do simple tasks on a handful of machines, Bolt is a really awesome tool for both of those cases. I think this workshop is a great use of time for learning how to make automating tasks and sharing those automated tasks with others that you work with a lot easier. So I think it's totally worth the investment of time and it will pay dividends in the time you save doing rote manual tasks.

Ben [00:13:56] That seems like a really good sort of value add to to Puppetize and doesn't cost anything so why not.

Lucy [00:14:02] Yeah totally. We've had a lot of really positive feedback from people who are already using Bolt. And so if you're really good about encouraging everyone to be able to go and learn for themselves if it's the right tool for them.

Ben [00:14:17] That sounds awesome. I hope to see you all there. All right. And next up I'll talk about something that I'm running I'm really really excited about. I've been doing a marathon kind of attached with Puppetconf for now Puppetize every year for quite a long time. And this year the Portland Marathon is kind of like reinvigorating itself with a with a new company and a new race director. And obviously we're sort of like Puppetize coming back home to Portland and we're sort of doing the same thing sort of reinvigorating and I thought that that was such a good match that I just sort of wanted to make this my race this year and I wanted to sort of invite community members.

Lucy [00:14:54] Yeah absolutely. I think running is like such a great way to build community and it's so kismet that the Portland Marathon is just the weekend before Puppetize this year and so it seemed totally perfect to have this like aligned, kind of goal of reinvigoration and then also just for the timing to be so perfect.

Ben [00:15:17] It's funny you say that I've been running for a very long time and I honestly I wasn't very, I was not in shape when I started. And it was I was sort of like brought into it by a friend who sort of bullied me a little bit into running a race with her. And it was really kind of empowering because I sort of discovered that this was this was not a thing that you had to like you didn't do it at anybody else's pace you ran at your own pace and you set your own goals and you set your own like limit. It's almost like building a sprint and whatnot and setting your own goals for that sprint. And it really worked well for me because I could decide if if I wanted to put in two minutes or whatever that number is. And it just sort of resonated. It's a great way for me to like think about the things that I'm working on and give me a mental space to come up with solutions where I'm not like actively staring at the computer and banging on the keyboard and getting more and more frustrated. It's sort of like meditation space a lot of people go out for a walk, I go out for a run.

Lucy [00:16:20] Absolutely. Oh my gosh I have so many thoughts running is absolutely a, like, I wouldn't call it an escape for me by a find that without running a search you just feel really like grouchy and claustrophobic and just really stressed like running is such a release for me of all that stress. And it's often where I have a lot of my best ideas. We ran like an internal Bolt Write the Docs where we spent a whole day just working on updating our documentation and thinking about how to improve our process and that whole workshop idea was born when I was on a run. I think it's a really great point that like you don't have to start being Usain Bolt like we - I think everyone has kind of this idea in their head of what it means to be a runner or what a runner looks like or what a runner acts like and they think well I don't run like that or I don't look like a gazelle as I run up the street and going to the marathon really makes you realize that there is no such thing as a runner or like a runner body type or a runner mentality or anything like that like if you're out there, you're an athlete, you're a runner.

Ben [00:17:37] I think it's kind of funny that the word Bolt has come up a couple of times and that sort of ties back in as I was coming up with the idea for this. I was thinking about what are we going to call this team. How is it going to work. And there was another tie in that I thought felt really well and that's Bolt is like Bolt is another name for running right? Running really fast and Bolt the product is a really fast kind of tool it's a fast way to get ramped up it's a fast way to get started managing your infrastructure. So we called this team Team Bolt and it is open for registration so if you go to the Portland Marathon web page you just say you're a member of a team type in Bolt and it'll come up and you can just select and join. We are providing running tees for anybody any attendees of PuppetizePDX who runs with a with Team Bolt. So just make sure you put in the email address so that I can contact you and get your t shirt size and everything and we'll get those printed up and ready for you as we get started with the race probably the night before or the day of. We'll figure that out.

Lucy [00:18:31] Yeah. We would love to have anyone join us. Like I think that it's easy to be like oh well it's for Puppet people and like if you're listening this you're a Puppet person - come run with us. You don't have to do the full marathon you can do the half. I think they might even have shorter distances as well. I don't think the distance you sign up for matters you can be a part of Team Bolt. And like we would absolutely love to have you participate with us. I think that's so much the point of running. It's not just the individual, it's like improving yourself, and like I said releasing that stress but it's also the community and the fun of like running with other people.

Ben [00:19:10] So you can either walk or you can run and there's two distances available there's the half and the full whichever one you select you can totally be just going up with a Team Bolt and run with us. I have put together a training plan. It's pretty easy to follow. It's actually even just a google calendar. So if you go to our Web site puppet.pt/PDXmarathon you'll find a description of like how the training plan runs what the different intensity levels are how to get involved and there's links directly to the calendars you can import them onto your own calendar and you can get reminders every day. And I'm also tweeting out everyday details about the run. There's a channel in our slack called very appropriate PDX Marathon where you can meet up with other people who are running and talk about it and just kind of share some of your wins and your losses and kind of encourage each other a little bit. And if you are hearing this late, then you can still get involved. You can still follow the training plan. Just don't like hop in and like start from today. You just kind of go back and look and see where you fit in your fitness level whether you're running a 5K or whether you're running a 10k now or whether you're starting from the couch and just kind of ramp up at about that same pace so that you get to the end at about the same time that we do. I don't feel like it requires a lot of fitness to begin with. I think that as long as you're running somewhere in between like a like a 5k which is about three miles or at 10k which obviously is double that I think that you are totally at a shape where you can join in and run the half. And like always I mean we have to put in this little disclaimer If you're worried about that you should check with your doctor and make sure. But if even if you're not if you'd like to walk it, if this sounds like something that you would want to get involved with. We would love to have you.

Lucy [00:20:50] Yeah all fitness levels I think should feel welcome like walking 13 miles or walking and running like you don't have to run the whole thing. I think all of that is totally doable. But again stay safe.

Ben [00:21:01] It sounds really intimidating. I think if you just look at the number 13 miles that's a lot. But in all honesty if you like if you've ever had like a step tracker or something and just like paid attention to how much you walk you probably walk about half of that any regular day. So it's sounds like a huge number but it's really just saying hey you know what. In this day I'm gonna do about twice what I did yesterday.

Lucy [00:21:24] I think the energy of the race really carries you too, like when I did my first half marathon I had only ever run 10 miles up to that point and found doing a half marathon like those last three miles felt I wouldn't say easy but it was definitely - but I don't think I could have run a half marathon just like on my own on a Tuesday morning. But at the race at the end like seeing everyone else running and hearing the music and getting the water and the gummy bears I totally had the energy.

Ben [00:21:55] When seeing everyone cheer as your go past, that's super energizing. My personal favorite is when the little kids come out they like stick their hands up and they all want high fives.

Lucy [00:22:04] Yeah. Yeah actually. So on my bib I put Batman last year and kids got really excited.

Ben [00:22:11] That's awesome. Well I hope to see you well there and I hope this has encouraged you a little bit to think about even just getting out and running and trying it out yourself. We would love to have you run with us.

Lucy [00:22:23] Absolutely. Run, walk, barrel role, however you want to get there.

Ben [00:22:26] And next up we'll be talking about our hackathon. The day before Puppetize will be running an all day hackathon here in the office. To help us with that, we have somebody here to talk about the hackathon that they did internally. Nick could you go ahead and introduce yourself. Yeah.

Nick [00:22:40] Hi Ben I'm Nick Lewis. I'm a senior software engineer on the Bolt team here at Puppet.

Ben [00:22:44] So yeah Nick how did how did that internal hackathon go? What what were you trying to get out of it?

Nick [00:22:49] So we're trying a couple of things. One is the hackathon in general and the other was the specific hackathon that we were doing this time. We're trying to start a program of doing periodic hackathons with the goal of taking some time every couple months or quarterly to step outside the box a little bit and stop focusing on our immediate work and look at other places that we can you know exercise our creativity or you know get an opportunity with a new technology or a new language and an opportunity to do something different and have a little fun. This hackathon in particular was focused on sort of flipping the bit on our open source contributions. So we we have a lot of open source projects that we steward here at Puppet and we get a lot of benefit from the open source community. And this was an opportunity for us to give back to the community in that way. So rather than working on our own open source projects we took the time to find other projects that are maybe related to our space or that we use as direct dependencies or even just things that people were interested in hacking on. And anyone who was interested could participate and we took a couple of days to try to find opportunities to make both code and non code contributions to various open source projects. So we had a number of people contribute to pull requests to a couple of libraries that we work on. Some people just did research on how to other open source projects run their pull requests their communities etc. And so I think we learned a lot about places that we can continue to contribute in the future and also what we got some experience and what it feels like to be on the other side as the Open Source contributor.

Ben [00:24:22] I like that a lot. I like the idea of participating rather than just being a member of the community. So moving on to our Puppetize hackathon once we heard about how successful this had gone for the engineering team we kind of thought that it might be really useful to ask them to help us out with this hackathon and to bring some of their expertise in running that. And so that's what Nick's here to talk about today. He'll tell us a little bit about some of the plans that we have for the upcoming hackathon. Maybe some of the ideas and some of the things that we're still working on fleshing out. So what do you think this will end up looking like, Nick?

Nick [00:24:55] So I think it's going to be a mix of concepts from open spaces and birds of a feather sessions with kind of an open space for anyone to talk about or hack on really any idea that they have. I think we will have tables set up staffed with employees for various projects that we have at Puppet. So if there's any, you know, if you're interested in Bolt or you want to know what Wash is or you want to hack on Puppet types and providers there will be someone there who is an expert in that topic who can talk to you about it and that might be you know bounce some questions off of them ask like, "hey why does it work this way" or "what if we did that.? Would it be possible to use Puppet like this?" Or maybe I have an idea for a new type that I want to write. Can you help me implement that or it might even be like I want to add this major new like integration point to a project. So whatever it is from like idea to implementation there will be people around who are interested in working on that with you or talking through it.

Ben [00:25:48] I really like kind of picture that it's pretty free flowing you can show up you can do what you want but there will be people there to support and help you if you need it or if you don't really know what you want to work on you can get a little bit of guidance you can gather and talk with people about ideas and sort of collaborate brainstorm on what possibilities are. The last config management camp I went to we actually ran a hack day the last day of the conference and somebody literally just sat down and asked Eric for a couple of ideas on how to do a thing. Some pointers to some docs maybe some questions that he could direct to engineers and came out with an entirely new feature for the puppet SSL SSH system. That was a pretty cool story and I kind of think that we're gonna get more stories like that come out of this hack day.

Nick [00:26:32] Yeah I agree. I think one of the things that we learned from the hackathon that we did internally is that people need some anchor points sometimes and I think if we say Okay come and it's a hackathon and you can do whatever you want then there's sort of the tyranny of the blank page which is I can do anything. First I have to figure out what I'm doing. But if you give people a little bit of a push which is like well here's you know a half dozen topics that you can discuss and you can talk about anything you want if you're if you're not interested in those. But go sit at one of those tables and start having a conversation. And we've done things like this in the past. At Contributor Summit some things like that and it's amazing how the conversations can evolve. People like someone will come up with maybe just a question the simple like how does this tool work. Like can I do X and then you'll give an answer and then someone else will be hey what if in addition to X you could do Y you go oh well that leads me to this other idea and then pretty soon you have someone start hacking on the thing that you just came up with.

Ben [00:27:28] And sometimes those questions are the most important thing. Like I've seen times where where an engineer was asked a question and they go oh hang on I never thought about that let me go write that down and then the you know a week or two or however many sprints later there's a new feature or something come out of it. So I'm really excited about this. I'm really excited to see what comes out. I'm excited to see if any new projects come out of this or if people just come up with new ideas and see what comes downstream. Do you have anything that you are hoping to see out of this?

Nick [00:27:56] I think the thing that I personally am most interested in these events is the conversation and what are the ideas that people are having? How are they viewing the tools? What do they wish that they could do or what what's the - it's exciting to see kind of the creativity that the users have for the tools that they're using. Beyond simply applying this to solve a problem immediately but I'm trying to look for how can I apply this these same patterns and these same tools to new problems.

Ben [00:28:24] It's almost like a giant Uback study just watching people try to solve a problem with the tool. Yeah definitely. So who do you think should come to the hackathon.

Nick [00:28:32] I would love it if everyone came to this hackathon honestly. I think that there's a lot of benefit that you can get out of it no matter what your what your relationship is with the projects or with the tools and with you know with programming even. Obviously if you want to come and literally hack on some code then this is a great event for you. But I think even if you want to come and say hack on ideas this is a great opportunity to come and talk to, both get some face time with the developers who work on the tools that you're using and to discuss problems that are relevant to you with other users who are experiencing those same problems.

Ben [00:29:03] Yeah with such a large open venue and so many people here kind of if it were milling around talking sharing ideas it's not like you'll be monopolizing somebodies time. It's not like a classroom where everybody has to stop and listen to you if you have a question. So yeah I think that sounds really great get everybody here. Everybody who is interested in hacking on things everybody who's just interested in ideas and people who'd want to just like pick the brains of our engineers. Well I'm excited to see what comes out of this. Thanks for coming in and giving us your expertise Nick and thanks for helping out.

Nick [00:29:33] Absolutely. Thank you Ben.

Ben [00:29:34] All right and next up we've got Ken here who's going to be talking about entirely different kind of experience here. So Ken what do you do here at Puppet?

Ken [00:29:43] So I'm a site reliability engineer here on the infrastructure core team. I herd a bunch of our servers for record virtualization behind our CI and development environments and do a lot of other ops stuff.

Ben [00:29:56] And what else do you do Puppet?

Ken [00:29:58] Yeah. So one of the other things that I do here is kind of like a hobby slash company cultural give that sort of thing is we have a corner of the office set aside for a few pinball and arcade machines that I've been maintaining here for about as long as I've been at the company.

Ben [00:30:14] Right on. That's pretty cool. And are these pinball games operational?

Ken [00:30:18] Yeah it would be kind of pointless if they weren't. It's all it's all stuff that I've generally bought as you know non-working project then restored and done the work to get them to spec again and brought them into the office and they're all playable and we've got a couple different groups of people who kind of you know will gather up at lunch for some kind of low key competitive place from time to time and throw a few games in between meetings stuff like that.

Ben [00:30:43] Well that sounds like a lot of fun. So if you've been following our podcast for very long you might remember Tim Vox Pupuli a project management coordinator talking about pinball at the Contributor Summit and he kind of like poked us a little bit to say hey that was a lot of fun. Maybe we should try doing that again. So we are. We'll be running a pinball tournament just kind of like low key after hours. Not very big kind of an intimate venue. Do you want to tell us a little bit about that, Ken?

Ken [00:31:09] Yeah. So it's going to be a small thing emceed by Erik Sorenson over at ground control which is one of the better pinball houses in Portland. They've got dozens of machines that they've been curating a pretty impressive collection.

Ben [00:31:22] Cool. So who do you think that this might appeal to is? Is this like anybody and everybody or?

Ken [00:31:28] Yeah I imagine it'll be like a similar crowd to some of the ongoing competitive events that run weekly in Portland. There's one called Flip city which is kind of a public tournament that has a lot of just walk-in players that they've never played competitively before and also has like the kind of people who will go on to like national competition and just everyone in between. But it's very accommodating and open to folks of all skill levels. So I'm hoping that's the thing we're putting together we'll have that same kind of vibe to it where you know it might be folks who play a lot or folks who have never looked at the machine before. But they can all come and you know have a good time a scheme where they end up on the bracket and playing some pinball.

Ben [00:32:06] That sounds really neat and I hear that there's something interesting about the software that Eric will be using to run this tournament.

Ken [00:32:13] Yeah. So it's called brackelope and it's something developed by a longtime pinball scene person in town. Isaac Ruiz he put it together I believe to run pinball tournaments initially but it's also ended up being used for all kinds of kind of bar League competition stuff if you like it like the tournament list on there there's hundreds of events now it's kind of fun to go through and see what else people have you know adopted this thing for. But he's really good guy. I see him a lot a lot of tournaments both as someone who who's running things and also you know square up against him. And it's always always good to run into him.

Ben [00:32:50] All right that sounds like a lot of fun. We'll make sure to get information out about the pinball tournament so you know when and where and how to get there. And I hope to see some of you there.

Ken [00:33:00] Yeah. Likewise it should be a good time.

Ben [00:33:02] All right. And there we have it folks a whole list of interesting fun exciting events that will be running alongside of PuppetizePDX PDX. We've got the birds of a feather session that Gene will be coordinating. We've got a a Bolt workshop the day before and we've got the Portland Marathon that we will be running in the weekend before. Nick Lewis is helping us do a an all day hackathon here in the office. And then there's the pinball tournament after hours down at ground control, nice little intimate venue. Maybe somebody will come out of this being the pinball champion. You can find all of this and more at the Puppetize agenda. Puppet.com/Puppetize. And then just click on the agenda page and you'll find all the information that you need. And with that I'll be signing off. And I hope to see you here.

Sometimes big challenges need to be tested and solved for at a much smaller scale. In this case, it’s the 1:87 model train scale with an interesting experiment coming out of Puppet.  Learn more about a new way to bring people together and automate with Bolt.

On this episode, Engineers Cas, Ken, Jacob, and Managing Editor Andrew chat about “bolting” remote task orchestration and automation onto a HO-scale model train set. The vision: sending Bolt plans to operate a model train at Puppet’s Portland headquarters.

Hear how this cross-functional team solves for the automation of remote devices, navigating network security requirements, possibly provisioning containers, and (most importantly) building a fun project that brings people together.

Andrew [00:00:00] Hey everyone. Welcome to today's podcast. It's a special one. We're going to talk to you today about the Puppet at scale project because sometimes big enterprise class challenges need to be solved at a smaller scale. In this case we are throwing Bolt one of Puppet's open source tools at an HO scale model train set. We'll be covering who is involved, we'll get to know some of the crew, we'll talk through some of the interesting discoveries, challenges, get a sense of where this project is heading and let you know how you can keep tabs on it. So first and foremost let's meet the crew.

Cas [00:00:49] Hello everyone. My name is Cas Donahue. I am an engineer on the Bolt team here at Puppet.

Ken [00:00:55] Hey my name's Ken Johnson. I'm an engineer on the infrastructure core team here at puppet.

Jacob [00:01:00] Jacob Helwig Coremunity team.

Andrew [00:01:02] And I'm Andrew Nhem, managing editor of puppet.com. We are an interesting crew, folks from different teams and whatnot coming together to work on this awesome project. We definitely want to give a shout out to folks who aren't at this table who are also giving a lot of time and energy to making Puppet at scale happen. Those are folks on our business technology operations team, the workplace team, our greater community, more engineering folks and the marketing team. So what is puppet at scale for those of you who have been paying attention, one of our open source projects is called Bolt and, Cas, Bolt is, you probably have the best definition of the four of us, go for it. Give your plug.

Cas [00:01:38] I think for this audience when I describe Bolt I'd like to frame it in sort of how it compares and extends Puppet. So with Puppet we have an amazing way to declare state and make sure that it's continuously enforced. Sometimes you need to do an imperative action, one off actions just to do a sanity check on something or something needs to happen right away. So what Bolt does is it allows you to make those imperative actions happen in a way that integrates well with what's happening with Puppet. We can affect change or make queries about your infrastructure using Bolt and we can use, we can talk to those pieces of infrastructure with different transports so instead of just the Puppet transport we can use SSH or WinRM or others. This allows us to have a bit more freedom and to play around and actually figure out how to solve things. The other great thing is that sometimes Bolt will allow you to orchestrate things by doing things in defined order and also allows you to declarative changes as well and keep your infrastructure in a good state.

Andrew [00:03:05] Thanks Cas, that wass spot on, definition of Bolt. Throughout the year we've been running workshops, spreading the good word about Bolt all around the world and randomly around the March timeframe in which every technology company is wondering what funny clever things should I do for April Fools, A group of us got together and you know those of you in the community, you know Nigel and Ben Ford they randomly kept talking about solving Puppet problems at a smaller scale. They kept having this idea around model trains and we'd be like yeah okay whatever. After about the fifth time that was brought up someone in a brainstorm session made the connection with Hey why don't we, could we possibly tell a train what to do with Bolt tasks? Obviously at that time the answer was I have no idea. And those are often the fun things to software: we have the time and at the time in March had no idea this would work but there is a lot of energy in forever around it and we kind of knew in our gut that we had to try to figure it out.

Ken [00:04:07] Yeah that's kind of a an idea that's been kicking around Puppet for a long time. I remember several years back when we moved from when we moved into the Lovejoy office I want to say, someone used their part of their departmental redecorating budget to buy some stuff to set up a model train set but it never really got off the ground then. But people have been saying we should do it for a while and it's really neat that we got the opportunity to do that and integrate it with one of our products.

Andrew [00:04:34] Yeah that's going to be neat. Whatever happened to that train V1?

Ken [00:04:40] You know I think it kind of just disappeared after a while of never being assembled. There were a couple of projects like that that sat around for a while collecting dust and then didn't make it through the next move.

Andrew [00:04:54] So hopefully we can make train V2 stick around.

Ken [00:04:57] This one already looks like it has a lot more traction to it.

Andrew [00:05:02] Well I wasn't really upon the traction track close enough. Whatever. For those of you listening I will throw in a couple of images and whatnot into the posts that holds this episode so you can get a sense of where we're at. Definitely work in progress. But a fun one at that. What's been pretty neat is that you know you probably have seen Puppet's State of DevOps Report. You've seen all the work that we do to empower DevOps managers or teams to do more to improve their agile practices. This was a pretty neat, different way to live a bit of that. We're bringing different teams together. We're trying to solve a big challenge together. And like Ken said it helps us integrate our into some of our new products and solve business call. And we want people to try Bolt and learn how to use it, and what better way than irreverently through a model train. Specifically HO scale. For the folks listening who think that all model trains are Fisher Price size that's definitely not the case. Can you weigh in a bit on how broad this hobby really is?

Jacob [00:06:04] A massive hobby. I mean there's, so you mentioned we're using an HO scale train. It's a good size and a typical layout on those is like maybe 15 by four feet. So they take they take up a fair amount of room to do like a small town or a couple of small connected towns. But I mean model railroading itself there's ones where a typical locomotive is like the size of your index finger as far as how long it is all the way up to ones you can ride around in people's backyard gardens. So there's a huge variety of scale and complexity in Model Railroad layouts. We're doing a pretty simple one to start out with because we wanted to have as few variables that we had to work out the kinks in as possible.

Andrew [00:07:03] I remember when we were first doing discovery on this project like not even the technical requirements we wanted to get some advice from Portland's local model train club, and this is a serious hobby. I mean they were actually concerned that we were going to start it incorrectly. We approached this career I think a handful, of Ben and I believe A.J. Johnson one of our Engineers, went over to give some advice and I mean they're also just very enthusiastic. I think they wanted to get involved as well but it is interesting to hear the reporting of hey you know make sure you kick this off properly and four months into it, I kind of get the sense of why they gave us that warning. Things are going smoothly but we've been learning a lot of things. The most surprising thing for me is obviously we've been testing the train and we've been experiencing some derailment. And obviously we're fixing that you know yay iderative improvement. Whoo. But I was surprised to see was a site that we found on HO scale train requirements. And at first glance it actually was well styled and written, I thought it was a like a real dot gov like Metro site.

Jacob [00:08:07] Oh you're talking about the NMRI the uh...

Andrew [00:08:10] Yes. Like I thought I was reading a site for civil engineers or like train engineers to like actually like architect real deal trains and layouts throughout like a metro area. But it's a well done model train site. This is how serious this is.

Jacob [00:08:24] Yeah, oh sorry. Not NMRI an NMRA, National Model Railroad Association. They have a lot of really good and really in-depth recommendations for how to do things at scale so that they look realistic. And it's kind of mind boggling the level of detail they go into in their standards and specifications.

Ken [00:08:46] Yeah I kind of like that we're going a slightly more whimsical with ours just the wildly out of scale bear stalking the countryside around the layout, kina. I just really appreciate that.

Andrew [00:08:58] Yeah, there's like a district of the train track that's like populated by Happy Meal toys I believe from the last 20 years. Someone donated all those

Ken [00:09:04] Oh man, I'll gonna have to go on eBay and get some good ones.

Andrew [00:09:07] Yeah. I think my mom might still have like all of ours in a hutch somewhere in her house.

Ken [00:09:14] I remember those like weird Transformers ones, so you would have your burger backs that turns into a pterodactyl and stuff.

Andrew [00:09:19] There you go.

Jacob [00:09:19] Like a box of fries.

Ken [00:09:21] Yeah that one.

Jacob [00:09:22] But yeah I definitely agree. I like that we're not doing like this is a hardcore super realistic model railroad layout. So yeah we're ignoring large swaths of what it NMRA has to say. But at the same time they have some really useful things like here's recommended wire gauges for doing the track wiring and here's recommended radiuses for curves to be able to do this type of railcar and have it be somewhat realistic. And probably not derail.

Ken [00:10:01] Mm hmm. Yeah. We've got the rail model railroading equivalent of best practices document for us setting up some new systems. It's very handy. Make sure we don't trip over our own feet. Trying to make this thing happen.

Andrew [00:10:15] Yeah we're doing it the right way but in a very welcoming inclusive way which I absolutely love seeing for those of you who've been paying attention at Puppet this year, you know we've got refreshed leadership. There's definitely been a resurgence of like wanting to work together towards toward big company goals. One of those goals is to get folks around the world to know about Bolt. Yeah sure we can come up with more workshops and webinars and different ways to promote ad campaigns and whatnot. We wanted to try something different. And it's been pretty magical to watch different teams come together to A. try and complete this train set for those of us who don't have the engineering chops then also B. To see folks come together to try and make this happen. So I'd love to hear around the table you know where are we at now what have you learned and what's keeping you excited about this project?

Jacob [00:11:10] Mostly I've been concentrating on getting the lay out talking to a computer so that we have some API to interact with from the Bolt side. And it's been really interesting learning what what software is already out there for doing things like that. So we're we're using a system called DCC, digital command control, for doing model railroad control and it lets us have multiple locomotives on the layout and control them independently instead of the really old classic straight up DC system where you just apply DC voltage to the rails and whatever voltage you apply is however fast whatever motors are hooked up to those rails is going to run. So DC lets us do things like have multiple locomotives on there running at different speeds in different directions and all of that, control switches, control lights and all kinds of other things from a central point. So from there we go into a computer which is running JMRI, which is one of the software packages that lets you do software control of a layout. And it's been interesting seeing exactly what the mindset they expect a user to have when interacting with it. They very much expect you to be running a scale railroad, so everything is kind of pre laid out where you have from point A to Point B is something that's predetermined because you have two industries that are connecting with each other and you want trains going between those two places. So there's not really a whole lot of on the fly dynamic creation of things like that. So one of the challenges we have ahead of us is figuring out how to get something that can do a little bit more of the dynamic route generation and scheduling of things. Talking with JMRI so that we don't have to build out a table of every possible point A to Point B combination and basically so we don't have to have people who are going to be interacting through this with Bolt have absolutely intimate knowledge of the layout itself so that they have something a little nicer to work with. So yeah that's that's one of the big challenges we've got ahead of us and what we've got going on.

Cas [00:13:37] Yeah in that sort of vein from from understanding how we would put an interface to this using Bolt, something that has become pretty clear to us is the system that controls the train is a very good use case for the concept of a remote transport that Bolt has. So when you have things like a network device or an HDGP endpoint, those things don't necessarily have a full shell environment like a server or something like that. So what the remote transport allows you to do is to run the logic bits on a proxy device. So if you had a fleet of network switches that you needed some kind of configuration on you could use the bolt remote transport to manage those devices from a proxy with just by using the endpoints that they have defined. So something that we're investigating is where do we put in the declarative, this needs to be this way, perhaps the layout of the switches on the track or where the train should be. Versus things that we just want to happen quickly like sounding the horn or turning on a light. So these problems are a perfect example of how DevOps people would be needing to control things that are both have a declared state on them versus stuff that they'd need to just do one off quickly and also to use something that is not traditionally a node in that it would have its full shell environment and you could interact with it with like package managers or service managers or something like that.

Andrew [00:15:36] It was cool to see the whole company get together, what we did was you know we didn't promise any kind of operational train but really like hey come hang out this come add some crazy art. Put some model train stuff together, paint the train. It wass really cool to see folks of all walks of life and all these stories pop up too. Apparently our CFO is huge into model trains, walked around upstairs, the sixth floor at Puppet HQ. Well that's misleading we only have two floors. I made Puppet HQ sound like massive.

Ken [00:16:10] Our downtown Portland Campus.

Jacob [00:16:10] Seven story Builiding.

Andrew [00:16:11] Yeah. So it's been really cool also is to have the blessing of the CEO. Yvonne was at the train lay out a couple of times taking pictures of the train. She really wants to learn how to use it apparently so she can impress her kids which I thought was pretty cool.

Ken [00:16:25] Yeah that getting the blessing of the CEO part is always a little bit important for these kind of little side project things. I was one of the instigators behind having arcade machines in the office and I was always a little like lukewarm on how I felt that company leadership thought about that until one day I was playing Samurai Shodown 11 on the Neo Geo and Luke Kanies walked up and tapped in on player 2 and just like spent 20 minutes kicking my butt in that game because apparently he had grown up playing that in arcades.

Andrew [00:16:59] A new challenger appears.

Ken [00:17:00] Right? It's true and I didn't expect it. It came out of nowhere but after that I figured he was probably pretty warm to the idea of having that stuff around. It was kind of fun.

Andrew [00:17:09] We'll throw a picture of the Puppet arcade did that thing is going strong. And I see it's pretty cool to watch people of just all walks of Puppet life come in and hang out and play. It's funny to see all the guests ask if they need quarters.

Ken [00:17:25] I found a few in a few of those machines. Every so often you'll get 50 cents in it and it's like oh someone's kid must have come in.

Andrew [00:17:31] Does that go to your maintenance fund?

Ken [00:17:32] Yeah. The arcade slush fund.

Andrew [00:17:36] But it's been great to have people ask questions and what not. We've had a couple of guests and whenever they walk by the train you know they always asked What is this what do we do with it and by the time we tell them the whole premise of what we've got going on what the big vision is they get really excited. Especially when those guests are Bolt users. We had a customer in last week and he was looking giddy and I'm like Hey this isn't ready yet. We'd love your help when it's done and like you know you can feel free to kick it around and test it but wee're not there yet. But he definitely wants to keep tabs. But yeah, International Train Day or the day before. It's just great to see different folks from different teams you wouldn't normally see. There was lot of painting going on. Someone had some like 70s jams going.

Jacob [00:18:18] It was actually a good time along the lines of the painting and the art. I really like a lot of really like a lot of the graffiti style art that people have been doing on the box cars and things like that. It looks it looks authentic without being like overly pedantic about what goes on there. It just it fits really well. I just think it looks really nice.

Ken [00:18:44] I like it. We get to see firsthand the flexibility of the DAG placements and layout designs. I love that we have a whole design document on that like with the rules of how an extension of the DAG logo can flow. It's kind of kind of interesting but it's clearly something people kind of intuitively grok and understand how to extend it in a way that makes sense. I don't know haven't have any of you read that thing when it launched?

Jacob [00:19:14] I saw the presentation when they were talking about it but I didn't dig into it. My art skills are anemic at best.

Ken [00:19:23] Mine are too, but it was fun to read this like clearly super thought out doc from people who like you know think about this stuff professionally and like you know for for being something that you know to someone who's not initiated into you kind of thinking about design and that way they like might kind of mock it like Oh you paid them how much to design that thing but then you read it and read the like thought process that went into it and kind of the way it's been that this simple image has been articulated in words and the instructions to expand it. And it's like Oh actually that yes that is that is worthy of the stature it has as it's own like special discipline.

Andrew [00:20:02] And the dag that we're talking about is the logo of Puppet. Dag. I always forget with the D is.

Ken [00:20:09] Directed acyclist graph. Yeah yeah.

Andrew [00:20:12] Directed acyclic graph. Check it out. Or just go to puppet.com you'll see a dag. You'll see many dags. I'm totally looking up train trivia right now. Did you all know that the term horsepower originated as a marketing tool.

Ken [00:20:25] That doesn't surprise me.

Andrew [00:20:27] Then back in the day when America was first testing locomotives. Guess who won the locomotive or the horse. It was racing against.

Jocob [00:20:34] I'm guessing the horse because the train broke down.

Andrew [00:20:38] Definitely the horse won.

Jacob [00:20:39] Maybe if it was a role race.

Andrew [00:20:41] And apparently the travel agency industry was kickstarted by a train trip.

Jacob [00:20:47] Yeah we have railroads to blame for time zones.

Andrew [00:20:50] Yikes yeah. The railroads also gave us standardized time zones.

Jacob [00:20:57] One thing that I actually found really amusing doing a lot of the train research is that a lot of model railroad information out there refers to prototype railroads. Prototype railroads are not the HO scale or the end or the Z or any of that. They are the full size ones that you get stuck behind at a railroad crossing. They refer to those as prototype railroads. They run the real railroads at their smaller scale. So I found that to be greatly amusing when looking through all of the documentation and everything else. Like I kept seeing this like prototype railroad prototype railroad. Like what are they talking about. And then I finally found something that actually put two and two together for me. It's like oh they mean the the real ones.

Andrew [00:21:43] Because it's easier to scale down right.

Jacob [00:21:46] Yeah I mean in this case, but they look at it as like the prototype railroads, the real railroads, went out there and they figured out all of the things that don't work so that when they go run their railroads on their layout they don't have to go relearn everything that the prototype railroads figured out. But also like with the expectations that JMRI has, like they expect like the typical thing that people do with Model Railroad layouts is they run a railroad. So they might have other people over and operate on that railroad as well. So they've got a dispatcher and people running individual trains and things like that. So they're running a railroad on the layout. So that's why all of JMRI expects that you've already you already know what the industries are you already know everything else you already know all the interconnects that are gonna be happening. So that's why they just kind of like well you just predefined those and then you'd say that well train number whatever is going to operate on that route or that transit right now. So you don't have to come up with one on the fly.

Ken [00:22:54] It's kind of taking this thing that in the real world is already pretty regulated and regimented and then like boiling it down and getting this even more perfect expression of the relationship between these things in the physical world and what's connecting these industries across this super complex Infrastructure we built for it.

Jacob [00:23:13] Yeah pretty pretty much exactly. And because that's not what we're trying to do. We're running into all kinds of edge cases where it's like well but we don't know where a person is going to want to send a train.

Ken [00:23:26] Right? They have no way to express the fact that someone might just decide to say Oh actually I wanted to go the other direction now midway through the run.

Jacob [00:23:34] Yeah. That's that's just not a thing that you would do on a railroad.

Ken [00:23:38] Yeah.

Andrew [00:23:39] But we're running into edge cases together.

Jacob [00:23:44] Yeah. A lot of it is like a lot of the challenge going forward is just figuring out how to smooth over those things so that people don't have to have intimate knowledge of our layout, intimate knowledge of how railroads work, intimate knowledge of all of the little sections of track and how they're labeled in the system. We don't want someone having to download a several hundred page PDF with different combinations of labels on it just to figure out like all right, so how do I get the train to go from this corner of the layout to that corner of the layout.

Cas [00:24:14] Yeah. What comes to mind for me relating it back to Puppet is something we haven't talked about yet which is Puppet as the sort of enforcement tool is one thing but then there's the module content available on forge which takes complex systems like a load balancer or a web server and we write an abstraction over it that anyone can use. And with everyone sort of like combined knowledge about the thing we make something that anybody can come and just set a couple parameters and boom they have it stood up. So it's really a nice sort of parallel here. We're taking a very complex piece of software and writing an interface to it, allows you know, just someone who's never operated a model train even with a controller let alone a computer, be able to issue some simple commands and see how that works.

Andrew [00:25:09] That's a great segway to a question that I want to ask you, Cas. I'd love to hear your thoughts on you know let's say and this is we're a bit a ways from this this greater vision near end state. But tell the listeners a bit about you know where you see this going, how you want folks to interact with this, with our model train.

Cas [00:25:31] Sure. So I would love for existing Bolt power users to find it interesting enough to write beautifully complex code that does amazing things. And I'd like it to be as easy enough that someone who's never even touched Bolt could install it and at least make the train go around the track. And so I think that one of our sort of design goals is to make this a a good reason to understand sort of how, why Bolt is so powerful. We'd like to use this as an opportunity to illustrate for both technical and non-technical users and a really powerful important part of bolt which is to effect change on systems that aren't classically a full functioning computer with a shell environment or a server. This could be like an HDPN point or a network device. And by relating that to a train that someone can see and anyone can understand. If I do X, Y happens. That will allow them to see, sort of make that comparison in their mind that while this tool can help me control things that aren't typically controlable by your traditional enforcement models or full, I guess computing environments. So yeah I think that the design goals are to get people familiar with Bolt and show them something that they can really start to appreciate what the tool can do for them.

Andrew [00:27:23] It's awesome. On the topic of learning what's been pretty exciting to me to watch our different teams. You know originally this this case was hey we want folks to learn about Bolt in a new way, a different way, a fun way. That was our only ask and we came up with all kinds of other use cases in which folks wanted to solve their very Puppety way. For instance you know in order to fulfill that greater vision we've got a couple network security hurdles to overcome. And it was awesome to loop in our business technology org, BTO, as it's known internally and they're all about it. They wanted to know what we're up to, what's going on. They're on their way to solving that problem.

Ken [00:28:03] I'm mostly just excited to see, as Jacob is and has been working on what the interface between Bolt and between this digital controller ends up looking like just because that kind of integration stuff is sort of the bread and butter of what I do in my own role and also generally the interesting part of the problem to me. So that'll be neat. And also just like once the thing is operating and you can throw a bolt plan at it and have it move around and do neat stuff like what's the next steps from there. Add on some additional things besides the training that you can you know execute these plans against and kind of make the landscape change around it. And also just like what is the artistic expression of the crew here at Puppet going to look like as we continue to build up the layout and apply everyone's you know talent and vision to creating the set around it.

Jacob [00:28:52] Might be a spoiler alert but the answer to one of your questions is yes we are actually planning on having like activate-able scenery and all kinds of other things like that.

Ken [00:29:00] Cool. Yeah I thought I thought I had heard of that and probably talked about it with a couple people.

Andrew [00:29:05] Yeah I heard the rumor of that sea monster or something like the Kraken or Cthulhu depending on which giant sea monster you subscribe to.

Jacob [00:29:15] I hadn't heard that one but that'd be cool to see.

Andrew [00:29:20] Popping up and saying hello. On the note of expression I'm pretty stoked from a communications standpoint to find different ways to package, it is probably a longer term goal but, a new way to teach folks about Puppet. In this case Bolt. Yeah you can sign up for training. Yes you can read our many blog posts. We're very education oriented here. You can check out the docs and get started in the plethora of ways. But this is just a different unique way to do it. It would be so fun to have classes predicated upon Hey can you send a task and our plan to tell that train what to do. Or perhaps tell the kraken to appear.

Ken [00:29:52] Yeah I think that kind of thing is super helpful to have, you know kind of a physical expression of what you've done in this digital format. Especially just with technologies like like ours and within the space we operate in. You have these super super complicated things you know hidden from kind of direct human understanding behind many layers of abstraction and you know being represented in this way that is you know somewhat unnatural. You know typing things on the screen. It's not the most intuitive. So being able to have that translate back into output that something that is very much in the physical world and you can you know see and touch it and observe what it is doing in reaction to what you've typed into the computer, that's a super powerful learning aid and way to you know make it clear hey actually you know this is still real stuff even though it's happening in a data center somewhere.

Jacob [00:30:48] Actually one thing that we haven't mentioned yet but it's kind of important is how people would know that what they've said to bolt or through to the layout through bolt it's actually doing what they wanted to do. And I mean we're planning on having a couple of cameras pointed at the layout with a livestream set up for it. And I mean there's a lot of details to work out but the idea is that there will be a livestream of the layout with some sort of identifier so that you can know that it is in fact your commands affecting the layout right now instead of whoever was playing with it just before just after you. The goal is eventually to have a way for you to see definitively that what you're doing through bolt is having an effect on the real world train layout.

Andrew [00:31:37] Perfect segway into us plugging where you can go to. For now just stay tuned and get some updates on what we've got going on with this Puppet at scale project. You can always check out the latest footage that we have from builds, pictures, commentary, what not, just go to puppet.com/puppet-at-scale. And with that I think we've covered all we need to today. There's so much more exploration to go. Cas, Ken, Jacob thank you so much for taking the time to hang out today on the podcast. Folks internally, externally, we're all really excited to see where this goes. I'm already proud of the crew. I mean just this is the first company I've been out where like different departments are all coming together. You know sans project management. There's no Jira involved. It's just straight up. You know let's hang out together, let's be accountable. Let's do some cool stuff together. It's been pretty awesome to watch and I'm looking forward to how the rest of the year plays out. We'll have another episode I think when we hit a new milestone and you can kind of keep folks updated on what's going on. But yeah check it out. Puppet.com/puppet-at-scale. You wanna learn more about Bolt, head over to puppet.com/bolt. And with that everyone thanks for listening. Have a great day.

Andrew [00:33:00] I keep calling the podcast "The podcast" but it actually has a name now. It is called pulling the strings and you can find it at puppet.com/podcast. You'll see all of our more recent 2019 episodes, some older ones and you'll start seeing that name throughout different ways to find our podcast iTunes etc..

Links to websites we mentioned: National Model Railroad Association and JMRI.

Andrew Nhem is the managing editor at Puppet.

Last month, we spent two and a half days in Budapest with some of our best community friends. I think the summary of Contributor Summit was the one common complaint I heard:

"There was too much good content and it was really hard to decide between workshops!" said everybody.

That's a really good problem to have and is the hallmark of a great experience. The key was in how the event was organized. Rather than tightly control the messaging, we simply asked our facilitators to build workshops that spoke to them and to just bring their best selves to the table ready to collaborate.

This was part of our new focus on the community contributor experience. The whole Summit practically crackled with electricity and what felt like every attendee came up to tell me how productive it was for them.

In this podcast, we talk about what made Contributor Summit special and some of the lessons we learned. We spoke with Tim Meusel, better known as bastelfreak, about the critical role that Vox Pupuli plays in our ecosystem. We even talked about marathon training, dispelled some misconceptions that Ben had about his mom's cooking, and established a clear winner in the German vs. Hungarian goulash rivalry.

What else? Well, you'll just have to listen to find out!

• Have opinions about the best goulash recipes?

Andrew [00:00:19] Hello everyone. Thank you for joining us today on Pulling the Strings. Today's episode we'll be going over the Contributor Summit that just passed recently — what went on, what people did, what people learned. And we have a couple special guests in the house. Feel free to introduce yourselves.

Ben [00:00:36] Hey everybody my name is Ben Ford here at Puppet. I'm our developer advocate, community manager, something like that. I mean I just do cool stuff here and I talk to a lot of cool people like Tim who is our other guest on the line.

Tim [00:00:49] Hi, I'm Tim. I'm a DevOps engineer at GoDaddy in Cologne, Germany and in my free time I spend a lot of time working on the Vox Pupuli and managing the Puppet community.

Andrew [00:01:02] Huge thank you to you Tim for making the time to join our podcast today. I know it's late over there. Are you well caffeinated? Are you ready to do this?

Tim [00:01:09] I'm 100 percent ready. Yes.

Andrew [00:01:11] Awesome. Before we dive in to post-Contributor Summit details I need to know what was the goulash game like in Budapest? Tell me all the details.

Ben [00:01:23] That's right. I told you about like the bike tour that like toured across Budapest and like sampled all of the different goulashes everywhere. And I'm sad to report that I never did make it to the bike tour, but I did try a lot of goulash and...

Andrew [00:01:38] It's still a win.

Ben [00:01:39] Still a win, always a win. We just like evolve a little bit like we change a scope and that way we can still call it a win, right. So I learned something very important about my own heritage I think and that is that what I grew up knowing as goulash was not actually goulash. They just call it beef stew over there. So goulash is one kind of soup and beef stew is another kind of soup. But the thing that my mother called goulash was just beef stew.

Andrew [00:02:12] Which is different than 'Murican beef stew.

Ben [00:02:14] Well yes very different from American beef stew. There's a lot of paprika in it. And it was delicious. It was absolutely delicious. And I actually learned that goulash is maybe not my thing. I mean it was all right. But but beef stew, that's my jam.

Andrew [00:02:29] Right on, right on. What about you Tim? Did you get in on any goulash slash beef stew?

Tim [00:02:34] Yes I tried a lot of it in the old town. We have a lot of different goulash sorts in Germany. They are very very similar and not much different to learn. But it's always nice to get to a different country and enjoy the different ingredients that they put into and how they spice it. Can highly recommend a trip to it.

Andrew [00:02:57] Right on. So is there a goulash turf war, kind of like in America we have like New York style pizza versus Chicago deep dish. Is there like German goulash versus rest of world?

Tim [00:03:09] Oh that's that's a very good question. We have some German specialties. Yes it's true. They add a huge amount of wine into it and cook it for a very long long time.

Andrew [00:03:23] You've won me over.

Tim [00:03:24] People've told me that that's the German version.

Andrew [00:03:26] I'll take it. Sounds delicious.

Ben [00:03:28] You've already got the points in Andrew's book. I think we have declared a German goulash as the winner.

Andrew [00:03:33] All right. And if you want to contest that you know where to find us on social media. And oh yeah, by the way, I'm you're easily distracted host, Andrew Nhem, managing editor of puppet.com. So while my heart wants to spend 90 more minutes, 120 minutes, who knows the rest of the day - yeah, and stomach - we want to talk about goulash. Let's get back to the main topic here which is to break down what went on at Contributor Summit. So guys tell me a little bit about the event. I didn't go. I had to stay around in Portland, you know, look after the show, but I would love to hear from folks with the feet on the street.

Ben [00:04:07] You really missed out. It was an incredible event in an incredible place. We had practically a whole week of people who like shared a lot of ideas and had a lot of things to talk about all in the same space. Some of those ideas were shared, some of those things were talked about like on the floor at Contributor Summit itself, but we also we went out for dinner every night and we hung out for another day afterwards and we played pinball. Tim can probably talk about the pinball museum in a little bit. And we just we spent a lot of time just sort of bouncing ideas off of each other and just like that's one of the things that I feel like is just incredibly important to building a strong vibrant community.

Andrew [00:04:54] That sounds amazing. Tim, I'd heard you know back here at Portland headquarters, we were looking at snippets and pictures from the event, the theme seem to be lots of hands on keyboards, smiles on faces. Tell me a bit about that, Tim.

Tim [00:05:07] Yes. I really enjoyed the event. The venue wasn't too big but it wasn't too small. We had a big open space many small tables and everybody could sit together and discuss and actually work on your laptops and develop some new code. And also listen to great talks. So I think it was was a very successful event for me as an attendee. Yeah I'm looking for new events that are coming up in the future. Speaking about the pinball event in the beginning there was an on our first night and in Budapest. We visited the biggest museum in Europe with a lot of pinball machines. I think they have over 50 or 60 machines and we could even play on every of them all of them were working and we did a little tournament against each other. It was a great way to get in touch with all the different people from all the different countries that all speak a different language and having the pinball machines around helped us to break the barrier and to get in touch more easily I think

Andrew [00:06:26] Pinball's the Rosetta Stone of games.

Ben [00:06:29] Yeah, totally. And kind of like the tournament really gave people like impetus to go talk to somebody that they may or may not have talked to before because we sort of had to like you had a name and you had to go figure out who that name was and then you had to decide which game you were going to play and then you had to like run through and play the game and take your turns and everything. And there was a lot of conversation that ended up just kind of organically coming out of that that maybe wouldn't have if we hadn't run that pinball tournament to begin with.

Andrew [00:07:01] That is awesome. I'm also a little disappointed Vox Pupuli held a pinball tournament, basically the World Cup of pinball and it wasn't live streamed. Really guys, really.

Ben [00:07:15] Well that's a growth opportunity for next year.

Andrew [00:07:18] Right on. Speaking of which, you know Portland is also a great scene for smaller independent events much like Contributor Summit. There's something about it. It's charming. It's very interesting when you get like-minded people together to think through difficult challenges, solve new problems. What was it about the event this year that made it so different and unique?

Ben [00:07:41] So the big change about this event from the events in the past was that we used to be like kind of married to PuppetConf. And that gave us some benefits that you didn't really have to get approval to go to another event. All you needed to do would just kind of show up a day early to PuppetConf and go to this. But it also like sort of set the scene as to like expectations of what the event was gonna be like. It was people who were going to PuppetConf who also felt like I mean I can show up to Contributor Summit as well. I think that separating it pulling it out let us focus a lot more on just the technical aspects and the community aspects. It let us just kind of like we didn't talk about roadmaps. We didn't talk about timelines. We didn't talk about hey here's a new release coming up in the future. We talked about here are some technologies, here are some really cool things you can do with those technologies, and let's kind of work together and brainstorm. Like what are what are some great ideas that can come out of like the synergy of you and I sitting together hacking on this. And I think that that's something that we've kind of been missing out on for a little bit. So having a venue that was tight like Tim was saying but wasn't like too small. It was it was like kind of like that right size. It was the perfect size enough people to tip critical mass but not so many that it was overwhelming. And we just, we hacked. We did amazing things.

Tim [00:09:06] Yeah. I can only agree on that. It was very good. In previous conferences most of them had multiple tracks at the time and you never knew which talk you should attend to. But this was the case for the last Contributor Summit. So this was the single slot event which was very very good I think. You know it's, there wasn't any sales pitch in any talk which is something I don't like as a technical person. I don't want anybody pitching to me, it's just all software. I would like to understand how technology works and what's coming up next and Puppet delivered this in a perfect way.

Andrew [00:09:44] That's awesome. You know when you go to an event like this it takes time. It takes energy. We live in an age where you can watch a lot of stuff on YouTube. You can take a training course for $9.99. But nothing really replaces an event like this where you get together and hack on projects together, solve problems together. Tell our newer listeners out there, perhaps folks who are wondering how do I learn more about DevOps practices, how do I learn more about config management, infrastructure automation, what is the value of getting together like this already and combining it with a curriculum that also includes you know stuff you learn on the interwebs?

Ben [00:10:28] That's a really really good question and I think what it what it comes down to especially as you're getting to know something, especially when you're starting out learning things, you don't always know what to ask. You don't always know what to look for, you don't know what to explore. And it's really easy to get caught up in the wrong thing or go down in the weeds in something that isn't quite as productive as it could be. And sitting in a room like in collaboration with other people that you're that you're working with, it lets you sort of like organically pick up on the things, the topics that are that are hot, the things that are of interest, like lets you sort of like learn from the conversations that other people are having and also gives you space to say "Wait, hang on. That's a thing I'm interested in. That's something that could solve this problem at my company that I didn't even really know that I had." And so it's just kind of like that organic growth of knowledge rather than prescriptive, here's a thing that you need to go like study ABC and D and then you're going to be an expert.

Andrew [00:11:30] I had the privilege of watching Ben, Kara Sowles and Anna Velasco basically coordinate an event on the other side of the world here from Portland. That is no easy task. I've heard many positive things coming out of Contributor Summit. There had to have been challenges along the way, right. I love to hear some of those from Ben, you know for an attendee like yourself Tim, to let people know out there about you know how these things are wildly complex things that need to be coordinated and what, you know, random things you may run into along the way.

Ben [00:12:00] Absolutely, totally. And in reality the thing is like no matter how big the event is that you're looking at no matter how well established it is, it's always a work in progress. It's always something that's growing and evolving. You go to a big AWS event, you go to Dreamforce or something, that's a thing that is growing and changing every single year. It's just that the changes are a lot more noticeable when when an event is smaller or when it's new. So one of the things that we honestly I think that we kind of miscalculated to begin with is how big of a change decoupling this from PuppetConf was going to be. In years past we sort of like had the privilege of being able to kind of coast along with PuppetConf and we didn't have to do like for example any of the promotion that goes into a new event or anything. And this year we had to start thinking about those things for the first time and we kind of misjudged how much that would carry over. And there were a couple of stumbling blocks along the way. And we had to learn a lot of hard lessons about what it means to build a new event what it means to take and stand something up brand new and pull something off. And I think that we had a couple of those sort of like humbling moments of "Hey, wait a minute. You know what, we had these grandiose dreams of what this was going to be. Let's kind of take that scope and pull it back down to the core and let's focus on what's important and not about what's grandiose." And I think that that's one of the things that let us pull off such a great event. At least I hope it was great, from my standpoint it felt like it was really good and I would love to hear like whether Tim felt that there were things that could have been done better at.

Tim [00:13:39] I don't think that there is anything that you could really improve without thinking about the event. Everything went fine. The venue was good. I think everything was properly organized. The food was good. We had huge luck with the weather, very very nice. And also the city that you choose, Budapest is pretty pretty cheap especially compared to San Francisco in 2017 for PuppetConf. One thing that I learned is, I live in Europe and we have a bunch of coins to pay with and paper money, but in Budapest I cannot pay with my Euros. They have a different currency, which is something that I learned while landing on their airport. So I wasn't as much prepared as I should have been. It's more something as my personal take away.

Ben [00:14:33] That's an interesting point actually because as an American traveling through like Europe and Asia and like other countries, I'm used to like having to change my currency into the local currency and then like that was an expected thing for me. So it's not often that I talk to a European and I have like I don't know what the leg up when it comes to currency.

Andrew [00:14:55] I have two recommendations for the event next year. Live stream the Vox Pupuli World Cup of Pinball. And also team jerseys by country. Thank you.

Tim [00:15:04] It's a nice idea, yeah.

Ben [00:15:08] But there are some really good ideas for things that we could do better next next year because I feel like where we are right now was it was kind of like on the lower end of the sweet spot of the size of the event. I'd love to grow it in size a little bit. I'd like to make it maybe maybe double the size that it was but I don't want to go to much bigger than that. But there are things that we could do that we to make that even I don't know make that work even better. I'd like to have like peripheral events so it's not just PuppetizePDX and Contributor Summit, but I'd like to also have like that like a sort of other events surrounding it and I'd like to get people more involved earlier so that when they when they come to the event they have some more expectations about like what we're going to be working on what we're gonna be talking about and we can present kind of like the culmination of some of these projects. So there are lots of things that we could do I think, and livestream. I think that's a given. The pinball, livestream the pinball.

Andrew [00:16:08] Let's talk a little bit about community. It is no joke. To be a part of a community like Puppet's. I mean, you're doing your day job and then you are spending your free time - this is time you could be using gardening, binge watching things on your favorite streaming...

Ben [00:16:22] Or if you're Tim, barbecuing.

Andrew [00:16:25] There you go. Always be barbecuing. But instead you are contributing back to modules, open source tools that help folks in their automation endeavor. I want to know a little more about Vox Pupuli, Tim. I mean you and the crew. This is a volunteer run organization that is a big part of the Puppet community.

Ben [00:16:44] I'd even go as far as to say that they're almost a pillar of the community.

Andrew [00:16:49] Just one pillar?

Ben [00:16:50] The foundation. There we go. They're the foundation of the community. Yeah. I don't know if we can overstate that.

Andrew [00:16:56] Titanium. Titanium foundation of the community, Vox Pupuli. Go for it, Tim.

Tim [00:17:02] Yeah sure. Just add more pressure on it. Yeah, Vox Pupuli. So I think it's pretty amazing how the community evolved in the past three or four years. We started as a very very small group with like 10 to 15 people contributing to Puppet modules from time to time. And if I check our current GitHub organization stats we have more than 120 modules, 160 GitHub modules in total and something around 170 people with merge access to all the repository stuff we own. Many of us are going to DevOps meetups or conferences and just speak about Vox Pupuli, how the community is organized, and what we do in our day to day work. Most of our people do this in their free time. They don't get paid to attend such conferences but it's a very nice feeling to work in a nice environment with friendly people that are open minded to new technology. And this gives us some power to to just keep working on it and makes a lot of fun.

[00:18:23] One thing that I think is a little bit interesting and to like make use of like one of those horrible buzzwords is that I kind of think that Vox Pupuli is one of those thought leaders in the DevOps world. People look up to Vox as being sort of like these are the people who know how this stuff works. I should pay attention to them and the people on the PMC or sort of like shepherding that. They're sort of like putting together those guardrails of late keeping this community going in a productive manner. I think it's amazing what they've been able to accomplish.

Andrew [00:18:57] And I agree. And Tim, if there are folks out there who are hearing what you're all about and wanting to get involved, how can they do so?

Tim [00:19:05] So we have a website voxpupuli.org that everybody can check out. It contains a lot of documentation about all the modules that we have and how people can contribute to the actual repositories, but also links to our IRC channel data we have on freenode and another link to a mailing list and also to our Slack channel. So there are many ways to get in touch with us. The community's spread around the globe and we almost cover every time zone, so whenever you have a question I would like to just reach out to us and chat. Somebody will be available in most cases.

Andrew [00:19:43] That's awesome. For those of you listening to the web site, we'll include links in the show notes.

Ben [00:19:50] Yeah and it's also worth pointing out that the friction for getting involved is actually very very low. If you happen to use one of Vox's modules or one of their tools and you find something that you want to add or fix or whatever, then you can just make a pull request. So you can just look at this and go "Huh, I could fix that thing" and make a pull request and that's sort of like your intro, your gateway into this organization. And maybe then next week you make another pull request and maybe down the road you'll start commenting on some of the issues and you just sort of like organically get welcomed into this group of people. It's not a very large barrier to entry.

Tim [00:20:35] I would like to add something to this. We have a bunch of open pull requests and many people think that to review such a request you need to be an actual member of GitHub organization, but that's not true. Like everybody can check out the open issues that we have and also the open pull requests and just see through the code, comment on it, or just click the approval button if you think the code is working, or if you've tested it. This makes it way easier for us to merge the code and to speed up the development process. And it's also a nice way to get in touch with the code base and the other developers that we have.

Ben [00:21:15] It's worth pointing out that if you go to the voxpupuli.org web page there's a link right in the header that will open up the pull request list and you can find those very easily and just list them out. Look at them, see what you think.

Andrew [00:21:27] Thanks you guys. I've got an interesting question here for both of you. It's pretty clear that the community, Vox Pupuli, you all are doing a bunch of work, heroics in a fly by night manner. There's got to be a way to tap into my resources. I mean a question that immediately pops in my mind is if all this goodness is happening you know what else could Puppet the organization be doing to support these efforts. Outside of fun podcast episodes and pinball tournaments and events in Europe.

Ben [00:21:56] But one thing that we're doing already is that we're helping out kind of like helping with some of their infrastructure needs. We're giving them spots, like Travis slots for testing and all of that. I would love to be able to expand that and I would love to be able to give them more access to some of our resources and build some tools that help not only us, but Vox Pupuli. I'm working on, like myself I'm working on some tools that will provide some insights, like development insights, into what sort of problems people are running into and what sort of impact potential changes are going to make towards downstream modules, various things like that. And I want to make Vox Pupuli like an equal partner and working on those and using those metrics and using those tools. But I am really curious to hear what Tim has to think about that.

Tim [00:22:50] I think development gets easier if you have very good defined standards and if there is only one standard to follow. So Puppet has many many people that develop Puppet modules. The same is valid for the formal organization and also for Vox Pupuli, for the CERN organization from Europe, so there are many many groups with multiple developers that all do the same stuff basically and all of us define their own development guidelines and rules and guidelines for review and pull requests and stuff like this. And I think if we all could work closer together to have less standards, but better standards for reviewing code and for style guides. This would help many of us and it would also make contributions from new people way easier.

Ben [00:23:54] Yeah I think that I would really like that a lot. Earlier today we were even talking about building like a reference architecture that Vox Pupuli could publish. I would love it if we sort of like worked with them and then we promoted that as well as like this is the standard open source reference architecture. I think there is a lot of opportunity there for like collaboratively building good standards.

Tim [00:24:17] Yes true. I check the Puppet documentation from time to time, basically after every release. The docs are pretty great and a very good example. It's about the actual Puppet language, about the DSL, about stuff that you can do and stuff that you can't do, but there aren't really recommendations. Like there are many many ways to achieve the same thing. And it's not really documented which way is the best and there are always some edge cases that you can't cover in every document. And on Vox Pupuli we notice that we have so many people that review pull requests and they own different time zones and we would like to have a high quality of our code base across all repositories. But discussing such things in Slack or on an email list is very complicated and takes a lot of time. So we just started to write down everything that we agreed on on our website and it started as a very very short document with like five bullet points and at the moment it's like two pages long and has 30 or 40 different items on it, and this document is growing over time. If we could make this on a long point of view an unofficial standard or an official document I think that would help many many people.

Ben [00:25:38] Yeah, we should totally get Jean [Bond] working with you to turn that into some more official recommendations and let you more or less kind of like own that as a reference.

Andrew [00:25:48] So folks. As you heard, community work, it's hard work, it's beneficial work. It often goes unthanked. Let's take a quick moment and applaud all the community members out there, whether you're a Puppet community member or a member of any other technological community where you're wanting to mentor, share ideas, best practices, you go. You are a winner in our books. Speaking of winners, one of us here has won an MVP award. Ben, who won?

Ben [00:26:17] Well it wasn't me and it wasn't you. So that leaves it down to Tim, who's won our MVP award again. I think this year we've decided that maybe we'll put term limits on that, so maybe that we know that Tim's not going to win the award again next year. So that gives us more difficult choices because honestly Tim was kind of like the shoe-in. It wasn't a who should be MVP this year was like, "Well is there anybody who can even approach like the amount of contributions that Tim does? No? Well I guess Tim's MVP again." So, no surprise. Tim you're awesome and you do great things for our community and I am honestly a little bit overawed at how much you end up doing. I'm kind of fascinated with how you're able to put so much work into the community and how you're able to do that and also maintain your sanity.

Tim [00:27:15] Thank you. Thank you so much for the nice words. It started out pretty pretty low with just a few reviews and merges on modules that I used on my own like three and a half years ago, and it just got more and more. I think one of the key facts here is that getting positive feedback for the work that you do is always awesome and most companies aren't really able to deliver this and because like well they already pay you money so separate appreciation is rare. But this is different in an open source community. Like people just sending out emails and say that the work that you do is pretty awesome and that just gives you more power to keep working on it.

Ben [00:28:05] That's something that I've seen quite a bit in Vox Pupuli. Like one of my earliest memories of working with Tim and Vox Pupuli is having a conversation about a brand new contributor, somebody who had just like stepped in and deciding what sort of gift that Tim was going to send this person to say thanks for contributing. And I was kind of blown away that he was willing to go to the effort of like even thinking of that. So I think that really says something about the kind of person that Tim is.

Andrew [00:28:34] You're a winner, Tim. You're a winner. So what are your hopes for the future of the org and for the community at large, Tim?

Tim [00:28:41] Tricky question. It would be nice to get more contributors. We have a bunch of open issues and it would be nice if we could clean up most of the backlog that we have. We're currently working on more optimization to make the work easier for all the people that we have, and that's looking good so far. And we just hope that the community keeps growing.

Ben [00:29:08] Yeah I think that that also that identifies another thing that we could collaborate with Vox Pupuli on, is that as far as I can tell one of the key things that keeps that organization working so effectively is how much they like ruthlessly and relentlessly automate as much as they can. And I think there is a large opportunity for all of us to sort of work together and build like standard automation for Puppet modules as a platform.

Andrew [00:29:35] So guys I got to know, what were your favorite projects coming out of Contributor Summit?

Ben [00:29:40] I think the thing that people were most excited about was Litmus, to be honest. Litmus is a testing tool that is sort of like the successor to some other tooling that we've had in the past. There's been a kind of a family of tools - there's Beaker and then a bunch of other tools around Beaker that just kind of we mishmash them together and use them for acceptance testing. Litmus is a new idea of saying, "Hey you know what. We have tooling in place to do a lot of the orchestration that we had to manually manage before. And we've got tooling that does really good unit testing and like integration acceptance testing. Why don't we just like take a fresh look at all of that. And then use this new simple glue layer to pull those together and make a very simple, very easy, very quick testing layer for Puppet modules?" And it's developed pretty quickly so it's brand new right now. And there was a little bit of interest a lot of people kind of like looking at it here and there but after Summit people came into that, looked at it, saw what it could do, kind of poked at it, got their fingers on it - like hands on keyboard time you were talking about - and like since then I don't think a single day has gone by without another pull request or a comment or an issue. It has turned into maybe our most like vibrant project in the Puppet ecosystem right now.

Andrew [00:31:09] You hear that, Bolt team? You got competition.

Ben [00:31:11] Well, Litus is using bolt on the back end. So they're also benefiting from this.

Andrew [00:31:16] True. So for folks out there, our module builders, our module makers, who are wanting a better means to test, where can they find Litmus, Ben?

Ben [00:31:26] That's a very good question because you can find it at its GitHub repository which is a simple github.com/puppetlabs/puppet_litmus but I believe that we also recently just pulled it up into our open source page which if you go to puppet.com/open-source.

Andrew [00:31:44] Awesome. What about you, Tim? What was your favorite project or thing coming out of the Contributor Summit?

Tim [00:31:50] I got two projects. One of them is Bolt. It's very very awesome to basically replace Ansible, and I'd say to run a masterless Puppet set up and it was nice that Lucy [Wyman] was there to chat with and also many many other people that are already using Bolt so I could get some actual hands-on experience from those people. And also I learned a lot of stuff about Puppet Debugger which was completely new to me. It's a nice interactive shell to really debug all your Puppet code to see which variables are available in your current scope and which values they have and to [identify] really complicated issues that you might experience from time to time.

Andrew [00:32:41] I want to acknowledge once again: Tim, you are a winner in my book. But you're not the only winner here. I want to acknowledge that producer Enya is wearing an awesome button-up shirt. She's showing up dressing for the role, outperforming us all. Enya you too are a winner. So Contributor Summit wasn't the only event going on this year. The community team at Puppet is also working on another one around the corner called PuppetizePDX. Ben, tell us a little bit about it.

Ben [00:33:12] Well I'm going to take a real short segue first. Coming back to a point you made earlier about pinball tournaments and about Portland and then relating it back to PuppetizePDX. Eric Sorenson or Eric0 as many people know him and I were talking about how well that tournament in Budapest went off and coincidentally we also have a pinball arcade. I don't know if I would go as far as to call them a museum, they're more like a bar. They don't have quite the same sort of like appeal of being down in the basement with like the dungeon arching over you and everything the Budapest Pinball Museum has, but there's still pretty good. We were talking about also running a pinball tournament here before PuppetizePDX. And by the look on your face I think that's like a vote in your book is to say yes we should do that.

Andrew [00:34:06] Yeah. Will there be jerseys? With teams? World Cup?

Ben [00:34:09] I'll leave that up to you.

Andrew [00:34:12] Oh, alright. Yeah that sounds like a great idea.

Ben [00:34:15] So PuppetizePDX is sort of like the other... we talked about how Summit was split off from PuppetConf. PuppetizePDX is the other half of that. So it is the spiritual successor of PuppetConf. So it'll have all the same sort of things. There will be of course like the roadmap talks and the new product announcements and a few of the sales-y things that Tim doesn't like, but it's also got all of the technical content that we've always had in PuppetConf. There'll be people presenting about solutions that they're working on, there'll be people presenting about problems that they've they've struggled with, about ideas that they have about tools that they've built, and there will be lots of time for collaboration. We'll be having a hackathon the day before sort of like free time in the office with subject matter experts freely available to help you solve problems that you're working on. You can use that time to contribute to Puppet, contribute to a module, or build a tool of your own and call on the expertise of people here who know the technologies inside and out. And we'll also be doing sort of bird of the feather conversations that are a little bit reminiscent of some of the things that we did at that Summit but more in the context of hey here's a lot of new tools that we've been talking about. There's a lot of roadmapping here. There's also this great big ecosystem, this DevOps ecosystem that is out beyond what what Puppet currently addresses. What are some things that we can do? Like how are we as community members addressing all the problems that we see here? How are we what what processes are we building what tools are we building. How are we using puppet to integrate and solve some of these problems? What are some things that Puppet could do in the future that it doesn't do yet? What are some other tools that Puppet could build to help answer some of these problems? And also all of the same product announcements and everything we've been talking about. Sorry, I know I'm a little bit excited about this. This is totally, this is a thing that we've been spending a lot of time in and it's something that I'm super super excited about.

Andrew [00:36:16] Yeah me too. PuppetizePDX it'll be October 9th through 10th 2019 this year. We've got events, trainings and whatnot tacking on to the days before. I'm not quite sure about the days after. You can learn a lot more though on puppet.com/puppetize. So everyone we've covered Contributor Summit, we've covered what it means to be a part of the community Vox Pupuli, Tim winning MVP once again, we gave you a preview of PuppetizePDX. Everyone, did we miss anything? Did we cover everything here? It's been amazing.

Ben [00:36:51] Oh that's a really open question. There is a lot of stuff out there to cover but I think we did cover everything that was in scope for this conversation.

Andrew [00:36:58] Right on.

Ben [00:36:58] But I mean there's always next time, right.

Andrew [00:37:01] Yeah, always next time. So I want to leave everyone with a bit of food for thought. During this awesome episode I pulled up Paula Dean's goulash recipe. So what I want to know is knowing the heritage, the deep heritage of goulash, how does a popular American chef pull it off. You think she does it? You think she doesn't? Maybe we should try it. Maybe we should report back in if you're interested. The show notes too. Need to bring it full circle everyone.

Ben [00:37:28] I think we should hashtag that.

Andrew [00:37:30] Hashtag goulash, powered by goulash.

Ben [00:37:32] Powered by goulash, I like that.

Andrew [00:37:36] That was a promise Ben made, Tim. I'm looking at the show notes from the previous episode. And he said he was going to run that whole marathon fueled by goulash. He is quoted on the air. It didn't happen. Maybe it needs to happen this year. There is a thing in Portland called the Portland Marathon that you are running, correct?

Ben [00:37:49] This is true. I'm actually even, I'm running a virtual training plan for it right now too so if you are interested in joining up with us and running the Portland Marathon or half marathon then you are absolutely 100 percent welcome. I haven't announced this yet but I just got approval to get team jerseys so we can get a team Bolt running shirt and you are invited to come run with us. If you go to pup.pt/pdxmarathon there's information there on how to get involved. Basically just sign up for the marathon. Join Team Bolt and show up.

Andrew [00:38:28] Let's do it, people. All right, and with that, thank you so much Tim for guest starring on our podcast here. Thank you so much, Ben. Always. Thank you, producer Anya in the awesome button-up shirt. You are rocking it today. And a big thank you to everyone listening. We really appreciate it, always. And with that catch y'all later.

In this podcast, Eric talks with Bryan Belanger from Fervid about working with Puppet in highly regulated compute environments. As a consultant for US Government agencies, Bryan's been working to get "Authority to Operate" for governmental cloud services, to speed up time to delivery and reduce compliance risk. The Puppet modules for STIG hardening, especially for Windows servers, let them build a baseline from existing systems and enforce that across the environment. We talk about the Forge, about Cloud, and about... Canada!

Windows hardening module: https://forge.puppet.com/autostructure/harden_windows_server

*Eric is a technical product manager at Puppet.*

In this podcast, Eric talks with Bryan Belanger from Fervid about working with Puppet in highly regulated compute environments. As a consultant for US Government agencies, Bryan's been working to get "Authority to Operate" for governmental cloud services, to speed up time to delivery and reduce compliance risk. The Puppet modules for STIG hardening, especially for Windows servers, let them build a baseline from existing systems and enforce that across the environment. We talk about the Forge, about Cloud, and about... Canada!

Windows hardening module: https://forge.puppet.com/autostructure/harden_windows_server

Eric is a technical product manager at Puppet.

Eric [00:00:38] Hey everyone! Welcome to another edition of the Puppet podcast. My name is Eric Sorenson. I'm a product manager here at Puppet and I have here in the studio with me today Bryan Belanger. Hi Bryan.

Bryan [00:00:30] How's it going, Eric?

Eric [00:00:32] Good. Did I say your name correctly?

Bryan [00:00:33] You did say it correctly. You know, Belanger - it's like the Americanized bastardized version of it. I tell people it's like it's like Smith in Canada, my ancestors are from you know the great white north of Canada. If you go in Canada, all the Canadian listeners: Belanger. So give a shout out to them, but it is Belanger since coming to the US.

Eric [00:01:01] Excellent. I wouldn't be surprised if they actually call it Belanger-eh in Canada.

Bryan [00:01:12] Where's our drum roll... that's the best I could do.

Eric [00:01:13] No Foley here today. Well, intros aside and terrible attempts at comedy aside we're here today to talk about some work that you've been doing with Puppet and some other related tools. I'm really excited to talk to you about some of the work that's been going on, particularly in the federal and security space. Can you talk a little bit about what your company does and kind of what you've been working on lately.

Bryan [00:01:39] Sure. Until recently, our company's name just changed. But we worked heavily in - well there was a need for our customer to move to cloud. In this case it was a federal space and there's a lot of concerns obviously you know whenever you move something as enormous as the federal government anywhere it takes a lot of time and effort, but security is a big thing. It's you know it's like those knee jerk reflexive reaction. So you know big solution to a lot of problems you know on the outside it's the cloud versus what we're spending internally - even though it's internal dollars, it's millions. We needed to save money, we needed to make things faster, we needed to get visibility but you know it's frustrating. There's customers you know who want to deliver you know I mean this time I mean internal customers to the federal government they want to deliver and they just can't. Cloud is going to be able to open up a lot of that you know saving money that was a byproduct there was no customer. It's hard to get customer visibility and a lot of these things but one of the big things is how do we get to the cloud. So the big thing in the federal government is authority to operate and the security wants to know hey that sounds great and all that but what we're going to need is we need a way to make things audible, trackable, you know, how do we make sure our stuff is. So there's just a general idea that if you're in what's called an ATO environment that everything is locked down to a certain extent which is acceptable you go to the great big cloud you know it's kind of scary. So we're trying to address a problem that is just in generally accepted internally for a while but is actually kind of like an elf in the room. In this case it's with Windows security. Windows has you know there are things called group policy. It does what supposed to do. But the big issue that security has is audit tracking so if you say you create a group policy rule to enforce this particular security thing it's going to do it. The problem is it's going to do it all behind the scenes. You just assume it's working well. What we needed is a way to say you know this event happened, this drift happened, and now you have a report, now it's audible, now you can even launch an investigation. We didn't have that before. There was actually a second byproduct. Well I get to that second. So what we needed is in the Windows space, we needed a way to enforce DOD STIG. DOD, kind of the Cadillac of you know security standard. Everybody wants to follow the DOD and some.

Eric [00:04:30] Is that the STIG? Not from Top Gear, but the STIG.

Bryan [00:04:37] Yes. So we would go in and you know our company developed a series of modules that enforce you know like 90 percent of it and that's where it began. So now we have a way, you know, if you are in the federal space and you want to move to the cloud and you likely have some IIS in house, you now have everything needed to meet you know. Usually your ATO submission whatever you want to call it, you now have your authority to operate. Another byproduct, you know, which kind of happened second hand but almost every company I should say almost everyone that does it they have their own flavor of security. So yeah, DOD is nice but we have you know our customized version security. Everything we did was generally written in Puppet resources. In this case they were - trying to find the word but - you can reverse engineer, so if you install Puppet on your Windows machine you say give me back all of this particular rule. You can reverse engineer what you're having so I can go in to your environment, install Puppet, install these modules, I can reverse engineer exactly what you have into a nice Puppet manifest I say I can pick it up I can move it to cloud I can stand up a brand new domain controller in my own VPC in the cloud. So those are the two big benefits we came out of it.

Eric [00:06:12] That does sound like a core Puppet use case. I mean that sort of hits on a lot of the strengths of Puppet's architecture as the ability to take those resources and present them in infrastructure as code just by discovering what's out there. If you have types and providers that use the instances method inside of them you can run Puppet resource against a running system and get back that text description of what the resources are that are available and how they're configured. So as you said you can just take that and apply it to a new system and have a high degree of assurance that you're introducing repeatability and you're sort of correcting drift as it happens. Sounds pretty awesome. Did you have to go in and write custom types inside of those or were you able to do most that stuff with the built in types that are in Puppet.

Bryan [00:07:07] Well, we had to write a lot of custom types. There's just no ifs, ands or buts about it. Yeah I like to say like yeah we can use Puppet to find types, or um, we are able to adopt that. But we had to do it in a lot of most of our custom we got we got really good at Ruby to say about we are really good at Ruby. But you know we're proud of the work, we're seeing adaptability, we actually hope more people pick it up. We've run across you know at least a couple service delivery providers that are planning on adopting in their own federal organizations. But you know I'd like to see it commoditized. I'd like to people just say pick it up and use it. There's no reason not to break a few things you know please please do testing. But but yeah yeah go ahead use it. It is one thing that's going to close down the layers of the onion right away. You've immediately made your environment, whatever it is, much safer.

Eric [00:08:07] So were you able to put those modules up on the Puppet Forge?

Bryan [00:08:10] Yep they've been out there. Yeah, I think, I want to say about a year and a half two years.

Eric [00:08:14] I'm sure we'll put a link in the show notes at the bottom here, but can you tell the listeners what name to search for on the Forge so they can find them easily.

Bryan [00:08:22] It will be under Fervid and there's gonna be one that's called 'Windows hardening' it is not out there yet. It'll be out there in the next two days.

Eric [00:08:31] Excellent. That's really cool. Forge is a great resource and hopefully we can get some more broad usage of that kind of code. The reusability of it is just a huge advantage as well as you know just having that crowd sourced kind of QA and testing and bug fixes and getting an active community around it is one of the things that I think Puppet does really really well in our community ecosystem. Can you talk about what - I know sometimes there's some sensitivity around government stuff - but can you talk about what the customer was, like what department it was and kind of what their main use case was for the servers that they're running? And if we have to redact this part and it ends up being just like 30 minutes of silence, you'll know why.

Bryan [00:09:21] I would love to share the customer. I can't and unfortunately there's various reasons for that.

Eric [00:09:29] Yeah, no problem. Just figured I'd ask the question. And were you using the modules in sort of a standalone mode or did you have them integrated into a larger Puppet Enterprise infrastructure or how did you work with, you mentioned the reporting and audit logs that came out of it. Can you talk a little bit about that stuff outside of the modules themselves and how that fit into a larger Puppet architecture?

Bryan [00:09:54] Yeah. One of the biggest things that our customer in the government or anything regulatory need is they need to audit reports. And the best way to do it, you know, we saw is through you know the enterprise so as always had the enterprise mind to be able to get your reports in a GUI that you can see you know you keep it tracked. There was these grandiose plans of you know integrating with other third party tools you know hopefully that'll happen someday hasn't happened yet but at least right now you have the ability to say something happened, it happened at this time, who was on there doing what when. And I believe, that was just a need that needed to be filled. Yeah it's completely open, completely open license, and it's intended to be used with Puppet Enterprise.

Eric [00:10:43] Awesome. Have you explored any of the other or were we able to integrate for this or maybe for other customers some of the capabilities in some of the newer Puppet Enterprise releases such as Bolt and tasks and task plans. Have you gotten into that kind of realm at all?

Bryan [00:10:58] Yeah same customer I talked a little bit of the story earlier. So they have kind of like this half pregnant situation. Typically you have a cloud provider which is being marketed as. But what the situation was there is this rule that the cloud provider in this case was going to manage the OS and below and then you guys manage everything OS and up and it wasn't a good setup. It caused a lot of issues. So one day we walk in and security is a big thing. This case they're a Red Hat customer you know you need to run your basic yum keeps everything patched at the OS level. That was on us. So walk in one morning you can't do anything the yum's broken. So it's like hey provider how about you know fix our stuff and I don't know why but they're like well no, what you can do is you can run these three commands on every like twelve hundred one thousand somewhere north of 1000 nodes you have. You know do the back of the math calculation we did this a little earlier it's where we spend it everybody gets on the computer and they log in one at a time to every single box. They can go ahead and fix this over 3, 4 days. I mean that was their answer.

Eric [00:12:28] Wow. Seems like there ought to be a better way.

Bryan [00:12:30] You think. You know, like, at the very least like you guys log in and fix it. But no, that's not how things are done in this case. Customer satisfaction I guess was not their thing. So the question is you know the operation like what do we do. I think we have a solution here. We took those roughly we took those three commands. We created a Bolt task for them and our job went down to like ten minutes. Something like that, 4 days the 10 minutes. It was a nice tackle thing when Bolt was early on. But it was an early success. Immediate impact. And we've done similar things since then, but that really saved a lot of headache and heartache.

Eric [00:13:14] That's pretty substantial time reduction if you can do it in 10 minutes and then spend the rest of the three days playing solitaire. That's in good shape. I haven't had too much experience working in government agencies, but I assume that's how that works.

Bryan [00:13:29] I'm going to shrug my shoulders.

Eric [00:13:30] Well, cool yeah. I'm a huge fan of Bolt. I mean I think it's - we did a podcast a month or so ago that was all about Bolt and that came out really good and you know I was involved in the project and we were just starting out and I still kind of keep a finger in there but it's not my primary day job anymore. And occasionally it just shocks me when I go away for a month or two and I'm working on something else and I come back and Bolt has gained all of these amazing capabilities in that intervening couple of months. It's just the project's moving really fast and it's just expanding in the amount of ecosystem and the kinds of tasks that people have written for it as well as the core capabilities of the tool. It's just really been kind of a runaway success. It just warms my heart to see that stuff out in the world actually solving people's problems.

Bryan [00:14:23] Yeah I'm not sure the impetus with it but it was something that was needed, without me even realizing it. But you saw the impact immediately. I mean Puppet was the best in the space. There's certain things that it didn't do well or couldn't do. I mean I would remember the old days, I and other providers we would spend hours writing you know Ruby stuff to do onetime things or would run all these execs. Execs are like fingernails across a chalkboard for me. So it was nice to do this and what I saw, the impact in some of the organizations I've been in, is you know I put on two big advantages the big one I saw right away is I'm able to create things that are parameter-ized. You know it's a semi formal language you know I know you can power shell things you can, you can write Linux shell scripts et cetera et cetera. But this allows you to formally have a parameter-ized strong check if you're using plans way of doing things. I'm able to take this, I'm able to handle it to some of the less technically astute operations team and they can start sysadmin-ing they don't have to learn the Linux stuff or the Windows stuff they get there and they're able to operate right away. And of course like I said earlier is what you know now with the Puppet Enterprise you know we now have auditability. So I have real pieces of code that act like real code that is we're able to hand off auditors - once again particularly in the regulatory space. It solved a big problem. I didn't see being solved before.

Eric [00:16:10] That's awesome. Yeah I think that type system is so powerful. It's one of the - ya know, Bolt kind of is independent from Puppet in a lot of ways but it's also closely related to Puppet. So if there's lessons that you learned, if there's techniques or are parts of the language syntax that you're used to from using Puppet you can translate those over into Bolt. And I think the type system is one of those examples where you can almost opt into it and you can just have very lightweight kind of type checking or you can get really pretty pretty intense with it and say like if you have it's a great example too of being able to hand something off to a NOC team or kind of a help desk kind of team and make sure that for each of the parameters that feed into a task or a plan that they, you know, if it's a service there's only a few things that you can do to a service. You can get its status, you can stop it, you can start it, you can reload it. And having those actions be well-defined inside of the type system means that if somebody tries to do something crazy to a service it just errors out even before it tries to run it's not even going to execute that command because it'll fail type checking before Bolt even executes the thing and I think that's that it provides the guardrails that lets those tasks be reusable and be shareable more broadly than just me running things out of my home directory which is I think kind of where people started out solving that problem.

Bryan [00:17:41] I think was open also open up and help us see more of this is. There's a lot of really good enterprise software out there that until I think Bolt came out which is not able to be installed and managed in the old way of doing this. To me this is going to open up the ability to automate these more complex platforms, these more complex installs. I think we're going to see bigger visibility across infrastructure as a whole with plans. You know, I've seen some of the some of the POC stuff I've seen out there, you know, pass a few parameters now manage the network and the thing. I mean Puppet tried to address this, but I think Bolt is going to knock that door down. So I think it's really just beginning.

Eric [00:18:28] Yeah I'm pretty excited about it too. There's lots of good stuff going on. Let's shift gears for a minute. We were talking about the cloud migration a little bit and I'm wondering if there are cloud or containerized kinds of infrastructure that you're also responsible for managing and how you're able to take some of the, you know, we mentioned taking some of the system descriptions for on-prem systems and applying them to new VPCs in the cloud. Are you also managing cloud resources themselves using any of the Puppet toolchain?

Bryan [00:19:02] No. I mean that's the short answer is no. I expect it's going to happen with future engagements but past engagements it really hasn't. We did a lot of POC stuff. But unfortunately I don't have any great stories around, you know, except getting some legacy applications to the cloud. That's what we've been focused.

Eric [00:19:28] I see. So it's more about my migrating the workloads over rather than actually managing the cloud infrastructure itself.

Bryan [00:19:33] That's correct. Yep.

Eric [00:19:36] That's that's perfectly all right. It's actually a pretty common story that we've heard. We're working on some new projects you may or may not have heard of that attempt to address some of those problems and try to bring some of the lessons from classical or traditional configuration management into some of these more cloud-native arenas for managing Kubernetes applications and cloud-native infrastructure. The project that I've been working on for the past few months called Lyra is a pretty interesting tool in that regard. It's just kind of had some of the concepts from Puppet. So again we bring over the type system, there's a Go Hiera implementation so if you have a investment or an existing Hiera hierarchy and yaml files the Go Hiera implementation can use those in a compatible way with the Ruby one. But the goal is really about orchestrating the different parts of a cloud-native application deployment rather than doing ongoing configuration management on systems because one of the paradigm shifts in cloud native is that the idea of like running an agent on a system and making changes to it live is kind of not really applicable anymore. It's kind of a different different mind shift where we're running immutable infrastructure and if I want to make a change I'll just make a new one upstream and let my continuous delivery pipeline bring that out into production.

Bryan [00:21:01] Yeah I know it's what's being seen by customers. We don't want agents running more and more we're becoming comfortable with the pets cattle model which I'm assuming people listening to this are familiar with.

Eric [00:21:12] I think that's a square on the Puppet bingo card to actually reference to cattle not pets.

Bryan [00:21:19] So it's not like a drinking game where we take a shot when...

Eric [00:21:23] You can play it - if you're playing along at home you probably can. Here in Portland it's 2:30 in the afternoon so it's a little early to get into that.

Bryan [00:21:32] OK so I was you know I was asked some things about Lyra ya know when it was being talked about and you know everyone's kind of familiar with you know what's the most popular tool in the space I'm kind of wondering you know you know I don't know if this can be talked about or not what Lyra's approach is. I mean I know what I don't like about the other tool but I do like about the other tool I'm hoping Lyra will address all sorts of things but I was wondering what where Lyra's trying to fit into the space.

Eric [00:22:01] Yeah I mean one of the main goals is that we want to be we want to enable reuse of existing content ecosystems. I think there's a lot of great work that people have done in application deployment on the Kubernetes with respect to Helm. It's kind of the tool that everybody's converged on and there's a rich library kind of like the Puppet Forge of Helm charts that describe how to put applications out on to Kubernetes as far as resource provisioning and kind of that initial setup. I think Terraform is a great tool. There's a lot of rich content out there that people have built either for their site specifically or they've built modules up out to Terraform and shared them on the Terraform registry. We want to enable reuse of those things without forcing people to rewrite stuff they've already done into a different syntax just to make use of Lyra itself. So the idea with Lyra is more that you'd be able to string those pieces together in a workflow rather than take over the role of being a Herraform or being a helm. So if your application requires like creating some infrastructure, deploying apps on to it, and then like sending some notifications you know a stopping a PagerDuty maintenance or sending a Slack or ChatOps notification once the deployment's all done we see Lyra as being the thing that links those pieces together and allows you to introduce some control flow and some logic and bind these different technologies together in a common language rather than trying to take the place of any one of those things that's already out there.

Bryan [00:23:33] Also sounds like Terraform will be possible integration point over there.

Eric [00:23:37] Absolutely.

Bryan [00:23:37] Okay, interesting.

Eric [00:23:39] But still really early we're still working on it but yeah that's why I was kind of asking the question too. I'm always trying to find people that have problems that we can maybe help solve because as I said it's a it's only been out for a month really in a usable form and I'm just you know like kind of rattling a cup around trying to find people to try my thing out and let me know if they like it or not so.

Bryan [00:24:07] Well, everyone hates TS state file.

[00:24:09] We certainly heard a lot about about Terraform state and you know I think one of the one of the users I talked to at KubeCon a couple of weeks ago said that at their site it's like a rite of passage that everybody at some point has to destroy everything and infrastructure using... destroy everything by messing up the Terraform state file and like once you do that then you're like part of the club and you can move on with your job. Well all that's good stuff. I hope you can give it a try and we can talk more and do a little demo offline or something like that I'd be be curious to hear your thoughts on where we're at. Any other kinds of it what's interesting to you like what's what's really sparked your interest in terms of technology it doesn't have to be Puppet related but is there anything that you've come across in the last three to six months that you're like Ah man I got a this is this is really something I'm interested and I want to learn more about this.

Bryan [00:25:07] Yeah. So the last three to six months have been fairly busy. I wish I could say yeah there's this shiny new key out there that I want to grab. I haven't been very closely you know watching Puppet's new products. And I know they're trying out new things. One thing I had been excited about for a while was the insight product. And it looks like it's going to go to some changes now. But what I had seen over and over again you know like in places I've worked before or been part of before is there's typically a strategic offering. Most of what I've seen in Puppet has been you know tactical and I understand why, but I'm hoping that this product moves forward. I see it as a possible strategic play. It's nice to be able to go in and transform companies, you know, yes you can add some automation here and I was there and they'd be better off for it. You know one reason we share in the community is because I want people to adopt it. We're all better off for it, I don't want to reinvent the wheels but you know I like the idea of being able to measure DevOps. And I'm hoping that allow real transformation to take place. So that's one of the things I've been watching closely, I'm talking to people today about that. I don't know if you have any other thoughts or insight about that.

Eric [00:26:30] Yeah, sure. I mean I think we're one of the things that we're really interested in is like just it's just like you said about making visible what's going on across the organization. And we think about Puppet as kind of like starting out and you know our heart is really with the person on the keyboard making changes and our tools are very squarely focused generally on making their lives easier. Yvonne our CEO has this phrase about eliminating soul-crushing work and that's kind of like where like I said where our heart is. But we know from the last ten years of working on this that just making one person more powerful and eliminating their job isn't really sufficient. You've got to spread that love across to the team that the person is working on and ultimately across the organization. And the broader you go with that, you also have to raise the elevation in which you're working and provide tools and visualizations that work not just for the person that's making the change but for the team and for that larger organization. One of the things that I'm really excited about with respect to our products recently has been the CD for PE tool which does provide that kind of visualization about what's going to change when you know what's going to happen as a consequence of this change. I think it's really pretty pretty powerful and it lets people feel more safe about making changes because they can more easily predict what's going to happen and the visualizations and kind of that audit log plus the predictive capabilities of it that say hey when when this change goes to production these are the nodes they're going to be affected and here's what's going to happen to them. I think that sort of up levels the conversation a little bit and lets people who are outside of the core team that's really in the guts of Puppet everyday, lets them make change with confidence. And we want to continue that and kind of continue to expand that focus, broaden out that focus and keep movement upwards in the organization and up the stack so that, like you said, you have a more comprehensive idea of how your organization is doing overall, not just how a particular server is doing or how your Puppet infrastructure is doing. So that's definitely an area that we're really interested in looking out over the next months and years.

Bryan [00:28:56] Yeah. No, it'll be nice to have. [inaudible] Yeah there's lack of visibility, just in the data center and I know where everyone's going is most people have multiple place in multiple providers. They need a way to glue it together and they need to do something more than just deliver applications in a CD manner. So yeah I've been able to look at it. I expect, I believe my next engagement will be touching on that so I'm excited to see what it's about.

Eric [00:29:31] So CD for PE is a tool for creating testing pipelines to get code out into production. But the actual tests that you perform in each of those stages is sort of left up to you. There are some templates and some other tools that are out there. I'm curious what kind of tests you run across your modules. Are you using any of the tools that are out in the community ecosystem like rspec and Beaker and serverspec and our new Litmus. I'm not sure if you had a chance to look at it but we have a new open source testing tool called Litmus. Can you talk a little bit about your kind of methodology or your philosophy of testing and what tools you use for testing Puppet code?

Bryan [00:30:15] My background is, I'm actually from the development side which is maybe, I seem to be in the minority here. Most of the people seem to come from the administration side. To me it's a natural fit but you know I already had years of leading teams and I've seen it over and over again where things would you know in development side you're expected to test. It's just the way it is. So when I saw that it became generally available for Puppet it became big. We had an issue which allowed me to bring it in, which I hate talking about but we had a you know one of the developers made a mistake. If we had had rspec it would have saved us a big problem. What we had is it was one of the modules on the Forge. I forgot exactly which one it was, but it handled ssh. We had a rule in there that basically says you know purge all the rules unless they're explicitly defined and that wasn't brought over correctly. If we had had rspec testing that would have been you know a five minute test and we wouldn't have literally locked ourselves out of the servers. You know it's just one of those stories. Fortunately, everything was OK. There's a back door, but it was scary for a long while. So rspec testing that's the waterline. I want basically every resource at least you know ensure or not insure anything critical. I mean maybe I'm going into a dialogue about how to test but we did that with our STIG modules. I believe every resource is tested and I'm hoping as that as it evolves in the user space that they add respective rspec tests to it. So I guess is that the other thing is is you know as a general rule you know from [inaudible] in the space once again because everyone's like what's the right level you know. Do I test every parameter? Do I test every particular thing that we can and can't do with rspec? I mean that's, of course the answer's no. But my recommendation you know when I do with the juniors is make sure everything exists or doesn't exist test what you think is critical. To me, testing is an evolutionary thing. What I say is I don't want you making the same mistake twice. Let's take our SSH general thing. It's never going to happen again because there is, in this case, it's once over on the control repo. Once over says make sure that these two parameters and these values are in place or you stop right now. It's going to save you. It's an investment, you're putting in a little bit more effort to make sure everything runs smoothly. So how does it evolve. You're not going to do it perfectly. You know no one is and you know I hear like people get upset about bugs you know over time I think it's just a wrong mindset. What I don't want to see is I don't wanna see a bug happen twice. So that's how you're testing suite gets larger. A bug happens. Number one you create a test that shows you the correct state. You run it and then it should fail again. If it doesn't, there's something wrong with your tests. Then you fix it. And if you've become a - I want to see you when I see a professional software engineer and I like to see this a Puppet space is you should get to the point where you're comfortable writing your tests first and then writing the code. You have a double check and you're going to see things run a lot smoother going forward.

Eric [00:34:03] I think that's one of the key - we mentioned infrastructure as code earlier and that's one of those key things that is a characteristic of a good system that allows you to represent infrastructure as code. It's not just is it text, can it be checked into version control, but can it be tested. Can you make assertions about what the code is going to do without actually running it and bringing that test driven development methodology into infrastructure code. It's one of the things that Puppet I think led the way on a long time ago and it's still like such such a core part of the experience of using it and writing it and running it in production is like getting that high degree of confidence before it actually goes out into the world that it's gonna do the right thing.

Bryan [00:34:46] So that's it, rspec was a minimum. We did our acceptance testing. I've done a fair amount of Beaker in Linux. I had trouble doing it in Windows. My expectation and I'm pretty excited about Litmus is I'm hoping that they ironed out some of the difficulties users have had with Beaker. So a solid acceptance testing there'll be the full scope and allow us to deliver and test cleanly. So that's, it's just it's it's a whole big, I guess what I was excited about. I'm excited about the promise of Litmus and I hope it's being realized.

[00:35:18] Yeah, again for the listeners. Bryan mentioned a couple of tools there. The first one is once over which is an open source toolkit that was originally developed by Dylan Ratcliffe, who's a professional services person here at Puppet, to do testing of control repositories like you talked about. And then Litmus is a new tool that is kind of an evolution for that acceptance level testing for modules so that you can fire off a server, fire off an agent, make sure that the agent gets a catalog correctly, and that the behavior once the catalog is applied is what you want. So that's kind of a more expensive test than just an rspec test which is more of a unit level test but it actually shows you what is going to happen on the systems when it's when it's live. And yeah, Litmus is definitely I would say informed by the lessons that the team that wrote it had from trying to run Beaker and you know it uses Bolt under the hood as the engine for going out and making stuff happen on systems so we get a little bit of reusability and you can plug into the Bolt content ecosystem as well through using the the Litmus testing tool. I think maybe we're at the end the conversation. Any parting words that you want to leave the listeners with?

Bryan [00:36:31] I'm just glad to be part of the Puppet sphere. Thank you for having me on. I appreciate it and I'm excited for Puppet.

Eric [00:36:39] Right on. Thanks for being on it. Safe travels back home.

Bryan [00:36:41] Thank you.

I hope to see you in Budapest, 4-5 June.

Contributor Summit is just around the corner and we can't wait to catch up with you. Budapest will be beautiful in June and we've got two days of workshops, collaboration, hacking, and discussions on the present and future of Puppet projects lined up for you. For some of you, this will be the first time we've met face to face, and that's pretty exciting!

And don't forget, Project Month is going on right now, and you'll have the opportunity to present your work at Summit. Maybe you'll even attract some contributors to help you out!

We realize that international travel might sound like it requires sacks of cash, but since hotel and food are so affordable in Budapest, the total cost of the trip might be far less than you expect. Try a few searches on your favorite travel website to see for yourself!

Don't forget to show up the evening before for the Vox Pupuli pinball party!

More Info:

• Contributor Summit

• CS Project Month

• Vox Pupuli pinball party

*Ben Ford is a developer advocate, principal software engineer at Puppet.*

I hope to see you in Budapest, 4-5 June.

Contributor Summit is just around the corner and we can't wait to catch up with you. Budapest will be beautiful in June and we've got two days of workshops, collaboration, hacking, and discussions on the present and future of Puppet projects lined up for you. For some of you, this will be the first time we've met face to face, and that's pretty exciting!

And don't forget, Project Month is going on right now, and you'll have the opportunity to present your work at Summit. Maybe you'll even attract some contributors to help you out!

We realize that international travel might sound like it requires sacks of cash, but since hotel and food are so affordable in Budapest, the total cost of the trip might be far less than you expect. Try a few searches on your favorite travel website to see for yourself!

Don't forget to show up the evening before for the Vox Pupuli pinball party!

• Contributor Summit

Ben Ford is a developer advocate, principal software engineer at Puppet.

Andrew [00:00:15] Hey everyone welcome to today's podcast episode. I am Andrew Nhem, managing editor of puppet.com. My guest today is Ben Ford, principal software engineer and developer advocate here at Puppet. Say hello to everyone, Ben.

Ben [00:00:34] Hey Andrew, how's it going today?

Andrew [00:00:36] It's going all right. We thought we had summer here in Portland but the rain came back but I'm cool with that because it's Portland.

Ben [00:00:42] It was a false summer, false start.

Andrew [00:00:43] Yeah, false start, false start. Well, we'll get there. We'll get there soon. Today's episode is pretty darn cool. We're talking about the upcoming Contributor Summit 2019.

Ben [00:00:54] Yeah it's definitely a thing. It's kind of been consuming my whole life for the last couple of weeks we're coming down to to the very end here. We'll be in Budapest, which the weather they're supposed to be a little bit like Portland I hope maybe it'll be a little bit warmer by then because we've got a lot of things planned out over there. I'm gonna do a 5k run alongside the event and there's some bike tours and some other fun things outside.

Andrew [00:01:22] Wow. Are you running a 5k to power the event, like on a treadmill?

Ben [00:01:26] Yeah we'll hook up a treadmill like and that's what's going to power our network.

Andrew [00:01:31] Are we gonna livestream that? Can we? Our producer's shaking - er, she's nodding. She's nodding. That means it's a go. Anyway, back to the subject at hand. For folks listening in who don't know what Contributor Summit is, can you tell us a little bit about it?

Ben [00:01:45] Yeah, so this is kind of in a way our very first Contributor Summit ever because in the past this has been attached to PuppetConf. And we are doing an experiment this year. We've kind of separated those out and what was PuppetConf in the past will be Puppetize PDX here in the fall in Portland. But this part - the Contributor sSummit we're doing that in Europe. This is our first Contributor Summit over in Europe. People over there have been asking for a very, very long time for us to to do some more events over there. We'll be in Budapest, like I said the first week of June and it's a pretty straightforward format. It's not going to be talk tracks the way that people are used to with conferences. It's going to be focusing on collaboration. So if you've ever gone to a Contributor Summit in the past you'll remember that like we had Eric stand up and talk about Bolt for example and there were a few other things where people gave presentations on the main stage but most of it was literally working with your peers, working with puppet engineers, and hacking on things, trying out new things, helping us sort of roadmap the future of different projects, and just collaborating with other people in the community. And we're really trying to keep that tradition real strong here. And we've got workshops on lots of the different products we've got going, lots of brand new projects and we've got plenty of time in the schedule for you to even bring your own. So if you have a project you want to share with people, we'll have slots in there you can just put in your workshop. And we've got lots of hack time, most of the second day is more or less free hack time.

Andrew [00:03:36] That sounds wonderful. It's great that it's a love letter to our European fans out there and it's great that the dial is turned more toward hands-on workshops, being able to meet and greet with like-minded folks, hack on your favorite Puppet project. That sounds amazing. I think it's exactly what the people have been clamoring for.

Ben [00:04:00] Absolutely. I mean it's it's great and all to sit in an audience and listen to somebody talk but it's even better if you can get your hands dirty and try out the thing that people are talking about on a similar note.

Andrew [00:04:12] I'd love to hear more about how people can participate. Where can they check out the projects in flight, pitch their own, join someone else's... where can we get more information?

Ben [00:04:21] So if you go to the Contributor Summit web page, and we've got a short URL pretty easy to remember: pup.pt/cs. It'll tell you all about it. We've got Project Month which is a way that you can start participating right now. And then there's also instructions and there's an agenda and kind of instructions on how to get started thinking about what sort of workshops you want to do and maybe some lightning talks you'd like to deliver. If you want to get started right now the Project Month is basically the full month of May. If you have something that you would like to contribute to sort of the world of DevOps - doesn't even necessarily have to be completely Puppet-related, just something that people in this space would find useful. Just sign up there, tell us what you're working on and kind of collaborate with us a little bit over the next month. We've got office hours running. We've got some different tutorials up and we've got engineers who are ready to answer questions if you need help getting started with like maybe the resource API or something. And then at the event itself I'll kind of run through all the projects from that month and describe them and tell everybody about sort of the outcomes, what people were able to build and put together. If you're at the event itself then you'll be able you'll have an opportunity to like stand up and pick your your project right on the main stage.

Andrew [00:05:56] That sounds great and I want to emphasize a couple of things that I heard out of that. As of recording this episode, we're halfway through May but as we all know projects never really end. So don't feel afraid to kick up a project now, pitch it, get some support, get some engagement. This is the reason why this event exists.

Ben [00:06:18] In all honesty what we what we're hoping comes out of Project Month is not just one month sprint and get something done, but a month sprint to get started on a project and get the first release out there for something that'll be sustainable so that you can continue working on it. You can continue building it maybe next year you can tell us something about it that's even bigger and better.

Andrew [00:06:38] Right on. The other thing that I heard out of that was also a huge nod to an inclusivity. One of the best parts about Puppet is that there are so many different ways to contribute, whether you're a documentarian, whether you have more of a UX / user user experience bend. Anyone can contribute, design. It's great in that Puppet projects are so inclusive in that regard and I hope that we invite a lot more folks who want to contribute more than code to the Contributor Summit and overall to our greater community.

Ben [00:07:11] So one thing that we're trying to really emphasize now - the word contributor is often thought of as being just like somebody who contributes code and that's true. It's like that's one of the ways you can contribute, but there are tons of ways you can be a contributor and participate in this community. If you contribute docs. If you file issues and help us fix things. Even if you just show up in Slack and help people answer questions. Or even just help help people clarify the questions that they're asking. All of these are ways to contribute.

Andrew [00:07:45] Well said my friend I'd love to hear more about the venue. Where are we holding Contributor Summit this year.

Ben [00:07:51] So I don't really know a whole town about the venue, but the pictures that I've seen are absolutely gorgeous. It's in the Budapest Music Center and it's basically it's like right downtown. It's just a block or two off the river there and all the pictures that I've seen just make me want to just like like wander around and just look at everything. And the space itself, the city and the neighborhood, there's so many things to do there. It is kind of crazy. I was going through a kind of a one of those travel guides to Budapest and trying to find out some of the fun things that we could do alongside the event and they've got everything from like boat trips on the Danube. I think that's how it's pronounced. And like tours and wine tastings and they've got the ruin pubs which is basically the idea of like putting together a bar in like a brand new upscale like really cool hip bar in the middle of a burned out hulk of a building. It's kind of alien to me but I'm like I want to go experience this it sounds like so much fun.

Andrew [00:08:59] Yeah. There's no better form of getting drunk than to drink anachronistically.

Ben [00:09:08] Budapest is really known for its spas. They've got apparently like this whole range of like spas - indoors, outdoors, hot, cold, whatever kind of a public bathhouse you want. And there's even this thing - you can sign up for a bike tour. So you rent a bike and you get a travel guide, like a tour guide to take you around and tour the city and all the historic spaces and eat goulash along the way. So it's like a goulash bike tour of the city.

Andrew [00:09:43] Fueled by goulash. I can get behind that.

Ben [00:09:47] I'm pretty stoked. That's definitely one of the things that I'm setting up for.

Andrew [00:09:51] When you power the entire Contributor Summit on the...

Ben [00:09:56] On the treadmill. It's going to be goulash power.

Andrew [00:09:59] Yeah, you're powered by goulash. That's what I like to hear. Powered by goulash and beer from the ruins. I see. And apparently we need to make sure according to the recent blog post by Deepak and Nigel we gotta make sure we say it Budapest (Budapesht).

Ben [00:10:16] Budapest (Budapesht). Apparently I mean I don't know. We could probably put the two of them in a room and just like have them bounce back and forth. Then maybe that's what we could power the treadmill with.

Andrew [00:10:26] Yeah. Or at least have it be the keynote for the event just two Puppet folks arguing for an hour. Unprecedented.

Ben [00:10:35] That'd never happen.

Andrew [00:10:39] I've heard rumors of an amazing pinball museum. Can you elaborate on that a bit?

Ben [00:10:44] So Vox Pupuli is our our community group. They are honestly pretty amazing. They maintain a huge number modules and they maintain an incredible velocity, keeping our community running. And kind of as a thank you to them we're helping them to host a party at the Budapest Pinball Museum. So the night before the event on the 3rd, we've rented the whole space and we've supplied drinks and snacks and all of that kind of stuff, you know the things you do at a party, and left it open for them. I don't know what they're going to do in the event. Tim's been talking about some really neat ideas maybe some like hack sessions right in the pinball museum, some prizes for reviewing PRs, but I think honestly the thing that's gonna happen the most is a lot of pinball going down.

Andrew [00:11:43] Yeah. Hacking on a pinball machine. Probably not. You get kicked out for that.

Ben [00:11:48] Somebody's certainly going to try though.

Andrew [00:11:50] Unless you're here at the Portland office where folks do hack on pinball machines. But that's another story for another time. So I'd love to know Ben, as developer advocate as beloved community member, what is the biggest thing you're looking forward to this year at the Contributor Summit?

Ben [00:12:07] In all honesty, the thing that I'm most excited about is the fact that I just got my daughter's passport expedited and so she's going to come with me. This will be her first international flight, which she has no idea how long this is gonna be, but she will in a couple of weeks. She's really excited to see someplace that she's never been before. She's even put together a like a shared note that she's like a list of all the fun things that she wants to do in Budapest. Of course there's no words on it is just pictures. So there's like 13 pictures of things that she wants to go see and do. So I'll have to do a little bit of sleuthing and figure out what those pictures actually point to.

Andrew [00:12:47] That's a pretty cool game. That is wonderful, Ben. All right everyone, so you've got a bunch of details from Ben Ford, developer advocate here at Puppet. If you'd like to learn more, where can folks go?

Ben [00:13:00] pup.pt/cs or they can hit me up on Slack - I'm binford2k. Ask me any questions you've got or even just drop in to the Contributor Summit room and see who's going and what else is happening there.

Andrew [00:13:14] So for everyone listening or who's decided to participate and listening at the end of this podcast, Ben give them the rundown of Contributor Summit 2019.

Ben [00:13:21] Contributor Summit 2019 in Budapest. This is the first ever standalone contributor summit. We really really want to see you there. This is going to be amazing and Budapest. First week of June - 4th and 5th and we're kicking things off with a pinball party the night before and Vox Pupuli would love to see you there.

Andrew [00:13:42] Very awesome. And if you just started paying attention it's mid-May. May is Project Month. Get those projects rolling people. We want to thank you all for listening as always, and we're signing off.

Ben [00:13:55] And I'm really excited to see what we put together here.

Andrew [00:13:59] Right on. Thank you so much Ben. Take it easy everyone.