'Tis the season for reflection, so Jen and Rebecca use Episode 6 as an opportunity to talk about what they learned this season and reflect on ways in which they can get better at their work, particularly with respect to approaching their work through a lens of inclusiveness. We hope you benefit from hearing all the ways in which we can improve in our work and hope our episodes this season can prompt some reflection for you as well.

Friends, this is our season 4 finale. Thank you for coming with us on this ride. To reiterate, we love the episode suggestions we receive from our listeners, so please keep them coming in. Until next season, stay well!

We had a great chat with Heather Krause, Founder of We All Count Project for Data Equity. Heather is a statistician by training and is presently using her skills and passion to improve equity in data science. She practiced data science in an international setting for many years before realizing many of her own practices were problematic, both reflecting and reinforcing western ways of thinking and often de-legitimizing the values of the people and groups she was studying. Rather than blow up her career, Heather thankfully imagined a new way of doing data science that would marry rigor with equity in the planning, execution, and communication of quantitative research. She now works with a range of clients from the public and nonprofit sectors, as well as academia and other research institutions, to help them identify areas for improvement and tangible practices to improve their own research.

We talk with Heather about what data equity means (and, for that matter, what data science means), the many hats she wears not only as a data scientist herself but as a therapist for researchers who are learning that their long-held practices and traditions may be inequitable in both obvious and not-so-obvious ways, and her philosophy that there is no trade off between rigor and equity in data science (and in fact they should be mutually reinforcing!). It was a great conversation, and we know our listeners will appreciate Heather's candor and thoughtfulness.

Friends, Heather brings The Goods. First of all, bless her soul, she's a disciplined blogger. Her website is also chock full of tools and resources which you can use to inform your approach to data equity. If you're just now starting to think about approaching your work through the lens of equity, we suggest you start with these resources. The website also offers a Data Equity Framework which researchers can use to think through some of the thorny issues that arise when planning a study, collecting data, analyzing data, and reporting on results. Check out the tools and framework, and if you're interested in digging deeper, please reach out to Heather for more information!

We talk this week with Martha Brown and Debbie Gowensmith about trauma-informed evaluation. It was a great chat, but, before we jump in, here's a little background on our guests. Our first guest is Martha Brown, president and founder of RJAE Consulting. A well-known evaluator and trainer, Martha's expertise is in the area of trauma-informed practices and restorative justice. She joins us alongside Debbie Gowensmith, vice-president of Groundswell Services, Inc., a consulting firm that specializes in strengthening equity-building movements. Debbie's expertise is in community-based research and program evaluation, and she also has a background in program management, which she brings to bear on her evaluation work.

We found our guests after reading Debbie's January 2021 AEA365 post on using Photovoice to support trauma-informed evaluation. Debbie introduced us to Martha, and the rest is history. As we mention in the introduction of the episode, we really only scratch the surface in this conversation and would probably benefit from another couple hours with Martha and Debbie! We consider ourselves newcomers to this work, so Debbie and Martha patiently covered some "Trauma 101" material with us before we moved on to discussing trauma-informed evaluation. We talk about different evaluation approaches and methodologies that are better suited for trauma-informed work, but really Martha and Debbie encourage evaluators to assume a trauma-informed lens irrespective of their methodology. Whether we're doing Photovoice or field work or surveys, we should be mindful of the prior experiences of both research participants and our staff and be careful not to engage in practices that re-traumatize individuals or communities.

Debbie and Martha referenced and shared a few resources, which we found helpful and will therefore pass along to our loyal listeners:

We really hope you enjoy the conversation as much as we did!

FYI, if you'd like to reach out to and/or learn from Martha, you can find her at:

Episode 3 brings us Jennifer Taylor and Sarah Goletz from the Indiana Area Health Education Center (AHEC) to talk about ripple effect mapping (REM). We knew very little about REM, so we sat like schoolchildren at the proverbial feet of Jennifer and Sarah as they patiently walked us through the basics. If you too are unfamiliar with REM, it's a method that combines elements of appreciative inquiry, mind mapping, and participatory evaluation to ascertain how participants and others involved in or affected by a program describe the program's outcomes and impact. It can also be used to identify a program's unintended consequences (both positive and negative) and inform recommendations. This page on the University of Minnesota Extension School's website has some great resources that help explain how it all works. At a high level, it involves gathering 12 to 20 stakeholders in a room, guiding them through a process where they talk in pairs about elements of the program they appreciate or find impactful, and then facilitating a discussion wherein participants share out their ideas while facilitators organize and map these ideas on a large sheet of paper, whiteboard, or projector at the front of the room. The result is a cool map that visually depicts a program's impacts (see example here).

Jennifer and  Sarah stumbled upon REM in their own work at AHEC, and now they not only use REM in their own research, they also happily teach others how to do so. Though COVID has put a temporary damper on some of this work, they were able to host a workshop in Fall 2020 and are happy to serve as evangelists for the method in any way they can. If you have questions, hit them up!

Friends, we have a doozy of an episode. Andy Johnson from Inspire to Change  joined us to talk about arts-based evaluation, creative evaluation, and applied neuroaesthetics. We could have talked to Andy all day, but professional norms dictated that we had to allow him to leave the meeting after our scheduled time expired. Bummer, but if we can have him back, we will!

Andy is an artist-turned-evaluator, so he brings an interesting perspective to his work with nonprofits. We wanted to chat with him because of this season's theme around inclusivity. Andy and his team have really unique ways of ensuring that their approach to building hypotheses and theories about program include all perspectives, and also that their way of presenting data reflects the many ways in which people prefer to access and interpret data in the real world.

We discuss about some of the unique ways in which evaluators integrate the arts into their work, with respect to both methodology and presentation. Andy introduced us to the visual matrix method, which is a way of using visual art to elicit research participants’ thoughts, sometimes subconscious thoughts, on a particular topic by having them freely respond to art work. We also talked about verbatim theater as a way of presenting qualitative data in a way that is accessible both to people who participated in the research and to consumers of the research. We added links here in case you want to check it out.

As a quick plug, Andy’s team is going to offer two classes this fall -- 1) creative evaluation for internal evaluators and 2) writing for social justice. You can visit the events page of their website to learn more about them.

We are excited to be kicking off Season 4 of the Glass Frog Podcast with  evaluator and author Chari Smith! Chari is the founder of Evaluation into Action,  which partners with nonprofit professionals to create realistic and  meaningful program evaluation processes. She teaches workshops aimed at  helping nonprofit professionals understand the value of program  evaluation, and recently published her first book that helps early  career evaluators, nonprofit professionals, and students learn how to  create and implement an evaluation plan. The book, Nonprofit Program Evaluation Made Simple: Get Your Data. Show Your Impact. Improve Your Practice, is on shelves, so order your copy today!

Chari is a proud member of the Oregon Program Evaluators Network  (OPEN) and the International Society of Evaluation Education (ISEE). She  is also an active member of the American Evaluation Association (AEA)  and serves on their Nonprofit and Foundations Topic Interest Group (TIG)  Leadership Council. As of summer 2021, she is also an adjunct faculty  member for the University of Portland’s Pamplin School of Business  Nonprofit MBA program. Chari is passionate about her work and brings her  enthusiasm, insight, and expertise to each project.

Chari is also available to speak on a range of topics that will both  inspire and inform. If you have an upcoming conference or event, please  reach out to her through her website. Feel free to view her opening plenary session at the American Evaluation Association conference 2020.

Participate in a workshop with Chari! On Thursday,  July 29 at 9:00 AM PST, Chari will be giving a workshop titled  Measurable Outcomes Made Simple. If you work with nonprofits or want to  learn a more collaborative approach to creating outcomes, please join  Chari. You can learn more and register for the workshop here.

You can connect with Chari on Twitter (@evalenthusiast), LinkedIn, or by e-mail at [email protected]. 

We talk with Chari about her artistic interests as well. She wrote a song about evaluation (!), which you can hear on her website. You can also learn more about her creative endeavors using the links below:

We are back for season 4! We have a great lineup this season, and can’t wait to learn from our guests alongside our listeners!

Welcome to the final episode of Season 3 of The Glass Frog Podcast!  This episode we got to talk with evaluator, consultant, cartoonist, and  all around nice guy Chris Lysy. We’ve been following Chris on and off  for a few years now through his blog. He recently published a book of evaluation-themed cartoons, Evaluation Illustrated, and we took that as sign that we should have him on our show.

We talked with Chris about digital evaluation, the non-predictable  and non-linear dimensions of his career as an evaluator, and of course  about his cartoons. If you haven’t checked out the book, you should. (It would be a great holiday gift for your evaluator friends and family members!) A few of our favorites are below.

Chris has an active newsletter and also a new Patreon where he shares more cartoons and keeps you up-to-speed on his work. Subscribe to both for a  regular dose of humor and thoughtfulness amidst these crazy times.

This is our final episode of the season. We’re already planning for  Season 4, so, listeners, please don’t hesitate to reach out if you have  ideas! It’s been fun being on this journey with you!