In PHQP 0022 Reliability And Validity Testing, Jeff reflects on the long, unstructured summer vacations of his childhood, contrasting them with today’s shorter, adult-scheduled breaks that limit children’s freedom to self-regulate and pursue their interests. Then, he critiques the lack of reliability and validity testing in early learning accreditation and quality rating programs.

Watch Now: PHQP_0022 Reliability And Validity Testing

Welcome to the Playvolution HQ podcast. I’m Jeff Johnson. Thanks for pushing play on with the show.

So, recently, coming back from Minnesota where I did a live weekend training, I had a great conversation with an Uber guy. We were about the same age and we got talking about summertime back when we were kids and going to the drive-in movies and jumping our big wheels off of things and fireflies and adults not really worrying about us and letting us play and have a good time and how that wasn’t always perfect. It was often wonderful playing outside till the streetlights came on, all of that kind of stuff, and it was a nice chance to reminisce with a peer, which got me thinking about topic one for this episode of a two-topic episode.

A couple weeks after that, just a couple days ago now, I run into a kid. He comes over to say hi to Gigi, my pup, when we’re out on a walk and I say, hey, I hear you. Yesterday was the last day of school.

He looks at me like I’m an idiot and he says, no, I have to keep going back until I’m almost an adult. He’s not off of school. He’s just on a short break until he’s got to go back and then go back and then go back until he’s almost an adult.

That kind of struck me as funny because back in the day, back when me and that Uber driver were kids, summer vacations seemed to last forever. I dug into this a little bit and it turns out that total school days hasn’t changed much since the 50s and maybe even earlier, 50s, 60s, 70s, 80s. It ends up being around 180 days of school and every state and every district’s got their own thing, but that seems to be about the average across the U.S. What has changed is how those days are disbursed.

We used to have a big block of time off during the summer, back in the day. It was almost from the end of May, beginning of June, to the end of August. Now it is often much, much shorter than that.

The days off hasn’t changed. They disperse them different because there’s more days off during the school year for teacher work days and slightly longer spring breaks and those kind of things. Maybe those things are good, but what those things are mostly for is to make things more convenient for adults.

Look, I get that, but what’s missing is those short summer vacations do kids a disservice because we don’t have the time to decompress and be away from adults and follow their own interests and have that freedom that we had back in the day. That’s something that’s missing because one of the troubling things about modern childhood is kids are always under the thumb of somebody else’s schedule and somebody else’s timeline. It’s really hard to learn how to self-regulate and know your own mind and follow your own interests when somebody’s always shuffling you from this activity to that activity.

Longer summers would do a lot for kids. Of course, that’s not going to happen. On topic two, that was kind of depressing, and this topic is going to be depressing too, if you want to take it that way.

It gets in my head every once in a while. I got to think about reliability and validity testing a couple weeks ago, and I decided we needed to do an episode about it because a lot of times in early learning settings, we don’t think about this. Reliability and validity testing is a thing.

Accreditation programs and quality rating systems should take advantage of this thing, but it turns out that many to most don’t. That should be a problem for the people that are living under those standards. Let’s dig into that a little bit.

My brain shut off there for a second. Excuse me. Reliability and validity testing validates the tool’s quality.

If accreditation and quality rating systems are asking caregivers to be parts of those programs and to see them as valuable and to trust the ratings they dish out, those same caregivers should be able to see those tools as valid in measuring quality or whatever else they’re attempting to measure. Reliability and validity testing is conducted by an independent third party, so the organization that sponsors the quality rating system or the accreditation program has a third party come in, usually a university or private company, come in and do a study to measure reliability and validity and see if the tool is actually measuring what it says it measures and is doing that reliably. Reliability and reliability and testing means consistent outcomes.

Does this tool have consistent outcomes? If I sent 10 people into this program to evaluate it, would they come out with the same general scores, the same general evaluation of that program? And that’s important because we want the people who are using this tool to all kind of be on the same page. We don’t want a big disparity in how they rate things. Validity measures whether the tool measures what it claims.

So if a tool is claiming to measure program quality, for example, validity testing tests to see if it actually measures that, which is also something caregivers should be concerned with because if the tool doesn’t actually measure what it claims, then it’s not a real valuable tool. Programs, like I said earlier, are lacking in quality assessment tools, this validity and reliability. There are a number of them who don’t do this testing when they should.

And the reason they’re lacking is this process is expensive and it’s time consuming. And it has to be repeated on a regular basis over time. It’s like every five, I think five years is what is the prime recommendation.

Maybe every 10 years the tool is reassessed for reliability and validity or every time they make changes to the tool. And it’s troubling when they don’t follow through with that. Should you trust a tool that has not been tested for reliability and validity? My response would be no, you shouldn’t.

You’ve got to judge this for yourself. And should you trust an organization that’s asking you to trust a tool that hasn’t been measured for reliability and validity? And again, my take would be no, no you shouldn’t. And yet these organizations are out there and I’ll name drop a couple of them.

One is the National Association for the Education of Young Children and the other is the National Family Child Care Association. Both of these programs, these organizations, have accreditation programs and it is very unlikely they’ve done reliability and validity testing because if they had, they would be touting this on their website as a feather in their cap. I’ve served on the National Board of the Family Child Care Association and on the State Board of the NAYC for the state of Iowa for a bunch of years and at no time in those positions could I find any proof that this kind of testing was done on a regular basis.

And since then, and before serving on those boards, I dug into it and could not find anything. I’ve recently done another big search, including using some deep search from some different AI models that can go through and find things a lot faster than I can, and this reliability and validity testing don’t exist for these accreditation programs. Now does that mean they’re bad programs? Absolutely not.

They may be very good at reliably measuring what they say they measure, but there’s no proof of that. And so caregivers who jump through all of the hoops for accreditation or quality rating stars or whatever it is, are maybe jumping through hoops for something that doesn’t mean what they think it means. And it seems kind of disingenuous for an organization to say, hey, you should, if you want to be a quality program, here’s what you need to do, when they haven’t, that organization hasn’t actually done what they need to do to prove that the tool they’ve come up with actually accomplishes that goal.

So some things to think about and maybe some questions to ask if you’re considering accreditation or being part of a quality rating system or already are, because I think a lot of caregivers aren’t aware of the idea of reliability, validity testing, and they go through these programs without actually knowing that they might not be accurate. There are some cool tools out there that do do this measuring, but there are many that don’t. And that’s a problem in this profession.

Wrapping up, go out and ask about reliability and validity testing if it’s something you don’t know about and you’re curious about. And if an organization gets back to you and says they haven’t done such testing, dig into the why. And look, if you dig into this, I’d love to hear what you find out.

Amazon idea for this month. Oh, we skipped. I’ve got to go back here.

We’ve got to transition over here. So this is somebody using my Amazon link bought this stuff. This is a six inch by 10 foot puncture resistant PVC dust collecting hose.

Now you can use this in a woodworking shop for collecting collecting sawdust, but they also work great for all kinds of stem play things going on in early learning settings, kind of mobile ramps. Back in the day, we used to keep sections of this outside on the playground all the time, and they’d be dumping sand and water down them and yelling through them and stretching them, and they would be giant snakes and all kinds of stuff. Real durable stuff lasts a long time and well worth the investment.

I don’t know if that’s what it’s going to be used for, but that sure is what we used it for. Good stuff. Transition back here.

Again, share the show if you like the show. Share the stuff on Playvelation HQ if you like that. I appreciate it.

Next week, we’re going to talk about fit for care. We’ll see what that means next week. Dad joke of the week.

Okay, here we go. Did you hear about the claustrophobic astronaut? He needed space. All right.

This has been the Playvelation HQ podcast. Back soon. Thanks for listening.

Bye-bye. This has been an Explorations Early Learning Upstairs Studio production. Oh, Jeeves, it’s hard to believe it is June of 2025 already.

Like, I know you’re young, but in my mind, it’s like September of 1973.

Contribute content to Playvolution HQ
Brought to you by Explorations Early Learning