If people aren't listening to your podcast then you're doing something wrong. Fix your episodes, reach more listeners, and build a show that you can be proud of.