How to measure social impact (and how NOT to)

I once had a job where one of the reportable metrics was simply “meetings attended”. Nothing created, nothing sold… just meetings.

I remember applying for a grant for this organisation, and feeling naturally bummed when I hadn’t been successful. But that didn’t matter… I was told to just pop it down on the “reportables”. I had ‘tried’... and that was enough.

Now this role might sound like bliss … where everyone gets a gold star, where you can turn up and achieve nothing and hey, you still get paid. But for me, it had the opposite effect. 

I wanted to contribute to the social good that we had been established for. I believed in the mission. So, I genuinely did ‘try’ for that grant. Now, I would hate to work somewhere where I was penalised for not being successful, but it was almost just as disheartening to know that it really didn’t matter if I put any effort into it at all. No one was looking.

What concerned me was not about trying and then failing. I am a big believer in “the fail fast philosophy” from the tech world and design thinking. That’s how innovation works. However, if you’re not learning anything in the failing process, then there’s a problem. And there’s a problem again if you get to report the ‘fail’ as a ‘win’... meaning you can do the same ineffective thing as a win over and over… and continue to be funded to do so.

It’s a trap some well meaning organisations can fall into, especially when they exist to tackle a big, systemic problem, involving social change;  such as homelessness, poverty, mental health or climate change. They get busy working… and need to report their efforts to their funders at regular intervals. Because they haven’t ‘solved’ the big bad problem… they don’t report outcomes, but rather the ‘doing things’. And can fail to see the difference between the two.

The other problems with the metrics for social change that I see, is that incentives are misaligned. For instance, when universities step into the social impact space, some will use the academic metric of ‘papers published’ as the gold standard measurement, without any concern about whether their research makes an impact on the ground. Worse still, is that career progression for an academic is based on their papers published - not any other work  which might have a bigger impact; such supporting their research to be enacted on the ground or even teaching. Some academics do this of course…but they aren’t rewarded for it. 

Thinking like a business, even if you’re not-for-profit

So what’s the solution? Think more like a business. Have a strategy… have KPIs…. Even if you hold them loosely. The good thing about business is that it’s generally easy to tell if something is working, when you look at the cold hard truth of the profit and loss spreadsheet. I’m not advocating for not-for-profit organisations to actually become businesses, but simply to borrow a few business-like mindsets and metrics.

Personally, I’m still developing my social impact metrics for Rural Podcasting Co. But I know my purpose - which is to help rural people tell their own story. And then ‘my why?’ (Thanks Simon Sinek) is because so many stories are told about us as rural people, not by us. And they’re not always accurate, fair or capture the nuance and diversity of rural lives.

So, in a year for instance, I can count, how many new rural podcasts did I help get off the ground? How many listeners did they reach? And also what did these podcasters go on to achieve?

This is where the impact and the ‘doing things’ can become confused. Firstly, podcast metrics on their own are crude and potentially misleading when it comes to impact. Some people become disheartened when they see what they think is a low listenership. However, if an episode gets an issue on the national or even local agenda, or gets you in front of the groups of people you want to be in front of… then it has an impact beyond the numbers. 

So, how then do you measure the social impact of a podcast?

I’ve been thinking about this for Ducks on the Pond, a podcast for rural women that I started in 2021. Its primary purpose is to raise the voices of rural women and the issues that are important to us. So yes, it does that - but how do I measure whether it is contributing to greater social change?

So far, I have come up with three key metrics.

1. Reach (Downloads + any other media coverage + key people or groups of influence)

2. Is it financially sustainable?

3. Have any of the episodes led to positive change or other outcomes of improvement?

Here’s the thinking behind it:

I will never know if someone hears an episode and feels inspired and goes on to do something great … or if an episode was one of the many things that made someone improve their life in some way. I do get messages about episodes that tell me it has helped or inspired a listener in some way (and thank you - they mean the world to me). 

Occasionally an episode goes on to do something bigger - like a recent episode about what NOT to say to a mother who has lost a baby (and how to best help her instead) about baby loss. It went on to become a story in the mainstream media, reaching a much wider audience. And the groups of mothers I spoke to are now using the original podcast episode as a resource - to be given to Warrnambool mums who have lost a baby.

Sustainability is also a metric for me. Rural podcasting Co. is a business, so I must also ask about my own podcasts … are they self sustaining, at the very least? But even if I became a not for profit organisation, finances can’t be ignored. Podcasting can’t generate social impact if episodes can’t be produced.

Finally, positive change or action. The science minded amongst you are cringing at metric number 3 right now. I hear you. How can you possibly know cause and causality? Well, you can’t 100 percent of the time, but welcome to the murky world of people, which is what social impact is all about. In the baby loss episode example… a resource has been created for grieving parents. So that’s an action or outcome that has come from the podcast episode.

I could simply count the number of podcast episodes I have produced at Ducks on the Pond. But that’s the ‘doing thing’... NOT the impact. The ‘doing things’ will of course look different, depending on your organisation. But being clear on what these are versus the impactful outcomes, is critical.

As for me, I’ll keep refining my own metrics. I’m not after a perfect, scientific answer… just an effective one.

Kirsten Diprose is the founder of the Rural Podcasting Co, which helps individuals and businesses tell their own stories through podcasting. Her podcast is called Ducks on the Pond. Kirsten is also studying a PhD in podcasting and local news sustainability. She lives on a farm in south-west Victoria with her husband and two children.

For podcasting tips and recommendations, sign up to my  Newsletter!