Shifting our world’s systems toward greater inclusion and equity is complex work. It requires developing a shared understanding of progress and clear signals to guide the way. What are the most effective tools and approaches we can use to measure systems change and promote learning?
A group of innovators and funders came together at the 2024 Skoll World Forum to share insights from their evolving journeys in measuring systems change. Collective Change Lab‘s John Kania framed systems change at multiple levels, from explicit policy changes to deeper relational and transformational shifts. Panelists noted the limitations of current measurement tools and the need for nuanced, qualitative approaches rooted in values. The discussion also turned to the practice of involving communities, addressing power dynamics, and clearly defining the nature of systems.
Watch the full session and read the transcript below to learn more from Kania, who spoke alongside Natasha Joshi, associate director at Rohini Nilekani Philanthropies, and Nivi Sharma, CEO of Bridges to Prosperity. Moderated by Søren Vester Haldrup, a transnational corruption and financial integrity specialist for the United Nations Development Programme.
Transcript from “Measuring Systems Change,” filmed on April 12, 2024 at the Skoll World Forum:
Søren Vester Haldrup: Welcome to this session on Measuring Systems Change. It’s really great to see so many of you here. Who would’ve known this would be such an interesting popular topic. I’m facilitating today’s session. And today’s session is not one where you’ll be talked at for two hours, but one where you will also get actively involved. But we’ll start out with a few presentations from an excellent panel. After that, we will break out into groups, and you all get an opportunity to reflect on what you’ve heard, to share your own experiences and to learn from others at your table. More instructions on that, will come in a bit, but let’s dive in.
Many of us, many organizations and people not least the ones at the Forum today, are working on catalyzing systems change. That’s what we have in our missions and what we’re aiming towards. Most of the big tough challenges these days can’t be solved by standalone, incremental, single point solutions. We need to work on how to transform complex systems, but doing that is not easy. And measuring it, knowing if change is happening and understanding why it’s happening and what role we might be playing is really tough because system change is a complex thing.
There are so many different things we could potentially look at to get a sense of where the change is happening. So what methods or frameworks are useful and when are they useful? What types of data do we need to do this type of work? What kind of time horizons should we work with given most of us have annual or five-year planning and funding cycles, but systems change might take decades to materialize? Who decides what is important to look at in a system, like who makes those decisions defining what good change looks like and how do we measure what matters rather than just what is easiest to measure? These are some of the questions we hope to cover in today’s session.
Fortunately, we have a great panel of speakers to help guide us through this and, and inspire us. So with that, I would like to bring our first speaker up on the stage. John Kania is perhaps the excellent, the perfect person to kick us off here. John has a wealth of experience. He’s the Executive Director at the Collective Change Lab and formerly the Global Managing Director at FSG. And you might know FSG from the Water Systems Change framework that some of you might have used. John, welcome to this panel and please.
John Kania: Thank you, Søren. Am I on? You guys hear me? Great. Well, hello everyone, good afternoon. I’m going to, sorry?
John Kania: Good afternoon. Thank you, yeah, I got to wake up. Yeah, so I’m going to start with a bit of a foundational perspective on systems change. It’s one, there are many, it’s one that I find helpful and then I’m going to talk about two approaches to measuring systems change that I have found to be both helpful and compelling. And I’m going to begin with this slide.
And I have been told that if you put a baby in your presentation, people will like you better. There’s actually legitimate research on this. So that’s one reason I put this in here. The other is to remind all of us, including myself, of what systems actually are. I think often systems can seem like they’re faceless, nameless institutions, but systems are made up of people, people who were once babies. And if you’re the parent of a baby, or an aunt, or the uncle of a baby, you want that, that baby to grow up healthy intellectually, physically, spiritually. But you can’t like grow for the baby, right? The best you can do is sort of like affect their condition, the conditions in the context around them. So I think that’s a pretty good metaphor for systems change because systems change is really about affecting conditions, actually trying to put better conditions in the way of people who are trying to create change or remove barriers.
My favorite definition of systems change, I don’t know how many have seen this, actually we had it in the Water of Systems Change, is that systems change is about shifting the conditions that hold the problem in place. Colleagues of mine and I a number of years ago said, we want to take this deeper and what sort of, when we look at the conditions of systems change, what are they? And we came up with this very poetically named framework called the Six Conditions of Systems Change. I’m curious how many people have seen this before? Okay, for those of you who haven’t, a couple of things to keep, sort of take away from this.
One is, systems change happens at three different levels. The first level is what we call structural. You see, it’s policies, practices and resource flows. Often when I talk to people about systems change, they’re either talking about policy change or sort of scaling practices or both, but they’re missing some dimensions that are critical when you think about systems change. So the second level is what we call the relational level. The first level you can see, that’s explicit.
The second we said is sort of semi explicit. And you know, that’s relationships and connections. You can’t shift a system if people aren’t aligned. So that’s where relationships come in. You can’t shift a system if people are poorly connected. They have to be connected. Steven Johnson said of individuals who are connected, he said chance favors the connected mind. You can say the same of systems.
So the second level also includes power dynamics. One way to think about power dynamics is that there’s sort of three things that you should always be considering. Who makes the decision? Who sets the agenda and who actually controls the narrative? Cynthia Rayner and François Bonnici in their excellent book on systems change wrote about that. The deepest level is what we call transformational systems change. And that’s mental models. So we can think about mental models as Indigenous, might be nature-based or oversimplify it in Western modern might be sort of science or technology-based. Those are two different mental models. When we think about sort of what’s going on, on the left and the right, we have two different mental models. So all of these, in fact, affect how you think about systems change.
Now, I could go on and on about this slide and I actually have, I could give an hour long talk just on this slide. But the one thing I want you to think about, and it’s the reason you’re here, you already know this, systems to measuring systems change is complex. Measuring that is complex, right? So how do we do this? There are a whole host of things that come up when you think about it. Change happens at multiple levels within a system. It defies simplistic quantification, distributed actions by multiple actors and sectors, lag time between critical activities, changes interact with each other. It’s typically not caused by a single actor. And the results are connected and cumulative and building on each other over time.
Again, we could spend a lot of time on this, but the important thing is, if you’re going to measure systems change, you have to take these into account. And if you’re not, then you’re in a bit of a denial about how well you’re actually measuring systems change. So I’m going to talk just about two ways of measuring change and I talk here about making meaning. To me, that’s what’s most important, that we’re trying to help the stakeholders in the system make meaning of what’s going on.
Robert Ricigliano at The Omidyar Group talks about, instead of evaluating or measuring systems, should we really need to think about how do we help people find their way? So I’m going to give you two very specific sort of approaches to helping people find their way and make meaning in the system.
The first is called outcomes harvesting. It was developed by Ricardo Wilson-Grau, sort of a international development guru in terms of evaluation. Outcomes harvesting, here’s his definition, it’s about collecting or harvesting evidence of what has changed. Outcomes are defined as what has changed and then working backwards to determine what and how the interventions contribute to this change. And Ricardo worked a lot in international development and Latin America around sort of food systems and agriculture and health and a whole bunch of different areas to actually come up with this process.
I’m not going to take you into this example. This is actually looking at corruption in Burundi. This was a World Bank example of where they did outcomes harvesting and they were looking at how quickly principles were getting paid because some weren’t getting paid for over a year. What I want to share with you in all of this, again it’s still in line, you’ll get it is sort of the structure. The structure that they thought about.
They actually said there are different types of outcomes, institutional changes, which would roughly equate with our top level that I just took you through. Learning capacity changes which would roughly equate to that relational. And then they said there are other outcomes and what they list under other outcomes is like mental model changes. And if you look at this picture and you say what, you can’t see it up close, but if you actually have a chance to look at this, what’s in blue is like the mental model change and it’s affecting everything else.
So the great thing about mapping like this is you can get a sense of time, what’s happening over time. You can get a sense of proportionality with the outcomes. Some outcomes are more important than others. It’s a really amazing way to try to make meaning of what’s going on. It’s a very straightforward process. You know, I invite you to, it’s basically we go out, we collect data from a lot of different stakeholders. We try to verify that data and then we harvest it. It’s not that complicated.
Couple of things that we get out of it, it’s qualitative and quantitative data. I can’t sort of emphasize that enough. If you want to measure systems change, you need both qualitative and quantitative. Qualitative to set the context, describe the quality and offer meaning. Quantitative to demonstrate scope scale and depth of change. And then if you go online, there’s a host of resources around this, just Google outcomes harvesting. And I just want to cite Mark Cabaj, who has been sort of my tutor on this, he’s an evaluator in Canada, an incredible guy. So his last name is C-A-B-A-J. And if you really want somebody who really knows their stuff, he’s great.
So the second, I’m going to switch a little bit and talk about a very different approach to thinking about systems change and that’s through storytelling. What do you get out of storytelling? One, you get emotion and feelings. Well, systems don’t have emotion and feelings. Of course, they do. They’re people, right? So how do we actually get at that? Okay, we also want to sort of really understand, sort of direction and momentum, et cetera, that’s hard to get out of sort of quantitative indicators. But a storytelling approach can take you there.
We developed approach and we had many people from around the world that were involved in developing this approach towards storytelling because we wanted to get it at the question of how do we tell systems stories in a way that actually approximates the way social change happens? Our definition is system storytelling is a process for pluralistic groups to come together and make meaning of systems they are seeking to positively influence toward equity and justice.
Why are we doing this? Well, systems require many different perspectives, right? And by and large, the stories that are told out there right now are, Joseph Campbell called them the hero’s journey. It’s one person who rises above the challenges and that’s not how systems change happen. So we have to find a different way and we can also through these storytelling practices, which are relational, keep people together.
You can’t see up here, I’m sorry, with the group that we worked with, which were storytellers from around the world that had at least one foot in social change, we came up with different themes of how to think about system storytelling. So one is multiple perspectives. You’ve got to go out and get multiple perspectives and bring them together. Another is thinking about intergenerationality. Systems don’t, just as we know from a standpoint of individuals, we need to think about intergenerationality and intergenerational trauma, we need to think about same way, what’s the backstory on the system? So we actually identified nine different themes that ask, really allow you to sort of take a lens, a systems lens into what’s going on in the system.
This is just sort of a look at sort of how we do this work. We do it in circle. So if you want to change the system, get the system in the room so we get representation of the system in the room to tell stories to each other and we start at a personal level. We go to an organizational level. We go to a systemic level and then we harvest. It is highly relational and just a really powerful experience in a different way of seeing into the system. I’m just about done.
One thing I will say is, I think, particularly in the western world when we talk about measurement and evaluation, we tend to cleave it off and say, “Let’s figure out what’s going on,” and then go do our other work strategy, et cetera. This sort of approach where you’re coming back together regularly telling stories, system stories to each other allows you to understand, cohere because nobody can change a system by themselves, imagine and re-narrate.
So if you want to learn more about this particular approach, you can go to our website, collectivechangelab.org and look at the system storytelling resources we have. I will just end with this. I’m going to bring back the image of babies. Babies become people and people make up systems, and I can think of no higher purpose that you can have to measuring systems change than to support the making of meaning within the systems by the people who occupy them. Thank you.
Søren Vester Haldrup: Thank you. Thank you, John. That was excellent. So many themes to pick up on here. I’m going to try to just pick up a few. I think one point on the fact that you have these different levels, you have many different causal factors and people and actors at play that all interact. So most of the ways that we are approaching monitoring or progress or results or change are based on very linear causal models. Those type of models are not necessarily very helpful if we deal with systems change where all these different things interact.
Also, very much like your point on making meaning because I think some of the things we sometimes conflate when we talk about measuring systems change is, are we interested in measuring change or evaluating whether good or bad change is happening? Are we interested in just learning about whether change is happening so that we and others can use that to make decisions and adapt? Or, are we interested in capturing results and evidencing our contribution to that, like a very much an accountability function and not a learning function? And clarifying some of those things I think are, is very useful and important.
Now, I’d like to bring in Nivi Sharma. Nivi is the CEO for Bridges to Prosperity, working on rural infrastructure development, currently executing a systems change strategy where they’re trying to look at how rural infrastructure can be a lever for water poverty eradication. Nivi, thank you for being with us today. Please tell us more about how you’re approaching that work.
Nivi Sharma: Thank you very much. I thought I’d start by getting a show of hands of other, who are the implementers in the room working on social impact? Oh good. I thought it was a room full of people with deep pockets. Well, everyone else has deep pockets and is hopefully funding systems change. But keep your hand up. Also show me if you are either on the journey to scale or the journey to systems change as an implementer. Okay, so lots of empathy on how hard this is.
My name is Nivi, I’m the third generation of leadership within this organization. In fact, today is our 23rd year anniversary of the organization. Yeah. So the organization started with this picture in the National Geographic landing on the lap of a wealthy American who said, “Well if someone needs to do that, either to themselves or their children to get from point A to point B, I bet whatever’s that point B is pretty important and there isn’t another alternative and I can go and build a bridge there.” So it started with a really simple story of direct implementation. A man who flew from Vermont to Ethiopia and built a bridge in Ethiopia. Lovely story, beautiful story. And from that grew the mission and vision of this organization.
Now, the mission started off as building bridges and the vision was unlike education or healthcare, the problem of infrastructure is a very unique one. And you’ll hear this throughout the thread of my story is it’s finite in nature. We know how many bridges we need to build and if we’re smart about it, we can fix this problem in our lifetimes. So that’s the vision is to end rural poverty caused by rural isolation in our lifetimes.
So I’ll start with the happy story. The happy story is how successful we’ve been. We’ve been around for 23 years. We’ve done this work all around the world in 21 different countries, built over 500 bridges, have been really diligent and thoughtful and almost horribly studious about the way that we’ve approached this. How we’re building bridges, how we’re documenting it, how efficient we are and what impact we’re having on the people that we serve. Interestingly, that 30,000 plus meters of bridge, bridge span, we also have another figure that we sometimes put in there in $30 million of economic activity generated through our bridges, which is again, a great thing to measure and track against.
So the sad story is that sometimes it feels like we’re boiling the ocean, we’re just doing our work, cracking away at it in one corner of the world whereas this is a huge problem. So we didn’t have the exact numbers, but just to get to the kind of magnitude of scale, about one in seven people don’t have safe, reliable year-round access to essential services, government services, schools, clinics, hospitals, markets. And out of those, about a hundred thousand bridges would fix a quarter of their problems.
We also got to a magnitude of scale on the solution. So I’m ramping up to how we got to systems changes. Like first figure out the magnitude of the problem and the solution. It’s only 5% of how much we spend in overseas development aid on infrastructure that would solve this problem and solve this problem for millions across the globe. So the sad story, a hundred thousand bridges need to be built. We have only built 500, a Swiss NGO has built, has gotten about 10,000 built, a lot more work to be done and so we need to be smart about it.
So this was where the big realization came is that we can’t do this alone. That the heroes of the story again is not us. We’re not going to, we’re not the top goal scorer anymore. We need to step back, be a coach, and really enable others. And this is where our pivot came from a direct implementer to becoming and thinking of ourselves as a behavior change organization that, who needs to be doing what differently? What do governments need to be doing differently about the way they think about the way they invest, the way they prioritize in infrastructure? What does aid need to do in order to funnel money to have the biggest bang for its buck that perhaps they don’t even know about yet? What does the private sector need?
You know, one of the first questions I asked when I joined B2P two years ago was why is American philanthropy building infrastructure in Africa? This is not complicated. We’re building foot bridges right now, vehicular bridges soon, but Africa builds expressways and methane extraction plants. This isn’t rocket science, but it’s a new tool in the toolkit and we need to really inculcate the manuals, the standards, the practices of building particular types of infrastructure that haven’t been built before.
So changing a landscape of a country, of a continent, to change that landscape and be driven and be informed by the people that matter most, the rural communities that we serve has really been central to all our thinking as we’ve moved through this journey into scale and systems change. So don’t look at our systems theory of change, but this is some of the ways in which we’re thinking about it.
Our strategy is based on let’s get the evidence, let’s get the irrefutable, undeniable proof that dollar for dollar this works. Let’s scream and shout about it from the rooftops. Let’s tell, let’s make sure that people thinking about development, whether you’re in government or in aid, you are thinking about rural infrastructure. And then let’s give you the technical tools you need in order to do it. If you’re willing to do it, are you able?
And that last box is really our end goal is governments and investors fund and implement innovative cost-effective and impactful rural transport infrastructure. It’s a mouthful, but systems change is a mouthful. It’s not an easy story to tell. So the topic of why we’re here today, how do we measure this? How do we know whether we’re getting closer to this, moving the needle on this? And unfortunately the answer is, I don’t have a great answer to it, but I do have an ideal answer.
There’s something in our world called the Rural Access Index. It pretty much measures the population of your rural community that lives within 30 meters of an all-season road. Now, this was, it’s an SDG, the World Bank is a custodian of it. And once in a while the system that you’re working in, that you’re implementing in will have a big brother umbrella of a measurement. And just looking at that number would be a great way of seeing whether you’re moving the needle along the way or not.
However, I’ll start with my challenges or excuses, but why this isn’t great and it’s hard, it’s complicated. We’re leaning into the complexity of it, but unfortunately the way that Rural Access Index is measured isn’t perfect. It’s not, we don’t measure 30 minutes of travel time. We say that’s about two kilometers. And then we say, well how do we measure two kilometers? We do it as the crow flies.
Now, children don’t fly to school, pregnant women don’t fly to clinics, farmers don’t fly to markets. And so, here we are as a small little NGO sitting in East Africa saying, this isn’t perfect, this isn’t being updated, it’s not even being measured. The custodians of this SDG are falling behind on this, they are struggling with it. So here we are saying, okay, we’ll develop our own measurement of this systems change. We’ll find a really easy globally applicable way using data that’s globally available and free, that’s scalable and we’ll try and use our own measurement of it. Now that’s a challenge. Like we shouldn’t be spending money and resource on it, but here we are.
The other challenges I have that I want to point out is, on the right is a trash can in Oklahoma and on the left is some maps of a place in Uganda. So there’s a town there called Bundimulinga, this is a woman called Caribbeo. She’s walking across a bridge we built with her two grandchildren in order to get this beautiful picture for a small BBC documentary that’s coming out about us, there are 20 to 40 boda bodas, motorbikes on either side being held off and honking.
So like this bridge serves hundreds of people, thousands of people in fact every day and it’s not even a dot on the map. So that’s a huge challenge. And when we’re solving big problems, the fact that this trash can has more pixels on the internet than a whole town with hopes, with dreams, with opportunities, they don’t even get a dot on the map, that’s a travesty.
And so, here we are building out, we’re mapping our own things and on over there you see not even the water is mapped. So as we’re thinking about climate change, we don’t even know where water is. We don’t even have the blue lines that show us where waterways are. And we’re building models to show, well, where is the water? Everywhere we build bridges, all those red dots, they don’t even show up with the blue line under them when you look them up on Google Maps.
There are other challenges, I won’t go into them for too long, but you know, thinking about our internal culture, we’re not bridge builders, we’re systems orchestrators or behavior change organizations. It’s expensive. I think Natasha will talk about how long-term horizons really affect things. But at the end of the day, what I want to leave you with is, we constantly think about what would happen if we were not there?
And then the best thing that I think we’ve done as an organization is rather than divided into our impact as direct implementers or systems change, we’ve broken that down a little further too. Is that we know how many bridges we can build. Okay, now we know how many bridges we can support, assist, or influence. Then we know how many bridges we can catalyze, which countries can we go in and then we can measure the longer-term systems change.
So it’s not a binary this or that. We can break systems change down into smaller steps, into rungs along this ladder and then that makes it more chewable and our general ledger is able to also be tagged out that way a little easier. But at the end of the day, internally as a culture, it really comes down to constantly asking ourselves, can we put our hands on our heart and say we are moving the needle on this? And that’s not an easy thing to measure. Thank you very much.
Søren Vester Haldrup: Thank you so much, Nivi. So many things to pick up on and I’m only going to pick up on one thing before I hand it over to the next person. But I think your point on the importance of culture and mindsets and trust is not to be underestimated when, not only in doing systems change work in general, but when it comes to how do we measure this stuff, we are very socialized into certain ways of thinking about what count as a good measure, what is good data, for what purpose are we doing something? How do we think about causal relationships and so on? Changing that is not just about getting some new skills, it is about changing our mindsets and how we look at the world and also how we relate between each other, such as how a funder relates to a grantee. I’ll leave it at that and want to bring in our next presenter.
We just heard a lot of interesting thoughts from someone who’s really doing the work on the ground. I’d like to bring in someone who has a bit of a perspective from a funder as well. Natasha Joshi, the Associate Director at Rohini Nilekani Philanthropies and Natasha’s been doing some great work in thinking on how to measure systems change in different types of change. Please tell us more.
Natasha Joshi: Hi, good afternoon. It’s really nice to see a full room for this topic. And I think especially when we think of measurement, I think the first thing, the first word that comes up in our minds is data. But I read this really nice definition of data in this book called Data Feminism, which said that data is information a computer can understand. And what that means is that, there is information that computers understand and that’s why we’ve created data in a certain way. But there is information that humans can understand and it doesn’t necessarily need to be only data. And that’s where our meaning making comes in because we have ways of perceiving and sensing many different things outside of what computers can understand.
So I think I want to start with that, because here’s a question for the room and everybody’s coming from many different contexts and I sort of want to understand that are we able to arrive at a credible, statistically valid definition for something like this or this or this? And all of these words, I think as you will read through them from our experience, are forces that significantly impact the progress and success or failure of programs on the ground. It affects the lives of communities and it’s this stuff that really influences whether things work or don’t work or how it happens. But do we have a way to measure this, and why did we come to this question?
I have a little story because this provocation came from one of our partner organizations, they support our livelihoods programs for Indigenous communities and so they’ve done all of the stuff. They’ve done the market linkages, they’ve done the scaling and then they, for one of our annual reports, they went around trying to ascertain the impact of their work. And they’d gotten all of the great metrics, household incomes had risen and market linkages had happened and products had proliferated, so on so forth.
But we got a letter from the founder saying that the thing that struck me was every time we spoke to anyone in the community saying, “Well, what has been our impact?” they didn’t say the household income and they didn’t say the products. They said this, “We’re not afraid, our children are not afraid. I’m not afraid that I won’t be able to survive.” And he wrote saying, “How do I measure this because it’s priceless?”
And that’s what got us started on this whole idea that there are things, there are forces that are operating, which are just so important. But the tools that we create to measure certain things in the process of measuring them, they actually create categories which might not even be real. So for instance, I think this is a really important insight for all of us have an experience of time. We all know what time feels like, but is a minute an actual piece of time or is it something we’ve created?
So when we look at what a clock does, a clock doesn’t sort of, it doesn’t just measure time, it produces it because time is actually a completely different dimension that all of us think about. And I think in different cultures for instance, there are ways in which they use the circadian rhythm to actually measure time. And then there are cultures that use memory and temporality to think of time. So therefore, I think the idea that the short point here is that, tools matter and what instruments we use to measure, they really, really matter. And the first thing to do is to acknowledge the fact that we have some tools and they are good, but they are nearly not enough for the kind of complex work that we are trying to do.
So we, at the philanthropy sort of set off in search of a different tool and the thing that our approach is always a learning approach. So instead of trying to do this research ourselves, what we did was we essentially tapped into the collective wisdom of 80 plus organizations who are our partner organizations. We asked them how and this is just a little snapshot of who they are. They work in rural, urban, tribal. These are the titans of different annual budgets, different thematic areas. But we asked all of them to say that how do you measure progress? Like if you had to not report it, but what do you do? And not just track it, but what is your sense of progress when you do your work? We heard this first.
So first thing that always comes up is we do this, this, this, we’ve vaccinated these many people, we’ve given this much food, et cetera. And we sort of said that there’s a set of concrete actions that comes up. There’s always something, all organizations reported what we called engagement, which is, who has downloaded the reports, how many people have liked things on their website, et cetera? But some measure of activity that they’re creating is within the ecosystem. And it feels important to partner organizations to report that as a measure of success.
Then collaboration of course comes up. I won’t spend too much time on that, it’s obvious. And finally, there is of course this idea of social change. So deeper things which are harder to move stuff that John talked about as well, mindsets, attitudes, et cetera. So I’m going to give an example just to kind of, because there are a lot of you who are implementing organizations, there’s three different organizations that we support.
So this is PRAYAAS and they work with people in custody, so in the prisons and they try to essentially, after they’ve come out of custody, rehabilitate them. So these are the kinds of, this is exactly in their words, I’ve just copy pasted it. But this is what they reported saying that they have lower recidivism that’s very concrete, you can measure it, there’s a number, you can say that you know this percentage and that works. And typically, concrete actions are very quantitative.
Engagement on the other hand is a little bit different people define it, but in their case, they said regularity of meetings. So how much is this stakeholder group engaging with me? Collaboration of course is this increased acceptance in society and family. So you have to kind of tie up with non-profits and other employment providers. So how many of those are in their network? And finally, freedom from addiction and violent behavior or something that they also kind of track, but I mean they don’t have a number for it, but they put it down saying that this is what we think of success.
Similarly, this is Civis and they do, they run an online platform where citizens can actually give comments on any legislation that’s in draft stage. So a lot of Indians don’t even know that this act exists actually, that you can give comments on draft legislation. But this is the kind of feedback they gave, and I’m going to sort of move to another organization working in the gender space so that you can just kind of start to see like how this data is being kind of reported and captured by us.
But the main point here is, when we plotted all of these four categories, I find that this is kind of the spectrum within which a lot of the data falls, it goes, there are things that are tangible, there are things that are intangible. Similarly, there’s stuff that moves fast that you can get quick results and there are things that move slow. And this is the sort of framework that we use because when we use this framework and plot everyone’s work, we don’t get one dot that says this is the impact. We get this beautiful like multi huge thing. We get a sense of where progress is, what’s happening. So much is going on. But I also want to move to the fact that we use the same framework to also measure our own impact.
When we said that if we, as a foundation, believe in sharing power, the first way to share it is towards a share being evaluated. So how do you, as a foundation, evaluate yourself? We found that we could do the same thing for ourselves, except that it just looked a little bit different in our case. But there are things that we do which are concrete actions, which is grants of course. And then there are things that we do for engaging our community. There’s collaboration, there’s social change, which is really actually informing philanthropy and informing narrative and things like that.
So the idea of putting this whole piece together is really to say that we are just starting out on this journey. We would like a lot more thought partnership on, how can we really start to capture qualitative data better? Qualitative data is not the same as anecdote, they’re two different things. But how do we build more momentum and more capacity to get to what we are calling more systems conscious, multi-method approaches? Because ultimately what we do want to do is we have to recognize that your current evaluations are limited. And I think that it starts there by recognizing that there is a limit and that what we really want is we want a higher resolution, a more better telescope. So thanks a lot.
Søren Vester Haldrup: Thank you so much. Again, so many things to unpack there and so many of you nodding. And I think one thing I think is particularly interesting here and relevant to bear in mind, also that clock analogy is excellent, is that what we measure matters. You know, we need to measure what matters because whatever we measure, that is what we end up focusing on achieving, that sets certain incentives that drives behaviors and so on. So we really need to be sure that we measure what we actually care about and that we are able to capture the things that are not as tangible, not as easy to measure, just like in John’s framework. Those bottom things, those things under the surface of the iceberg I think is so important to bear in mind.
Now, I think have time for maybe just one question. I’d like to ask you just pose one question for you to reflect a bit and then we’re going to break out into groups. One question I’ve been grappling with a lot, I know many others are, is about, how do we, given systems change is collective work as you also said, John, it’s something not one actor can drive systems change, not one actor is the one causing it. How do we make sure that we are humble and don’t overestimate our own importance when we do this type of work and when we try to measure it in particularly if we’re trying to get a sense of what role we have played? Any thoughts on how to deal with that easy, easy question?
John Kania: Yeah, I mean there are a lot of ways you could go with that. I think it’s a really important observation that you made Søren that, in almost any evaluation you’re privileging certain perspectives. So I think the first point vis-a-vis consciousness is, are you aware of the perspectives that you’re privileging? You know, I talked about system storytelling and doing that work in a circle and the intent there is that while you can’t totally get rid of the power dynamics and power differentials, doing circle work well does in fact mitigate the power differentials and just bringing multiple perspectives together.
I can’t emphasize that enough that we really need in systems change work to hear from all the various different stakeholders, and that’s going to give us a richer understanding of what’s going on. And while we’re doing that, to again have the consciousness of where formal authority and power lie and sort of take that into account.
Søren Vester Haldrup: Any other thoughts?
Natasha Joshi: Could you sort of rephrase that question?
Søren Vester Haldrup: Yeah, I think it’s around, well, one aspect is, if we try to just understand where the change is happening, I think that’s one question. But we often tend to measure, aim to measure systems change because we want to get a sense of what role we are playing, what contribution we are making and does that even make sense to ask that question?
Natasha Joshi: Yeah, I mean I think so, so the whole contribution, attribution debate is also quite well established in terms of just how can you sort of put your stamp on anything that you’re doing when things are so, so complex? You know, something that really struck me with Nivi’s photo when she showed it, the first one, the one where the man is, I was thinking like who was the first person who got across, who threw the–
Nivi Sharma: Right.
Natasha Joshi: the pipe to the other one? And so, it’s, so attribution, contribution sort of falls on that kind of, it’s complicated for that reason, but I do agree that I don’t, I think one other thing to think about when it comes to change, right, is this, everyone says change is constant and I think what we are trying to understand is essentially, change will be eternal in some sense. It’ll keep changing. So there is no kind of static point to which we are saying that if all of us do this, we will get to this point and then we can all stop doing something, you know?
So I think, I do think some of these things have to be contended with, but I do think they lie at the level of you have to dig deeper, just in terms of values and why you are arriving at this work, why you’re doing it. And I think that conversation needs to be brought into our field a lot more. Especially like, and I can say this is on the funding side that I think we do need to think a little bit about some of that stuff. Like what has brought us into this world of work, into this world of applied social science, why do we do it? You know, why is it interesting? Because it is rooted in like, it is rooted in values and something that’s very, very human, so.
Søren Vester Haldrup: Yeah, yeah, we could keep talking, but I’m keen to shift gears now and give you all an opportunity to reflect on what you’ve heard, reflect on your own experience and listen to others. And I think we will get some questions up on the screens here. So I’d like you now to, in your groups, at your tables, take turns and share. You don’t have to cover all of these questions. Maybe pick the one or the ones that are most relevant or to you or that you feel you have something to share on and go around the table and discuss.
You have around 20 minutes in total, you get a two-minute warning before we go back into plenary. So make sure you give everyone time to share and our panelists and myself, we will kind of join you, we’ll walk around and, and listen in and contribute. So we won’t attempt to do a thorough debrief by table, but simply would like to ask you all two questions. You could choose to just pick one of them.
Was there some interesting point, anecdote, something that came up in the discussions around the table that you think are particularly interesting? That’s one question. Another question is, is there something, a piece of advice, some sort of insight from today that you are walking away with that you want to apply in your own work or in your own organization? Feel free to pick either these two, any reflections. We have some people with microphones, they’re going to run around to give you a microphone so please raise your hand.
Renée: Hi everyone, my name is Renée, I’m the founder of OpenEmbassy and we had a, I think like the common thread between our shared experiences is the question of, how do you navigate between the impact you want to make as an organization versus the people that fund you and how they think about system change? So I see people nodding, we had several discussions. That’s nice. And then I think like related to this is the importance of not doing it by yourself, but working in coalitions to demonstrate as civic actors that you know what the change needs to be and that others like governments and funders should more or less follow your evidence and experiences.
Søren Vester Haldrup: Such a good point. And I know, I think I heard a table over here that touched on some of the things, same things. We tend to work in this principle agent type of logic. Someone implements work on behalf of someone else. How do we move towards more collaborative logic? How do we move towards working as an ecosystem and how do we do that if we are facing scarce resources and we’re all struggling to get the same small pot of funding? A really good point, any other reflections?
Phil: Hi, I’m Phil. I run the Center for Policing Equity and other things. I’m also Julie’s hype man for those who don’t know, Julie from Thorn, one of the things that we talked about was a word that doesn’t get to get spoken in these kinds of spaces very often, which is power and how very difficult it is if your organization is set to address, address systems. But you’re either unwilling to speak about power to challenge it directly or the philanthropic space is unwilling to speak about or engage with that.
And so, having more direct conversations about power, enlisting folks who are not so fluent in the language of social entrepreneurship so that they can reveal their experiences of power, integrating that into the process is necessary for certain forms of systems change, particularly the ones that tend to be at the margins and are most under-resourced.
Søren Vester Haldrup: Really good point, I think we have another hand here.
Unknown Speaker 2: Hi, I was egged to, to give this one. So one of the things that was a question over here was talking about like how we define systems. That was one of the things that came up for me personally. So I will just bring this, I think that we also come to this, to this thinking about systems differently depending on what your area of social justice so however you are. And I think it’s really important and would’ve been really important to hear like when we say measuring systems change, what systems are we talking about?
Some of us are fighting really large systems and others are in different types of systems and I think it’s really important to be clear about that because the measurement is quite different and the thinking process is quite different. And also, if you have interlocking systems, the measurement is also different. So I think those are questions that we still have to grapple with and untangle.
Søren Vester Haldrup: Really good point. I like that. There are so many, talking about boundaries is when you get into the question of measuring systems, it is like where does it stop and where does some other system begin? It’s artificial constructs where we use to make sense of something, but it’s really difficult to set those boundaries of this is what the system is that we are going to measure. And really good point again on the power and who defines what we look at and we tend to look at M&E as a very technical activity. It’s like, oh, you have KPIs and baselines and you collect data. No, it’s deeply political and there’s power and who defines all these types of things. Any other thoughts?
Unknown Speaker 3: Something I’ve been reflecting on as I was listening to the speakers and my colleagues at the table is, a dimension that we’ve missed talking about is time. Systems change needs time and therefore patient philanthropic capital, which is in very, I mean I don’t think we see it often enough. And so, if you’re talking big changes then we really need to start thinking about what patient capital really means.
Søren Vester Haldrup: Absolutely.
Unknown Speaker 4: And are people wait ready to wait?
Søren Vester Haldrup: Yeah, such a good point. As again like we all work in certain time horizons and we are not set up to work on very long-term change and we all have people we are accountable towards and need to report to. So we need to rethink, what are those interim things we’re going to look at to know if we are on track, at the same time as we hold the space to work towards like a decade or two-decade sort of time horizon for systems change. We have a microphone coming up to you.
Unknown Speaker 5: So I don’t think I could do justice to the discussion at our table, but I will say it raised really important points including that all of this has to be grounded in what the people in the communities we’re trying to help really want. And that there’s, that can be very difficult to do, that funders often aren’t listening to that and other issues of power imbalances. But I have a very specific question.
The presentations really resonated for me because we fund a lot of systems change and storytelling is something that folks we have funded spontaneously have provided us with. And the four points that Natasha raised in her things, that’s what I have much less articulately called the how of systems change when most people focus on the what. Where’s the money going, what are those things? The challenge I have is that, I can get the benefactor that I work for to understand that. But if I want to get the World Bank or UNICEF or USAID or FCDO to fund systems change work, I sort of got to be able to say to them, this is a more effective way to spend your money in the sense that you will get these outcomes and this is a more cost-effective way to spend your money. You will get a better bang for your buck.
And I am really struggling how to move from the qualitative approach, which makes deep sense and I completely agree with it and it’s great, but I guess I was hoping here to get that. So if anybody’s got the answer, please tell me.
Søren Vester Haldrup: Thank you very much. Let me just check the clock. I think we might just have time for, for one more.
Unknown Speaker 6: Sorry, thanks for this session. I would, I tend to answer your question really quickly from the point of prioritization. I think I’ll use an example maybe from some of the work we’re doing in Nigeria. So you have different development partners working with DFIs and they’ve, with the way things are happening, all the tailwinds going on and some of the inflationary pressures because like we’re talking about systems change, right? There are more discussions around what real outcomes are going to look like and aligning that with the government priorities.
So the priorities of the government are central, whether it’s on national, sub-national level. And so, getting the DFIs, development partners, donors together in one room and saying, “How are we going to align and come up with real outcomes?” It’s forcing the folks like the World Bank and the UNICEF to align with those priorities so you can have a win. That’s one. I hope, but this is in the African context, right? So, but in our group, I just want to give kudos to Maria who is our leader in the group here. So as she was speaking, there are common themes that came out in terms of system change.
Number one is data-driven evidence. Number two is the importance of communication. Number three is around monitoring and evaluation. Number four, which is really key is around ownership. If people don’t really, if there’s no true ownership then there’s no real system change. Somebody has to own that process. And number five is commitments, number six is diversity. So there has to be understanding diversity and diversity must be incorporated in systems change. Number six is around the legal aspect, which is typically forgotten, whether it’s the legal framework play in systems change. And the last one is around sustainability, over.
Søren Vester Haldrup: Thank you so much and I think we’ll have to end it there. We are at time. Thank you so much all of you for coming. Thank you to our panelists for taking the time and share your experiences and the organizer.
