Webinar Resources

Kien Lee: Good morning, good afternoon, and good evening to everybody. Whichever time zone you’re in, I want to say hello, I’m Kian Lee, I’m President of Community Science, and I have the privilege of working with these three amazing people on this call with me today.

Kien Lee: Jane Erickson and Sabine Moniz.

Kien Lee: from the Learning and Impact team of the Ripple Foundation, and Sanjeev Sidharan, who brings a lot of expertise in complex systems evaluations and currently teaches at the University of Hawaii.

Kien Lee: Throughout the presentation today, please feel free to drop your questions in the chat. I think there’s a space for you to drop your questions, and we will answer them as we go along, if they’re a clarifying questions. If not, we will try and leave them to the end during the discussion.

Kien Lee: Just a note that we will also share the slides, as well as any other resources that we reference throughout the presentation today, after… and a recording of this, actually, following, a few days after this session.

Kien Lee: A very, I think, Sabine, if you want to share the slides now and go to the second slide, just a very quick overview of how we will flow today.

Kien Lee: Jane’s gonna provide some background information to set the stage.

Kien Lee: Sabine and I will talk about the reasons for the enterprise evaluation that Ripple decided to do, and the components for which Community Science, was responsible for.

Kien Lee: Sanjeev will then share some of his insights and experiences about this whole endeavor, and then I’ll facilitate a discussion at the end of this. And with that, I’m going to turn it over to Jane.

Jane Erickson: Thank you so much. Hi, everyone. Really appreciate you taking time out of a

Jane Erickson: busy season to be with us. I’m Jane, talking with you from Syracuse, New York, where I’m looking outside at a very beautiful fall day. We have peak foliage right now. I lead the Ripple Foundation’s work around measurement, evaluation, and learning, and

Jane Erickson: This is a really meaningful milestone for our evaluation team at Ripple to share this story with you today. We’re really proud of this work.

Jane Erickson: And especially grateful to do it with such committed and thoughtful colleagues, so extra thanks to Sabine and Ken and Sanjeev.

Jane Erickson: So to start us off, I’m going to give a little bit of context about Ripple’s journey to how we’ve been rethinking our evaluation work. Go to the next slide.

Jane Erickson: So a little bit of background about Ripple. We are a nationally focused operating foundation that works as a catalyst for equitable health and well-being in the U.S. We mainly work through our flagship initiative, Rethink Health.

Jane Erickson: Where we partner with folks across the country to help them do the work of equitable system change.

Jane Erickson: We work locally with organizations and partnerships who are supporting long-term change in their communities.

Jane Erickson: And we also do a lot of nationwide field building, sharing stories, resources, advising. And this kind of bifocal lens of local and national support is a really important part of how we work. You’ll see that

Jane Erickson: through the ways we’ve built our measurement, evaluation, and learning approaches throughout the hour that we have together today. So just a little bit about the why behind this work for us.

Jane Erickson: For me and my colleagues at Ripple, the thing that really motivates us the most is a deep belief that we can all thrive together through shared stewardship.

Jane Erickson: We are joining with others all across the country to really live into the unifying expectation of all people and places thriving, no exceptions. And we know that,

Jane Erickson: Even with systems that are fraught with the injustices that we see everywhere, and the divides that are fracturing

Jane Erickson: how we understand one another, especially right now, that we can transform these systems if enough of us commit to thinking and working in different ways together. This is what we call shared stewardship.

Jane Erickson: You can go to the next slide.

Jane Erickson: We have devoted our team and our tools to this for over 15 years, and through all of that work, our field work, and in really deep partnership with others, we’ve learned that 3 things matter the most when the goal is equitable system change.

Jane Erickson: The first is a relentless focus on the belief and the value of the North Star goal of all people and places thriving. This is unifying, it is something we all want, and it’s something that’s measurable.

Jane Erickson: We also have learned that in order to thrive together, we must expand vital conditions in communities. So, understanding whether and how things are in place in our communities

Jane Erickson: That foster thriving, like meaningful work, a thriving natural world, and basic needs for health and safety, and a sense of belonging.

Jane Erickson: And the civic muscle, this is the pink circle that you see in the middle there, the civic muscle that we know is so important for people to build these conditions together, especially right now. And if we’re going to do that.

Jane Erickson: We need shared stewardship. Ways of thinking and acting together that help us to build those vital conditions.

Jane Erickson: This is about understanding how well we’re collaborating and adapting, and really building the relationships and trust that we know are so foundational to making these kinds of

Jane Erickson: changes. A couple more points here. This really is…

Jane Erickson: From my perspective, making the work of social change more human and humane and less Technical and technocratic.

Jane Erickson: All of this requires that we, as a country, move from fragmentation in how we see one another and work with one another to greater interdependence. And at Ripple, we believe it’s really important to be clear about these big aspirations and to not shy away from them.

Jane Erickson: Because this is the work that’s most necessary right now. One other point, kind of tying this very explicitly to evaluation stuff, for us, this really is about the discovery of, like, directional norms. So, understanding the extent to which these things are taking hold.

Jane Erickson: And we think about our measurement, evaluation, and learning work as, like, rocket fuel to help us and others do that.

Jane Erickson: We can go to the next slide.

Jane Erickson: So, we’ve studied this alongside many others for over 15 years now, and

Jane Erickson: Have had the privilege of seeing the incredible work that’s happening across the country, and the real momentum that’s growing. Being able to kind of bear witness to more and more organizations and partnerships.

Jane Erickson: That are stepping in to this commitment, seeing themselves as stewards for Thriving Together, working to expand vital conditions in their communities.

Jane Erickson: And it’s a big deal, and it isn’t yet the norm, of course, in how we are all showing up as a field.

Jane Erickson: So in 2023, let me go to the next slide… we, we took a step back to develop a 10-year strategic roadmap.

Jane Erickson: Really centering this commitment to help grow a rising movement to thrive together, and knowing that it’s going to take longer than 10 years to reach that goal, but also really appreciating that we can and need to do a lot in that time period.

Jane Erickson: So as you can see, the roadmap really centers this commitment to help grow Rising Movement to Thrive Together, and there are 4 really important parts to that.

Jane Erickson: We see these as needs in the field, and also where Ripple can play an important role as a catalyst

Jane Erickson: really knowing that we’re one of many doing this work. So those things are reinforcing a unifying narrative, really making a unifying case about how we can all thrive together.

Jane Erickson: Investing in vital conditions, especially belonging and civic muscle, measuring progress towards thriving together, and perhaps most important right now is bridging differences, really drawing on our diversity as an asset to help us heal and move forward together.

Jane Erickson: Can go to the next slide. So, just a couple more points here, and then I’m going to hand it over to Ken. So, we’ve always done our work at Ripple in 3-year cycles of action learning, and kind of since, since the,

Jane Erickson: roadmap was really put in place, there is now an explicit focus on diffusion. So, working with others to deepen and spread stewardship, vital conditions, and the North Star goal of all people and places thriving.

Jane Erickson: And so the question now for us is, knowing that

Jane Erickson: we don’t have… no one has all of the answers to this. We have to learn our way into it. The things we’re really holding central is how are these shifts becoming normative?

Jane Erickson: how can we really diffuse in deep partnership with others? We’ve always used our evaluation work as a way for us to understand how we can navigate change as a field.

Jane Erickson: and as an organization. So, this, kind of the establishment of our roadmap strategy and the rooting in these questions really gave us the opportunity to take a step back and rethink our measurement, evaluation, and learning approaches to center the movement-oriented strategy

Jane Erickson: And also the relational way that we know this work has to be done.

Jane Erickson: So, last point I’ll make here, we can go to the next slide. So, we knew going into this upgrade project that, centering nested theories of change was going to be really important.

Jane Erickson: To help us not lose sight of this larger movement.

Jane Erickson: And really hold,

Jane Erickson: positional humility about Ripple’s role. Thank you to Sanjeev for the gift of that lovely framing. So a few years before we started this evaluation project, we had built, with one of our, long-term

Jane Erickson: thought leaders, Bobby Milstein, a theory of system change with hundreds of partners across the country about what it takes for stewards everywhere to get to thriving.

Jane Erickson: It’s a shared vision that centers the North Star goal, and really importantly, also lays out all the dynamics we, as a field, have to hold in navigating towards that. So, kind of holding that theory of system change that is a shared thing with many others as foundational.

Jane Erickson: We knew that we wanted to center our new organizational theory of change in that broader, kind of widely shared theory of system change. And in doing that.

Jane Erickson: We really wanted to make our organizational theory of change a tool for Ripple to help navigate choices about supporting the movement to thrive together, and a resource to understand, you know, how we can be doing that well through our evaluation work.

Jane Erickson: We didn’t want to build a logic model or an ecosystem model. We really wanted, kind of a navigational tool, which is how we’ve been using it. So, to give this a little bit more color, Ken will share some perspectives from a group of local and national thought leaders, that we brought together to help shape this.

Kien Lee: I am muted. Thank you, Jane. So this process, we worked with

Kien Lee: Ripple to bring in an amazing group of partners, people who are on the ground working on this work, as well as people who wrestle with this at the national level, from evaluators to, you know, doers. Like, it was the whole range, and these are the amazing people. Sanjeev was one of them, hence his involvement in this call, because he brings his perspective and his experience with

Kien Lee: That whole endeavor. Next slide, please, Sabine.

Kien Lee: We got a lot of amazing feedback from everybody when we drafted the first sort of theory of organizational change, what you saw,

Kien Lee: Jane had presented with the hexagons, and we will go into way more detail about that a few slides later, but just to give you an idea, you know, what the thought leaders that we work with really loved it was just how the humility that Ripple had in saying, like, it’s among many other stewards trying to move this work forward, and so how do you illustrate that?

Kien Lee: in a diagram that you’re driving it for yourselves, you’re one of many, and you’re working towards this change collectively. And then making sure that it’s explicit that, you know, equity is center… is the center of all of this as well.

Kien Lee: There’s a lot of other amazing feedback we got, and I won’t go through the bullets, but essentially what it came up… the comments really reflected the complexity of this work, right? How do you account for the resistance? You know, just because you have this beautiful illustration on paper doesn’t mean it’s gonna roll like that, and it’s gonna be perfect. So, how do you account for that resistance? How do you learn from it? Where do you learn from it? How are you ready for it?

Kien Lee: Also capturing that it’s not linear, right? So there are multiple forces going on, you know, you can make 5 steps forward, and then 10 steps backwards, and then 2 steps forward again, so it’s not linear.

Kien Lee: Again, we can sometimes get caught up in the very technical jargon of this work, right? So how do you make it real? How do you use stories to bring this come to life? Like, this is really what change looks like in reality, and in very different contexts. So really, the comments that we got from everybody was revolved around these themes, and so we had the very challenging work of figuring out, like, how do you make this show up?

Kien Lee: In an illustration and in the words and the language that Ripple especially chooses to use to be able to convey its organizational theory of change.

Kien Lee: With that, I’m going to stop. Next slide, please, Sabine, and then just invite Sanjeev to say a few words about the insights, experiences that, you know, he brought to this, as well as what he gained from it.

Sanjeev Sridharan: Thank you.

Sanjeev Sridharan: for me, as I… firstly, I just love the Kate Sibley question. I’m going to start there. We grow in the directions of the questions we ask. Just to make things more concrete for myself, I work on brain health, which is the number one condition of ill health these days.

Sanjeev Sridharan: The questions I’m asking is, it’s…

Sanjeev Sridharan: It’s not as though a single intervention… by the way, I see a few people who work on brain health on this call, it’s not a single intervention that brings about a change. You really need a collection of people working together that can bring about change. So just to frame this, and as I understand Ripple’s rather different theory of change, the questions I ask

Sanjeev Sridharan: in my own work on brain health is, in what ways are our evaluations incomplete? I think we’re just too fixated at the project level. Even at the systems level, I think the part of where we are going with stewardship is beyond the intervention as the unit of analysis, and I think I’d sort of push you in those directions itself.

Sanjeev Sridharan: I think this idea of actually studying context, I mean, I’m speaking to you from Hawaii, which is the home of some of the great navigators. I love what Nainoa Thompson says. You know where you are in the ocean by knowing where you came from. And I think we have not taken context seriously in our work. How do we bring context, make context much, much more central in our work?

Sanjeev Sridharan: I loved your focus when you talk about stewardship, around the connections function of evaluation. How do you connect across differences? The learning function.

Sanjeev Sridharan: But also the discovery function. How do you create opportunities? So, I worked on Parkinson’s disease for many years. I think we’re asking the wrong question if you say the only object of evaluation is something working. What are you learning, and what are the opportunities you are actually creating in the act of doing the evaluation itself?

Sanjeev Sridharan: I’ll… I’ll wrap it up in just 30 more seconds.

Sanjeev Sridharan: I think we have to ask, as a field, are our tools good enough?

Sanjeev Sridharan: are the way we practice theories of change, for example, good enough? We’ve just written a paper critiquing the evaluation criteria. I’m not sure that actually is good enough to actually… so as a field, I think I want to kind of stress that, I want to frame evaluation as a fairly new science itself. How do we do a better job?

Sanjeev Sridharan: much of the work that Ripple is doing really forces us to think about different evaluation purposes. How do you mix developmental with realist, with other kinds of positions itself, too? I think I’ll wrap it up with this last comment.

Sanjeev Sridharan: In the way we think about evaluation, our language is rather remarkably stale. I haven’t seen too many evaluations that use love.

Sanjeev Sridharan: that use healing, that use loneliness. And you are not going to address mental health and brain health without actually having a much more richer language. So anyway, all of this is simply to say, part of the dialogue is really how do we up our own game to actually be helpful in this journey. Over to you.

Kien Lee: Thank you, Sanjeev. Those are great food for thoughts. Jane, let me turn to you and see if you have any response and kind of your comments and insights from this endeavor.

Jane Erickson: Mmm, I…

Jane Erickson: I will share kind of one follow-on thought to, one of the points that Sanjeev raised around the,

Jane Erickson: limits that… Happen when we focus just on, is something working?

Jane Erickson: And also thinking about a…

Jane Erickson: The work that we do only in service to solving problems.

Jane Erickson: The thing that I am so energized by around how we approach our work at Ripple and the way that we have built our evaluation stuff is that it is not about

Jane Erickson: just solving a discrete problem, or understanding whether or not something is working. It’s about making space to really imagine what it looks like to build a fundamentally different future from where we are now. And…

Jane Erickson: Just focusing on projects or programs, certainly there is a role, an important role for that, but it is necessary and not sufficient.

Jane Erickson: For us to get to where we need to go as a field.

Jane Erickson: So for me, this is really kind of positioning evaluation and learning and measurement

Jane Erickson: as a really critical function in social change work, and that if we’re going to do that, it means, also to Sanjeev’s point, thinking really differently about our approaches, even the way that we look

Jane Erickson: At the purpose and the kinds of tools that we’re building to be supportive.

Kien Lee: Thank you, Jane. I will pick up a little bit on that, too, in terms of, like, the… as an organization working with you all on this, is how… what is… what is necessary and not sufficient, and as we help you all develop the tools.

Kien Lee: I know that one of the things we had to do was constantly go up to that, you know, that 100,000 feet level or more all the time to see the big picture, and where this is all gonna go, and then have to come down to almost, like, 10 feet above ground to go, what’s concrete and what needs to happen, and we were constantly fluctuating that way to be able to make

Kien Lee: this all work, right? And so that was one of the both the rewarding experiences and the challenging experiences we had on this project working with you all. Sabina, I’m going to turn to you to see, because you were also on the ground helping to make this come alive and to implement it. So yeah.

Sabine Monice: Yeah, thank you. I think something, I would like to kind of share is, you know, in you talking a little bit about

Sabine Monice: the thought leaders and, the local partners that we engaged with, thinking about just the relational aspect of this work that,

Sabine Monice: something that we’d like to say at Ripple is not about you without you, so, just really focusing in and honing in on the importance of storytelling, and why we chose such a participatory approach to do this work, as stewardship is kind of the heart of everything that we do here at Ripple.

Sabine Monice: And, I’ll go into a little bit deeper about some of the more processed pieces.

Sabine Monice: of the work, and, you know, this was one of the first projects that I had the privilege of managing, at my time at Ripple. But yeah, I think that’s so important as,

Sabine Monice: to Sanjeev’s point about just loneliness and kind of asking some of those questions, and maybe we’re focusing on the wrong things as we kind of think about systems change and what it takes to solve some of those wicked problems, you know, it starts a lot with relationships. So,

Sabine Monice: I will go ahead and think about some of the whys, some, so, and some of the shifts that we… we did, as we looked to, kind of, what we like to call an upgrade to our evaluation.

Sabine Monice: So, throughout this project, we… we really wanted to shift the way we thought about evaluation. So.

Sabine Monice: I think Jane alluded to this a little bit, but instead of solely focusing on evaluating individual projects, we took a step back to look across everything that we were doing towards a more portfolio approach, and with a clear eye on how it contributes to the broader movement that we’re a part of.

Sabine Monice: And this wasn’t an accidental shift, it was in… guided by a core set of intentions that, you see here, and again, at the center of it was, the roadmap that Jane shared a little bit about.

Sabine Monice: You know, the roadmap really has been our anchor, helping us knit together years of past work into a more cohesive, forward-looking approach.

Sabine Monice: And here’s a little bit more, about these intentions that you see. We wanted to build evidence for what works, for whom, and in what context. We wanted to really prioritize what matters, not just for us, but, to the field at large. And again, thinking about who we would engage, not only at the local level, but for our colleagues at the national level.

Sabine Monice: And really thinking about embedding, practices of emergent learning, you know, not only the things that we can control, but the things that we may not be able to control. So that looks like asking powerful questions, making learning visible, and making sure that we’re, contributing, involving communities that are at the forefront of some of the issues that we’re trying to solve for.

Sabine Monice: And the value of the changes that are most significant to the people, communities involved, and ultimately, fuel internal effectiveness and growth. So all of this, has led to

Sabine Monice: a change in how we evaluate our impact. In doing so, we all… we recognize that this is all a work in progress. This was a really, really big endeavor that we, did in partnership with Community Science, and, you know, we would be remiss to not name that this is still a work in progress. We…

Sabine Monice: Are today gonna focus on, some of the main components of what we did, but, here, you know, there are just so many pieces that make up the enterprise evaluation.

Sabine Monice: But we’re going to be focusing today on our guiding questions, our theory of change, and our measurement framework. But internally, we’re doing a lot of cross-project learning to kind of help tie things together. There’s data collection tools that we’re creating, synthesis and sense-making that is on an iterative basis.

Sabine Monice: Again, there’s strategic… the roadmap that holds everything together, but there’s strategic planning that goes on, and the action learning cycles that happen. But yeah, these are things that are, again, emergent and happening on an ongoing basis, so…

Sabine Monice: We wanted to just kind of highlight just 3 pieces, and again, we did these in partnership with,

Sabine Monice: community science. So, and we talked about, I want to talk… highlight guiding questions first. This was a really important piece for us, as these questions highlight, areas at the edge of our collective knowledge.

Sabine Monice: They help us reflect what’s most relevant to both our local and national leaders, and internally, what we want to focus on. And, they are grounded in our commitment to shared stewardship and thriving together. These weren’t created in a vacuum.

Sabine Monice: They were developed in a participatory, emergent learning process over 6 months, so we really took our time, to do these right and collectively. We engaged Rippled staff and external partners for feedback and refinement.

Sabine Monice: And, our theory of change. It’s closely aligned with our guiding questions, and it flows directly from our Theories of systems change, in the Thriving Together diagram.

Sabine Monice: And it clarifies assumptions of our strategies that will lead to desired outcomes, and the full range of outcomes that we’re committing to tracking and being held accountable for over time.

Sabine Monice: And, these outcomes form the foundation for the measures indicated in our measurement framework that will guide both our evaluation and our organizational learning. And we also have a fine set of measures.

Sabine Monice: And, these have been defined… designed to track our progress towards those identified outcomes, inform what data to collect and how to collect it.

Sabine Monice: and ensure that we have a strong, practical foundation for ongoing learning. And again, this would not have been possible without Kian’s leadership. The process that they, really spelled out for us

Sabine Monice: They did extensive document review and kind of learning a lot, about how we work at Ripple.

Sabine Monice: And they supported the development of the theory of change, all those things that I mentioned, the evaluation framework, some of the data collection tools, and just our overall approach. So, I’m going to turn it over to Kien so she can get into the very, very,

Sabine Monice: Really, really riveting details of the nitty-gritty of what… of those, products.

Kien Lee: Sabine, if you could just stay on the slide for a second, thank you so much. I think one of the things I want to note is that this really shows the complexity of the whole process, and one of the things that we had to do, we came into…

Kien Lee: focus on those three hexagons that are very colorful, but we had to be mindful of all the other things that were happening, and constantly figure out how does what we’re doing for Ripple fit into all these pieces, and how they all come together. So it wasn’t as if we had… we could do this in isolation, but we had to be so mindful of what was there to build on, right? We weren’t throwing things out the window, what was important to keep

Kien Lee: keep what was important to enhance, and then what was important to fit the rest of it together. So that was an ongoing, really, partnership with the Ripple staff to figure this out.

Kien Lee: Sabine, if you can… if you can go… actually, not the next slide, which I’m… you can go to the next slide, but I’m not going to cover it, because I think we’ve just talked about it.

Kien Lee: most important point to make here is that all of this is in service of that learning, right? Like, the learning, the futuristic learning, the learning so that the movement to drive together can really grow, and that was really the primary purpose of this entire piece of work.

Kien Lee: The next slide, please.

Kien Lee: What I will say about this next slide, is, as we move.

Kien Lee: mind sneaking Fasten at technology here, unusual, is that there are some foundational concepts that really inform this work, right? And I think, given who we see on the call, most of you are probably very familiar with this.

Kien Lee: you have to have that developmental aspect of it, because nothing is… nothing is complete. As Sanjeev would say, there’s no completeness to this kind of evaluation, this kind of work. It’s work in progress, you gotta keep… it’s iterative.

Kien Lee: There’s a realistic aspect to this, it’s sort of… it’s very driven by context. Context is really important, as every B has emphasized here. What works for whom, how, what doesn’t work for whom, how it’s all very… shaped by the context, and then the context is also shaped by the work, and so it’s really, really understanding that.

Kien Lee: The other piece of this is that, you know, what’s changed? What’s the most significant change? A significant change in one context may not necessarily be the same level of significance in another context. So again, understanding what is significant, in that context and centering participants,

Kien Lee: experiences, and last but not least, I mean, there are all kinds of complex dynamic forces going around that we have to be aware of, that this whole, movement has to be aware of, and where does that resistance come in? Where is there an opportunity? Where is there not an opportunity? All of this have to be considered.

Kien Lee: These tools are not enough, and I think Sanjeev mentioned earlier before, right? Like, the tools are not enough, the language may not be enough. So again, these are just things that help us make… that remind us of really what we need to think about as we’re trying to create a learning system for this. But it’s not the end-all, be-all of everything, because what should really drive the learning and the evaluation

Kien Lee: It’s really not the models, not the tools.

Kien Lee: But how are we going to solve the problems we have today so that people are thriving, and not suffering? And that question should drive the entire design and process forward.

Kien Lee: Next slide, please.

Kien Lee: So this is the organizational theory of change, that, Jane had brought up in more detail, here. It’s not a logic model, it’s not an ecosystem model, it’s really a tool. It’s simply that. It’s a tool to… so that Ripple can have in front of it to be able to navigate choices about how to support the movement thriving together, about its role and its, its,

Kien Lee: its contribution to this movement. On the left, what you see is the proximal, and, you know, this is where Ripple’s role is. This is how it contributes. This is where it has… it makes catalytic actions, right? And then all the way to the right is the distal, and this is the movement. This is where all the stewards, in addition to Ripple, is contributing to moving this movement forward.

Kien Lee: Now, the key aspect of all of this is that middle thing that we always call the S-curve.

Kien Lee: Because that is where the diffusion happens. That’s where a lot of learning needs to take place, because how do you go from the catalytic actions on the left, right, to diffuse it to the point where it’s contributing, it’s driving, it’s growing that movement on the right? And so we really spent a lot of time thinking about that, and it was the thought leaders that we engaged that really helped us understand, like.

Kien Lee: While you don’t always know what’s happening in there, you gotta call it out, and you gotta put it in front of you so that you remind yourself that there is something happening there that allows you to grow the movement.

Kien Lee: Next slide, please.

Kien Lee: And then, you know, sort of the measurement of this is this, right? So, like, in the proximal side, this is where sometimes we have to get to that, you know, 2 feet above the ground. I think it’s just animated, Sabine? So, like, you can just probably just animate it. Yeah. So, you know, so that the left, like, you know, when, when

Kien Lee: Ripple is thinking about its contributions and its catalytic actions. It’s holding itself accountable to those things. So what are those measures? How are they looking at holding themselves accountable to these things? Like, yeah, we’re doing the coaching, we’re building the partnerships, we’re supporting our stewards, what are they doing there? And then those outcomes that come out of that, right? So this is kind of where they have a lot more control.

Kien Lee: And thinking about those, immediate, changes that they can see with those that they engage and the communities in which they work.

Kien Lee: And then there’s the diffusion, like we said, this is where there needs to be, sort of, the learning, and a little bit more thinking about how do you even assess that diffusion. Then the distal outcomes, which, Ripple has been very good, has been collecting some of the data, doing national polls of, this is the movement measure, this is what’s happening in the movement. And this kind of outcomes and these kind of measures that it’s not always about, like, you know, is the movement right or wrong? Is the movement good

Kien Lee: or bad, but it’s about, is the movement growing? What are we learning from it so that we can go back and be better at what we do, and go back so that, you know, Ripple can say, this is where we can be better, this is where we can contribute some more. So this sort of gives you a taste of sort of what we were wrestling with and what we were thinking about.

Kien Lee: And with that, I’m going to turn it back to Jane.

Jane Erickson: Thank you. One thing to… to kind of double-click on around all of the

Jane Erickson: stuff that Ken just shared is the… in those hexagons, the kind of proximal, more near-term things that we’re tending to in our field work and in our evaluation work, and in those… those kind of longer-term

Jane Erickson: Distal elements, there is a lot of consistency in what we are paying attention to doing and measuring.

Jane Erickson: So it’s all rooted in the stuff that we talked about at the beginning of the talk, around shared stewardship, around vital conditions, and around thriving. Just looking at that in some different ways, and of course, in the proximal, more near-term stuff.

Jane Erickson: really trying to understand not only the extent to which these shifts are happening with the practitioners and the communities that we support, but also trying to discern, to the extent that we can, what Ripple’s contribution is to those things.

Jane Erickson: So, I will, a few more points, and then we’re gonna open it up to, discussion, and I’m really excited to hear more thoughts from Sanjeev.

Jane Erickson: This evaluation becomes enterprise through our organizational learning work, so our norms and systems for learning within projects and across them. One of the biggest breakthroughs that Ken and the community science crew helped us with is orienting

Jane Erickson: all of this through a cluster evaluation approach, really, again, rooted in action learning. In philanthropy, cluster evaluations are usually focused on, you know, a part of a portfolio, and then looking at projects within that, and thinking about the evaluation of those things as a suite.

Jane Erickson: Ripple is taking that and applying it across all of our work, again, in service to learning and navigation towards that North Star. So what does that actually look like in practice? It looks like

Jane Erickson: learning and using evaluation insights for that learning that happens within individual projects. It happens across groups of like projects.

Jane Erickson: And then laddering up everything across the organization. We’re still very much in the early days of implementing this. Our colleague Sierra Bryant, who I know is listening right now.

Jane Erickson: And is an incredible emergent learning practitioner, is leading that work.

Jane Erickson: And the hope is that all of this is, you know, not only helps us to do our work and orient our, choices internally, but also that it can help inform resources and publications that we share with the field.

Jane Erickson: More broadly, sort of that returning learning to the system is really, really important.

Jane Erickson: And how we think about the role of evaluation in supporting movement building. We can go to the last slide here. So, as evaluators in all of this.

Jane Erickson: Our team at Ripple really sees our role as supporting learning and creativity and kind of co-discerning value with our program teams.

Jane Erickson: Rather than, and with our partners, rather than kind of being

Jane Erickson: outside experts determining value. We wanted to share this quote because it really lifts up

Jane Erickson: what feels meaningful about these beliefs and approaches to the people and organizations that we support. These are the kinds of changes that we hope to see, you know, more broadly across the country. So I’ll paraphrase here.

Jane Erickson: that vital conditions and shared stewardship gave me a whole new vocabulary. That vocabulary is a part of our organization, and has given us a way to shift our culture.

Jane Erickson: I, Sanjeev, I love that you brought, Nanoa Thompson into this at the very beginning. Your remarks there makes me, kind of, just in bookending this, makes me think about his work around navigation.

Jane Erickson: And how he asks us to think about the idea of social navigation.

Jane Erickson: Can you see your future? Do you believe you can navigate there? And we take that as sort of core to both our strategy at Ripple, how we do our work, and why we do our evaluation work. So the hope is that…

Jane Erickson: Through the design and delivery of the,

Jane Erickson: of the work that we do, that it helps to lift these things up and aid in the discovery process for stewards around the country. And is certainly how we aim to use our kind of evaluation work for ourselves as a navigational tool.

Jane Erickson: So with that, I will hand it back to Ken for some discussion.

Kien Lee: Thank you. Thank you, everybody. I’m going to first turn to Sanjeev, again, so that he can share some thoughts as a discussion on this. And Sanjeev, also, if you, as you’re discussing, there’s a question in the chat just about the

Kien Lee: Your comment about, needing a better vocabulary that is intentionally curated for this kind of directed social change.

Sanjeev Sridharan: Super. I’ll try to be quick. Just to fix the problem, I just want to say your focus on thriving, suffering, and struggling, and there are real metrics, and Printful takes it super seriously, really helped me as we worked on brain health to actually help

Sanjeev Sridharan: move away from clinical versus community intervention. It really provides an integrated frame in a very real way. I recommend you check out the Ripple reports on this. Just to frame my comments, I’m really going to be thinking about Parkinson’s disease. People are suffering at a time of COVID.

Sanjeev Sridharan: How can evaluations help? The North Star is we want people to lead better quality of life, so to be more precise, to be thriving individuals. Right now, during… when we started the evaluations 2020, things were not looking great. People were… people were depressed, they were lonely, etc.

Sanjeev Sridharan: So, what would be… what would be evaluation’s contribution to this dialogue? I’ll ask these following 7 or 8 questions. Number one, in what ways… I’m no longer that impressed by theories of change, I’ve got to say. I asked the question, in what ways does the evaluation highlight that the theory of change in itself is incomplete?

Sanjeev Sridharan: I think more often than not, we are sort of reifying theories of change instead of really interrogating them. So I really use incomplete… the fact is, there’s no way a brain health intervention or dance intervention can address everybody’s needs. It surfaces in completeness. We are rather remarkably poor in doing that, number one.

Sanjeev Sridharan: Two, I don’t think that context is taken seriously in a majority of evaluations. It’s a buzzword.

Sanjeev Sridharan: I like asking, in what ways did this evaluation take context seriously? Whose context? Etc. And so, to echo Sabine, did you actually talk to people, nothing about me without me itself, too? How does that come in? Or is it… is there a distant remote expert sitting… sitting far away deciding what our context is? So, how do we actually use context itself as a measure of rig… as a measure of rigor of an evaluation?

Sanjeev Sridharan: Three, much of this work, brain health, again, is just a fixer in your mind, mental health, there are multiple values at play. How does the evaluation itself surface values, but surface a heterogeneity of values? And what’s the kind of space in which these different values play out? That’s three.

Sanjeev Sridharan: Fourth is quite critical. It’s not really learning, but it’s dynamic learning. That’s where the social navigation bit comes in, kind of stuff, to where you’re actually kind of not… it’s not a learning framework. You’ve got to be learning as you go along. You’re trying to fly… you’re trying to… you’re trying to navigate from Hawaii to Tahiti. How do you actually kind of adapt to changing conditions, etc. That’s four.

Sanjeev Sridharan: Father, it’s really critical to Ripple’s work, but I want to push on this. I think as a field, we’re not good. This idea of refining a unifying narrative around stories, I think as an evaluation field, we’ve been remarkably poor in building platforms of stories. How do we actually take stories, not in a way of collecting stories, but how does that story itself become your guide for what’s coming next? And that’s why I’m asking one of the questions, you’ll see words like love, words like healing.

Sanjeev Sridharan: You see the vocabulary that comes out of stories.

Sanjeev Sridharan: trillion times richer than our original theories of change. But again, the issue is not reporting on the stories, the issue is so what. How are you doing taking the stories to challenge yourself to have a much more emergent theory of change?

Sanjeev Sridharan: I love Ripple’s focus on invest in vital conditions, and I would include ideas of loneliness, etc. How do you make people… But that actually throws in quite a few evaluation challenges that in our great flag waving on outcomes, that I think we might forget that actually getting these vital conditions right becomes important. It actually throws a value to challenges, I don’t want to take too much time discussing that.

Sanjeev Sridharan: Eighth, it’s not just measurement, but measurement for navigation. What is the measurement? And again, a navigator kind of knows you’ve got to kind of actually mix measures. So really, in a sense, we’re not after evidence, we’re after an ecology of evidence that kind of moves us in the right direction. Story for another day. Last two comments?

Sanjeev Sridharan: really critical in Ripple’s work on fostering inclusiveness. I love the saying, diversity is being invited to a party, inclusion is being asked to dance. Nine times out of 10, or 9 times out of 100, in evaluations are… we talk to all these different people, look at all the people around the table. It’s silly. There’s almost never an evaluation that actually says this is how, as a result of the evaluative process, people got up and danced.

Sanjeev Sridharan: And I think that’s really important, that we don’t use words like inclusion if we are not serious about it.

Sanjeev Sridharan: I think the last comment is critical. My friend Michael Wolcock, who’s somewhere in Boston, likes to say we should be in the business of solving problems, not in the business of selling solutions. I mean, too often, evaluations focused with a project really is in the business of selling a solution. The problem we are trying to solve is to work on people

Sanjeev Sridharan: people, as they define it, how they define what they mean thriving means to them, how do we address that problem? All right, I’ll stop it there, but my point is, as a field, we have a lot of work to do to step up to the plate. I’ll stop it there.

Kien Lee: Thank you, Sanjeev. That’s a lot, and, incredibly rich, a lot of things for all of us to think about, and very timely at a time like this, as we ask ourselves what our roles are to help solve problems, that we’re facing today, big problems.

Kien Lee: And, Ajain, I wanted… I saw a question in there, and I wanted to check in with you. Sanjeev said a little bit about, navigational learning and being able to use measurement for navigation. Do you want to add a little bit more to that comment, since that was a question in the chat?

Jane Erickson: Boom.

Jane Erickson: Sure, and I can give, I’ll share this through kind of a very specific example, and then, I would love to hand it over to Sabine to share a little bit more about how we’ve been

Jane Erickson: kind of implementing a lot of this from a kind of day-to-day standpoint, because I think it gives a really good… kind of brings this to life.

Jane Erickson: So,

Jane Erickson: we are, you know, I think most important is holding that, accountability is very important, understanding, the shifts that are happening and aren’t

Jane Erickson: And why? Also really, really important. And positioning those things

Jane Erickson: Not just, as it ends to themselves, but as ways that we can help to understand the extent to which, as an organization and a field, we are really making progress towards very ambitious aspirations.

Jane Erickson: And not doing that just within the context of a specific project, but looking at this from the perspective of how an organization is showing up effectively in its role, and more broadly, how all of it is kind of knitting together as a field.

Jane Erickson: So we… I see this as, as, kind of repositioning one of the main goals of evaluation and thinking about it as a navigational tool. We are,

Jane Erickson: just starting a strategic planning process, for the next 3 years of our work at Ripple, and all of that kind of is situated within this 10-year roadmap. And one of the first things that we did, was to essentially do a retrospective of

Jane Erickson: all of the lessons that we have learned over all of our work, but position the… that kind of

Jane Erickson: inquiry that we did across staff, looking at all of our evaluation publications and whatnot, really centering this question of what does it take to diffuse? How do we do that? What does it look like to do that in deep partnership with others?

Jane Erickson: So that’s an example of how we use, kind of, the brain trust of our past work and our evaluation work, and also kind of lessons from others in the field to help us make decisions as an organization about the direction that we want to be going in.

Jane Erickson: And I’ll hand it over to Sabine to share some… some other examples and other thoughts about what this kind of navigation looks like in practice.

Sabine Monice: Thanks, Jane. So, there are some, like, we have some external examples, like, with our partners,

Sabine Monice: we, recently launched the Movement to Thrive Together hub, so, in…

Sabine Monice: a way where folks can, you know, there’s some… there’s a data element to that, if, I think there will be some resources that we’ll share. So beginning to track progress within the movement and kind of see, through quantitative and qualitative data resources, just, like, understanding the state of, like, thriving together and vital conditions and stewardship, so kind of how those pieces all together, like, work together.

Sabine Monice: And then with, our work with our strategic partnerships team, just, like, helping our partners kind of understand a little bit more of monitoring and evaluation work. That’s a little bit about how we’re kind of use… utilizing some pieces of this project.

Sabine Monice: And then, kind of, internally facing, when we help support our strategic initiatives team, just doing just basic, kind of, the nuts and bolts of.

Sabine Monice: evaluation work, so project evaluation, so kind of the… the not-sexy stuff, but the things that are really foundational to kind of monitoring our progress and how do we do better, right? So just the data collection tools and things like that.

Sabine Monice: again, to Jane’s point, she’s already mentioned, but strategic planning on a higher level, and then how we communicate with our board, and how we effectively communicate our mission, to them. So those are just some of the practical ways that we’ve been using, the tools from this work on a day-to-day basis.

Kien Lee: Thank you.

Kien Lee: I, you know, want to move to some questions that are in the chat, and just want to acknowledge that some of the questions, are more concrete, direct questions that I know Jane can answer, and I know she’s typing an answer, and then some, are bigger questions that I think would be great to hear from everybody on this call. So I’m going to drop the first one in the chat. I’m going to read it so everybody hears it, but I’m going to

Kien Lee: Drop it, because some of us are visual, it helps to see it.

Kien Lee: The question is, I would appreciate the panelists’ reflections on the links between the positional humility you mentioned and a power lens. How do you understand power, how to navigate power differences, and how to circulate power productively?

Kien Lee: I drop it in the chat if anybody needs to read it as well. Zay, anybody want to go for it first?

Kien Lee: Go, Sungji.

Sanjeev Sridharan: I don’t mind. I used to be with the Gates Foundation, And…

Sanjeev Sridharan: Too much of the evaluation’s job was show us the proof that the Gates Foundation is making a difference in mental mortality in India.

Sanjeev Sridharan: Knowing fully well Gates Foundation is going to leave India in a few years, it made little sense. The story and all our theories have changed, including the way we

Sanjeev Sridharan: we represented partners was literally Gates as the big… the big… the big… the big, bear in the room here itself, to everybody else being, like, sidelings itself. So, first comment.

Sanjeev Sridharan: Evaluations do things that reify power, that amplify the power differentials by including our theories of change. We don’t… so, that’s one. Two, it’s not terribly hard to invest $100 million and have impacts on maternal mortality. It’s harder to have sustainable impacts. So our theories of change of…

Sanjeev Sridharan: thinking through what happens when Gates leaves is quite different from a theory of change that focuses on impact. So once again, we are so fixated on our vocabulary of impacts itself, too.

Sanjeev Sridharan: So, three is, what are evaluation tools, coming back to Bridget’s question, that actually kind of gets Gates to say, actually, let’s start… let’s start talking to the community, let’s start talking to UNICEF and a bunch of other organizations that are… that are focused on similar kinds of problems here in the community, how we talk to the state, in a way that actually our own theories of change are co-constructed. There’s almost never that kind of process happening. So, Bridget, good question.

Sanjeev Sridharan: The quick answer is, I actually think our tools amplify a lack of humility, rather than actually, thinking of ways that actually mitigate it. Oh.

Kien Lee: Thank you, Sanjeev. I would add to that, too, that I think what we end up doing too much, and spending too much time, is defining power. We spend so much time to figure out how do you define it, how do you measure it, rather than just understanding it and acting on it. And so we… we… I don’t think power is that difficult. Power means people get what they want.

Kien Lee: There are some people who’ve never gotten what they want, and they should have the power, and then there’s some people who always get what they want, and still have the power, and how do we deal with that? And so, you know, I… we at Community Science and my former partner and I, David Chavez, have always talked about, like, you know, it’s really figuring out what do we do with it, and spend less time trying to understand it philosophically and define it, because we’ll never come up with

Kien Lee: a definition that everyone will agree with. I’ll turn it over to Jane and Sabine.

Jane Erickson: I’ll just add kind of a, operational comment. In… so, so…

Jane Erickson: the role of positional humility and thinking about power and all of the things that are connected to it show up in some very explicit ways. I’ll just share in our theory of change and in how we measure. One really specific example, so we’ve talked a lot about shifting stewardship practices and then building vital conditions. There’s a whole lot of stuff in between

Jane Erickson: The kind of shifting how we work together in order to shift conditions in communities.

Jane Erickson: Obviously, Power building and the role of, authenticity and relationship.

Jane Erickson: is really critical in that. Those are questions that we think about in our project evaluation work, and then kind of across the enterprise more broadly. So, how are… to what extent are… is power shifting as a part of a change process, and with whom? How are relationships strengthening, and in service to whom?

Jane Erickson: And then, you know, being able to share those kind of stories of significant change in a way that make those emerging shifts visible and accessible to the partners that we’re supporting, to help their movement building locally.

Kien Lee: Anything to add, Sabine?

Sabine Monice: I’m just reflecting on your comment of, like, the definition of power. I think sometimes getting caught up in

Sabine Monice: and defining things. I think there are pros and cons to that. So…

Sabine Monice: Yeah, I think it is… yeah, I think it’s very good advice to just kind of…

Sabine Monice: not let the definitions kind of, break… break… kind of be… be a barrier to the work, but I do also want to reflect on a moment in this project

Sabine Monice: where kind of the word movement was a bit of a tension, in the theory of change, right? And how we needed to kind of coalesce around what are we defining movement for. So, I think

Sabine Monice: There is a time and a place for definition, especially when you have a group of evaluators, right? So being very clear about the intention. So as long as

Sabine Monice: You’re clear about the intention, I think.

Sabine Monice: That can kind of help navigate, the work and where that energy needs to go. I don’t know if that is helpful, but, because sometimes you can get caught up in the language, but then it’s like, what is the intention about the group of people that you’re bringing together? If it’s all for a similar cause, I think that can be helpful, so…

Kien Lee: Thank you. Thank you. We’re about 3 minutes to the end of the webinar, and I want to respect everybody’s time, and there’s… there are still questions coming in. We also received a long list of questions during registration, so we’re going to do our best to answer the ones that are pretty straightforward to answer without discussion, and perhaps even, put them together and share them with everybody, so everybody can actually see the questions that are coming out.

Kien Lee: And really gives you a sense of where we are as a field and where people’s thinking are. I think there may be no answers, but the questions do give you that sense as well. So for the last 3 minutes, I just want to ask Sanjeev, Jane, and Sabine if you have any other comments you want to add before you… we close.

Sanjeev Sridharan: I don’t mind, but first, let’s wrap it up with… I think it’s sort of important for us to recognize

Sanjeev Sridharan: that we are a evolutionary… I’m talking about evaluation… is a fairly… it’s still early days in our own field itself. Rather than get stuck into our ways of thinking, whether it’s developmental, summative, or formative, or our tools, how do we come to the party in a way that’s helpful, that’s humble?

Sanjeev Sridharan: That’s part… one… one simple chair in a large table. How do we actually

Sanjeev Sridharan: in a sense, be part of the problem solving in a much more dynamic way. I’ll leave it there.

Kien Lee: Thank you.

Jane Erickson: I’ll just say, kind of very briefly, speaking of how…

Jane Erickson: all of this was made possible at Ripple. The… kind of where we started and… and our history is really rooted… our DNA is about organizational learning and about systems change, and so the…

Jane Erickson: In some ways, there was not really a buy-in process internally with our board and leadership and staff. The question was how.

Jane Erickson: And that really allowed us to be creative and ambitious, and to have the gift of working with all of these thought leaders locally and nationally to help contribute and build this. So, I feel like we were able to do something and go far because we were starting from a really solid place in terms of

Jane Erickson: The appreciation and the value of doing this work in this way.

Jane Erickson: Sabine.

Sabine Monice: I’m just grateful for, community science for helping, navigate what could have been a very,

Sabine Monice: tumultuous process to bring out some… a really great starting point for… for Ripple, and us as we continue our journey to,

Sabine Monice: the North Star with, everybody, having access to a thriving future. So, thank you, Kien, and… and the leadership, and the thought partnership you brought, and Sanjeev, and… and, it’s been… it’s been a great pleasure working on this project, and… and we are such a better organization, being rooted in such… in such great, and having such a great foundation in this evaluation for… for it, so…

Sabine Monice: really grateful.

Kien Lee: Great, thank you. We’re right on time, so I want to say I have the pleasure of being the last person to wrap it up, so I can say I agree with everything my wonderful colleagues here said, and if nothing else to add, thank you everybody for being part of this, for staying on till the end, and, look forward to sharing all the information with you. Have a wonderful rest of the day, or evening.

Kien Lee: Thank you.

Sanjeev Sridharan: Thanks a lot.