The recent tragic incident at the Pulse nightclub in Orlando and the many similar incidents before that continue to bring to the surface issues about mental health (feelings of hopelessness, sense of isolation, and mismanaged anger), broken systems (ability of individuals with mental health issues to purchase assault weapons so easily), sense of community (individuals’ need to belong to a group of people with shared experiences), and intersectionality (people who struggle with multiple social identities, but societal and group norms prescribe which singular identity they should have). The incident was triggered by hatred toward a group of people with different sexual orientations and gender identities; there has been speculation that the perpetrator himself may have been struggling with multiple social identities. The incident unleashed more fear and prejudice against an entire community of people with a different faith. The incident also fueled national debate about gun control, causing some to sharpen their focus on stopping the violence and others to become more adamant about the ability to purchase weapons to protect themselves and their loved ones. The entire situation reflects a complex web of biases, misconceptions, systemic issues, and the human need to belong, that combined, can create inequitable outcomes for certain communities.

Inequitable. Inequity. Equity. Equality. Disparity. Unequal. These terms have become increasingly common in communities and national discussions about fighting poverty, lowering the rates of obesity and diabetes, keeping young men of color in school and out of jail, and assuring that everyone has healthcare coverage. Inequity refers to lack of fairness and justice, while equity means that everyone has fair opportunity to attain their fullest potential. The absence of disparities—health, economic, education, and any other condition—should not be mistaken for equity. Elimination of such disparities is merely one step toward achieving equity. But do we really understand what these terms mean? More important, do we really understand the capacities and circumstances needed to achieve equity, what is it we are trying to achieve concretely, and how do we measure our progress toward the “it”?

Evaluating Equity-Promoting Efforts

As evaluators of equity-promoting efforts, we frequently come across people responsible for implementing these efforts, and they are often overwhelmed by the idea of measuring their outcomes. Indeed, it is not a simple feat, and there is no magical answer. Current advancements in data visualization techniques and Geographic Information Systems (GIS) technology mask the thoughtfulness and perseverance and sometimes even infuriation it takes to evaluate equity-promoting initiatives and how we are making progress toward equity. These tools can support the evaluation approach, but they are not the answer. Instead, the best possible answer lies in: 1) clarifying what equity means for the effort, 2) asking the right questions (i.e., challenging assumptions that reflect the status quo or that well-intentioned initiatives will lead to the desired equity outcomes), 3) using the basic tools of evaluation, 4) managing expectations, and 5) communicating effectively. All five principles must be combined to form the best approach to evaluating equity-promoting initiatives; one principle without the other will not be sufficient.

Clarifying what equity means for the effort concretely. The meaning of equity is often taken for granted, especially among progressive individuals who are committed to promoting it. Consequently, extensive discussion about its meaning, particularly within the context of the effort, does not occur, which in turn contributes to misaligned expectations and inaccurate measures of change. As mentioned before, equity means that everyone has fair opportunity to attain his or her fullest potential. In terms of an equity-promoting initiative, it is important to clarify what the concrete outcomes are that can be observed about fair opportunity, fullest potential, and everyone. It is imperative that all stakeholders clarify the entry point for promoting equity (e.g., local or state government, school district, workforce pipeline) and the type of inequity that leads the work. As mentioned before, individuals are not made of a singular but multiple social identities. However, it is hard enough to reduce the inequity for one population experiencing inequity due to a particularly salient demographic characteristic (e.g., race), let alone several populations with other salient demographic labels. Therefore, stakeholders must be clear about the population for which they expect to see change, so that the evaluation treats the outcomes for that population as intentional, while outcomes for other groups as unintentional, but nevertheless equally important to document and understand.

Asking the right questions. Developing evaluation questions is an essential and universal step in the evaluation process. In an equity-promoting effort, attention must be paid to:

  • Asking questions that are feasible to answer within the initiative’s timeframe;
  • Ensuring that the questions are not framed in a way that perpetuates stereotypes about a particular group of people or assumes that equity will be achieved because that is the initiative’s intent; and
  • Being clear about who benefits and who can be potentially harmed by the answers to the questions.

For example, a good evaluation question would be, “To what extent was an equity lens implemented and what conditions were necessary to implement it?” versus “Was equity achieved by implementing the initiative?” Also, youth of color are frequently lumped together and labeled at-risk youth. With such an assumption, the evaluation could end up looking only at negative behaviors and how the initiative improved the behaviors, and neglect to identify positive behaviors, how the initiative helped to reinforce these behaviors, and the role of social capital and community in the youths’ lives. If such assumptions go unchallenged, evaluations will produce outcomes that continue to perpetuate biases about certain populations.

Using the basic tools of evaluation. A strong theory of change and a graphic illustration of how the theory is operationalized and expected to unfold, accompanied by a sound measurement framework (variables, data collection methods, data sources and frequency of data collection) and analysis plan, are essential. The graphic illustration is an especially useful tool to facilitate and push discussions among stakeholders about their vision for the path of change toward equity. The illustration must be explicit about the following:

  • Flow of the incremental outcomes (or immediate, intermediate, and long-term outcomes) that will lead to equity as the impact (e.g., reduced poverty rates). Equity will not likely be achieved by the end of the effort. Nevertheless, it is important to know if the effort is on track to bring about the desired change.
  • The impact desired—even though it is not likely to be achieved within the initiative’s lifetime—must be visible in the illustration to remind all stakeholders to stay focused on the ultimate goal and not become lost in or derailed by the ups and downs of the work, and lose sight of why the effort was important in the first place.
  • The historical, community, and systemic conditions that can potentially affect the effort’s implementation or outcomes are also reminders to all stakeholders to contextualize the change process and outcomes.

The measurement framework, as mentioned before, is a matrix that lays out the variables, data collection methods, data sources, and frequency of data collection based on the illustrated pathway of change. It needs to include process and outcome measures, and collection of qualitative and quantitative data, in order to capture the complexities of the change process. The complexities must be documented and interpreted accurately, because sometimes the outcomes reflect a step in the right direction and other times they signal a step away from the anticipated direction for a variety of reasons, such as changes in the political landscape, economy, or people’s mindset about an issue. While this does not necessarily mean that the equity-promoting effort failed, it does require the evaluator and stakeholders to examine what happened, including what went wrong in the initiative’s implementation, or if external conditions such as national, state, or local elections and economic forces created a window of opportunity or put up a barrier that caused stakeholders to temporarily deviate from the original path of change. Quantitative or qualitative data alone will not fully capture these events and their contributing factors and consequences.

Managing expectations. Some federal and private funders underestimate the complexity of efforts designed to promote equity and, consequently, may have unrealistic expectations about the outcomes. These efforts take time to plan, implement, and evaluate, and they typically involve a whole host of organizations from across sectors to untangle and begin to dismantle the structures in place that perpetuate inequity. Evaluators must play an active role in helping funders and implementers have realistic and aligned expectations in order to ensure the use of right measures. For instance, in an initiative designed to promote economic equity, it is not realistic to see the needle move on poverty rates at the population level in three to five years—the typical grant period for community and systems change initiatives. What is more feasible are changes in local government policies and practices that increase access to job training and jobs for a particular group of people. An example of an outcome here would be increased pay, job promotion, or job retention for the intended group due to new skills or certification for a particular skill.

Also, in the evaluation of equity-promoting initiatives, the paradoxes of cross-sector collaboration and the complexity of understanding and dealing with power dynamics are part of the process (see Community Science’s earlier piece on “It Is Time to Take a Look at Collaborating for Equity and Justice”). Depending on how power is used or exploited, conflict should be an expected short-term outcome and therefore measured and documented.

With such complexities, the results are not so conclusive, and definitive studies are not possible. Funders, grantees, and evaluators should establish a process for feedback, learning, and adapting, which takes time and resources. All stakeholders should be clear from the outset about the amount of time and resources they have to dedicate to the process.

Communicating effectively. Finally, communication about the equity-promoting effort and its goals, evaluating findings, and lessons learned is usually left to the program staff and communications staff without any involvement from evaluators or other experts in strategic communications regarding equity. Equity is a complex concept, and if there is not a communications strategy to message it properly or manage the media, equity-promoting initiatives can create backlash, fear, or leave behind a negative sentiment that will make it harder to implement similar initiatives in the future. An effective communications strategy for equity-promoting efforts must take into consideration the precision of the language used to convey the goals and findings (so that expectations are managed appropriately) as well as the sensitivity of bringing attention to the experiences of certain communities without perpetuating biases about them. The communication also has to help people understand the distinction between individual-, institution-, and community-level changes.

Sounds Simple, but Remains a Challenge

After reading the above, you might ask, “Aren’t the above a part of any good evaluation and not just evaluation of equity-promoting efforts?” And you are right. The above principles should be integrated into the evaluation design for any initiative. However, certain professional traditions, norms, and fads challenge their implementation and hinder the advancement of our collective knowledge about equity work. These include:

  • The methodology for evaluating equity does not sound objective for two reasons. First, it is not an experimental or quasi-experimental design, which is considered the gold standard in evaluation. Yet, these methodologies are not suited for evaluating complex initiatives aimed at promoting equity. Second, when evaluators are direct and candid about their values for social justice, they are perceived as not objective. (The issue of objectivity in research and evaluation will be covered in another newsletter issue by Community Science.)
  • Equity is perceived as a political concept unrelated to science, which implies that it cannot be evaluated or demonstrated. Yet science has demonstrated the presence of inequity and its consequences on individuals, families, and communities.
  • Evaluators must have the will and the capacity to call out inequity and how it affects an effort— from the way it is designed to the way it is evaluated. This expectation is part of the American Evaluation Association’s five guiding principles that require evaluators to respect people and take into account the diversity of general and public interests and values.
  • There is a fad in the research and evaluation consulting industry that seeks to rename and repackage scientific traditions and methods as so-called new solutions for researching or evaluating equity-promoting efforts. Combined with advancements in technology, these new solutions become very attractive to funders and implementers of these efforts and detract from the basic and essential tools of research and evaluation.

Here are initial practices that evaluators can adopt to begin to counter the above traditions, norms, and fads:

  • Educate funders about other methodologies, such as the cross-case study, that are more appropriate for evaluating equity-promoting initiatives and still scientifically rigorous;
  • Set straight that the integrity of the evaluation (rules of evidence, replicable methods, appropriate access to data, and overall transparency) and not objectivity is what is most important;
  • Develop the courage, will, and skills to identify and name the inequity in the assumptions underlying an initiative’s theory of change, and facilitate discussions with funders and implementers to connect the dots between the assumptions, interventions, desired outcomes, and proposed measures;
  • Promote a process of learning and adaptation through a developmental evaluation approach and be clear from the outset about the complexity of the work and the amount of time required of funders and implementers to participate in such a process;
  • Build the capacity of community leaders and decisionmakers to demystify new and emerging methodologies that boast quicker and better solutions to measuring equity by equipping them with skills to break the methodologies down to the basic and essential tools of evaluation and determine the added value of the methodology; and
  • Be explicit about racism and oppression as legitimate subjects for scientific study and discussion.

Ultimately, evaluators have to see themselves as change agents who are responsible for ensuring that their work contributes to attaining equity for all. The above recommended practices will not be easy to adopt if evaluators do not see themselves as holding a piece of the puzzle or having the responsibility to help us all understand what it takes to make progress toward equity.