As part of its mission to conduct objective, rigorous research of community change processes, in 2016 a Community Science team completed and published a new valid and reliable index of collective community capacity. This article describes the new ACEs and Resilience Collective Community Capacity (ARC3) Survey in detail.

ACEs. Adverse Childhood Experiences (ACEs) are a complex population health problem with significant detrimental outcomes. ACEs are commonly defined as 10 types of child abuse, neglect, and family exposure to toxic stress. ACEs are (1) emotional abuse, (2) physical abuse, (3) sexual abuse, (4) emotional neglect, (5) physical neglect, (6) mother treated violently, (7) household substance abuse, (8) household mental illness, (9) parental separation or divorce, and (10) incarcerated household member (American Academy of Pediatrics, 2014).

National and state government leaders, foundations, researchers, social service agencies, and concerned communities are working to (a) increase awareness and understanding of the impact of ACEs, (b) develop effective strategies to prevent ACEs, increase resilience, alleviate trauma, break the complex cycle of intergenerational transfer of ACEs from parents to their children, and (c) support communities as they promote healthy child and adult development (Robert Wood Johnson Foundation, 2015). There is an allied movement to increase resilience at both individual and community levels (Pinderhughes et al., 2015).

APPI Project. In 2012, the ACEs Public-Private Initiative (APPI), a Washington state consortium of public agencies, private foundations, and local networks, was formed to study interventions to prevent and mitigate adverse childhood experiences and facilitate statewide learning and dialogue on these topics. Using a competitive process, APPI selected five communities throughout the state to participate in the evaluation: the Skagit County Child and Family Consortium; the Whatcom Family & Community Network; the Okanogan County Community Coalition; the Coalition for Children and Families of North Central Washington; and the Walla Walla County Community Network.

Community Science worked with researchers from Mathematica Policy Research to evaluate the APPI project. The evaluation team assessed the extent to which five community sites developed sufficient capacity to achieve their goals and examined the relationship of the sites’ capacity to selected site efforts and their impact on ACEs-related outcomes. To measure the community capacity of the APPI sites, Community Science led the development, design, implementation, and analysis of the ARC3 survey to measure the APPI sites’ collective community capacity to address ACEs and increase resilience in their communities.

Community Capacity. The ARC3 survey is grounded in collective community capacity building theory and practice. Community capacity is commonly defined as “the interaction of human, organizational, and social capacity existing within a given community that can be leveraged to solve collective problems and improve or maintain the well-being of a given community” (Chaskin, 1999). To develop community capacity measures for the ARC3 survey, Community Science and the evaluation team worked with the APPI leadership and APPI sites to identify collective community capacity concepts and measures that (1) differentiated between coalition-, network-, and community-wide levels of capacity; (2) were shared by multiple capacity building models; (3) were associated with positive outcomes; (4) were relevant to ACEs and resilience; and (5) were measured through valid and reliable survey instruments.

To identify community capacity concepts and measures that fit these criteria, the evaluation team reviewed the research literature from five community capacity building models that were common across the APPI sites. The models were (1) prevention coalitions, (2) community collaborations, (3) comprehensive community initiatives, (4) community capacity development, and (5) collective impact. The evaluation team also looked for valid and reliable survey measures that were able to (1) differentiate between coalition-, network-, and community-wide levels of capacity; (2) be shared by multiple capacity building models; (3) be associated with positive outcomes; and (4) be relevant to ACEs and resilience. The team found five survey instruments that fit some criteria and adapted items from the tools to create the ARC3 survey.

Measuring Four Levels of Capacity. The ARC3 survey was designed to gather data at four nested levels or layers of capacity: (1) coalition capacity to develop and sustain a strong infrastructure; (2) network capacity to work collectively across sectors on community change; (3) capacity to plan and implement community-based solutions addressing ACEs and resilience; and (4) community-wide capacity to empower the entire community to work at scale to achieve community-wide results.
These levels of capacity map onto 11 ARC3 capacity domains (see Table 1).

  • At the core team or coalition level, the survey collects information about the strength and sustainability of the site’s leadership, infrastructure, and communications functions.
  • At the network level, the survey collects information about the site’s ability to develop a network structure of community partners who can work collectively across sectors on community change.
  • The survey measures the community’s capacity to address ACEs through community problem-solving processes that focus on equity and are informed by data.
  • At the level of community-wide impact, the survey collects information about site-specific strategies to empower the community to work at multiple levels at sufficient scale (breadth) and scope (depth) to achieve community-wide results.

Survey Implementation. The survey was drafted and tested in three pilot sites. The pilot survey originally included 10 measurement domains. The eleventh domain (network structure) was added to the final instrument. The pilot survey was administered to members and community partners of three (non-APPI) community coalitions in Washington state. Analysis of the pilot survey results showed that the 10 domains of the Collective Community Capacity Index ranged from acceptable to excellent in their internal consistency. Based on the feedback obtained during the pilot, the evaluation team shortened the instrument to 56 questions. The final survey was administered over a five-week period during February and March 2016. All individuals included on the sites’ member and partner lists were asked to respond to the survey. The survey’s overall response rate was 84.4 percent.

Survey Results. The sites received their highest scores in five domains: (1) developing community cross-sector partnerships addressing ACEs, (2) implementing evidence-based, community problem-solving processes, (3) developing shared goals targeting ACEs and resilience, (4) communicating effectively with their partners, and (5) addressing equity. The sites reported moderate capacity in (1) developing a sustainable infrastructure, (2) engaging and mobilizing large numbers of community residents, (3) implementing programs, policies, and practices at multiple levels, and (4) increasing their capacity to use data to document and evaluate their results. The lowest score was obtained for sites’ capacity to work at sufficient scale to achieve community-wide change. These scores reflect the site capacities described in the APPI interim evaluation report and in the site profiles in the final report (Hargreaves et al., 2015; Verbitsky-Savitz et al., 2016). This corroborative evidence supports the validity of the survey’s results.

Key Findings. Two sites with the highest collective capacity index scores, on average, were among the three top sites with demonstrated evidence of effectiveness in the final report’s outcome study. Their coalition capacities, community change activities, and network structures were quite different than those of the third site. The first two sites focused more on evidence-based, universal prevention programs and were supported by dense partner networks. In contrast, the third site operated more like an entrepreneurial business, and it created a larger, more diverse, and less dense network to work with more diverse community partners on a broader range of community awareness efforts and more experimental trauma-informed pilot projects.

This is an important finding—community capacity is not the only driver of community impact. There is not just one “best practice” for creating community coalitions or networks. Rather, it is the alignment of (1) collective community capacity, (2) network characteristics, and (3) choice of community change strategies that drives community change. Together, these elements form a locally based theory of change for achieving community-wide impact. Optimal alignment varies, based on community needs and conditions.

References
American Academy of Pediatrics (2014). Adverse childhood experiences and the lifelong consequences of trauma. Available at https://www.aap.org/en-us/Documents/ttb_aces_consequences.pdf. Accessed November 1, 2016.

Bess, K. D. (2015). Reframing coalitions as systems interventions: A network study exploring the contribution of a youth violence prevention coalition to broader system capacity. American Journal of Community Psychology, 55, 381–395.

Bush, R., Dower, J., & Mutch, A. (2002). Community capacity index manual: Version 2. Brisbane, Australia: University of Queensland Centre for Primary Health Care.

Butterfoss, F. D. (2007). Coalitions and partnerships in community health. San Francisco: John Wiley and Sons.

Chaskin, R. (1999). Defining community capacity: A framework and implications from a comprehensive community initiative. Chicago: Chapin Hall Center for Children, University of Chicago.

Hargreaves, M., Verbitsky-Savitz, N., Penoyer, S., Vine, M., Ruttner, L., & Davidoff-Gore, A. (2015). APPI cross-site evaluation: Interim report. Cambridge, MA: Mathematica Policy Research. Available at https://www.mathematica-mpr.com/our-publications-and-findings/publications/appi-crosssite-evaluation-interim-report

Pinderhughes, H., Davis, R., & Williams, M. (2015). Adverse community experiences and resilience: A framework for addressing and preventing community trauma. Oakland CA: Prevention Institute.

Robert Wood Johnson Foundation (2015). ACEs: Early life events that can damage our adult health. Available at http://www.rwjf.org/en/library/collections/aces.html

Verbitsky-Savitz, N., Hargreaves, M., Penoyer, S., Morales, N., Coffee-Borden, B., & Whitesell. E. (2016). Preventing and mitigating the effects of ACEs by building community capacity and resilience: The APPI cross-site evaluation findings. July 2016. Washington, DC: Mathematica Policy Research. Available at http://www.appi-wa.org/evaluation/evaluation-reports