Ideas

This post explains how foundations, nonprofits, and public agencies can use AI-powered analytics to strengthen evaluation and continuous learning while keeping human judgment at the center. Drawing on lessons from practitioners, it shows how organizations can use AI to analyze data faster, support real-time learning, build staff capacity, and make better use of evaluation resources. It also explains why responsible AI use requires clear learning questions, human validation, attention to equity, and openness to evidence that challenges assumptions.

Artificial intelligence (AI) is everywhere. We use the term for technologies ranging from Claude, ChatGPT, and machine learning to self-driving cars and facial recognition software. It has become shorthand for almost any technology that appears to think, predict, generate, or analyze.

That broad definition carries into research, evaluation, and philanthropy. Organizations increasingly use AI to analyze evaluation findings, grantee reports, survey responses, administrative records, and other data. The question can quickly become: If the technology can analyze it, why not use it?

AI has unquestionably expanded what’s possible. It can process large volumes of information far faster than conventional approaches, creating new opportunities to generate and use evidence. But greater capacity introduces a new risk: treating every available data point as worth analyzing simply because we can. Just because we can, should we?

To explore the use of AI-powered analytics among people working in philanthropy, human services, leadership development, and evaluation, my colleague Stephen Shimshock from BCT Partners and I recently hosted the webinar Can We Evaluate and Learn Faster with AI-Powered Analytics? The discussion featured Kyle Hanson of the Idaho Department of Health and Welfare, Patrick Kirby of the Institute for Nonprofit Practice, Kate Ryan of Gemma Services, and Anita Patel of the Bush Foundation. Together, we examined how organizations are or are not integrating AI into their work and the practical and ethical questions that may raise.

Here are six lessons from the conversation.

1. Human Judgment Is Still Essential

AI-powered tools from natural language processing to machine-learning systems cannot replace people in generating credible insights and evidence. The three panelists already using these tools described how staff remain directly involved in developing the criteria and rubrics used to organize and analyze case-management and leadership-course data. BCT and Community Science reinforced that human judgment is needed at every stage: defining a schema to structure unstructured data, manually analyzing a sample using the schema, comparing the results with AI-generated results, refining the schema, and reviewing the final analysis for accuracy and context. AI can accelerate analysis, but it cannot supply the expertise, accountability, or interpretation that makes the analysis useful.

2. Ground AI Use in Evaluative Thinking and Continuous Learning

AI makes it tempting to analyze large volumes of data simply because we can. However, the most useful questions are still: What are we trying to learn, and which data will help us answer that question?

Community Science and BCT developed the framework below to ground that inquiry. It builds on an earlier Community Science model and connects evaluative thinking with continuous learning and strategy improvement.

The framework treats evaluative thinking as an ongoing management and learning practice instead of an exercise reserved for the end of a program. We look backward to understand what happened and forward to test what might happen next. Organizations make the problem explicit; take small, adaptive steps; learn from the results; and build the capacity to decide what to continue, change, scale, or stop. Repeating the cycle allows evidence to shape strategy in real time while also supporting board reporting, community accountability, field building, and other external needs.

Equity should be embedded throughout the cycle:

• At the outset, examine available data through a systems lens to identify disparities and ask what has worked, for whom, and under what circumstances.
• As the solution evolves, ensure that measures and data collection and analysis methods reflect the cultures, histories, and contexts in which the work operates.
• During learning and action, intentionally include people who are often excluded from receiving findings or making decisions about how those findings will be used.

3. Aim for More Learning from the Same Investment

A common question is: Can AI reduce evaluation costs?

AI can stretch a budget, but it doesn’t necessarily make evaluation cheaper. Its value may lie in using the same resources differently, for example, analyzing more data without substantially increasing the budget. Kate Ryan from Gemma Services and Patrick Kirby from INP shared that some efficiencies are possible, particularly in analysis time. Many essential costs still remain including: engaging participants, developing a theory of change, determining evaluation and learning questions, identifying indicators and measures, designing data collection and analysis, validating AI-generated analysis, and interpreting findings. AI can also require substantial upfront investment in systems, staff capacity, and software. Kate and Patrick emphasized that over time, however, those investments help embed evaluation into day-to-day operations and enable staff to conduct more analysis themselves, resulting in consistent adaptation and improvement. The goal isn’t necessarily evaluation for less; it’s more evaluation and learning from the same investment.

4. More Data Isn’t Always Better

The ability to analyze more data doesn’t mean that more analysis will produce better decisions. Start by using the framework above to determine what you need to learn, which data can answer those questions, and how much evidence is sufficient. Then assess whether an AI-powered approach is worth the investment.
As the panelists’ experiences showed, the technology doesn’t eliminate the upfront work required for a sound evaluation; it changes where time and resources are spent.

5. Faster Iteration Can Engage Program Staff

Evaluation is often burdened by myths, judgment, and anxiety. AI-powered tools can make analysis, adaptation, and reflection more immediate, creating opportunities for program staff to build evaluative thinking, reflective learning, and strategy skills. Used well, this faster feedback loop can:

• Build staff skills through repeated practice and timely feedback;
• Create an organizational rhythm for sharing results, reflecting, and generating evidence without waiting months for an external evaluator’s report; and
• Help staff examine their work creatively. Kyle Hanson from the Idaho Department of Health and Welfare described how they developed a strengths-based case concept that helped staff use data to identify the factors supporting young people’s success, and then consider how to cultivate those strengths with other youth.

6. Responsible Use Requires More Than Accuracy

Panelists raised several interconnected concerns about AI adoption. First, its use should align with an organization’s mission and values. For mission-driven organizations, according to Anita Patel from the Bush Foundation, that means considering broader consequences, for instance, AI’s environmental footprint and whether they conflict with a charitable purpose.

Second, AI-generated analysis cannot be accepted at face value. People must validate how data are coded, assess whether results make sense, and investigate problems in the underlying algorithm or model. Human oversight, however, brings its own risk: because AI makes it easy to iterate across large datasets, users may keep rerunning analyses until the results confirm what they already believe. Responsible use therefore requires transparency, critical judgment, and a willingness to let evidence challenge existing assumptions.

Conclusion: Use AI to Strengthen Human Capacity for Decisionmaking

There is no single perfect approach to AI-powered evaluation. Responsible practice will emerge through thoughtful experimentation, reflection, and shared learning. Stephen emphasized that the central aim should be to strengthen human capacity to make better decisions. We should use AI to accelerate feedback and expand what teams can learn and adapt, while keeping people accountable for the questions, methods, validation, interpretation, and decisions.

Ground the work in an explicit learning framework, test outputs against evidence and lived expertise, and remain open to findings that challenge what you think you know. As organizations learn, sharing what works, what fails, what concerns us, and what excites us will help the field move forward without losing the human judgment at the heart of meaningful evaluation.

About the Author

• How can organizations use AI-powered analytics in evaluation and learning?
• Why does human judgment remain essential when AI analyzes evaluation data?
• How can evaluative thinking guide decisions about when and how to use AI?
• Can AI reduce evaluation costs or help organizations get more learning from the same investment?
• Why is analyzing more data not always better?
• How can faster AI-supported feedback engage program staff in continuous learning?
• What does responsible use of AI-powered analytics require?