Sections of this topic
Free Nonprofit Micro-eMBA Module #11: Designing Your Program Evaluation Plans
© Copyright Carter
McNamara, MBA, PhD, Authenticity Consulting, LLC.
Much of the information in this module is adapted from Field Guide to Nonprofit Program Design, Marketing
, and Evaluation.) This learning module is in the nonprofit organization development program.
However, this module can also be used by anyone as a self-study
exercise to learn more about nonprofit program evaluation.
Sections of This Module Include the Following
Introduction
Outcomes
Materials for Review
Suggested Topics for Reflection and Discussion
Activities to Build Systems and Practices
Assessments
Tracking Open Action Items
INTRODUCTION
Nonprofits deliver their services to their communities
in the form of programs. There is a common misconception
that a program is an activity of delivering services — but
that’s not correct. A program is
- a highly integrated set of activities
- that aims to meet a verified, unmet need in the community
- by accomplishing certain outcomes
- among certain groups of clients and
- by evaluating that those outcomes are being achieved among
those clients.
Thus, the design of a program should include verifying that
there indeed is an unmet need in the community. That has to include
more than the passionate assertions of the founders of the nonprofit
— it has to include objective, verifiable evidence of the unmet
need. The design should include verifying what outcomes need to
be achieved, including what new knowledge, skills, and abilities
among the clients to be served. Program evaluation should regularly
be conducted to verify that those outcomes are being achieved.
The activities of program evaluation can include a wide
variety of methods to evaluate many aspects of programs in nonprofits.
As with any evaluation activity, the type of evaluation used
depends on what the organization wants to be able to do as a result
of the evaluation, whether it’s, for example, to improve the program’s
processes or measure the results of the program. There are many types
of program evaluations. For example, a goals-based evaluation
examines if the goals of a program were achieved or not. A process
evaluation examines the strengths and weaknesses of the program’s
processes. An outcomes evaluation examines what outcomes
were achieved and the extent to which desired outcomes were achieved.
There are numerous books and other materials that provide in-depth
analysis of evaluations, their designs, methods, a combination of
methods, and techniques of analysis. However, personnel do not
have to be experts in these topics to carry out a useful program
evaluation. The “20-80” rule applies here, that 20%
of effort generates 80% of the needed results. It’s better to
do what might turnouts be an average effort at evaluation
than to do no evaluation at all because of concerns about not doing
it perfectly. Far too many program evaluations become extended,
scientifically-based, data-driven efforts that generate information
that is either impractical or irrelevant — if the information
is understood at all. This learning module orients personnel to
the nature of program evaluation and how it can be carried out
in a realistic and practical fashion.
NOTE ABOUT BOARD COMMITTEES: Consider using the Board Programs
and Marketing Committee to review and help guide implementation
the information in this learning module. Major activities and
goals from this learning module could be incorporated into that
Committee’s Committee Work Plan.
OUTCOMES
General Program Evaluation
- Understanding Program Evaluation, Goals and Types
- Gain Overview of Data Collection Methods
- Understand Guidelines for Analyzing, Interpreting and Reporting
Information - Recognize Pitfalls to Avoid
- Evaluate Your Program Planning and Evaluation Practices
Outcomes-Based Evaluations
- Get Ready for Your Outcomes Planning
- Select Your Outcomes and Target Outcomes
- Develop Your Logic Model
- Identify Indicators for Each of Your Target Outcomes
- Decide What Data You Need and How You Will Collect It
- Gain Guidelines for Your Data Analyses
- Draft Your Outcomes-Based Evaluation Report
MATERIALS FOR REVIEW
- The following materials will help you address each of
the topics and learning activities in this module.
Basic
Guide to Program Evaluation (about evaluation, in general)
— read all of each of the following sections:
– – – Program
Evaluation
– – – Where
Program Evaluation is Helpful
– – – Basic
Ingredients: Organization and Program(s)
– – – Planning
Your Program Evaluation
– – – Some
Major Types of Program Evaluation
– – – Overview
of Methods to Collect Information
– – – Selecting
Which Methods to Use
– – – Analyzing
and Interpreting Information
– – – Reporting
Evaluation Results
– – – Who
Should Carry Out the Evaluation?
– – – Contents
of an Evaluation Plan
– – – Pitfalls
to Avoid
Outcomes Evaluation — read all of each of the following sections:
– – – Reasons
for Priority on Outcomes-Based Evaluation
– – – Basic
Principles for Small Nonprofits to Remember Before Starting
– – – What
is Outcomes-Based Evaluation?
– – – Common
Myths to Get Out of the Way Before You Start Your Outcomes Planning
– – – Planning
Any Type of Evaluation Includes Answers to These Very Basic Questions
– – – Planning
Your Outcomes Evaluation — Step 1: Getting Ready
– – – Planning
Your Outcomes Evaluation — Step 2: Choosing Outcomes
– – – Planning
Your Outcomes Evaluation — Step 3: Selecting Indicators
– – – Planning
Your Outcomes Evaluation — Step 4: Planning Data/Info Collection
– – – Planning
Your Outcomes Evaluation — Step 5: Piloting/Testing
– – – Planning
Your Outcomes Evaluation — Step 6: Analyzing/Reporting Results
SUGGESTED TOPICS FOR REFLECTION
AND DISCUSSION
- Learners are strongly encouraged to discuss the following
questions with peers, board members, management and staff, as
appropriate.
Program Evaluation in General
1. What is “program evaluation”? Name at least three
of its benefits — or three situations when it is helpful? (See
Program Evaluation and Where Program Evaluation is Helpful.)
2. What are the two key ingredients you need to have before
you can conduct program evaluation? (HINT: The answer may be simpler
than you think, but it’s surprising how many nonprofit do not
have both of these ingredients before they set out to conduct
program evaluation!) (See Basic Ingredients.)
3. What determines how you will conduct your program evaluation?
(HINT: Your program evaluation plans depend on what _____ you
need to collect in order to make major_____.) (See Planning Your Program Evaluation.)
4. Name at least four of the seven key considerations when
designing program evaluations. (See Planning Your Program Evaluation.)
5. What are three major types of evaluations typically done
when carrying out nonprofit program evaluations? (There are many
types of evaluations. This question asks about major types of
evaluation used in program evaluation in nonprofits.) (See Some Major Types of Program Evaluation.)
6. Name at least four of the six major methods of data collection
as listed in the information about program evaluation. (See Overview of Methods to Collect Information .)
7. Name at least five of the nine questions that should be
considered when selecting the evaluation method (when conducting
program evaluation). (See Selecting Which Methods to Use.)
8. What are the four levels of information that can be gathered
from people during a program evaluation? (See Selecting Which Methods to Use.)
9. Name at least three of the five pitfalls of program evaluation
planning. (See Pitfalls to Avoid.)
Outcomes-Based Evaluation
1. What are the major reasons that outcomes-based evaluation
is a top priority among nonprofit funders and service providers
today. (See Reasons for Priority on Implementing Outcomes-Based
Evaluation.)
2. What are program inputs? Activities? Outputs? Outcomes?
Outcome targets? Indicators? (See What is Outcomes-Based Evaluation? )
3. What is the difference between outputs and outcomes? Outputs
and indicators? Indicators and outcome targets? (See What is Outcomes-Based Evaluation? )
4. Name at least three of the six myths listed in the suggested
materials for review. (See Common Myths to Get Out of the Way Before You
Start Planning)
5. Name at least five of the typical questions answered in
almost any type of evaluation. (See Planning Any Type of Evaluation Includes Answers
to These Very Basic Questions.)
6. Name the guidelines for picking which program to evaluate.
(See Planning Your Outcomes Evaluation — Step 1:
Getting Ready.)
7. What is a logic model? (See Planning Your Outcomes Evaluation — Step 2:
Choosing Outcomes.)
8. What should you look for in short-term outcomes? Intermediate
outcomes? Long-term outcomes? (See Planning Your Outcomes Evaluation — Step 2:
Choosing Outcomes.)
9. What is a questionnaire? Interview? Case study? (See Planning Your Outcomes Evaluation — Step 4:
Planning Data/Info Collection.)
10. What are some practical considerations when selecting which
data collection method to use? (See Planning Your Outcomes Evaluation — Step 4:
Planning Data/Info Collection.)
11. What should you always do with the data that you’ve
collected before you analyze it? (See Planning Your Outcomes Evaluation — Step 6:
Analyzing/Reporting Results.)
12. What ultimately determines the level and scope of the evaluation
report? (See Planning Your Outcomes Evaluation — Step 6:
Analyzing/Reporting Results.)
13. Name at least five of the sections of a typical evaluation
report. (See Planning Your Outcomes Evaluation — Step 6:
Analyzing/Reporting Results.)
ACTIVITIES TO BUILD SYSTEMS AND
PRACTICES
- Learners are strongly encouraged to complete the following
activities, and share and discuss results with peers, board members,
management and staff, as appropriate. - As you proceed through the following activities, be sure
to note any incomplete actions in the Action Item Planning List.
Writing Your Overall Program Evaluation Plan
Complete the following checklist about each of the program
that you would like to evaluate.
Purpose of Evaluation?
What do you want to be able to decide as a result of the evaluation?
For example:
__ Understand, verify or increase impact of products or services
on customers/clients (eg, outcomes evaluation)
__ Improve delivery mechanisms to be more efficient and less costly
(eg, process evaluation)
__ Verify that we’re doing what we think we’re doing (eg, process
evaluation)
__ Clarify program goals, processes and outcomes for management
planning
__ Public relations
__ Program comparisons, eg., to decide which should be retained
__ Fully examine and describe effective programs for duplication
elsewhere
__ Other reason(s)
Audience(s) for the Evaluation?
Who are the audiences for the information from the evaluation,
for example:
__ Clients/customers
__ Funders/Investors
__ Board members
__ Management
__ Staff/employees
__ Other(s) _______________
What Kinds of Information Are Needed?
What kinds of information are needed to make the decision you
need to make and/or enlighten your intended audiences, for example,
information to understand:
__ The process of the product or service delivery (its inputs,
activities and outputs)
__ The customers/clients who experience the product or service
__ Strengths and weaknesses of the product or service
__ Benefits to customers/clients (outcomes)
__ How the product or service failed and why, etc.
__ Other type(s) of information?
Type of Evaluation?
Based on the purpose of the evaluation and the kinds of information
needed, what types of evaluation is being planned?
__ Goal-based?
__ Process-based?
__ Outcomes-based?
__ Other(s)? ___________________________________________________
Where Should Information Be Collected From?
__ Staff/employees
__ Clients/customers
__ Program documentation
__ Funders/Investors
__ Other(s) _____________
How Can Information Be Collected in Reasonable and Realistic
Fashion?
__ Questionnaires
__ Interviews
__ Documentation
__ Observing clients/customers
__ Observing staff/employees
__ Conducting focus groups among ______
When is the Information Needed?
What Resources Are Available to Collect the Information?
Writing Your Outcomes-Based Evaluation Plan
Complete each of the steps specified in the following sections.
You will be asked to include some of your answers in the Framework for a Basic Outcomes-Based Evaluation
Plan, which will culminate in your developing your basic outcomes-based
evaluation plan. (You might want to convert the Framework to your
preferred word-processing software to make it easier to modify.)
1. Planning Your Outcomes Evaluation — Step 1:
Getting Ready
2. Planning Your Outcomes Evaluation — Step 2:
Choosing Outcomes
3. Planning Your Outcomes Evaluation — Step 3:
Selecting Indicators
4. Planning Your Outcomes Evaluation — Step 4:
Planning Data/Info Collection
5. Planning Your Outcomes Evaluation — Step 5:
Piloting/Testing
6. Planning Your Outcomes Evaluation — Step 6:
Analyzing/Reporting Results
ASSESSMENTS
1. Planning
Indicators (Best Practices) (includes program planning and evaluation)
REMINDERS FOR THOSE IN THE ON-LINE DEVELOPMENT PROGRAM
1. Are you exchanging feedback with others about what you’re
learning in this program?
2. Are you sticking to your study schedule for this program?
3. Are you practicing your basic skills in management and leadership,
including in problem solving and decision making, planning and
meeting management?
4. Are you communicating throughout your organization by using
your skills in internal communications?
5. Are you managing yourself? How many hours a week are you
working? Are you noticing any signs of stress? If so, what are
you doing about it?
6. One of the ways you might be able to tell if you’re stressed
out and/or losing perspective might be whether you’re tracking
details or not. Are you using the action item list referenced
above?
7. Are you reflecting on learnings from past modules and how they build on the learning
in this module? For example, are you seeing your organization from a systems view,
as explained in the module “Starting and Understanding Your Nonprofit?”
TRACKING OPEN ACTION ITEMS
1. One of the first indicators that an organization is struggling
is that open action items are not tracked and reviewed. (Open
action items are required actions that have not yet been completed.)
Instead, organization members only see and react to the latest
“fires in the workplace”. Whether open action items
are critical to address now or not, they should not entirely be
forgotten. Therefore, update and regularly review a list of open
action items that includes listing each open action item, who
is responsible to complete it, when it should be completed and
any associated comments. When updating the list, consider action
items as identified during discussions, learning activities and
assessments in this module. Share and regularly review this action
item list with the appropriate board, management and staff members
in your nonprofit. You can use the following Action Item Planning List.
2. If you have questions, consider posing them in the national,
free, online discussion group hr.com, which is attended
by many human resource and organization development experts.
(Learners in the nonprofit organization development
program can return to the nonprofit organization development program.)
For the Category of Evaluations (Many Kinds):
To round out your knowledge of this Library topic, you may
want to review some related topics, available from the link below.
Each of the related topics includes free, online resources.
Also, scan the Recommended Books listed below. They have been
selected for their relevance and highly practical nature.
