Contents | 1. Essence | 2. Aproaches | 3. Process | 4. Measuring outcomes | 5. Paradoxes
6. Examples | 7. Jargon | 8. Checklist | 9. Practice tips | 10. Resources

1. The essence of evaluation?

The essence

Evaluation is a process of asking and answering questions about worthwhileness.

We often judge worthwhileness in terms of effectiveness, efficiency, adequacy and appropriateness.

In answering evaluation questions we need to use the tools most appropriate to the change processes we are evaluating. So evaluating an administrative process like doing the payroll is different to evaluating a community development process.

The size of the evaluation needs to be in proportion to purpose of the evaluation. This often means a small project will have a small evaluation compared with a larger project.

Planning and evaluation are integrally connected. They are like two sides of the one coin. When projects are being planned it is also the time to plan the evaluation. See the Section Planning on page 19.

Worthwhileness

Evaluation is a process of asking and answering questions about worthwhileness.
Is something worthwhile happening?
Could something more worthwhile be happening?
There are no value free evaluations. Worthwhileness implies values. A key question in any evaluation is: Whose values will be used to judge the worth of what is happening?

Evaluation questions

We often judge worthwhileness in terms of effectiveness, efficiency, adequacy and appropriateness.

• Effectiveness, e.g. are we making a difference?
• Efficiency, e.g. can we achieve more with less?
• Adequacy, e.g. are we adequately meeting the needs of the client? Are we adequately meeting the needs of the community?
• Appropriateness, e.g. is what we do appropriate in relation to the funding guidelines?

An evaluation process

Evaluation is a process:

• Having a purpose
• Asking a question
• Identifying the information needed to answer the question
• Designing and testing a method for collecting the information
• Collecting the information
• Analysing the information
• Determining the answer to the question
• Using the answer.

Key question: Who are the users of the evaluation? What will they find useful?

Evaluation strategies and tools

Many different strategies and tools are used in evaluation processes, e.g.

• Interviews
• Questionnaires
• Focus groups
• Peer review
• Statistical analysis
• Report writing

Key question: What is the most appropriate mix of strategies and tools for any particular evaluation process?

Types of change processes

There are many different types of change processes, e.g.

• Manufacturing processes (e.g. making light bulbs)
• Administrative processes (e.g. doing the payroll)
• Service processes (e.g. banking)
• Human service processes (i.e. processes where people change during the process, e.g. counselling)
• Community development processes (e.g. identifying community needs and working out ways to meet them).

The characteristics of these change processes are different, for example, manufacturing and administrative processes compared with human service and community development processes:

• are more precisely defined
• are more standardised
• have clearer cause and effect links
• are often made up of countable and measurable steps.

Human service and community development processes compared with manufacturing and administrative processes:

• are less precisely defined
• are more individualised
• have multiple cause and multiple effects - it is hard to show cause and effect links
• involve people making choices to participate.

In community development processes:

• Each process is unique - it is not a standardised process
• The specific goals to be achieved may not be known at the beginning of the processes - it is an open-ended process not a pre-determined one
• Often the goals to be achieved are difficult to precisely define
• The steps in the process are often not known in advance; nor are they precisely defined
• People in the processes are part of families, friends, neighbourhoods, work teams, communities
• There are many causes and many effects and so it is hard to show cause and effect relationships
• People in the processes make choices about their commitment and participation
• People in the processes may want different things.

These characteristics affect the appropriateness of the strategies and tools. For example it is not appropriate to use unit costing in community development processes because unit costing requires standardised processes whereas community development processes are individualised and open-ended.

Key question: Are the evaluation strategies and tools appropriate for the kinds of processes being evaluated?

Uncertainty and rigour

The differences in characteristics of the various types of processes mean there is more uncertainty in evaluating human service and community development processes than there is in evaluating manufacturing or administrative processes.
For the same reasons, there is more possibility of staff in human services and community development processes thinking they are doing a good job when they are not (self-delusion), compared with staff in manufacturing and administrative processes. It is obvious if the payroll is not completed on time or a television is not functioning. It is often not so obvious that a community development process is not working well.

More rigour is required in the evaluation of human services and community development processes (compared with manufacturing or administrative processes) to ensure we are reasonably certain of what is happening in the change processes being evaluated.

Rules of thumb

Some rules of thumb for rigour are:

• Use many different kinds of evaluation strategies
• Always use:
  - collaborative reflection and dialogue strategies
  - listening to peoples ‘experiences’ strategies
  - facts and figures strategies
• Ask different people the same things
• Involve people with contrary views.

Rules of thumb for use of data

• do not use "performance indicators" to be the judge of your performance
• use "performance indicators" to help you ask good questions.

Key question: Given the nature of the change processes we are evaluating, have we sufficient evaluation strategies in place to convince a reasonable person about the worth of what we are doing?