Contents | 1. Essence | 2. Aproaches | 3. Process | 4. Measuring outcomes | 5. Paradoxes
6. Examples | 7. Jargon | 8. Checklist | 9. Practice tips | 10. Resources

In practice there are numerous paradoxes and dilemmas in evaluating human services and community development. These paradoxes and dilemmas create enormous difficulties. We must chose between two responses:

• The difficulties are insurmountable. There is no point in evaluating services. We would be wasting our time - lets keep our energies for direct service.
• The difficulties are many and the results will be imperfect but we do need to do something to deal with the potential self-delusion - thinking we are doing a good job when we are not. The same reasons that make evaluation so difficult also create room for self-delusion so we need to be even more rigorous in our evaluation of services.

Some of the paradoxes and dilemmas are:

Outcomes

1. The outcomes to be achieved may not be known in advance.
Often clarity about the outcomes to be achieved only emerges during the course of the service delivery process or the community development process. This means that we may not have made the right observations or measurements at the beginning of the service process to know whether there has been any change during the course of the service delivery or community development.

2. The outcomes to be achieved may be difficult to precisely define
In human services and community development many of the outcomes are inherently difficult to precisely define and so they are also inherently difficult to measure, for example: good parenting, self esteem, agency networks, friendship networks, family networks and reduced risk of child abuse are all difficult to precisely define and measure.

3. The specific outcomes we are wanting to achieve may differ depending on the values we hold.
Where there is not consensus among all stakeholders about the values underpinning the service provision there will not be agreement on the specific outcomes we are looking for. For example, we may all agree that good parenting is an outcome to be achieved. But there are many views of good parenting based on different values. These different values can lead to different judgments about the specific outcomes to be achieved.

4. People involved with the service processes may have different views about what outcomes have been achieved and the extent to which they have been achieved.
In child and family services if we ask the worker, the client and another family member about the outcomes achieved they may say quite different things. It is not uncommon for a client to say that a lot has been achieved and the worker to be concerned at the lack of real progress and a family member to have a different view.

 The nature of the service process - from a client viewpoint

5. There are different views of who the client is.
The client could be:

• an adult family member
• the child
• the immediate family
• the extended family.

Different stakeholders may have different views of who the client is.
This means that when we are talking outcomes for clients we can be talking about very different outcomes. And the data to be collected to measure outcomes can be very different depending on how the client is identified.

6. The steps in the service process can differ from one service to another.
There are often differences between services about the nature of the service process. This means it is often difficult for several services to use the same tools. For example one service may use an assessment process where the worker and the client need to agree on what is written on the assessment form. Another service provider may use a separate form for each of the client and the worker.

7. People in the service processes may want different things.
For example the client may want to separate from her partner. The partner may want to remain together with the client. The children may want something different.

8. People in the processes make choices about their commitment and participation
People may not want to continue to be involved with the service process. How is a client withdrawing to be interpreted? It could be viewed as inappropriate service delivery because it could be seen that the service has failed to keep the client engaged; or it could be viewed as the client exercising their free choice.
These different views lead to different descriptions of outcomes achieved.

9. People in the service process are part of families, friends, neighbourhoods, work teams and communities
This means that it is difficult to show cause and effect relationships because the service is not the only cause at work in the life of the client. If the client is achieving outcomes is this because of the service? Because of support from family members? Support from neighbours? How could we know what the service’s contribution is?

10. Services are provided short term. We are hoping for long term outcomes.
This means that the clients may need to have left the service for many years before we can know whether or not the service has had a real positive impact. But we have to make the decisions about service delivery now.
Extensive longitudinal research can help with this issue, but never solve it because society is changing, and so our knowledge about society is always, to some extent, out of date.

The nature of the service process - from a service viewpoint

11. The service is part of a service network.
This means that it is difficult to show cause and effect relationships between what this service has done and the outcomes achieved because the outcomes may have been achieved through the service provision of the entire network.

12. Service staff are often focussed on individual clients and do not have the time or expertise to build a picture of all their clients as a group
Most human services staff are trained to work with clients. Their work is client focussed and the day to day demands are client focussed. Their day to day work is interrupted with client demands.
These realities often make it difficult for agencies to build pictures of their clients as a group because this task requires different skills (research and data analysis) and different time pressures (eg staff being able to work for longer periods uninterrupted).
Measurement

13. There are different approaches to how to measure outcomes.
For example will we measure outcomes according to:

• An agreed behavioural standard (eg. this is a skill required by all parents)
• The relative change in the client (and their situation) (eg. the client's housing situation has improved in...)
• What we had intended to achieve (eg. we achieved what we said we would).

These approaches lead to the development of different kinds of tools, for example:

• Standardised assessment tools (which can show us how the clients’s responses compare with the population at large)
• Non - standardised questionnaires (which can show us relative change in this client - but not comparisons with the population at large)
• Goal attainment scaling (which can show us the extent to which we achieved the goals we set out to achieve, but not measure the unintended consequences).

In addition, each of these three approaches to measuring outcomes can be developed using different methods of measuring, eg, independent observation and judgment, subjective observation and judgment or subjective judgement.

These various approaches lead to the development of very different tools.

Mixing good service delivery and good measurement

14. There are conflicting demands between good service delivery practice and good measurement and data collection practice.
For example in many human services the service process typically starts with building rapport and works from a basis of building on strengths. To present a long detailed questionnaire to the client at the beginning of the service process may be intrusive and so reduce rapport; if it is comprehensive it is likely to highlight deficits as well as strengths and the client may feel they have less strengths to build on than they had previously felt.

 15. There are conflicting demands between resource use for direct services and resource use for measurement and data collection.
For example should a service use staff time in doing detailed data collection and analysis when the time could alternatively be spent on direct service delivery?

16. There are conflicting demands between client confidentiality and data collection
Clients have a right to confidentiality and services should not provide information about clients to other services without their permission. However, because clients use a variety of services and services are part of service networks data is ideally needed about all the services a client uses.

Ideally what is required is client focussed databases that can gather data on all services provided, rather than service focussed databases.
However services would typically see it an invasion of client privacy if they asked their client for permission to contact all the other service providers for relevant service provision data, especially if all services asked every client. However to paint a picture of services work with clients would require all this data.

17. Human services work with clients in many languages. Measurement tools are often only produced in one or two languages.
This means that the tools may not be available in the language needed. Tools are also often culturally biased. Tools and documentation are often only available in English because of the costs associated with doing translations and assessing cultural bias.

18. Human services work with clients with low literacy levels, physical and intellectual disabilities and mental health issues. Many measurement tools are often not suited for use with these people
This means that the client may not be able to complete a questionnaire at all or without support. Providing support may be a conflict of interest if the support is provided by a staff member. For example the tool may be asking the client to give views on the staff member’s work. Providing independent support may cost too much. For example, employing an independent person to work with the client to complete the tool or survey.

The nature of being human

19. There are characteristics of being human that mean that we don't always know that we think we know.

For example:

• We interpret ambiguous information in ways that fit our preconceptions.
• We are more critical of information that contradicts what we already believe than we are of information that supports our existing beliefs
• We don't always tell the truth
• We tend to see what we do in a positive light
• We are social beings and can create groups or sub-cultures such that everyone in the group reinforces each others views
• We can create self-fulfilling prophesies

How do we respond?

Our attitudes
These paradoxes and dilemmas create enormous difficulties in evaluating human services and community development. We must chose between two responses:

• The difficulties are insurmountable. There is no point in evaluating human services. We would be wasting our time - lets keep our energies for direct service.
• The difficulties are many and the results will be imperfect but we do need to do something to deal with the potential self-delusion - thinking we are doing a good job when we are not. The same reasons that make evaluating human services so difficult also create room for self-delusion. We want to minimize the room for self-delusion and so we take what steps we can. We are more rigorous than we would otherwise be.

Our practice
If our attitude is one of: there are difficulties but we need to do something to minimise the potential self-delusion there is a role for each of:

• Clients and their families
• Human services staff and volunteers
• Human services and agencies
• Peak organisations
• Government
• Other organisations and agencies.

It is only when all parts of the human service system are working together that we will be able to make a reasonable attempt at evaluating human services and community development. It is not just an issue for individual workers or services.

It requires resources. It will require we understand the nature of the human service and community development processes and the inherent paradoxes and dilemmas. It will have an impact on how each of us works.