 |
|
|
|
|
|
|
|
|
|
| search | feedback | |
|
|
| thriving organisations |
|
| governance | |
| planning | |
| evaluation | |
| quality improvement |
|
| evidence based practice |
|
| risk management | |
| writing
policy & organisational manuals |
|
| questionnaires | |
| data analysis | |
| social capital | |
| facilitation | |
| training workshops | |
| ideas & articles | |
| about us | |
|
|
| disclaimer | copyright | |
Management |
|
Contents |
1. Essence | 2.
Aproaches | 3. Process | 4.
Measuring outcomes | 5. Paradoxes
6.
Examples | 7.
Jargon | 8.
Checklist | 9.
Practice tips | 10.
Resources
Some questions to consider in measuring outcomes are:
-
Where does measuring outcomes fit (in evaluation)?
- Why measure outcomes?
- How in theory can we measure outcomes?
- What are some of the paradoxes and dilemmas in practice?
- What is realistic? Who can do what?
- Practice tips?
1. Where does measuring outcomes fit?
In essence, in measuring outcomes in human services we want to describe:
-
The outcomes we are trying to achieve (and any unintended outcomes)
-
The extent to which we are achieving these outcomes (including showing a cause and effect link between the services provided and the outcomes achieved).
Measuring outcomes in human services is one small part of evaluating human services.
Measuring outcomes in human services does not replace other essential approaches to evaluation, for example:
-
Program evaluation
- Program monitoring and review
- Service network capacity evaluation
- Continuous process improvement
- Etc
Even if we are able to measure outcomes we also need to know the answers to other questions such as:I
-
Are we using an appropriate service model?
- Do we have adequate referral networks?
- Are case conferences working?
- Is the staff peer review process appropriate?
- Etc.
If human services outcomes could be measured:
Clients
Clients will have a more accurate and reliable picture of what has been achieved by a particular service. This is likely to allow them to make better judgments about the value of the service and also make better choices about services.
Practitioners
Practitioners will be better able to monitor and reflect on their work because they will have measures of what has been achieved.
Services
Services that want to continuously improve the quality of their services will have information about the effectiveness of the services provided. This information can be used to monitor the effects of improvements to service processes.
Services that want to reflect on the different kinds of services will be able to see the relative effectiveness of the different kinds of programs they are offering. For example is family work in home more or less effective than group work? Under what circumstances?
Peak organisations
Peak organisations will have the evidence necessary to:
- develop standards of practice based on what standards actually get better results for clients
- develop suggested tools and databases for services
- participate in policy discussions and debates with evidence about the costs and benefits of services.
Government
Government will be able to determine the costs and benefits of human
services and make better policy decisions about prevention
services.
For example if
family support can be shown to achieve outcomes and those outcomes
lead to less child abuse it may be far more cost effective to pay
for prevention
services than for services that have to work with problems after
they have arisen.
3.
How “in theory” can we measure outcomes
in human services?
There is a difference between measuring outcomes in human services
“in
theory” and “in practice”.
It is useful to ask the question: How could we measure outcomes in
human services “in theory” if adequate
resources were given to the task and the practice paradoxes and dilemmas
were
not
there to complicate the
task.
The answer can provide a starting point for thinking about measuring
outcomes in human services.
“In theory” we can measure outcomes in human services
by
describing:
The outcomes we are trying to achieve
The extent to which we are achieving these outcomes(and unintended outcomes).
To do this we
would:
A. Agree on the specific outcomes
to be achieved. For
example in family support services outcomes may include: improved
self esteem, improved family and friends networks, better parenting
practices.
B. Develop valid and reliable
measurement tools for these outcomes,
eg, for self-esteem, family and friends networks and parenting
practices. (Valid meaning that the measurement tool measures
what it says it measures and reliable meaning that if we
made the same measurement twice we get the same result.)
C.
Develop valid and reliable measurement tools for unintended
outcomes
D. Use the tools to measure
change in clients over time (including
collecting and analysing the data). For example
by measuring the clients situation at the beginning of
service, at the
end of service and at a point in time after the
completion of service and analysing the results.
E. Show a cause and
effect connection between the service provision and the outcomes. This
could be done in one or more of several ways, for example:
Research design such as random assignment of clients to different types of service provision
Connecting research findings with practice measures. For example in measuring health outcomes we can monitor the number of people who are smoking or non smoking because there is separate research to show the connections between smoking and health outcomes. If there were adequate research into family support outcomes we could connect our service measures with the findings of that research.
4. What are the paradoxes and dilemmas in practice?
In practice there are numerous paradoxes and dilemmas in evaluating human services and community development, and in particular measuring outcomes in human services. These paradoxes and dilemmas create enormous difficulties in practice.
Some of the key paradoxes and dilemmas are:
-
The outcomes to be achieved may not be known in advance.
-
The outcomes to be achieved may be difficult to precisely define.
-
The specific outcomes we are wanting to achieve may differ depending on the values we hold (clients, worker, funding body)
-
People involved with the service processes may have different views about what outcomes have been achieved and the extent to which they have been achieved.
-
People in the service process are part of families, friends, neighbourhoods, work teams and communities (and so there multiple causes and multiple effects on clients)
-
Services are provided short term. We are hoping for long term outcomes.
-
The service is part of a service network (and so is only one part of the human services system)
-
There are different approaches to how to measure outcomes.
-
There are conflicting demands between good service delivery practice and good measurement and data collection practice.
-
There are conflicting demands between resource use for direct services and resource use for measurement and data collection.
-
There are conflicting demands between client confidentiality and data collection
-
Human services work with clients with low literacy levels, physical and intellectual disabilities and mental health issues. Many measurement tools are often not suited for use with these people.
-
There are characteristics of being human that mean that we don't always know that we think we know.
For full details see Paradoxes and dilemmas in practice.
The overall consequences are that:
a)
it is often difficult to measure outcomes in human services
b)
it is difficult to show cause and effect
relationships between
services provided and outcomes for clients or communities.
5. What is realistic? Who can do what?
Within
the context of all the paradoxes and dilemmas if outcomes in
human services are to be measured then everyone needs to play
their part.
Clients and
practitioners
Clients and practitioners will be explicit
about the client’s situation
and what they are working on. This involves practitioners
making explicit:
- the service models they are using
- the judgments they make about the client and the client’s situation.
This will require practitioners
- using appropriate assessment tools and
- gathering relevant data
- reflecting on the analysis of the data to improve service practice.
It will require clients
- completing an assessment tool or survey
- completing follow up surveys or other tools
- reflecting on the results.
Services
Services will gather data on who receives what services.
They will systematically review the evidence for change taking place with
clients and their situations.
In addition to the points above for practitioners and clients,
service staff will:
Collect data on all clients
Analyse data
Report on the analyses
Use the analysis in reflection on service practice to improve service practice.
Peak
bodies (for
service providers)
Peak
bodies will:
Gather monitoring and review data
Coordinate the development of the measurement tools
Develop a research agenda in collaboration with other relevant organisations to work specifically on showing cause and effect links between services and outcomes
Government
Government will have program monitoring and review information
available at a regional and state level. This will require support
from government
for services programs:
Minimum data sets
Data collation tools
Data collation and analysis systems.
Government will
establish frameworks and systems for data to be gathered from the
services network about clients, not just about individual services.
Government will fund services for the time and resources they spend on data
collection, collation and analysis.
Government will fund research into human services, particularly
research to show cause and effect linkages between
service delivery
and outcomes.
Other
organisations and agencies
Universities and other academic organisations will undertake
the necessary research to provide a sound basis for the use and
interpretation of evaluation
tools in human services.
6. Practice tips
Balancing the intrinsic appeal of measuring outcomes in human services with the enormous difficulties cause by the complexity, paradoxes and dilemmas a few practical tips are:
- Where it is not possible to prove cause and effect relationships do not use outcome measure to judge your performance. Rather use outcome measures to help you ask good questions.
- Don't just focus on the outcomes to be achieved, have processes in place to identify and document unintended outcomes.