4. Evaluation issues: The intrinsic appeal of
the theoretically impossible
I have been working with human service organisations since 1988 on
planning and evaluating human services and community development processes.
Over that time I have noticed a series of ideas, that can't work because
they are theoretically impossible, permeate the sector in waves.
Judging the effectiveness of human services through key performance indicators
has a simple logic and elegance. But it is not theoretically possible. In
human services performance indicators can provide a starting point for asking
good questions but they cannot be the judge of performance and effectiveness.
For example, if we are getting better results for clients (key performance
indicators up) is it because we are more effective in our work or is it because
we are now only working with the easy to work with clients (and so less effective
in our work)?
Judging the effectiveness of human services through monitoring clients progress
in achieving goals also has a common sense logic - if clients are progressing
we must be doing a good job. But monitoring clients progress does not show
the cause and effect linkages between services and results. It doesn't show
what would have happened if there had been no intervention. Monitoring change
in clients may be useful but it is not a judge of program effectiveness.
Accountability through contracts, service specifications and KPIs has intrinsic
appeal in a contracting and tendering world - are we getting what we paid
for? We want to hold you accountable to deliver on what we have paid for.
An admirable thought. But not everything is known in advance. Not everything
stays static over the period of a contract. Dialogue and reflection are key
ingredients for accountability in times of uncertainty and change.
Why do these ideas (and others like them) have so much intrinsic appeal?
Is it because we find living with uncertainty too difficult? So we pretend
there is more certainty that there is. We ignore the causes of uncertainty?
Is it because the alternatives seem too complicated - how could we really
understand what is happening in a community development process or service
provision in a services network?
Is it because we are using conceptual models and tools that work well in
manufacturing production lines and transfer them to human services without
systematically analysing the differences in the nature of the processes involved
- particularly the nature of the cause and effect linkages, the complexity
of measurement and the impact of measurement on the service delivery process
Is it because we get caught up in the dominant discourse - a cultural 'group
think'? We unthinking go with the flow. We don't think we can swim against
Is it because the ideas have administrative convenience? It is easier to
report on KPIs and service specifications than tell the story of a changing
community and evolving service provision.
It is because ministers and others need to reduce what is to be said to a
20 second grab?
Are the ideas memes* that are good at getting themselves copied?
Is it because we don't want to tell those in authority it doesn't work? Is
it a case of the emperor's new clothes?