search  | feedback
  based practice
  risk management
  writing policy
  & organisational
  data analysis
  social capital
  training workshops
  ideas & articles
  about us
  disclaimer | copyright

  Alternatives Pty Ltd
  ABN 23 050 334 435



Evaluation issues: The intrinsic appeal of the theoretically impossible

I have been working with human service organisations since 1988 on planning and evaluating human services and community development processes.

Over that time I have noticed a series of ideas, that can't work because they are theoretically impossible, permeate the sector in waves.

Judging the effectiveness of human services through key performance indicators has a simple logic and elegance. But it is not theoretically possible. In human services performance indicators can provide a starting point for asking good questions but they cannot be the judge of performance and effectiveness. For example, if we are getting better results for clients (key performance indicators up) is it because we are more effective in our work or is it because we are now only working with the easy to work with clients (and so less effective in our work)?

Judging the effectiveness of human services through monitoring clients progress in achieving goals also has a common sense logic - if clients are progressing we must be doing a good job. But monitoring clients progress does not show the cause and effect linkages between services and results. It doesn't show what would have happened if there had been no intervention. Monitoring change in clients may be useful but it is not a judge of program effectiveness.

Accountability through contracts, service specifications and KPIs has intrinsic appeal in a contracting and tendering world - are we getting what we paid for? We want to hold you accountable to deliver on what we have paid for. An admirable thought. But not everything is known in advance. Not everything stays static over the period of a contract. Dialogue and reflection are key ingredients for accountability in times of uncertainty and change.

Why do these ideas (and others like them) have so much intrinsic appeal?

Is it because we find living with uncertainty too difficult? So we pretend there is more certainty that there is. We ignore the causes of uncertainty?

Is it because the alternatives seem too complicated - how could we really understand what is happening in a community development process or service provision in a services network?

Is it because we are using conceptual models and tools that work well in manufacturing production lines and transfer them to human services without systematically analysing the differences in the nature of the processes involved - particularly the nature of the cause and effect linkages, the complexity of measurement and the impact of measurement on the service delivery process itself.

Is it because we get caught up in the dominant discourse - a cultural 'group think'? We unthinking go with the flow. We don't think we can swim against the tide.

Is it because the ideas have administrative convenience? It is easier to report on KPIs and service specifications than tell the story of a changing community and evolving service provision.

It is because ministers and others need to reduce what is to be said to a 20 second grab?

Are the ideas memes* that are good at getting themselves copied?

Is it because we don't want to tell those in authority it doesn't work? Is it a case of the emperor's new clothes?

Paul Bullen