Counting and Measuring in Community Services
|People change in the process||People don't change in the process|
|Intended outcomes not known in advance||Outcomes known in advance|
|Outcomes not precisely defined||Outcomes precisely defined|
|Processes not well defined||Processes well defined|
|Processes individualised||Processes standardised|
|Outcomes individualised||Outcomes standardised|
|Multiple causes and multiple effects - hard to show cause and effect relationships||Processes have established links between causes and effects|
|The process is often a person-person||Process is often a person-object|
|People make choices about the process||Products don't make choices|
|Usefully described as an open system||Usefully described as a closed system|
|Often looking for long term outcomes||Often looking for short-term outcomes|
Each type of process has a unique mix of X and Y characteristics. For example community development processes have more X characteristics than administrative processes.
In manufacturing processes the product is defined in advance and the process for manufacturing the product is usually precisely defined (Y characteristics). Whereas in a human service intended outcomes are not necessarily known in advance and the process is individualised rather than standardised (X characteristics).
See Table A below for one interpretation of the predominant characteristics of each of the six types of processes noted above.
The X characteristics increase the levels of uncertainty associated with counting and measuring the process. The Y characteristics increase the levels of certainty associated with counting and measuring the process.
The more X characteristics in your process the more you will need to use numbers to help you ask good questions rather than to provide the answers. In processes with many Y characteristics numbers are often used as the judge of performance - this is not possible in processes with many X characteristics.
There are many purposes for counting and measuring processes. Tools are often designed with specific purposes in mind. For example the following purpose and tools often go together:
Counting and measuring tools are based on assumptions, for example:
processes with countable/measurable steps and components
Benchmarking Standardised processes; standardised outcomes; countable and measurable steps
Quality control Standardised processes, standardised outcomes and identifiable cause and effect chains
Cost-benefit analysis Costs and benefits must be able to be quantified in monetary units
Goal attainment scaling individualised outcomes which can be determined in advance
A key question is: Are the assumptions on which the tool is based consistent with the characteristics of the processes in which you are using the tool.
Many counting and measuring tools have been designed for processes with Y characteristics. People then try to use these tools in processes with X characteristics. This inevitably leads to difficulties.
For example applying unit costing to administrative processes is likely to be successful because unit costing is based on principles of standardised processes with steps which can be counted and measured. Whereas using unit costing in human services or community development processes is likely to be problematic because the processes are more individualised and open-ended.
To be effective and useful count and measure tools must be matched to the characteristics of the processes they are being used in.
Sometimes count and measure tools developed for one kind of process are used in another type of process and the tool is subsequently modified to make it appropriate to the new type of process.
In 1992 NCOSS published a booklet on Performance Indicators and included the definition of a performance indicator as:
A numerical measure of the degree to which the objective is being achieved.
Human services have consistently had difficulties developing numerical measures for the degree to which objectives are being achieved. In the manufacturing sector a rule of thumb in relation to performance indicators is one only counts what one has control over. This highlights the difficulty of developing performance indicators in human services because often one does not have control over what one is trying to achieve.
In 1994 the Department of Finance in their publication Doing Evaluations A Practical Guide did not even include the term performance indicator, preferring to replace it with performance information which it defined as:
Evidence about performance that is collected and used systematically.
This change in approach from numerical measures of achievements to systematically collected evidence is an attempt to deal with the issues associated with the original nature of performance indicators as a count and measure tool in a manufacturing or administrative process and its application in other kinds of processes.
There is substantial evidence that count and measure tools can be very effective. Japanese products in the 1950s were often considered to be cheap junk. They are now considered to be high quality. One of the reasons is the use of total quality management and its range of count and measure tools.
There is substantial evidence unit costing can be used to improve an organisation's efficiency (in administrative and manufacturing processes). However there is negligible evidence that unit costing has made community development processes more efficient. There is negligible evidence that performance indicators (as defined as numerical measures of what has been achieved) have been useful in measuring outcomes in human services.
The key question is not: Is there evidence that this tool worked? but rather Is there evidence the tool worked for this purpose in this kind of process?
The papers at the NCOSS Made to Measure Conference described a variety of count and measure tools for different processes and purposes. For example:
Richard Eckersley used a variety of social indicators to discuss issues around well-being material progress and quality of life. Indicators included Gross Domestic Product (GDP) , surveys of public perceptions, Living Planet Index (LPI) and Genuine Progress Indicator (GPI). This paper is example of using social indicators to identify and evaluate social change.
The Australian Institute of Health and Welfare is responsible for developing National Community Services information agreements, information development plans and data dictionaries. This work provides a national framework for collecting data about community services. The framework is designed for multiple users and uses. In theory it provides a framework for a variety of processes in community services. In practice the framework is less developed in the area of social change processes and community development processes.
Brian Elton gave a paper on Unit costing and output based funding in human services and Lyla Rogan of PRP consulting presented some case studies of unit costing of human services.
Some of the limitations of unit costing Rogan identified were:
Unit costing is only meaningful where outputs can be defined. This is likely where service is predictable and regular. It is not possible where an agency is providing a multi-faceted service (different needs, intensity of services, length of service, etc).
These papers are examples of the use of unit costing in human services and the difficulties that are involved. It is no accident that when Elton was asked whether he knew of any organisations that had successfully used unit costing in human services such as family support services he replied "No". Unit costing has been around for decades. Is it that no one has thought of using it in human services? Is it that using it effectively in human services like family support services is an impossibility? It is that the tool if significantly modified to take account of the characteristics of human services will be useful?
Gillian McFee from the Department of community Services presented a case study on costing the Half-fare Transport Concession Scheme. This involved costing direct costs such as tasks undertaken by customer service staff and intangible costs such as opportunity costs and fraud. This is an example of a cost benefit tool used in administrative processes. While effective in this case it is unlikely the same tools would work in a community development process or a human service process such as family support services.
There are many concerns in community services about the use and misuse of count and measure tools. The concern is often that the tools being proposed are not appropriate for their purpose or the process in which they are being used, for example,
In processes with more X characteristics than Y characteristics there are many difficulties with using count and measure tools. However services do need to be able to answer the questions: Are we doing a good job? How do we know? To answer these questions services will need to use a wide range of strategies and not rely on one or two count and measure tools.
If you are considering using count and measure tools in community services some key questions are:
The following six processes have different characteristics. Some processes have predominantly X Characteristics, some have predominantly Y characteristics. The following is one interpretation of the predominant characteristics of each type of processes.
|X Characteristics||Social Change Processes||Community Development Processes||Human Service Processes||Service Processes||Administrative Processes||Manufacturing processes||Y Characteristics|
|People change in the process||X||X||X||XY||Y||Y||People don't change in the process|
|Intended outcomes not known in advance||X||X||XY||Y||Y||Y||Outcomes known in advance|
|Outcomes not precisely defined||X||X||X||XY||Y||Y||Outcomes precisely defined|
|Processes not well defined||X||X||X||XY||Y||Y||Processes well defined|
|Processes individualised||X||X||XY||XY||Y||Y||Processes standardised|
|Outcomes individualised||X||X||XY||XY||Y||Y||Outcomes standardised|
|Multiple causes and multiple effects - hard to show cause and effect links||X||X||X||XY||Y||Y||Processes have established links between causes and effects|
|The process is often a person-person relationship||X||X||X||X||Y||Y||Process is often a person-object relationship|
|People make choices about the process||X||X||X||X||Y||Y||Products don't make choices|
|Usefully described as an open system||X||X||X||XY||Y||Y||Usefully described as a closed system|
|Often looking for long term outcomes||X||X||X||Y||Y||Y||Often looking for short-term outcomes|
Manufacturing process, eg, making talking books, disability aids
Administrative processes, eg, doing the accounts, the payroll
Service processes, eg, accommodation, information, banking
Human service processes, eg, counselling, family support services
Community development processes, eg, getting a bus route changed
Social change processes in society, eg, social change from the 1950s to the 1990s in Australian society.
Manufacturing processes are usually standardised processes to create standardised products.
Human services are often individualised processes to achieve individualised outcomes.
Describe what is happening
Allocate funds equitably
Compare your processes with another organisations processes
For example, Total Quality Management has been designed to improve processes within organisations. It is based on assumptions that there are standardised processes to achieve standardised outcomes and the steps in the processes can be counted and measured.
for example, unit-costing may be a useful tool in the process of improving efficiency for standardised processes. It will be an inappropriate tool for improving efficiency in community development processes.
For example, Who has used the tool in this way in this kind of process and found it useful?