Counting and Measuring in Community Services

PDF Version

Paul Bullen
October 1999

A. The Context
B. Different Types of Processes
C. Characteristics of Processes
D. Purpose for Counting and Measuring
E. Counting and Measuring Tools
F. Does the Count and Measure tool work for your purpose in your process
G. The Count and Measure Conference
H. Misusing count and measure tools
I. Conclusions

This paper was first published in "Made to Measure: Counting and costing in human services,
Conference Papers, Volume 2" published by New South Wales Council of Social Service, October 1999. It explores the relationship between count and measure tools being used in community services and different types of processes. Tools need to be appropriate for the processes in which they are being used.

Other papers given at the Made to Measure Conference can be found at NCOSS Conference Papers.

A. The Context

Management literature, the social sciences and community services practice are packed with count and measure tools.

Management literature talks about tools like performance indicators, benchmarking, quality control, unit costing and cost-benefit analysis to name a few. The social sciences talk about tools like socioeconomic indices, scales of attitudes and indicators of well being. Community service practice includes everything from 'stats' to goal attainment scaling.

We are part of a count and measure culture. There are spoken and unspoken assumptions: if it can be measured its real; and if it can't be measured it's not so real.

There are many concerns in community services about the use of count and measure tools. Some of these concerns are justified. Others are not.

In working through the issues involved in using count and measure tools in community services one needs to understand different processes have different characteristics and that tools are based on assumptions that may not apply in all processes.

B. Different Types of Processes

There are many different types of processes in community services. Each type of process has different characteristics, eg,

Manufacturing Processes, eg, making talking books or disability aids. These processes are usually standardised processes to create standardised products.

Administrative Processes, eg, doing the accounts. These processes are usually standardised processes to create standardised outputs (eg, staff pays, financial reports).

Service Processes, eg, accommodation, information, (banking and airlines are examples outside community services). These processes have some elements of standardisation and some elements of individualisation.

Human Service Processes (where the client changes through the service process). These processes are usually individualised (eg, counselling, family support services, developing independent living skills).

Community Development Processes (where a group of people take action to meet a community need) , eg, getting a bus route changed. These processes are open-ended; the outcomes are not pre-determined.

Social Change Processes (where society itself is in the process of changing) eg, the changes in Australian society from the 1950s to the 1990s. These processes operate across the whole of society over long time frames and the outcomes of the processes are usually unknown - we describe what has happened after the event rather than predict it before hand.

C. Characteristics of Processes

There are many ways of describing processes. One way is to think in terms of the relative abundance of X characteristics and Y characteristics.

X Characteristics	Y Characteristics
People change in the process	People don't change in the process
Intended outcomes not known in advance	Outcomes known in advance
Outcomes not precisely defined	Outcomes precisely defined
Processes not well defined	Processes well defined
Processes individualised	Processes standardised
Outcomes individualised	Outcomes standardised
Multiple causes and multiple effects - hard to show cause and effect relationships	Processes have established links between causes and effects
The process is often a person-person	Process is often a person-object
People make choices about the process	Products don't make choices
Usefully described as an open system	Usefully described as a closed system
Often looking for long term outcomes	Often looking for short-term outcomes

Each type of process has a unique mix of X and Y characteristics. For example community development processes have more X characteristics than administrative processes.

In manufacturing processes the product is defined in advance and the process for manufacturing the product is usually precisely defined (Y characteristics). Whereas in a human service intended outcomes are not necessarily known in advance and the process is individualised rather than standardised (X characteristics).

See Table A below for one interpretation of the predominant characteristics of each of the six types of processes noted above.

The X characteristics increase the levels of uncertainty associated with counting and measuring the process. The Y characteristics increase the levels of certainty associated with counting and measuring the process.

The more X characteristics in your process the more you will need to use numbers to help you ask good questions rather than to provide the answers. In processes with many Y characteristics numbers are often used as the judge of performance - this is not possible in processes with many X characteristics.

D. Purpose for Counting and Measuring

There are many purposes for counting and measuring processes. Tools are often designed with specific purposes in mind. For example the following purpose and tools often go together:

• Improve the efficiency of the process (unit costing)
• Monitoring the process to ensure it stays within predetermined limits (quality control)
• Measure outcomes from processes (performance indicators)
• Compare your processes with another organisations processes (benchmarking)
• Improve processes so as to improve the quality the outcomes from the process (total quality management and its associated count and measure tools)

E. Counting and Measuring Tools

Counting and measuring tools are based on assumptions, for example:

Tool Assumptions

Unit costing Standardised processes with countable/measurable steps and components
Benchmarking Standardised processes; standardised outcomes; countable and measurable steps
Quality control Standardised processes, standardised outcomes and identifiable cause and effect chains
Cost-benefit analysis Costs and benefits must be able to be quantified in monetary units
Goal attainment scaling individualised outcomes which can be determined in advance

A key question is: Are the assumptions on which the tool is based consistent with the characteristics of the processes in which you are using the tool.

Many counting and measuring tools have been designed for processes with Y characteristics. People then try to use these tools in processes with X characteristics. This inevitably leads to difficulties.

For example applying unit costing to administrative processes is likely to be successful because unit costing is based on principles of standardised processes with steps which can be counted and measured. Whereas using unit costing in human services or community development processes is likely to be problematic because the processes are more individualised and open-ended.

To be effective and useful count and measure tools must be matched to the characteristics of the processes they are being used in.

Sometimes count and measure tools developed for one kind of process are used in another type of process and the tool is subsequently modified to make it appropriate to the new type of process.

In 1992 NCOSS published a booklet on Performance Indicators and included the definition of a performance indicator as:

A numerical measure of the degree to which the objective is being achieved.

Human services have consistently had difficulties developing numerical measures for the degree to which objectives are being achieved. In the manufacturing sector a rule of thumb in relation to performance indicators is one only counts what one has control over. This highlights the difficulty of developing performance indicators in human services because often one does not have control over what one is trying to achieve.

In 1994 the Department of Finance in their publication Doing Evaluations A Practical Guide did not even include the term performance indicator, preferring to replace it with performance information which it defined as:

Evidence about performance that is collected and used systematically.

This change in approach from numerical measures of achievements to systematically collected evidence is an attempt to deal with the issues associated with the original nature of performance indicators as a count and measure tool in a manufacturing or administrative process and its application in other kinds of processes.

F. Does the Count and Measure tool work for your purpose in your process?

There is substantial evidence that count and measure tools can be very effective. Japanese products in the 1950s were often considered to be cheap junk. They are now considered to be high quality. One of the reasons is the use of total quality management and its range of count and measure tools.

There is substantial evidence unit costing can be used to improve an organisation's efficiency (in administrative and manufacturing processes). However there is negligible evidence that unit costing has made community development processes more efficient. There is negligible evidence that performance indicators (as defined as numerical measures of what has been achieved) have been useful in measuring outcomes in human services.

The key question is not: Is there evidence that this tool worked? but rather Is there evidence the tool worked for this purpose in this kind of process?

G. The Count and Measure Conference

The papers at the NCOSS Made to Measure Conference described a variety of count and measure tools for different processes and purposes. For example:

Quality of life

Richard Eckersley used a variety of social indicators to discuss issues around well-being material progress and quality of life. Indicators included Gross Domestic Product (GDP) , surveys of public perceptions, Living Planet Index (LPI) and Genuine Progress Indicator (GPI). This paper is example of using social indicators to identify and evaluate social change.

National Community Services Data Development

The Australian Institute of Health and Welfare is responsible for developing National Community Services information agreements, information development plans and data dictionaries. This work provides a national framework for collecting data about community services. The framework is designed for multiple users and uses. In theory it provides a framework for a variety of processes in community services. In practice the framework is less developed in the area of social change processes and community development processes.

Unit costing

Brian Elton gave a paper on Unit costing and output based funding in human services and Lyla Rogan of PRP consulting presented some case studies of unit costing of human services.

Some of the limitations of unit costing Rogan identified were:

Unit costing is only meaningful where outputs can be defined. This is likely where service is predictable and regular. It is not possible where an agency is providing a multi-faceted service (different needs, intensity of services, length of service, etc).

These papers are examples of the use of unit costing in human services and the difficulties that are involved. It is no accident that when Elton was asked whether he knew of any organisations that had successfully used unit costing in human services such as family support services he replied "No". Unit costing has been around for decades. Is it that no one has thought of using it in human services? Is it that using it effectively in human services like family support services is an impossibility? It is that the tool if significantly modified to take account of the characteristics of human services will be useful?

Half-Fare Transport Concession Scheme

Gillian McFee from the Department of community Services presented a case study on costing the Half-fare Transport Concession Scheme. This involved costing direct costs such as tasks undertaken by customer service staff and intangible costs such as opportunity costs and fraud. This is an example of a cost benefit tool used in administrative processes. While effective in this case it is unlikely the same tools would work in a community development process or a human service process such as family support services.

H. Misusing count and measure tools

There are many concerns in community services about the use and misuse of count and measure tools. The concern is often that the tools being proposed are not appropriate for their purpose or the process in which they are being used, for example,

• the use of unit costing in community development and some human service processes (this is not appropriate because some of the assumptions on which unit costing is based will not hold true in community development and some human service processes)

• the use of numerical performance indicators in human services as a measure of outcomes (this is often not appropriate because of the difficulties in measuring the real outcomes and showing the cause and effect relationships).

In processes with more X characteristics than Y characteristics there are many difficulties with using count and measure tools. However services do need to be able to answer the questions: Are we doing a good job? How do we know? To answer these questions services will need to use a wide range of strategies and not rely on one or two count and measure tools.

I. Conclusions

If you are considering using count and measure tools in community services some key questions are:

• What kind of process are you counting and measuring?
• What are its characteristics?
• What is your purpose - what are you trying to achieve through counting and measuring?
• What tools are available for this purpose?
• On what assumptions are they based?
• What tool is appropriate for your purpose and your process?
• Is there evidence this tool works for this purpose in this process?

Table A

Characteristics of Processes

The following six processes have different characteristics. Some processes have predominantly X Characteristics, some have predominantly Y characteristics. The following is one interpretation of the predominant characteristics of each type of processes.

X Characteristics	Social Change Processes	Community Development Processes	Human Service Processes	Service Processes	Administrative Processes	Manufacturing processes	Y Characteristics
People change in the process	X	X	X	XY	Y	Y	People don't change in the process
Intended outcomes not known in advance	X	X	XY	Y	Y	Y	Outcomes known in advance
Outcomes not precisely defined	X	X	X	XY	Y	Y	Outcomes precisely defined
Processes not well defined	X	X	X	XY	Y	Y	Processes well defined
Processes individualised	X	X	XY	XY	Y	Y	Processes standardised
Outcomes individualised	X	X	XY	XY	Y	Y	Outcomes standardised
Multiple causes and multiple effects - hard to show cause and effect links	X	X	X	XY	Y	Y	Processes have established links between causes and effects
The process is often a person-person relationship	X	X	X	X	Y	Y	Process is often a person-object relationship
People make choices about the process	X	X	X	X	Y	Y	Products don't make choices
Usefully described as an open system	X	X	X	XY	Y	Y	Usefully described as a closed system
Often looking for long term outcomes	X	X	X	Y	Y	Y	Often looking for short-term outcomes

---

Table B

Six VIP Questions

1. Where are you doing the counting and measuring?

For example:

Manufacturing process, eg, making talking books, disability aids
Administrative processes, eg, doing the accounts, the payroll
Service processes, eg, accommodation, information, banking
Human service processes, eg, counselling, family support services
Community development processes, eg, getting a bus route changed
Social change processes in society, eg, social change from the 1950s to the 1990s in Australian society.

2. What is the nature of the processes where you are counting and measuring?

For example:

Manufacturing processes are usually standardised processes to create standardised products.
Human services are often individualised processes to achieve individualised outcomes.

3. What is your purpose - What are you trying to achieve through counting and measuring?

For example:

Describe what is happening
Improve efficiency
Allocate funds equitably
Measure outcomes
Improve processes
Compare your processes with another organisations processes

4. What tools are available for this purpose? On what assumptions are they based?

For example, Total Quality Management has been designed to improve processes within organisations. It is based on assumptions that there are standardised processes to achieve standardised outcomes and the steps in the processes can be counted and measured.

5. What tool is appropriate for your purpose in your process?

for example, unit-costing may be a useful tool in the process of improving efficiency for standardised processes. It will be an inappropriate tool for improving efficiency in community development processes.

6. Is there evidence this tool works for this purpose in this process?

For example, Who has used the tool in this way in this kind of process and found it useful?