Performance Indicators

New management jargon, political marketing,
or one small element in developing quality services?

Editorial comment
What are performance indicators?
Putting performance indicators in their place
Performance indicators - The FINE print
Is your organisation ready for performance indicators?
Will you use performance indicators wisely?

This article was originally published in Caring, the Association of Children's Welfare Agencies Newsletter (September 1991).

Editorial Comment

Performance indicators have reached plague proportions in NSW. They are overrated by the NSW Premier and Government. They are misunderstood and misused by Government departments. They are undervalued by the welfare sector. They are over-promoted by consultants. They are also one small important element in the development of quality services.

The trends within NSW in the past few years can be crystallised by the State Government's slogan "Putting People First by Managing Better". With the slogan has come management by objectives, demands for higher levels of accountability and the need to sell to the electorate the fact that the state is managed better and is performing better. The need to sell the idea that performance is improving has overshadowed the need to actually improve performance.

There is a genuine need for NSW Government Departments, funding programs and welfare organisations to focus on performance, ie what they are achieving, rather than strategies, ie, the activities that are undertaken to achieve results. The emphasis on performance is very useful in bureaucracies and organisations that have tended to justify their existence by the mere fact that everyone is busy rather than what they are actually achieving.

Performance indicators are one way of reinforcing a legitimate emphasis on performance however their implementation in Government departments and the welfare sector leaves much to be desired. Some Government Departments just don't understand how to set objectives and develop performance indicators (See for example the Corporate Plan of the Department of Planning 1990-1993).

Government departments wanting to measure performance of funded organisations are now starting to specify in their funding agreements that performance indicators have to be used and/or developed. Many community organisations funded through the Community Services Grants Program are developing performance indicators to keep the funding body happy rather than to improve performance.

For funded bodies increasing demands for performance indicators are made in a climate of no growth, government funding cut backs, and the rationalisation of services to distribute the funding dollar 'equitably'. Funded organisations are often feeling powerless in the face of the decisions of the bureaucracy and there is a growing breakdown in trust between the funded organisations and their funding bodies.

An emphasis in organisations on what is being achieved is important. However to actually improve performance many other things besides performance indicators are needed as well including: commitment to providing and developing quality services; staff who are willing and encouraged to ask questions about why they do what they do; Management Committee members and staff who are focussed on the clients' need and meeting their clients' needs; performance indicators that are seen for what they are, just one small element in developing quality services; and adequate support services for example staff training and administrative support systems.

Performance indicators in services that aim at changing the person receiving the service are more difficult to develop and use effectively than performance indicators in for example the manufacturing sector. In human services there are complex measurement and cause and effect issues to be considered in developing and using performance indicators.

In human services if performance indicators are used as the judges of performance then they will be misused and abused and it is likely that performance will remain static or even decline.

If performance indicators are seen as some clues among many to asking questions about how performance can be improved then it is possible that performance will improve.

The challenge is for the Government, Government Departments, funding programs and funded organisations to actually improve their performance. To do this may require a new attitude to performance indicators and a much better understanding of the issues involved.

What are Performance Indicators?

Definitions

Performance indicators are one of many tools to help answer the question: How do you know what you are achieving?

One definition of a performance indicator from an NCOSS publication is: A numerical measure of the degree to which the objective is being achieved.

Performance indicators are usually seen as numerical measures of achievement that are easy to collect and use. In theory they can only be derived for things over which you have control, however in practice people don't have absolute control over anything and so 'having control' is really a matter of whether there is enough control for your purpose.

A more sophisticated definition from the Office of Public Management is: A performance indicator defines the measurement of a piece of important and useful information about the performance of a program expressed as a percentage, index, rate or other comparison which is monitored at regular intervals and is compared to one or more criterion.

Examples

If one of your objectives is to train people so they are able to gain employment then one performance indicator can be the proportion of trainees that gain employment after the training is completed.

If one of your objectives is to refer people to services appropriate to their needs then one performance indicator could be the proportion of people who contacted you that considered the referred service relevant to themselves and their needs.

If one of your objectives is to improve the parenting skills of parents then one performance indicator could be the proportion of parents who feel they are coping better with parenting their children at the end of the program (than they were at the beginning).

Development

In order to develop performance indicators it is essential to:

• Clearly identify the organisations values and philosophy;
• Clearly identify the clients and their needs;
• Have aims and objectives which specifically state what is to be achieved in relation to these needs;
• Identify each step in the service delivery process and what is to be achieved at each step in the process as well as how it is to be achieved.

So, for example, an organisation committed to the empowerment of families may have an aim of improving the functioning of families and a service delivery process based around the following outcomes:

• For the client to acknowledge their need;
• For the client to articulate the need;
• For the client to identify what needs to be done to meet the need;
• For the client to increase their self esteem / confidence so it is possible for them to act to meet their needs;
• For the client to increase their skills and develop appropriate attitudes so the client can do what they see they need to do;
• For the client to do what needs to be done to meet their needs.

A performance indicator for each of these steps could be the proportion of total clients where both the worker and the client agree that the outcome has been successfully achieved.

Strategies that could be used to achieve these outcomes could include providing information to clients, interviews, group work, training and so on.

Abuses

There is a commonly held view that performance indicators are used to judge performance. While this may seem self evident it leads to many difficulties in human service organisations. If performance indicators are used as the judge of performance in an unthinking way, wrong conclusions will be drawn.

For example, one of the performance indicators for labour market programs in TAFE is whether or not people in the program gain employment at the end of the course. If the number of people gaining employment at the end of the course reduces has the performance of the course reduced? Possibly, but not necessarily. Alternative explanations are that: there could be less jobs available now than there were previously; the people coming into the course may be less skilled than previous intakes; or a larger proportion than previously may be going on to further education.

Some people have the view that performance indicators should be used as the basis for rewards (and punishments) for individuals or organisations. This may also seem self evident but it is even more seriously flawed than the previous proposition. It assumes that performance indicators and performance are the same, and in most cases they are not.

So where rewards or funding is based on performance indicators the individuals and organisations being judged and rewarded typically start to fudge the figures to obtain the rewards irrespective of performance. The larger the rewards the greater the pressures to fudge the figures.

Fudging the figures can be done in a variety of ways. Sometimes borderline or doubtful cases are reclassified as successful outcomes. "That client really did achieve something didn't they?"

Sometimes the way in which clients are counted is not appropriate. Some organisation have been know to count clients who come to groups each time they come. So if 10 clients come to a group that meets once a week for 10 weeks there are 100 clients counted not 10.

Sometimes the definition of a successful client outcome is changed. For example last year we only counted the outcome as successful when both the worker and the client agreed that the service was successful where as this year we count the success whenever the client thinks there was a success irrespective of the workers views.

Uses

For performance indicators to be used wisely they need to be seen as numerical indicators that require interpretation. They need to be used as some of the many clues available to help in asking questions about performance and the improvement of performance.

Government Departments, Management Committees and staff who wish to use performance indicators wisely will need to be firmly committed to provision and development of quality services and see mistakes (including poor performance) as an opportunity for improving performance rather than an opportunity for 'punishment'(e.g. reprimanding staff or cutting funds). This orientation is only possible where there is a basic trust between the key players (for example staff management committees and funding bodies).

Putting Performance Indicators in their Place

In the last few years in NSW performance indicators appear to have been over-rated. They have been blow up out of proportion. Their rightful place is as one small element in the process of planning for quality services.

With the reviewing of funding programs there has been strong presses for funded organisations to develop performance indicators. Performance indicators cannot be developed in a vacuum. To effectively develop and wisely use performance indicators it is probably essential to deal with each of the following steps that are required in planning for quality services.

Committed to Quality Services

All Management Committee Members and staff need to be committed to providing quality services and the continual improvement of the quality of those services if there is to be an appropriate context for the development of performance indicators.

In the current political climate this commitment to quality service provision must be able to withstand the pressures that will be felt to 'just keep the funding body happy'.

Values

The organisations values must be made explicit if there are to be firm foundations for the organisation.

The values the organisation adopts will affect the needs it wishes to address and clients to whom it will give priority.

Values will affect approaches to service delivery and answers to questions such as:

• What are quality services?
• How do you get quality services?
• How do you know you are providing quality services?

For example if a child care centre's values were built around a core value of 'children should be seen and not heard' then a quality service might include teaching children to be quiet and orderly, service strategies might be highly structured and controlling and a quiet, clean child care centre could be seen as a 'measure' of success.

Alternatively if a child care centre's values were based around the core value of 'individual creativity' then a quality service might include helping children explore, the services would be provided by children being encouraged to express themselves and a measure of success might be the level of noise, fun and mess during the day.

Needs

Peoples needs are the reason for existence for human service organisations. Despite this organisations often have difficulties clearly identifying the needs they wish to meet.

If your organisation is a direct service organisation the need will be a client's need, for example, the need for food, shelter, friends or self-esteem. If it is a need it is probably something you have experienced or can imagine experiencing.

Training is not a need; we may have or feel a need to understand and undertake some training as a way of meeting the need. Counselling is not a need; we may have or feel a need to get on better with those around us and go to a counsellor to help us meet our need.

Some examples of client needs are:

• Need for friends and social contacts;
• Need for a safe environment;
• Need to have an advocate;
• Need to know;
• Need to be able to relate well with those around me;
• Need for parents to have a break from their children.

If the need is clearly identified then a lot can be learnt about what the aims and objectives of the organisation should be, what sort of services are appropriate, where they might be located and so on.

By way of analogy imagine yourself as a 'seller of hats'. What need are you meeting for the people who buy your hats? Is it the need for a hat?

If you sell hats there are many needs you could be meeting. People may have a need to feel good in front of others; they may have a need for protection from the sun; they may have a need for protection from the snow; they may have a need for disguise; they may have a need to meet work regulations (for uniforms).

If you were meeting the need of people to feel good in from of others (fashion) this description of the need would give you a lot of clues about what hats you would sell, to whom, at what price, in what location. If you were meeting people needs for hats for uniforms you would be selling different hats, probably to different people, at a different price, in a different place.

In the same way if an organisation is able to clearly and specifically identify the need it is meeting there will be many clues about the clients, the services they might be require, the location and so on.

Aims and Objectives

Once the clients and their needs are clearly identify you will need to ask: What do we want to achieve in relation to those needs? The answer to this question will be your aims and objectives.

Aims are general statements of what you intend to achieve. In direct service organisations aims will often include or imply what is to be achieved in relation to client needs.

Some examples of service aims are: to strengthen and enhance the capacity of families to meet their needs in child rearing; to enhance the capacity of older people to live full and independent lives within the community; to increase the accessibility and quality of social infrastructure for disadvantaged groups.

Some organisations make a serious error when they are writing their aims and objectives. They write strategies but call them aims and objectives. The following are not examples of aims: to run parenting groups (this is a strategy that would meet the first aim above); to run independent living courses for those over 65 (this is a strategy that could be used to meet the second aim above). These aims do not tell you what the organisation intends to achieve in relation to client needs.

Objectives are specific statements of what you intend to achieve. It is very useful if objectives are written so they are

• Specific;
• Measurable;
• Achievable;
• Results oriented (ie written as something to be achieved);
• Time bound.

Some examples of objectives are: to provide 35 parents each week with a 2 hour break from their children for the next 12 months; within 18 months to significantly increase the level of self-esteem of all the parents of children (within such and such a geographic area) who are at high risk of child abuse.

Sometimes the specific needs of clients are very different from one client to the next. In these services each clients' need has to be assessed on an individual basis. An alternative approach to the above is to write objectives such as:

• For the client to articulate the need;
• For the client to identify what needs to be done to meet the need;
• For the client to increase their self esteem / confidence so it is possible for them to act to meet their needs;
• For the client to do what needs to be done to meet their needs.

Service Delivery Process

Once the aims and objectives are identified a service delivery process needs to be developed. The service delivery process will include:

* A list of what is to be achieved at each step in the process;

For example three initial outcomes in a counselling service delivery process might be:

• Clients are identified
• Client needs are identified
• Clients are prioritised

* Explicit assumptions that you are making about how the outcomes in your service delivery process are linked.

For example in the above process the assumption could be that clients will be prioritised on the basis of their needs and that is why the needs have to be identified before clients are prioritised.

* A list of the strategies that will be used to achieve each outcome in the service delivery process.

For example the strategy used to identify clients could be to wait for referrals;

The strategy used to identify client needs could be an interview and completing of an assessment form;

The strategy used to prioritise clients could be a staff meeting.

* Explicit assumptions that you make when you choose a particular strategy for each outcome in your service delivery process.

For example waiting for clients to be referred assumes that referring agencies know your services are available and that clients in need will contact either your or the agencies that refer clients to you.

Knowing What's Going On

Organisations need information that can help them know both whether their aims and objectives are being achieved and where their strategies are being effectively implemented.

Performance information is information you can use to help answer the question: Are we achieving our aims and objectives?

Activity information is information you can use to answer the question: What do we know about what are we doing? Are we carrying out our strategies?

Performance information can be quantitative or qualitative. Performance indicators are one kind of quantitative performance information.

Activity information can be qualitative or quantitative.

Many of the statistics that organisations typically keep are quantitative activity information. For example, the number of clients, the number of groups run, the number of referrals. This is not primarily performance information. Performance information would include information about what the clients achieved and what the groups achieved.

Performance Indicators - The FINE Print

When you read the fine print things often turn out not to be as simple as they first appear. Performance indicators require a lot of fine print. Here is some of it.

Performance of Activity/Workload Indicators

By definition performance indicators must be about performance rather than the activities undertaken or the level of workload. Occasionally activities and performance are directly related, for example, the number of people vaccinated is a good indicator of both the performance achieved (people vaccinated) and the activities undertaken by medical staff (the number of injections given), however often outcomes and activities are not so directly related.

Unfortunately many organisations and government departments either do not write their aims and objectives as statements of what they will achieve or do not develop performance indicators that relate to achievements. For example the Department of Planning has as one of its objectives "To encourage community participation in environmental planning and assessment process, and to disseminate information about the work of the department to the public" and the performance indicators are "Number of publications produced, exhibitions held, articles published, public enquires handled..." Where are the performance indicators of actual community participation? What is the information dissemination designed to achieve?

How reliable is the indicator?

Reliability is about asking the question if I measured the same thing again would I get the same result?

If I were to measure the floor area of a room, got out a tape measure, measured the size of the room and calculated the area and the room was 20.35 square metres what is the chance that if you measure the same room with the same tape measure that you will get 20.35 square metres? The chance of that would be remote. There is always error in measurement. It is simply a question of how much error, not whether there is any.

If your organisation develops performance indicators you need to make an estimate of how reliable they are and decide whether they are reliable enough for your purposes.

In human services the outcomes are often changes in the clients. Measures of these changes, for example, changes to the level of self-esteem are likely to be far less reliable than measuring the size of a room.

How valid is the indicator?

Validity is about answering the question: Are we measuring what we say we are measuring?

Do you know how valid your performance indicators are? Do you know how valid they need to be for the use you are going to put them to?

For example if I were to measure the height of 20 children aged from 6 months to 10 years by putting them on a set of bathroom scales my measure would not be valid. However if I only wanted to identify the short and taller children weighing them might be adequate for my purposes because there is a reasonable degree of relationship between weight and height in children. The measure might be valid enough for my purpose.

Degree of Control and Cause and Effect

In theory you need to have control over the outcome you are intending to achieve if you are to develop a performance indicator for the outcome.

The NCOSS publication "Performance Indicators for Community Organisations" in Chapter 2 The Rule" states "There are five basic rules which need to be remembered about performance indicators...3. Performance indicators can only be derived for things over which YOU have control."

However in practice people don't have absolute control over anything and so 'having control' is really a matter of whether there is enough control for your purpose.

For example most family support services would have as an objective for many of their clients an increase in the level of self-esteem for the parent(s). Even assuming that one could develop a valid and reliable measure for self-esteem it must be asked whether self-esteem would be a good performance indicator.

The difficulty arises because there are a multitude of causes that affect the clients self-esteem that do not relate to services provided by family support services. How can a counselling service show that there is a cause and effect relationship between the service provided and the changes in the client. For most services this would be almost impossible to prove with absolute certainty. Even with major research efforts proving cause and effect relationships can be very difficult if you have to convince people who hold a different view from yours. Look at the history of the debate about whether smoking causes cancer. What chance has a small community based organisation got to prove that its service delivery process is the cause of the change in the client, especially when the people needing to be convinced have a different value base or don't trust you?

This difficulty is intrinsic to all human service organisations. In human service organisations the service is intended to have an impact on the person receiving services. However the service is only one among many causes effecting the client and often the real outcome that is being sought will only be observed years or more after the service has been provided.

Equating Performance with Performance Indicators

Here is one example of what can happen when performance is equated with the performance indicator.

"The Newcastle Commission, 1861, represents an early attempt to use evaluation to change learning patterns. The commission had been set up in order to raise the standards of achievement in the basic subjects. It reached the following conclusion:

There is only on way of achieving this result, which is to institute a searching examination by competent authorities of every child in every school to which grants are to be paid, with a view of ascertaining whether these indispensable elements of knowledge are thoroughly acquired, and to make the prospects and position of the teacher dependent, to a considerable extent, on the results of this examination. (Newcastle Commission 1861)

"The report of this commission led to the institution in England of the 'payment by results' system which was to last forty years in formal terms but whose effects are still visible. The examination in the schools were conducted by inspectors, who tested in a very limited area of computation together with one problem. The grants were paid on the basis of the number of pupils attaining a pass mark. The consequences were disastrous: the already limited teaching became narrower still, concentrating on the rote learning of simple number facts and ignoring and applications. There was no encouragement for higher achievement and thus the main effort was spent on cramming the weaker children to get a pass mark and neglecting the needs of the average and more able pupils. The effects were thus quite reverse of the intentions, contributing to a very limited achievement coupled with a lasting distaste for learning."

The performance indicator was the pass mark. Schools were rewarded for an increase in the performance indicator. Teachers taught to the test. However education which was not the same as the test result suffered.

Inputs and Outcomes

Outcomes (and performance indicators) can be affected by inputs into a program. How clients are selected can change an organisations outcomes even if the processes of service delivery remain effectively the same.

For example in the New Enterprise Incentive Scheme Program unemployed people are provided with training and an allowances approximately equal to the unemployment benefit to help them set up new businesses. A performance indicator is the number that are still in business at the end of 12 months.

While the target group of this program includes migrants, single parents and long term unemployed so long as the performance indicator is used as a reward people will be accepted into the program on their ability to set up the business. One may ask would they have been able to set up the businesses anyway and may it have not been more useful to work with those on the margins so that people who might not otherwise set up a small business do so. In this latter case the performance indicator would be lower but the overall gain to society would be greater.

Conclusion

Performance indicators can be very useful tools in helping you answer the questions: How do you know what you are achieving and how can your service improve its performance?

However if they are used simplistically the results can be disastrous.

Is Your Organisation ready for Performance Indicators?

1. Are your staff and management Committee all
   committed to providing quality services?
   

   [ ] 1. Yes [ ] 0. No

2. Do you have a written statement of the needs
   your organisation is wanting to meet and a
   statement of whose needs they are?
   

   [ ] 1. Yes [ ] 0. No

3. Do you have a written statement of your
   organisations values and philosophy that has
   been formally adopted by the organisation?
   

   [ ] 1. Yes [ ] 0. No

4. Do you have a written statement of aims (what
   you intend to achieve in relation to the needs
   you wish to meet)?
   

   [ ] 1. Yes [ ] 0. No

5. Do you have a written statement of objectives
   (specific, measurable, achievable, results
   oriented and time bound statements of what
   you intend to achieve in relation to the clients
   needs)?
   

   [ ] 1. Yes [ ] 0. No

6. Do you have a written statement of the key
   steps in the service delivery process and what
   you are trying to achieve in each step of the
   process?
   

   [ ] 1. Yes [ ] 0. No

7. Do you have a written statement of the key
   assumptions that you are making in your service
   delivery process; for example assumptions
   about the logic behind your service delivery
   process or assumptions about the strategies you
   are choosing to achieve a particular end?
   

   [ ] 1. Yes [ ] 0. No

8. Do you a systems for collecting statistics
   that everyone in the organisation uses both in
   terms of putting information into the system and
   getting useful information out of it?
   

   [ ] 1. Yes [ ] 0. No

9. Do the members of your organisation like asking
   questions about why they do things?
   

   [ ] 1. Yes [ ] 0. No

10. Do your staff and management committee members
    see mistakes as opportunities for improving your
    services?
    

    [ ] 1. Yes [ ] 0. No

SCORE

1-3 Your organisation cannot develop useful performance indicators; the ground work is not in place.

4-6 Don't give up yet but you have a lot of ground work to do before getting onto performance indicators.

7-9 With a little more ground work you might be in a position where you could develop performance indicators.

10 Your organisation probably has a sound basis for developing performance indicators

Will Your Organisation use Performance Indicators Wisely?

1. Has your Board or Committee and staff discussed and agreed on answers to:
   • What is a quality service?
   • How do we get a quality service?
   • How do we know we have a quality service?
     

     [ ] 1. Yes [ ] 0. No

2. Will performance indicators be used as the judge of your organisation's success or failure?
   

   [ ] 1. Yes [ ] 0. No

3. Will performance indicators be used as the basis for rewards to staff?
   

   [ ] 1. Yes [ ] 0. No

4. Will performance indicators be used as clues to asking questions about performance and its improvement?
   

   [ ] 1. Yes [ ] 0. No

5. Will you be using performance indicators for each outcome in each key step in your service delivery process?
   

   [ ] 1. Yes [ ] 0. No

6. Will staff, client and committee perceptions be as important as performance indicators?
   

   [ ] 1. Yes [ ] 0. No

7. Will your organisation be rewarded by the funding body for improving performance indicators?
   

   [ ] 1. Yes [ ] 0. No

8. Do you know how reliable your performance indicators are?
   

   [ ] 1. Yes [ ] 0. No

9. Do you know how valid your performance indicators are?
   

   [ ] 1. Yes [ ] 0. No

10. Could you convince a reasonable person that what you do causes what you are trying to achieve? (Why?)
    

    [ ] 1. Yes [ ] 0. No

SCORE

1-3 Don't use performance indicators you will only make things worse.

4-6 don't use performance indicators unless you come to a much better understanding of the issues?

7-9 Don't use performance indicators yet, but don't give up hope, there is a chance you might use performance indicators wisely if you improve your understanding of them a little more.

10 There is a good chance you might use performance indicators wisely in your organisation.

References and Further Reading

Donovan, F and Alun C. Managing Human Service Organisations. Prentice Hall. Sydney, 1991

Mayo, Toni. Performance Indicators for Community Organisations, NCOSS, 1990

Gilbert, Roy. Reglomania, The Curse of Organisational Reform and how to Cure it. Prentice Hall. Sydney, 1991

Hughes, Phillip. Two Roles of Evaluation: Compatible or Conflicting? Evaluation Journal of Australasia Volume 2, Number 3, 1990.

The Article

Paul Bullen explores the issues in four articles in this issue of Caring. The first article is a provocative Editorial looking at the politics of performance indicators. The second article answers the question What Are They? The third article Putting Them in Their Place locates them as one small element in the process of planning and developing quality services. The fourth article, The Fine Print explores some of the complexities of using performance indicators. There is also two self rating Questionnaires to give you clues on whether your organisation is ready for performance indicators and whether or not you will use them wisely.

The four articles are a reflection on practice. They arise out of consultancy experiences within NSW over the past three years. They are written specifically in relation to human service organisations within NSW, particularly those organisations that are providing services to change the clients themselves.