7. What are some of the paradoxes and dilemmas in practice? How do we respond?

In practice there are numerous paradoxes and dilemmas in measuring outcomes in family support. So many, in fact, that we may be tempted to do nothing.

Some of the paradoxes and dilemmas are:

Outcomes

1. The outcomes to be achieved may not be known in advance.

Often clarity about the outcomes to be achieved only emerges during the course of service delivery. This means that we may not have taken the right measurements at the beginning of the service process to know whether there has been any change during the course of service delivery.

2. The outcomes to be achieved may be difficult to precisely define

In family support many of the outcomes are inherently difficult to precisely define and so they are also inherently difficult to measure, for example: good parenting, self esteem, agency, friendship networks, family networks and reduced risk of child abuse are all difficult to precisely define and measure.

3. The specific outcomes we are wanting to achieve may differ depending on the values we hold.

Where there is not consensus among all stakeholders about the values underpinning the service provision there will not be agreement on the specific outcomes we are looking for. For example, we may all agree that good parenting is an outcome to be achieved. But there are many views of good parenting based on different values. These different values can lead to different judgments about the specific outcomes to be achieved.

4. People involved with the service processes may have different views about what outcomes have been achieved and the extent to which they have been achieved.

If we ask the family worker, the client and another family member about the outcomes achieved they may say quite different things. It is not uncommon for a client to say that a lot has been achieved and the worker to be concerned at the lack of real progress and a family member to have an alternative view.

The nature of the service process - from a client viewpoint

5. There are different views of who the client is.

The client could be:

an adult family member
the child
the immediate family
the extended family.

This means that when we are talking outcomes for clients we can be talking about very different outcomes. And the data to be collected to measure outcomes can be very different depending on how the client is identified.

6. The steps in the service process can differ from one service to another.

There are often differences between services about the nature of the service process. This means it is often difficult for several services to use the same tools. For example one service may use an assessment process where the worker and the client need to agree on what is written on the assessment form. Another service provider may use a separate form for each of the client and the worker.

7. People in the service processes may want different things.

For example the client may want to separate from her partner. The partner may want to remain together with the client. The children may want something different.

8. People in the processes make choices about their commitment and participation

People may not want to continue to be involved with the service process. How is a client withdrawing to be interpreted? It could be viewed as inappropriate service delivery because it could be seen that the service has failed to keep the client engaged; or it could be viewed as the client exercising their free choice.

These different views lead to different descriptions of outcomes achieved.

9. People in the service process are part of families, friends, neighbourhoods, work teams, communities

This means that it is difficult to show cause and effect relationships because the service is not the only cause at work in the life of the client. If the client is achieving outcomes is this because of the service? Because of support from family members? Support from neighbours? How could we know what the service’s contribution is?

10. Services are provided short term. We are hoping for long term outcomes.

This means that the clients may need to have left the service for many years before we can know whether or not the service has had a real positive impact. But we have to make the decisions about service delivery now.

Extensive longitudinal research can help with this issue, but never solve it because society is changing, and so our knowledge about society is always, to some extent, out of date.

The nature of the service process - from a service viewpoint

11. The family support service is part of a service network.

This means that it is difficult to show cause and effect relationships between what this service has done and the outcomes achieved because the outcomes may have been achieved through the service provision of the entire network.

12. Service staff are often focused on individual clients and do not have the time or expertise to build a picture of all their clients as a group

Most family support staff are trained to work with clients. Their work is client focused and the day to day demands are client focused. Their day to day work is interrupted with client demands.

These realities often make it difficult for agencies to build pictures of their clients as a group because this task requires different skills (research and data analysis) and different time pressures (eg staff being able to work for longer periods uninterrupted).

Measurement

13. There are different approaches to how to measure outcomes.

For example will we measure outcomes according to:

An agreed behavioural standard (eg. this is a skill required by all parents)
The relative change in the client (and their situation) (eg. the client's housing situation has improved in...)
What we had intended to achieve (eg. we achieved what we said we would).

These approach lead to the development of different kinds of tools, for example:

Standardised assessment tools (which can show us how the clients’s responses compare with the population at large)
Non - standardised questionnaires (which can show us relative change in this client - but not comparisons with the population at large)
Goal attainment scaling (which can show us the extent to which we achieved the goals we set out to achieve, but not measure the unintended consequences).

In addition, each of these three approaches to measuring outcomes can be developed using different methods of measuring, eg, independent observation and judgment, subjective observation and judgment or subjective judgement.

These various approaches lead to the development of very different tools.

Mixing good service delivery and good measurement

14. There are conflicting demands between good service delivery practice and good measurement and data collection practice.

For example in family support the service process typically starts with building rapport and works from a basis of building on strengths. To present a long detailed questionnaire to the client at the beginning of the service process may be intrusive and so reduce rapport; if it is comprehensive it is likely to highlight deficits as well as strengths and the client may feel they have less strengths to build on than they had previously felt.

15. There are conflicting demands between resource use for direct services and resource use for measurement and data collection.

For example should a service use staff time in doing detailed data collection and analysis when the time could alternatively be spent on direct service delivery.

16. There are conflicting demands between client confidentiality and data collection

Clients have a right to confidentiality and services should not provide information about clients to other services without their permission. However, because clients use a variety of services and services are part of service networks data is ideally needed about all the services a client uses.

Ideally what is required is client focused databases that can gather data on all services provided, rather than service focused databases.

However services would typically see it an invasion of client privacy if they asked their client for permission to contact all the other service providers for relevant service provision data, especially if all services asked every client. But to paint a picture of what services are doing with the client would require all this data.

17. Family support services work with clients in many languages. Measurement tools are often only produced in one or two languages.

This means that the tools may not be available in the language needed. Tools are also often culturally biased. Tools and documentation are often only available in English because of the costs associated with doing translations and assessing cultural bias.

18. Family support services work with clients with low literacy levels, physical and intellectual disabilities and mental health issues. Many measurement tools are often not suited for use with these people

This means that the client may not be able to compete a questionnaire at all or without support. Providing support may be a conflict of interest if the support is provided by a family worker. For example the tool may be asking the client to give views on the family worker’s work.

Providing independent support may cost too much. For example, employing an independent person to work with the client to complete the tool or survey.

The nature of being human

19. There are characteristics of being human that mean that we don't always know that we think we know.

For example:

• We interpret ambiguous information in ways that fit our preconceptions.
• We are more critical of information that contradicts what we already believe than we are of information that supports our existing beliefs
• We don't always tell the truth
• We tend to see what we do in a positive light
• We are social beings and can create groups or sub-cultures such that everyone in the group reinforces each others views
• We can create self-fulfilling prophesies.

Paradoxes and dilemmas and the four key steps

What do these paradoxes and dilemmas mean for our four key steps in measuring outcomes

A. We may not agree on the specific outcomes to be achieved.

B. It is difficult to develop valid and reliable measurement tools for these outcomes.

C. Using the tools to measure change in clients over time may conflict with direct service practice.

D. Showing a cause and effect connection between the service provision and the outcomes is exceedingly difficult.

How do we respond?

Our attitudes

These paradoxes and dilemmas create enormous difficulties in measuring outcomes in family support. We must chose between two responses:

• The difficulties are insurmountable. There is no point in measuring outcomes in family support. We would be wasting our time - lets keep our energies for direct service.
• The difficulties are many and the results will be imperfect but we do need to do something to deal with the potential self-delusion - thinking we are doing a good job when we are not. The same reasons that make measuring outcomes so difficult also create room for self-delusion. We want to minimize the room for self-delusion and so we take what steps we can.

Our practice

If our attitude is one of: there are difficulties but we need to do something to minimise the potential self-delusion there is a role for each of:

• Clients and their families
• Family workers
• Family Support Services
• Peak organisations such as NSW Family Services
• Government
• Other organisations and agencies.

It is only when all parts of the human service system are working together that we will be able to make a reasonable attempt at measuring outcomes in family support. It is not just an issue for individual workers or services.

It will require resources.

It will require we understand the nature of the human service and community development processes and the inherent paradoxes and dilemmas.

It will have an impact on how each of us works.