Contents | 1. Introduction | 2. Steps | 3. Practicalities | 4. Examples | 5. Client questionnaires
6. Standardized tools | 7. On-line surveys | 8. Checklists | 9. Resources

3. Practicalities

Some of the recurring practical issues are:

1. How long should the questionnaire be? What's the right length?

The length of a questionnaire will be determined by many things including:

• its purpose
• the questions to be included to achieve the purpose
• the motivation of the respondents to answer it
• the ability of the respondents to complete it
• the design and layout for ease of use
• the resources available for collation and analysis

For example, a questionnaire to staff about working conditions could include 100 questions and be seen as 'short' if staff agree with the purpose, are motivated and the questionnaire is well designed for ease of use. A one page 5 question survey asking open-ended questions to human service clients may be seen to be too long because of the barrier of writing open-ended answers; whereas a 20 question tick the box survey may be seen by the same clients as short because it is easier to do.

There is no one right answer to the question: What is the right length? The answer will be affected by all the points above.

2. Are my questions good questions?

Many people drafting questionnaires worry (rightly) about the wording of the questions. There are many books on surveys and questionnaires. They often include chapters or sections on composing questions.

A most useful book is The Survey Research Handbook by Pamela L Alreck and Robert B. Settle Irwin McGraw Hill. It includes Chapter 4 Composing questions. Here are seven of their many examples of questions showing 'right' and 'wrong' question composition.

Focus of the question
Wrong: When do you usually go to work?
Right: What time do you ordinarily leave home for work?

Brevity of the question
Wrong: When was the last time you went to the doctor for a physical examination on your own or because you had to?
Right: How many months ago was your last physical examination?

The use of core vocabulary
Wrong: What emotions were evoked by perceiving the spectacle?
Right: What kinds of feelings did you have when you saw it?

The use of over demanding recall
Wrong: How many times did you go out on a date with your spouse before you were married?
Right: How many months were you dating your spouse before you were married?

The use of double barreled questions
Wrong: Do you regularly take vitamins to avoid getting sick
Right: Do you regularly take vitamins? Why or why not?

The use of loaded questions
Wrong: Do you advocate a lower speed limit to save human lives?
Right: Does traffic safety require a lower speed limit?

The use of leading questions
Wrong: Don't you see some danger in the new policy?
Right: Do you see any danger in the new policy?

3. How many completed questionnaires do I need to have an adequate response rate?

Sampling and response rates

Compare the following two scenarios.

Scenario A: I send out 1000 questionnaires and get 200 back (20%) without any reminders.

Scenario B: I take a random sample of 150 of the 1000 people who could be surveyed, send out 150 questionnaires and get 105 back (70%) by using up to three follow up reminders.

Which scenario will provide the more adequate data? For most purposes Scenario B. Scenario A will not usually provide adequate information in human services.

In human services there is considerable research to show that those who complete questionnaires are different to those who do not complete questionnaires/or are late in responding and so it is especially important to have a high response rate (more than two thirds). In human services this is a very achievable goal with surveys to clients, staff, management committee members and other stakeholders.

It is often preferable to survey a sample of clients and do several reminders in order to get a high response rate rather than send out larger numbers of questionnaires with no follow up and achieve a lower response rate.

What kinds of analysis

A further issue to consider is the type of analysis that will be required. For example if one wanted to compare differences between male and female clients and differences between younger and older clients one would need a larger sample than if one only wanted to describe the group of clients (without sub-groups).

It is very important to know what kinds of analyses will be required before the questionnaire is sent out so that the sample size can be made sufficiently large to actually do the analysis required. There are many stories of staff starting to analyse questionnaire data only to realize that the sample is not big enough to do the analysis they had been hoping for. Sometimes technical statistical advice will be required to work out an adequate sample size.

4. How do I guarantee that completed questionnaires can be anonymous and also know who has completed questionnaires so I can follow up those who have not returned questionnaires on time.

When you mail out the questionnaires include a postcard as well as the questionnaire. The questionnaire will be anonymous and be returned to one postal address. The postcard will include the name of the person completing the questionnaire and be returned to a different postal address. The postcard will say "I my name have completed my questionnaire and returned it". The returned postcards can then be used to identify who has completed the questionnaire but the researcher will not know who completed any particular questionnaire.

This approach ensures that reminders are not sent to people who have already completed questionnaires.

5. How long would a mail survey of clients typically take.

Once the questionnaire is designed the process of getting the questionnaires out and back (by mail) will still take six to 8 weeks if three reminder notices are allowed for.

One needs to mail out the questionnaire and set a deadline for completion about 10 days after the mailout. Within a few days of the deadline passing a reminder needs to be sent to those who have not returned a completed questionnaire indicating that the deadline has been extended to a day about 10 days after the reminder mailout. Within a few days of the second deadline a further reminder needs to be send out with a another new deadline.

Once all the completed questionnaires are returned the collation and analysis can be done within a few days.