Some of the recurring practical issues are:
1. How long should the questionnaire be? What's the right length?
The length of a questionnaire will be determined by many things including:
For example, a questionnaire to staff about working conditions could include 100 questions and be seen as 'short' if staff agree with the purpose, are motivated and the questionnaire is well designed for ease of use. A one page 5 question survey asking open-ended questions to human service clients may be seen to be too long because of the barrier of writing open-ended answers; whereas a 20 question tick the box survey may be seen by the same clients as short because it is easier to do.
There is no one right answer to the question: What is the right length? The answer will be affected by all the points above.
2. Are my questions good questions?
Many people drafting questionnaires worry (rightly) about the wording of the questions. There are many books on surveys and questionnaires. They often include chapters or sections on composing questions.
A most useful book is The Survey Research Handbook by Pamela L Alreck and Robert B. Settle Irwin McGraw Hill. It includes Chapter 4 Composing questions. Here are seven of their many examples of questions showing 'right' and 'wrong' question composition.
Focus of the question
Brevity of the question
The use of core vocabulary
use of over demanding recall
The use of double barreled questions
The use of loaded questions
use of leading questions
3. How many completed questionnaires do I need to have an adequate response rate?
Sampling and response rates
Compare the following two scenarios.
Scenario A: I send out 1000 questionnaires and get 200 back (20%) without any reminders.
Scenario B: I take a random sample of 150 of the 1000 people who could be surveyed, send out 150 questionnaires and get 105 back (70%) by using up to three follow up reminders.
Which scenario will provide the more adequate data? For most purposes Scenario B. Scenario A will not usually provide adequate information in human services.
In human services there is considerable research to show that those who complete questionnaires are different to those who do not complete questionnaires/or are late in responding and so it is especially important to have a high response rate (more than two thirds). In human services this is a very achievable goal with surveys to clients, staff, management committee members and other stakeholders.
It is often preferable to survey a sample of clients and do several reminders in order to get a high response rate rather than send out larger numbers of questionnaires with no follow up and achieve a lower response rate.
What kinds of analysis
A further issue to consider is the type of analysis that will be required. For example if one wanted to compare differences between male and female clients and differences between younger and older clients one would need a larger sample than if one only wanted to describe the group of clients (without sub-groups).
It is very important to know what kinds of analyses will be required before the questionnaire is sent out so that the sample size can be made sufficiently large to actually do the analysis required. There are many stories of staff starting to analyse questionnaire data only to realize that the sample is not big enough to do the analysis they had been hoping for. Sometimes technical statistical advice will be required to work out an adequate sample size.
4. How do I guarantee that completed questionnaires can be anonymous and also know who has completed questionnaires so I can follow up those who have not returned questionnaires on time.
When you mail out the questionnaires include a postcard as well as the questionnaire. The questionnaire will be anonymous and be returned to one postal address. The postcard will include the name of the person completing the questionnaire and be returned to a different postal address. The postcard will say "I my name have completed my questionnaire and returned it". The returned postcards can then be used to identify who has completed the questionnaire but the researcher will not know who completed any particular questionnaire.
This approach ensures that reminders are not sent to people who have already completed questionnaires.
5. How long would a mail survey of clients typically take.
Once the questionnaire is designed the process of getting the questionnaires out and back (by mail) will still take six to 8 weeks if three reminder notices are allowed for.
One needs to mail out the questionnaire and set a deadline for completion about 10 days after the mailout. Within a few days of the deadline passing a reminder needs to be sent to those who have not returned a completed questionnaire indicating that the deadline has been extended to a day about 10 days after the reminder mailout. Within a few days of the second deadline a further reminder needs to be send out with a another new deadline.
Once all the completed questionnaires are returned the collation and analysis can be done within a few days.