Questionnaires - Steps, Paul Bullen, Management Alternatives Pty Ltd

Questionnaires can vary from very simple to very complex. Some questionnaires are intended to be given to 30 people. Others are intended to be given to thousands of people. Some questionnaires are short. Some are long. So the steps in the process of developing and using questionnaires can vary from one questionnaire to the next.

Below is a thumbnail sketch of a process for developing and using a questionnaire. It reads like a linear process beginning at step 1 and going to step 14 however it is really a circular iterative process.

For example the purpose affects the questions to be asked. The analysis you want to do also affects the choice of the questions to ask. The analysis you want to do also affects the sampling.

It is essential in developing a quality questionnaire to have conceptually worked through all the steps in the list below and come back to the beginning and gone though a second time so that it is possible to see the implications for each step on all the other steps. For example if you want to analyse the data for different geographic areas do you have a sample of respondents big enough for each geographic area to be able to show differences between geographic areas (if there were any).

What is the purpose of the questionnaire?
For what purpose will the collated and analyzed results be used?

For example: To find out what out clients think about our services so we can improve them.

2. Who are the users? What will the users find useful?

Who will use the results from the questionnaire?
What will these users find useful?

For example: Our senior managers will use the results and will want to get pointers on how to improve the services

3. What broad questions are we hoping to answer through the questionnaire?

Given the purpose of the questionnaire what questions are we hoping to answer through the questionnaire. Initially these may be broad questions such as:

Who are our clients? What do they think of our services? How can our services be improved?

4. What specific questions do we need to include in the questionnaire?

Given the purpose of the questionnaire and the broad questions we hoping to answer through the questionnaire what are the specific questions to include.

Initially these may be a brainstormed list. They will be refined as the process proceeds.

For example:

5. Who will be completing the questionnaire? What do they think about the issues and questions

Who is the target group for the questionnaire? What does the target group think about the issues and questions being explored in the questionnaire?

It will be useful to run a focus group with the target group prior to drafting the questionnaire. What issues would they explore to achieve the purpose? What questions would they ask? How would they ask them?

How many questionnaires will you send out? If you are going to choose a sample, how will you choose the sample? See practicalities for more details.

Develop a first draft and get stakeholders and users to comment on it. In developing the draft you will need to:

Pilot it. Piloting is different from getting people to comment on the questionnaire.

Ask some people in the target group to complete the questionnaire. It is often in the process of trying to complete the questionnaire that difficulties emerge that were not picked up when people comment on the questionnaire.

Do cognitive testing - ask some people to complete the questionnaire in a face to face interview where they think out aloud and tell you their responses to the questions as they attempt to answer them.

For example to the question: How many times have you been to the doctor in the past three months, someone may say to themselves: "I don't remember...it is probably between 2 and five times, I'll write four - probably close enough - I did go more often but maybe that was more than three months ago." The conclusion you may reach is that this is not a good question because people are not able to remember with sufficient accuracy.

Pilot the collation and analysis. Check that you can actually collate and analyse the data in the ways you are hoping to. Many questionnaires are used that cannot be appropriately analysed because of poor design that was not identified in the process of piloting the collation and analysis.

If you are using a method for getting the questionnaires out and back that you have not used before - then pilot that method as well to make sure it works in the way you are hoping.

Revise the questionnaire in the light of piloting. If there have been significant changes - re-pilot it.

Use an appropriate method for getting the questionnaires out and back to the target group. This is easy for a 'captive group' such as group participants. It is more difficult for participants such as a random sample of people living in the community. In this situation there may need to be several reminders before an acceptable response rate is achieved. See practicalities for more details.

Collate the completed questionnaires. For very small numbers of questionnaires (eg less than 15) this could be done manually. Otherwise it should be done using appropriate computer packages. See analysis for details.

Analysis of anything but the simplest of questionnaires will probably need to be done by computer. Basic collation and cross-tabulation can be done by spreadsheets such as Excel and Quatro Pro however more complex analysis and graphing will require specialised statistical packages, for example Statistica. For a range of statistical products try Hearne Marketing (and choose to browse statistics). Trial packages can be downloaded from these sites.

The report needs to be more than the collated results. It needs to answer the question: What have we learnt? What do we need to do about it?

Use the information for the original purpose (and other purposes that may have emerged during the process).