|
4.7 Gathering Survey Data
 Once
a questionnaire is developed and one or more pretests or pilot
studies have been conducted, the next step is to gather data
from an appropriate group of respondents.
There are four basic methods for doing this: the mail survey,
the telephone survey, the personal interview, and group
administration. Researchers can also use variations and
combinations of these four methods, such as disk-by-mail surveys
and mall interviews. Each procedure has definite advantages and
disadvantages that must be considered before a choice is made.
The remainder of this chapter highlights the characteristics of
each method.
4.7.1 Mail Surveys
Mail surveys involve mailing self-administrable questionnaires
to a sample of individuals. Stamped reply envelopes are enclosed
to encourage respondents to mail completed questionnaires back
to the researcher. Mail surveys are popular because they can
secure a great deal of data with a minimum expenditure of time
and money. At the outset, however, researchers should be aware
that respondents are busy people with many demands on their
time. Consequently, many people do not share the researcher's
enthusiasm for questionnaires and often simply throw them away.
The general stages of a mail survey are discussed below.
Even though the steps are listed in numerical sequence, many of
these tasks are often accomplished in a different order or even
simultaneously.
1. Select a sample:
Sampling is generally done from a prepared frame (Chapter 4)
that contains the names and addresses of potential respondents.
The most common sampling frame used is the mailing list, a
compilation of names and addresses in narrowly defined groupings
that commercial firms sometimes prepare (see accompanying
boxed material).
2. Construct the questionnaire:
As discussed earlier, mail survey
questionnaires must be concise and specific, since no
interviewer is present to alleviate misunderstandings, answer
questions, or give directions.
3. Write a cover letter:
A brief note explaining the purpose and
importance of the questionnaire usually increases response
rates.
4. Assemble the package:
The questionnaires, cover letters, and return envelopes are
stuffed into mailing envelopes. Researchers sometimes choose
to use bulk mail with first-class return envelopes. An
alternate method is to send questionnaires first class and use
business reply envelopes for responses. This method allows
researchers to pay postage only for the questionnaires actually
returned. Postal options always depend on the research budget.
5. Mail the surveys.
6. Closely monitor the return rates.
7. Send follow-up mailings:
The first follow-up should be sent 2 weeks after the initial
mailing, and a second (if necessary) 2 weeks after the first.
The follow-up letters can be sent to
the entire sample or only the subjects who failed to answer.
8. Tabulate and analyze the data.
A) Advantages
 Mail
surveys cover a wide geographic area for a rather reasonable
cost. They are often the only way to gather information from
people who live in hard-to-reach areas of the country (or in
other countries). Mail surveys also allow for selective sampling
through the use of specialized mailing lists.
In addition to those mentioned, lists are available that
include only people with annual incomes exceeding $50,000,
or consumers who have bought a car within the past year, or
subscribers to a particular magazine, or residents of a specific
zip code area. If researchers need to collect information from a
highly specialized audience, the mail technique can be quite
attractive.
Another advantage of the mail survey is that it provides
anonymity, so that subjects are more likely to answer sensitive
questions candidly.
Questionnaires can be completed at home or in the office,
affording subjects a certain sense of privacy. People can answer
questions at their own pace and have an opportunity to look up
facts or check past information. Mail surveys also eliminate
interviewer bias, since there is no personal contact.
Probably the biggest advantage of this method, however, is its
relatively low cost. Mail surveys do not require a large staff
of trained workers.
The only costs are for printing, mailing lists, envelopes, and
postage. If the cost per completed questionnaire were to be
computed, it is likely that the mail survey would prove to be
the most inexpensive of all the survey methods. At a minimum, it
can be said that researchers who are willing to spend time,
energy, and money in a mail survey can usually ensure an
above-average return rate.
B) Disadvantages
 First,
mail questionnaires must be self-explanatory.
There is no interviewer present to answer questions or to clear
up misunderstandings. Mail surveys are also the slowest form
of data collection. Returns start to trickle in around a
week or so after the initial mailing and continue to arrive for
several weeks thereafter. In fact, it may be months before some
responses are returned. Many researchers simply set a cutoff
date, after which returns are not included in the analysis.
Another problem with mail surveys is that researchers never know
exactly who answers the questions.
A survey sent to corporate executives, for example, may be
completed by assistants. Furthermore, replies are often
received only from people who are interested in the survey, and
this injects bias into the results. Most researchers agree,
however, that the biggest disadvantage of the mail survey is the
typically low return rate. A typical survey (depending on the
area and type of survey) will achieve a response rate of 20% -
40%. This low return casts doubt on the reliability of the
findings.
C) Increasing Response Rates
 A
number of procedures for improving return rates have been
investigated by survey researchers. There are no hard and fast
guarantees, however, in a meta-analysis (the findings of
several studies are treated as independent observations and
combined to calculate an overall or average effect) of numerous
studies concerning mail surveys. Previous studies have shown
that on the average, response rates can be increased in a
variety of ways. In descending order of importance. It was
also found that following procedures to increase mail survey
response rates: university sponsorship, stamped return postage
as opposed to business reply, written prenotification of the
survey sent to the respondent, postcard follow-up, first-class
outgoing postage, questionnaire color (green paper as opposed to
white), notification of cutoff date, and stamped outgoing
postage as compared to metered stamping.
Offering monetary incentives also increases
response rates, but the authors did not pursue this area since
only a few studies offering incentives were available to them.
The authors further suggest that additional research is required
to determine which combinations of the procedures, if any, can
have an interactive effect to increase response rates even more
than any single element does alone.
4.7.2 Telephone Surveys
 Telephone
surveys and personal interviews must employ trained members of a
research team to ask questions orally and record the responses.
The respondents generally do not get a chance to see the actual
questionnaire. Since telephone and personal interviewing
techniques have certain similarities, much of what follows
applies to personal interviews as well.
Telephone surveys seem to fill a middle ground between mail
surveys and personal interviews.
They offer more control and higher response rates than most
mail surveys but are limited in the types of questions that can
be used. They are generally more expensive than mail surveys but
less expensive than face-to-face interviews. Because of
these factors, telephone surveys seem to represent a compromise
between the other two techniques, and this may account for their
growing popularity in mass media research.
Interviewers are extremely important to both telephone and
personal surveys. An interviewer ideally should function as a
neutral medium through which the respondents' answers are
communicated to the researcher. The interviewer's presence and
manner of speaking should not influence respondents' answers in
any way. Adequate training and instruction can minimize bias
that the interviewer might inject into the data. For example, if
he or she shows disdain or shock over an answer, it is unlikely
that the respondent will continue to answer questions in a
totally honest manner. Showing agreement with certain responses
might prompt similar answers to other questions.
Skipping questions, carelessly asking questions,
and being impatient with the respondent might also cause
problems. To minimize interviewer bias,
the interviewers should follow the following recommendations:
-
Read the questions exactly
as worded. Ask
them in the exact order listed. Skip questions only when the
instructions on the questionnaire tell you to. There are no
exceptions to this.
-
Never suggest an answer,
try to explain a question, or imply what kind of reply is
wanted. Don't prompt in any way.
-
If a question is not
understood, say, "Let me read it again," and repeat it
slowly and clearly.
If it is still not understood, report a "no answer."
-
Report answers and comments
exactly as given,
writing fully. If an answer seems vague or incomplete, probe
with neutral questions, such as, "Will you explain that?"
or, "How do you mean that?" Sometimes just waiting a bit
will tell the respondent you want more information.
-
Act interested, alert, and
appreciative of the respondent's cooperation.
But never comment on his or her replies. Never
express approval, disapproval, or surprise. Even an "Oh" can
cause a respondent to hesitate or refuse to answer further
questions. Never talk up or down to a respondent.
-
Follow all instructions
carefully,
whether you agree with them or not.
-
Thank each respondent.
Leave a good impression for the next interviewer.
A general procedure for conducting a telephone survey follows.
Again, the steps are presented in numerical order, but it is
possible to address many tasks simultaneously.
1. Select a sample.
Telephone surveys require researchers to specify clearly the
geographic area to be covered and to identify the type of
respondent to be interviewed in each household contacted.
Many surveys are restricted to people over 18, heads of
households, and so forth. The sampling procedure used depends on
the purpose of the study.
2. Construct the questionnaire.
Phone surveys require straightforward and uncomplicated
response options. Ranking a long list of items is especially
difficult over the telephone, and this task should be avoided.
In addition, the length of the survey should not exceed 10
minutes for nonprofessional interviewers. Longer interviews
require professionals who are capable of keeping people on the
telephone.
3. Prepare an interviewer instruction manual.
This document should cover the basic mechanics of the survey
(what numbers to call, when to call, how to record times, and so
on). It should also specify which
household member to interview and should provide general
guidelines on how to ask the questions and how to record the
responses.
4. Train the interviewers.
Interviewers need to practice going through the questionnaire to
become familiar with all the items, response options, and
instructions. It is best to train
interviewers in a group using interview simulations that allow
each person to practice asking questions. It is advisable to
pretest interviewers as well as the questionnaire.
5. Collect the data.
Data collection is most efficient when conducted from one
central location (assuming enough telephone lines are
available). Problems that develop are easier to remedy, and
important questions raised by one interviewer can easily be
communicated to the rest of the group. A central location also
makes it easier for researchers to check (validate) the
interviewers' work. The completion rate should also be monitored
during this stage.
6. Make necessary callbacks.
Additional calls (usually no more than two) should be made to
respondents whose lines were busy or who did not answer during
the first session. Callbacks done on a
different day or night tend to have a greater chance of success
in reaching someone willing to be interviewed.
When the first call produces a busy signal, the rule is to wait
one-half hour before calling again. If the first call produced a
"no answer," wait 2 to 3 hours before calling again, assuming it
will still be a reasonable hour to call. If evening calls
produce no answer, call during the following day.
In
addition, interviewers should keep track of the disposition or
status of their sample numbers. Figure
4.2 contains a sample disposition sheet.
Figure 4.2: Sample disposition sheet
|
Sample Telephone
Interview Disposition Sheet
Phone number _________________
Call #1 ___ #2 ___ #3 ___ #4 ___ #5
___
Date ___ Date ___ Date ___ Date ___
Date ___
Time ___ Time ___ Time ___ Time ___
Time ___
Code
1 Completed interview
2 Answering machine
3 Busy
4 No answer
5 Refusal
6 Appointment to call again
(when _________________)
7 Nonworking number (out of order,
disconnected, nonexistent)
8 Nonresidential number
9 Reached but respondent not
available (out of town, hospital, etc.)
10 Reached but not interviewed
(ineligible household, speech or physical problem,
age disqualification) |
7. Verify the results.
When all questionnaires have been completed, a small sub
sample of each interviewer's respondents should be called again
to check that the information they provided was accurately
recorded. Respondents should be told during the initial
survey that they may receive an additional call at a later date.
This tends to eliminate any confusion when subjects receive a
second call. A typical procedure is to ask the subject's first
name in the interview so that it can be used later. The
interviewer should ask, "Was James called a few days ago and
asked questions about television viewing?" The verification can
begin from there, and need consist of only two or three of the
original questions (preferably open-ended and sensitive
questions, since interviewers are most likely to omit these).
8. Tabulate the data.
Along with the normal data analysis, telephone researchers
generally compute a response rate: how many completed
interviews, how many refusals, how many no-answers, and how many
disconnects.
A) Advantages
The
cost of telephone surveys tends to be reasonable.
The sampling involves minimal expense, and there are no
elaborate transportation costs. Callbacks are simple and
economical. Wide Area Telephone Service (W\TS) enables
researchers to conduct telephone surveys on a nationwide basis
from any location.
Compared to mail surveys, telephone surveys can include more
detailed questions, and, as stated earlier, interviewers can
clarify misunderstandings that might arise during the
administration of the questionnaire.
The nonresponse rate of a telephone survey is generally low,
especially when multiple callbacks are employed.
In addition, phone surveys are much faster
than mail. A large staff of interviewers can collect the
data from the designated sample in a relatively short time.
In summary, phone surveys tend to be fast, easy, and relatively
inexpensive.
B) Disadvantages
First
of all, researchers must recognize that much of what is called
survey "research" by telephone is not research at all, but an
attempt to sell people something. Unfortunately, many companies
disguise their sales pitch as a "survey," and this has made
respondents suspicious and even prompts some to terminate an
interview before it has gotten started. Additionally,
visual questions are prohibited. A researcher cannot, for
example, hold up a picture of a product and ask if the
respondent remembers seeing it advertised. A potentially
severe problem is that not everyone in a community is listed in
the telephone directory, the most often used sampling frame.
Not everyone has a phone, and many people have unlisted phone
numbers; also, some numbers are listed incorrectly, and others
are too new to be listed. These problems would not be serious if
the people with no phones or unlisted numbers were just like
those listed in the phone book. Unfortunately,
researchers generally have no way of checking for such
similarities or differences, so it is possible that a sample
obtained from a telephone directory may be significantly
different from the population.
4.7.3 Personal Interviews
 Personal interviews usually involve inviting a respondent to a
field service location or research office (called a one-on-one
interview).
Sometimes interviews are conducted at a person's place of work
or at home. There are two basic types of interviews,
structured and unstructured. In a
structured interview, standardized questions are asked in
a predetermined order; relatively
little
freedom is given to interviewers. In an
unstructured interview, broad questions are asked, which allows
interviewers freedom in determining what
further questions to ask to obtain the required information.
Structured interviews are easy to tabulate and analyze but do
not achieve the depth or expanse of unstructured interviews.
Conversely, the unstructured type elicits more detail but
takes a great deal of time to score and analyze.
The
steps in constructing a personal interview survey are similar to
those for a telephone survey. The list
below discusses instances in which the personal interview
differs substantially from the telephone method.
1. Select a sample.
Drawing a sample for a personal interview is essentially the
same as sample selection in any other research method.
In one-on-one interviews, respondents are
selected on the basis of a predetermined set of screening
requirements. In door-to-door interviews, a multistage sample is
used to first select a general area, then a block or
neighborhood, and finally randomly select a household from which
a person will be chosen.
2. Construct the questionnaire.
Personal interviews are flexible: detailed questions are
easy to ask, and the time taken to complete the survey can be
greatly extended (many personal interviews last 30-60 minutes).
Researchers can also make use of visual exhibits, lists, and
photographs to ask questions, and respondents can be asked to
sort photos or materials into categories, or to point to their
answers on printed cards. Respondents
can have privacy and anonymity by marking ballots, which can
then be slipped into envelopes and sealed.
3. Prepare an interviewer instruction guide.
The detail of an instruction guide depends on the
type of interview. One-on-one interviewer guides are not very
detailed because there is only one location, respondents are
prerecruited by a field service, and times are arranged.
Door-to-door interviewer guides contain information about the
household to select, the respondent to select, and what to do in
the event the target respondent is not at home.
Interviewer guides often contain information about how to
conduct the interview, how to dress, how to record data, and how
questions should be asked.
4. Train the interviewers.
Training is important because the questionnaires are longer
and more detailed. Interviewers should receive instruction on
establishing a rapport with subjects, administrative details
(when to conduct the interviews, how long each will take, and
how much the interviewers will be paid), and follow-up
questions. Several practice sessions
are necessary to ensure that the goal of the project is met and
that interviewers follow the established guidelines.
5. Collect the data.
Personal interviews are both labor and cost intensive.
These problems are why most researchers prefer to use telephone
or mail surveys. A personal interview project can take several
days to several weeks to complete because turnaround is slow.
One interviewer can only complete a handful of surveys each day.
In addition, costs for salaries and expenses escalate quickly.
It is not uncommon for some research companies to charge as much
as $1,000 per respondent in a one-on-one situation.
Data gathering is accomplished by either writing down answers or
by audio taping or videotaping the respondents' answers. Both
methods are slow and detailed transcriptions and editing are
often necessary.
6. Make necessary callbacks.
Each callback requires an interviewer
to return to a household originally selected or the location
used for the original interview. Additional salary, expenses,
and time are required.
7. Verify the results.
As with telephone surveys, a sub sample of each interviewer's
completed questionnaires is selected for verification.
Respondents can be called on the phone or
re-interviewed in person.
8. Tabulate the data.
Data tabulation procedures for personal interviews are
essentially the same as with any other research method.
A codebook must be designed, questionnaires
are coded, and data input into a computer.
A) Advantages
Many
of the advantages of the personal interview technique have
already been mentioned. It is the most flexible means of
obtaining information, since the face-to-face situation lends
itself easily to questioning in greater depth and detail.
Furthermore, some information can be observed by the
interviewer during the interview without adding to the length of
the questionnaire. Additionally,
the interviewers can develop a rapport with the respondents and
may be able to get replies to sensitive questions that would
remain unanswered in a mail or phone survey.
The identity of the respondent is known or can be controlled in
the personal interview survey.
Whereas in a mail survey it is possible that all members of a
family might confer on an answer, in a face-to-face interview,
this can usually be avoided. Finally, once an interview has
begun, it is harder for respondents to terminate the interview
before all the questions have been asked.
In a phone survey, all the subject needs to do is
to hang up.
B) Disadvantages
As
mentioned, time and costs are the major drawbacks to the
personal interview technique.
Another major disadvantage is the problem of interviewer
bias. The physical appearance, age, race, sex, dress,
nonverbal behavior, and/or comments of the interviewer may
prompt respondents to answer questions untruthfully. Moreover,
the organization necessary for recruiting, training, and
administering a field staff of interviewers is much greater than
that required for other data collection procedures. If large
numbers of interviewers are needed, it is usually necessary to
employ field supervisors to coordinate their work, which in turn
will make the survey even more expensive. Finally, if personal
interviews are conducted during the day, most of the respondents
will not be employed outside the home. If it is desirable to
interview respondents with jobs outside the home, it is
necessary to schedule interviews on the weekends or during the
evening. A hybrid of personal
interviewing is intensive or in-depth interviewing.
4.7.4 Mail Interviews
 Although mail interviews are essentially a form of personal
interview as just discussed, their recent popularity and
widespread use warrant individual consideration.
During the late 1980s, mall intercepts became one of the most
popular research approaches among marketing and consumer
researchers. Studies found that of all people who participated
in a survey in 1984, 33% were mall intercepts.
Although mall intercepts use convenience samples and sampling
error cannot be determined, the method has become the standard
for many researchers.
It is rare to go into a shopping mall without seeing a man or
woman with a clipboard trying to interview a shopper. The method
has become commonplace, and some shoppers resent the intrusion.
In fact, it is common for shoppers to take paths to avoid the
interviewers they can so easily detect.
By
the way, purposely avoiding an interviewer isn't necessary.
There is another way out if you don't wish to take the time for
the interview. Remember from previous discussions that all
research requires specific types of people — a screener is
developed to eliminate respondents who do not qualify.
Nearly every questionnaire has security screening questions to
eliminate respondents who work for a company in any way related
to the company sponsoring the study, or anyone who works for a
marketing research firm. The last part of the security screener
is your way out. When the interviewer stops you, simply say, "I
work for a marketing research company." Your chances of being
recruited are very slim. We're not advocating the practice of
lying here, just offering a suggestion. Mall interviewers are
generally nice people. It's easier for them to hear the security
bail-out than a caustic remark about their presence in the mall.
The procedures involved in conducting mail intercepts are the
same as those for personal interviews. The only major difference
is that it is necessary to locate the field service that
conducts research in the particular mall of interest. Field
services pay license fees to mall owners to allow them to
conduct research on the premises. Not just any field service can
conduct research in any mall.
A) Advantages
 Mail
intercepts are a quick and inexpensive way to collect personal
interview data.
B) Disadvantages
S ome
of the major problems are: convenience sampling restricts the
generalizability of the results, the length of interviews must
be short; and there is no control over data collection
(researchers are at the mercy of the field
service to conduct a proper job).
Disk-By-Mail Surveys
 During the late 1980s, a high-tech form of mail surveys has
been used that appears to offer promise in the future. The
procedure is called disk-by-mail surveys, or DBM. The name
of the survey approach essentially explains the procedure:
respondents
are sent computer disks that contain a self-administered
questionnaire, and are asked to complete it by using a personal
computer. This method obviously involves several new areas to
consider when conducting a research project.
DBM surveys are essentially the same as a typical
self-administered mail survey. The normal steps involved in
problem definition, questionnaire design, and pretesting are
used. However, there are several unique
considerations researchers must address when using DBM.
 Type of Study
Most DBM surveys are conducted with professionals or other
business related samples. The reason is simple. Only about 20%
of
American households have personal computers. Sample selection
would be time-consuming and costly. However, computer ownership
will certainly increase in the future, and in-home DBM surveys
may become commonplace. For the time being, DBM surveys are
conducted with professionals who generally have access to
personal computers in their workplace.
Sample Selection
 Locating qualified respondents for DBM surveys is the same as
for any other research project, except that in addition to the
other
screener questions, there must be one
about the availability of a personal computer.
Computer Hardware
 A typical self-administered mail survey requires that the
respondent only have a writing instrument.
DBM surveys complicate the process in several ways. First of
all, computers can use one of several different operating
systems, or languages, which run the
computer
(Chapter 17). Fortunately, the systems used by IBM and Apple are
the most widely used. The problems with the two operating
systems can be solved by preparing two different DBM disks, or
by asking one of the groups of users to try and locate the other
type of computer to complete the survey.
A second problem with the DBM method is whether to use a color
or monochrome display to present the questionnaire.
Not all color monitors are equal, and the color appearance may
be drastically different from one monitor to another.
A monochrome display is best to avoid problems.
The type of disk drive is a third problem.
The screener must include questions about the type of drive (for
example, 5.25 or 3.5) so respondents receive the correct disk
format.
Another problem, and not necessarily the last, relates to
problems respondents may have with the computer disks. Disks are
fragile and may be damaged in the disk duplication process, in
shipment, or by the respondent. Replacement disks may have to be
sent to some respondents.
Support
 Because
computer problems may occur, or respondents may be unable to
complete the survey, most DBM surveys offer respondents a toll
free number to call for assistance. This adds further costs to
the project.
Reliability and Validity
 Significant questions are raised about these two areas in
relation to DBM surveys. Who actually completes the surveys? Are
responses
more or less accurate than those provided to interviewers or in
typical mail interviews? Does the novelty of the approach have
any effect on respondents?
As
mentioned earlier, DBM surveys are a totally new approach in
research. Not much is known about the procedure, but in all
likelihood, DBM surveys will be used more frequently in the
future.
4.7.5 Group Administration
Group administration combines the features of mail surveys and
personal interviews.
The group-administered survey takes place when a group of
respondents is gathered together (pre-recruited by a field
service) and given individual copies of a questionnaire, or
asked to participate in a group interview (a large focus group).
The session can take place in a natural setting, but is
usually held at a field service location or a hotel ballroom.
For example, respondents may be recruited to complete
questionnaires about radio or television stations, students in a
classroom may complete questionnaires about their newspaper
reading habits, or an audience may be asked to answer questions
after viewing a sneak preview of a new film.
The
interviewer in charge of the session may or may not read
questions to respondents. Reading questions aloud may help
respondents who have reading problems, but this is not always
necessary (it is possible to screen respondents for reading
and/or language skills). The best
approach is to have several interviewers present in the room so
individual problems can be resolved without disturbing the other
respondents.
Some group-administered sessions include audio and/or video
materials for respondents to analyze.
The session allows respondents to proceed at their own pace, and
in most cases, interviewers allow respondents to ask questions,
although this is not a requirement.
A) Advantages
The
group administration technique has certain advantages. In the
first place, a group-administered questionnaire can be longer
than the typical questionnaire used in a mail survey. Since
the respondents are usually assembled for the express purpose of
completing the questionnaire, the response rates are almost
always quite high. The opportunity for researchers to answer
questions and handle problems that might arise generally means
that fewer items are left blank or answered incorrectly.
B) Disadvantages
On
the negative side, if a group-administered survey leads to
the perception that the study is sanctioned by some authority,
suspicion or uneasiness on the part of respondents might result.
For example, if a group of teachers is brought together to fill
out a questionnaire, some might think that the survey has the
approval of the local school administration and that the results
will be made available to their superiors. Also, the group
environment makes it possible for interaction among the
respondents; this has the potential for making the situation
more difficult for the researcher to control. In addition,
not all surveys can use samples that
can be tested together in a group. Surveys often require
responses from a wide variety of people, and mixing respondents
together may bias the results.
Finally, group administration can be expensive. Costs
usually include recruiting fees, coop payments, hotel rental,
refreshments, and salaries for interviewers.
4.7.6 Achieving a Reasonable Response Rate
 No
matter what type of survey is conducted, it is virtually
impossible to get a 100% response rate. Researchers have more
control over the situation in some types of surveys (such as the
personal interview) and less in others (such as the mail
survey). But no matter what the situation, not all respondents
will be available for interviews and not all will cooperate.
Consequently, the researcher must try to achieve the highest
response rate possible under the circumstances.
What constitutes an acceptable response rate?
Obviously, the higher the response rate the better, since as
more respondents are sampled, it becomes less likely that
response bias is present. But is there a minimum rate
that should be achieved? Not everyone would agree on an
answer to this question, but there are some helpful data
available. Several studies have calculated the average response
rates for surveys of various kinds. A comparison with these
figures can at least tell a researcher if a given response rate
is above or below the norm. For example, Dillman (1978) noted
that response rates for face-to-face interviews have dropped
sharply in recent years. In the 1960s, the average rate was
80%-85%. More recently, the completion rates of general
population samples interviewed by the face-to-face technique is
about 60%-65%. Yu and Cooper (1983) studied the completion rates
reported in 93 social science journal articles from 1965 to
1981. They found the completion rate for personal interviews to
be 82% and for telephone surveys about 72%. Mail surveys had an
average completion rate of about 47%. (Note that many of the
personal interviews included in this study were done in the
1960s and early 1970s. This should be kept in mind when
comparing these figures to Dillman's data mentioned above.)
Regardless of how good the response rate, the researcher is
responsible for examining any possible biases in response
patterns. Were females more likely to respond than males? Older
respondents more likely than younger ones? Whites more likely
than nonwhites? A significant lack of response from a particular
group might weaken the strength of any inferences from the data
to the population under study. To be on the safe side, the
researcher should attempt to gather information from other
sources about the people who did not respond; by comparing such
additional data with those from respondents, it should be
possible to determine whether under representation introduced
any bias into the results.
Using common sense will help increase the response rate.
In phone surveys, respondents should be called when they are
likely to be at home and receptive to interviewing. Don't call
when people are likely to be eating or asleep. In a one-on-one
situation, the interviewer should be appropriately attired. In
addition, the researcher should spend time tracking down some of
the nonrespondents and asking them why they refused to be
interviewed or did not fill out the questionnaire. Responses
such as "The interviewer was insensitive and pushy," "The
questionnaire was delivered with postage due," and "The survey
sounded like a ploy to sell something" can be quite
illuminating.
Along with common sense, certain elements of the research design
can have a significant impact on response rates.
Yu and Cooper (1983) in their survey of 93
published studies discovered the following.
-
Monetary incentives
increased the response,
with larger incentives being the most effective. Nonmonetary
incentives (for example, ballpoint pens) were also helpful.
-
Preliminary notification,
personalization of the questionnaire, follow-up letter, and
assertive "foot-in-the-door" personal interview techniques
all significantly increased the response rate.
-
Things that were not
significantly related to an increased response rate were a
cover letter, assurance of anonymity, and stating a
deadline.
-
Stressing the social
utility of the study and appealing to the respondent to help
out the researcher did not affect response rates.
|
One
way to increase response rate in any type of survey is to
prepare a persuasive introduction to the survey.
All
instructions necessary to complete the questionnaire should be
clearly stated for respondents or interviewers. These
instructions vary depending on the type of survey conducted.
Mail surveys usually require the most
specific instructions, since respondents are not able to ask
questions about the survey. Respondents and interviewers should
understand whether the correct response consists of circling,
checking, placing in a specific order, or skipping an item.
All
surveys flow better when the initial questions are simple and
easy to answer.
The
physical design of the questionnaire is another important factor
in survey research. A badly typed, poorly reproduced
questionnaire is not likely to attract many responses in a mail
survey. Nor does a cramped questionnaire with 40 questions
to a page help to instill respondents with a positive attitude.
Response categories should be adequately spaced and presented in
a nonconfusing manner. For example, the following format might
lead to problems:
Questionnaire
length is an important concern in any type of survey. One basic
reason is that questionnaire length is directly related to
completion rate. Long questionnaires cause fatigue and
respondent mortality, and low completion rates. Shorter
questionnaires guarantee higher completion rates.
Although surveys are valuable tools in mass media research,
there are problems present in any survey.
Experience in survey