Survey Response Rates: Web vs. Mail
Institutional researchers face a choice between implementing student surveys through the e-mail system or through the traditional surface mail system. Three researchers (Kaplowitz, Hadlock, and Levine or KHL) recently published a study that explores the differences in response rate that may occur with these two modes of survey administration. They made the following conclusions, among others:
- "This study found that a Web survey application achieved a comparable response rate to a mail hard copy questionnaire when both were preceded by an advance mail notification [hardcopy notice]. A reminder mail notification had a positive effect on response rate for the Web survey application compared to a treatment in which respondents only received an e-mail containing a link to the Web survey. Reminder mail notifications did not produce higher response rates to the Web survey for respondents who had received a pre-notice. The cost differential between the mailed hard copy questionnaire treatment and the Web survey treatments with mailed advance notice was substantial..." [p.100]
- "The findings...suggest that, in a population in which each member has Web access, a Web survey application can achieve a comparable response rate to a questionnaire delivered by surface mail if the Web version is preceded by a surface mail notification..." [p.100]
- "The mean age of respondents in the four Web groups of 24.14 years old differed significantly, at the 0.05 level, from the 30.55 years old mean age of mail survey respondents..." [p.97] [Note: So resulting response distributions for specific survey items may vary between mail and survey mode if these responses correlate with age.]
- Mail mode (pre-survey postcard, survey, follow-up postcard, and a final reminder with a replacement survey---all hardcopy) had a 31.5% response rate at a cost of $10.97 per respondent. Web group 2 (hardcopy postcard pre-survey notice and actual e-mail survey) had a 29.7% response rate at a cost of $1.31 per respondent. [pp. 96-98]
KHL analyzed data from a designed survey experiment using five random samples that included 17,296 students at Michigan State University. Each sample, which contained a mutually exclusive set of students, received a different "treatment" or survey administration protocol. The sample design stratified each treatment's sample to capture equal proportions of undergraduate, graduate and professional students. The survey's topic was student attitude, knowledge, and use of watershed (an environmental topic).
This study may have more relevance to researchers who will survey students for their attitudes about topics of general interest (such as environment or politics) than to researchers who will survey students about institutional quality or educational experiences. Survey topic tends to have some influence upon response rates, independent of survey administration mode. Michael D. Kaplowitz, Timothy D. Hadlock, and Ralph Levine (all of Michigan State University) document their study in the article ("A Comparison of Web and Mail Survey Response Rates") in Public Opinion Quarterly (Vol.68, No.1, Spring 2004, pp. 94-101).
[Abstract done by Willard Hom, Director, Research & Planning Unit, System Office, California Community Colleges, 1/23/05]
|
