Survey Anxiety

So the survey design phase is well underway. We are currently pilot testing by asking Rutgers University Librarians to take the survey as if they were a student. As of this writing, 54 have taken the pilot, and half of them have left comments for possible improvement. The next step will be too consider those suggestions and redraft the instrument. The final survey is slated to be posted on May 1 and run through the 21st. For incentive, one Macbook computer and one iPod touch will be awarded by random drawing to one undergraduate and one graduate respondent.

The survey questions have been derived through a collaborative process with every member of our project team encouraged to pose or edit questions. Discussions via Sakai were insightful and and very helpful—I hope that every person who contributed an opinion or perspective will see their ideas reflected in the final survey draft. Though I am observing from a distance—I am writing this from my campus office in Harrisonburg, VA—it seems that enthusiasm for this project is growing, and we are all getting excited to see how our instrument works and how effective this process will be.

While we have been concentrating on the design of the survey, I have been anticipating the pending results. Our purpose is to collect data that will help in the redesign of the RU Libraries web interface. Ideally,we hope to be able to generalize our findings onto the entire student and graduate student population. How, when, and why do they access the Libraries’ website? Do they ever log on through their iPhones? What browser to they regularly use, and what operating system? Do they usually log on from home or from campus? How often to they come directly to the RU Libraries main page, and how often do they go first to Google? These are a few of the questions we have to ask. But once the survey is complete, the question will remain: how close are we to valid and reliable answers?

I am mindful of the words of one of the grand gurus of survey design, Don Dillman. Dillman et al. (1998, 2001) articulate four errors that survey designers need to beware: coverage error, sampling error, measurement error, and nonresponse error. In order for a survey to be scientifically valid, each error must be minimized. As I think about they survey design process, I am concerned, of course, with all of these possible sources of error. But I am most concerned about sampling error.

As Dillman and Bowker (2001) define them, coverage error is “the result of all units in a defined population not having a known nonzero probability of being included in the sample drawn to represent the population”; sampling error is “the result of surveying a sample of the population rather than the entire population.” Measurement error is “the result of inaccurate responses that stem from poor question wording, poor interviewing, survey mode effects and/or some aspect of the respondent’s behavior,” and nonresponse error is “the result of nonresponse from people in the sample, who, if they had responded, would have provided different answers to the survey questions than those who did respond to the survey.”

While I am of necessity concerned with all of these types of errors, the one I am most concerned with for our purposes is coverage error. Hard-core statisticians such as Dillman insist that for a survey to be valid, the respondents must be randomly selected from a sampling frame. Without this random selection, it is not possible, according to the laws of statistics, for us to generalize onto the population in question.

Of course, web surveys frequently are NOT random, and thus many will say that surveys such as ours are just not scientific. As a social scientist, this of course concerns me. But as a cultural anthropologist, it is not, perhaps such a big deal. While I use the hard-core survey methodology as an ideal model to draw from, I am not entirely married to it. There are other factors that weigh in equally. For instance, rather than make sure that every student has an equal chance of being selected from our survey, I am more concerned that each student has an equal chance of winning one of our incentive prizes. This makes a random survey impossible: it would not be fair for us to offer the incentive to some students but not to others, even if our survey pool had been randomly selected. But what does this say about our ability to project our findings onto the general population?

In my previous two studies, at the University of Rochester and Colorado State, I used a chi square analysis to see how my survey response population matched my actual student population (of graduate students) by comparing the ratios of students in the sciences, social sciences, and humanities in each group. In both studies, the two “pies” were similar enough to suggest that the sample group was a fair representation of the total student population. The question remains, will I be able to do the same with our survey?

The first problem is that we are not directly and systematically contacting every undergraduate student. We are promoting the survey through other ways; only grad students will be contacted directly. I don’t know if we should even worry about coverage error. Perhaps we should not be concerned about projecting our findings onto the entire student body? The difference will be saying “45% of respondents use…” instead of “45% of our students (with a 3% margin of error) use…” The trick, I think, is in being clear that we are not attempting a rigorous scientific analysis. We are instead trying to get a close enough idea as to what students are up to to inform our redesign. I think that the strength of our study will be in our ability to combine the survey results with the results of the interviews and future focus groups. Rather than be totally vested in one research methodology, we are here creating our own research design based on other designs; we are thus using a pastiche of methods all focused on a particular purpose through a process of triangulation. If we keep our eyes firmly on that goal and purpose, then we can compensate our lack of scientific validity with a good measure of qualitative research strategies.

Well, that is enough for this blog entry. But this has been on my mind quite a bit the past few weeks, and I wanted to share these thoughts with the team. I would be curious to hear the thoughts of others on this subject.

References:

Dillman, D. A., Tortora, R. D., & Bowker, D. K. (1998). Principles for Costructing Web Surveys (No. 98-50). Pullman, Washington: Social and Economic Sciences Research Centero.


Dillman, D. A., & Bowker, D. K. (2001). The Web Questionnaire Challenge to Survey Methodolgists. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science. Lengerich, Germany: Pabst Science Publishers.