RU Libraries Ethnography Research Project

From Behind the Camera

2009-04-17T07:17:00.000-07:00

The graduate survey ran successfully from April 7 - April 14. I think we had a decent response rate - more than 2500 - and now the task of making sense of the results. The interviews have also begun; I worked with Todd on my first one and had to remember to stay focused on running the camera -very easy to be completely absorbed in what the interviewee/student is saying! The one interview I did was rather fascinating - a transfer student who gave high marks to an information literacy session she had attended (as part of the Students in Transition classes) led by one of our librarians. She even had the handout from the class with her. Running the camera was also fun - brought back memories of my days in film school as an undergraduate, when a video camera was just like another appendage for me. The way of the ethnographic interview is very different from scripted interviews that all follow the same path - they are geared towards engaging the interviewee, making the most of each persons' individual differences, letting them share their own thoughts and ideas, no matter how tangential they may seem at first. Fact is, everything they say or do is significant in some way. The trick is to capture it all, then relate it back to the research query (in this case, the connection between online research habits and the use of the libraries' web site). The team working on this project will certainly have a lot to work with when it comes time to review these rich interviews.

Valeda Dent Goodman
Associate University Librarian
Research and Instructional Services
Rutgers University

Survey Anxiety

2009-03-30T15:02:00.000-07:00

So the survey design phase is well underway. We are currently pilot testing by asking Rutgers University Librarians to take the survey as if they were a student. As of this writing, 54 have taken the pilot, and half of them have left comments for possible improvement. The next step will be too consider those suggestions and redraft the instrument. The final survey is slated to be posted on May 1 and run through the 21st. For incentive, one Macbook computer and one iPod touch will be awarded by random drawing to one undergraduate and one graduate respondent.

The survey questions have been derived through a collaborative process with every member of our project team encouraged to pose or edit questions. Discussions via Sakai were insightful and and very helpful—I hope that every person who contributed an opinion or perspective will see their ideas reflected in the final survey draft. Though I am observing from a distance—I am writing this from my campus office in Harrisonburg, VA—it seems that enthusiasm for this project is growing, and we are all getting excited to see how our instrument works and how effective this process will be.

While we have been concentrating on the design of the survey, I have been anticipating the pending results. Our purpose is to collect data that will help in the redesign of the RU Libraries web interface. Ideally,we hope to be able to generalize our findings onto the entire student and graduate student population. How, when, and why do they access the Libraries’ website? Do they ever log on through their iPhones? What browser to they regularly use, and what operating system? Do they usually log on from home or from campus? How often to they come directly to the RU Libraries main page, and how often do they go first to Google? These are a few of the questions we have to ask. But once the survey is complete, the question will remain: how close are we to valid and reliable answers?

I am mindful of the words of one of the grand gurus of survey design, Don Dillman. Dillman et al. (1998, 2001) articulate four errors that survey designers need to beware: coverage error, sampling error, measurement error, and nonresponse error. In order for a survey to be scientifically valid, each error must be minimized. As I think about they survey design process, I am concerned, of course, with all of these possible sources of error. But I am most concerned about sampling error.

As Dillman and Bowker (2001) define them, coverage error is “the result of all units in a defined population not having a known nonzero probability of being included in the sample drawn to represent the population”; sampling error is “the result of surveying a sample of the population rather than the entire population.” Measurement error is “the result of inaccurate responses that stem from poor question wording, poor interviewing, survey mode effects and/or some aspect of the respondent’s behavior,” and nonresponse error is “the result of nonresponse from people in the sample, who, if they had responded, would have provided different answers to the survey questions than those who did respond to the survey.”

While I am of necessity concerned with all of these types of errors, the one I am most concerned with for our purposes is coverage error. Hard-core statisticians such as Dillman insist that for a survey to be valid, the respondents must be randomly selected from a sampling frame. Without this random selection, it is not possible, according to the laws of statistics, for us to generalize onto the population in question.

Of course, web surveys frequently are NOT random, and thus many will say that surveys such as ours are just not scientific. As a social scientist, this of course concerns me. But as a cultural anthropologist, it is not, perhaps such a big deal. While I use the hard-core survey methodology as an ideal model to draw from, I am not entirely married to it. There are other factors that weigh in equally. For instance, rather than make sure that every student has an equal chance of being selected from our survey, I am more concerned that each student has an equal chance of winning one of our incentive prizes. This makes a random survey impossible: it would not be fair for us to offer the incentive to some students but not to others, even if our survey pool had been randomly selected. But what does this say about our ability to project our findings onto the general population?

In my previous two studies, at the University of Rochester and Colorado State, I used a chi square analysis to see how my survey response population matched my actual student population (of graduate students) by comparing the ratios of students in the sciences, social sciences, and humanities in each group. In both studies, the two “pies” were similar enough to suggest that the sample group was a fair representation of the total student population. The question remains, will I be able to do the same with our survey?

The first problem is that we are not directly and systematically contacting every undergraduate student. We are promoting the survey through other ways; only grad students will be contacted directly. I don’t know if we should even worry about coverage error. Perhaps we should not be concerned about projecting our findings onto the entire student body? The difference will be saying “45% of respondents use…” instead of “45% of our students (with a 3% margin of error) use…” The trick, I think, is in being clear that we are not attempting a rigorous scientific analysis. We are instead trying to get a close enough idea as to what students are up to to inform our redesign. I think that the strength of our study will be in our ability to combine the survey results with the results of the interviews and future focus groups. Rather than be totally vested in one research methodology, we are here creating our own research design based on other designs; we are thus using a pastiche of methods all focused on a particular purpose through a process of triangulation. If we keep our eyes firmly on that goal and purpose, then we can compensate our lack of scientific validity with a good measure of qualitative research strategies.

Well, that is enough for this blog entry. But this has been on my mind quite a bit the past few weeks, and I wanted to share these thoughts with the team. I would be curious to hear the thoughts of others on this subject.

References:

Dillman, D. A., Tortora, R. D., & Bowker, D. K. (1998). Principles for Costructing Web Surveys (No. 98-50). Pullman, Washington: Social and Economic Sciences Research Centero.

Dillman, D. A., & Bowker, D. K. (2001). The Web Questionnaire Challenge to Survey Methodolgists. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet Science. Lengerich, Germany: Pabst Science Publishers.

Drafting the survey --

2009-03-25T12:39:00.000-07:00

As a member of the committee that will likely be responsible for deriving something useful from the results of the ethnography project, it's been interesting being involved in the discussions about the types of questions we should be asking in our initial survey and how we should be asking them. Since the ultimate goal of the project is to redesign the Rutgers University Libraries' website, it's been a challenge to come up with questions that will actually help us toward that goal. Being neither an ethnographer nor a statistician, my perspective on the questions is based on my experience as an information services librarian and as a member of the committee that works with the webmaster to design and maintain the website.

The first consideration in any survey should probably be to try not to scare people away from responding by asking too many questions. I've visited lots of survey sites only to find that they were asking too much of my time for little or no reward. We're entering respondents in a drawing for some nice prizes, so there is an incentive for them to complete the survey, but that's not a license to overwhelm them with questions.

The second potential pitfall to keep in mind is “library speak.” Librarians as a group tend to use terminology that may not be familiar to the average library user. Survey questions need to be written in plain English. If they can't understand the questions, they won't be able to answer them in a useful manner.

So what kinds of questions can we ask students that will be meaningful when we begin the actual redesign process? There are some basics that need to be included along with the questions that I consider the real focus of the survey. We’ve included questions that will:

identify the status of respondents (e.g., class year, campus, discipline, etc.), both so that we can look for perspectives from different audiences, and so that we can select a representative sample for the interviews that will follow;
determine how experienced respondents are at doing research/using the existing Libraries' website;
find out what technologies students are using so we know what we're designing for;
find out what students are using and not using on the existing site (or if they're using it at all); and,
find out what they'd like to see that's not on the site (or find out what's already on the site that they don't know about).

We’re also using a variety of question formats – pick lists, Likert scales, yes/no, comment boxes, etc. – that I think keep the survey from being too tedious.

That's a very general picture of the survey instrument, not yet in final form as of this writing. The intent is to make the most of the limited number of questions we're using in this first stage of the project. We'll get feedback from the largest number of users here, but supplementary interviews of some students will allow us to follow-up with more in-depth questions.

I'm still not sure how all of this will translate into a website redesign. At this point it really seems to be as much about figuring out the process as it is about looking ahead to achieving a useful end result.

Learning on the Job

2009-03-11T06:18:00.000-07:00

I am both a MLIS student at SCILS and a Web Developer for Rutgers University Libraries, and I intend to share my perspective on this project from those lenses in future posts. But for this, my first post, I would like to share some thoughts as the father of a toddler and a soon-to-be toddler.

I remember feeling anxiety, hope, and a daunting sense of responsibility when my first son was born. I didn't know how to be a father. As in parenthood, I find myself in a moment of great expectations as part of a team embarking on an ethnographic study, another new frontier for me. In both cases, I'm doing my best to "learn on the job."

One thing I had to realize rather quickly about parenthood is that little kids can't tell you what they need in words. You have to intuitively understand how to address their needs by reading visual cues, interpreting partial words, and otherwise avoiding meltdown by whatever means necessary. The pediatrician, Dr. Harvey Karp (a.k.a. "the baby whisperer"), tells us meltdown usually occurs when the child feels like you're just not "getting it" (Parker-Pope, 2008). More than once I've had to reveal my inner caveman to get the point across that yes, "I know you don't want to get dressed!!!" To my astonishment, simply reiterating their feelings on their level is often more effective than using the "calm parent voice" while weathering a sonic blast that would make Pete Townshend cringe.

Librarians have to intuitively interpret information needs everyday. In the ideal world, every student would have their own librarian to interpret their needs, soothe their anxieties, and teach them to gracefully navigate the frightening maze of scholarly information. Instead, we must do our best tailor our Web site to be that "surrogate" co-pilot, knowing it's a poor substitute, but hoping it will point students we may never meet in the right general direction.

Undergraduates are new to the research process and learning this new "language" can be very frustrating to them. And when we don't "get it", they don't kick and scream. OK, sometimes they kick and scream, but more often they say "whatever" and go away with a bad experience. Unlike the parent-child relationship, they can survive well enough in the wild.

Likewise, it can be frustrating and overwhelming for librarians and staff to keep up with the rapid changes in information technology we've been experiencing. How do we decide which technologies are the most important to focus on? How do we understand what our appropriate place is in the social frameworks each new technology enables? How do we let them know we "get it"?

An ethnographic study is a sensitizing process (Crabtree et al, 2000). We hope it will allow us to see the world through others' eyes. It can open doors for understanding and collaboration, and ultimately we are collaborators in each student's education and each scholar's research. As a Web developer, I am most interested in how we can practically apply the student narratives we collect to real-world decisions about interface design, usability, and investment in specific technologies. That will come, but I do believe those narratives will reveal communication problems we were previously not aware of, and improvements in communication will eventually lead us to the place that other parents tell me about when they say, "It gets easier."

The shift to parenthood was not easy for me because it required a fundamental shift in my own identity. I believe that libraries are going through a similar phase as many of the values they've traditionally provided are shifting in focus. While some anxiety is natural and understandable, a deep understanding of our students' perspective gives us the opportunity to both find new needs to fill, and reinforce our traditional services in words they will understand.

References:

Crabtree, Andy, David M. Nichols, Jon O'Brien, Mark Rouncefield, Michael B. Twidale. 2000. "Ethnomethodologically informed ethnography and information system design." Journal of the American Society for Information Science 51:7, pp. 666-682. John Wiley & Sons, Inc. external link: http://dx.doi.org/10.1002/(SICI)1097-4571(2000)51:7<666::aid-asi8>3.0.CO;2-5

Parker-Pope, Tara. (2008). Coping with the caveman in the crib. Retrieved March 10, 2009, from The New York Times Web site: http://www.nytimes.com/2008/02/05/health/05well.html

Getting Started - RUL Ethnographic Web Project Underway

2009-03-02T05:36:00.000-08:00

Back in January, the Rutgers University Libraries began an interesting research project - to examine the online research habits of students and to use those findings to inform the eventual re-design of the libraries' website down the road. Two teams were formed, a Core Team and a Project Team, and there is already growing interest in the project both at Rutgers and within the larger academic library community. It is a unique approach to a website re-design, and there is a lot to learn about the collaborative nature of team design. We expect the project to take a number of months, and that doesn't include the website work itself, which is yet another phase. Besides the stuff one would expect to encounter and learn with any research project, we are hoping to have our assumptions about how the Internet and the website are used, and the ways our users conduct their work, challenged. The Rochester study of undergraduates was eye-opening, and subsequent studies of both faculty and graduate student research habits have been just as interesting. We hope our project will allow us to step into the shoes of our users, and engage the entire library community to set the stage for a groundbreaking look at one of the libraries' most important access points - the website.

Valeda Dent Goodman
Associate University Librarian
Research and Instructional Services
Rutgers University