How Online Communities and Flawed Reasoning Sound a Death Knell for Qualitative Methods

Yesterday, I participated in a panel discussion in Second Life, with Celia Pearce, Thomas Malaby and Tom Boellstorff, on the roles and merits of qualitative and quantitative methods in cultural anthropology.  The audio and a text transcript will be available soon.  (I will provide a link here, as soon as they are.)  UPDATE:  You can find the transcript here.  Instead of rehashing the arguments, I would like to simply spell out my rather bleak prediction of what data-rich and easily-manipulated online communities (like virtual worlds and social networking sites) mean for the future of qualitative research in areas like anthropology that study culture.

Ted Castronova and I are hardly the only ones to note that virtual worlds are great tools for running experiments that we couldn't run otherwise, allowing theory-testing in fields like economics and anthropology.  And for those who prefer exploring data archives (using econometric methods instead of experimental design to draw clean inferences), they might follow a model more like Dmitri Williams and his colleagues, and get data from virtual worlds or social networking platforms.  

So here is the prediction:

Enterprising young scholars who are interested in cultural anthropology and are also trained in statistical methods are going to draw out testable predictions from the body of existing qualitative work, and test those predictions by applying experimental or econometric methods to data extracted from virtual worlds and social media.  They will garner funding and publicity in the areas where they compete head to head with qualitative researchers, and the latter will be forced to defend their methods and conclusions.  Some schools will conclude that they can make a bigger impact in the field by hiring faculty trained in these methods.  Several decades later, the top departments and journals studying the ideas of cultural anthropology will be dominated by quantitative methods.  Qualitative methods will either be relegated to less-prestigious schools and special-interest journals in cultural anthropology, or else cultural anthropology will decline in influence relative to other departments (like psychology) that embrace quantitative methods to study similar questions.

I am not arguing that this is a desirable outcome, but it isn't a hard prediction to make.  Economics and psychology are already in the final stages of this process, sociologists are losing prestige to psychologists as they resist the trend, and anthropologists are already starting down the path. I think my fellow panelists see the writing on the wall, which explains their impassioned arguments on the merits of qualitative research (everyone), the folly of objectivity (esp. Thomas), and the need to secure a place for all methods (Celia and Tom).  

Having thought more about this since the panel, I am concluding that quantitative research drives out qualitative research in a two-stage process.  In cases when quantitative and qualitative methods go head to head--where the same research question is amenable to both methods--the quantitative researchers have a real advantage in persuading skeptics, getting funding and influence.  While there is room for debate about whether this is the best outcome, it is certainly a defensible one.

Where Celia, Thomas and Tom have a real cause for concern--and where the academy makes a far less defensible judgment--is in reasoning like this:

Golly, if quantitative research is preferable to qualitative research when the two go head to head on the same research question, then qualitative research must not be worthy of respect even when applied to questions that are not amenable to quantitative methods.

That type of flawed generalization is pretty common (as shown in repeated experiments in marketing and psychology)--but it is flawed, nevertheless.  The research questions become no less important just because they can't be addressed with controlled experiments or fancy econometrics.  Neither do the qualitative methods become any less rigorous.  So Celia, Thomas and Tom are right to fight this unwarranted conclusion and make sure it doesn't eliminate qualitative methods from the best departments and journals in their field, even in the cases where there is no quantitative alternative.

However, I think it is counterproductive to criticize the follies of objectivity, the oversimplified philosophy of science held by most practicing quantitative researchers, the limits of hypothesis testing, and the like (which featured prominently in yesterday's discussion).  While some philosophers and scientists might be on your side, these arguments give the appearance of protectionism and resistance to new methods, while not actually addressing the flawed reasoning that I see as truly sounding the death knell for qualitative methods when technology and data make quantitative methods.