« Twixt | Main | The virtual census »

Jul 17, 2009



On a small but snarky note, I would say that most of his criticisms can be leveled at micro level experiments in the real world. Most of them are of limited value and are filled with contrivance. And this limitation doesn't impact large scale field effects research.


I completely agree with your first point (that replicating real-world things in virtual worlds don't usually work out well) and wanted to point out how true this has been particularly with advertising.
A lot of the backlash towards Second Life seemed to be centered around this fact - companies tried advertising in-game in ways that were analogous to real-world methods without realizing that, as an entirely new medium, new strategy around advertising was needed. The same concept - new media call for new tactics - is just as valid with academic experimentation.


The number one point researchers need to remember is that every decision in an experiment needs to be justified, including the decision of where to conduct it. Choosing to leave the lab for a virtual world is a big decision that dramatically reduces the control the experimenter has over the setting, and reduces their ability to know who their subjects are.

This doesn't mean 'don't do it.' Instead, do it only if there is a compensating benefit. Ted's idea of macro experiments is one reason to go into the virtual world. Virtual worlds can have other benefits as well, such as allowing the experimenter to manipulate the appearance of the avatars representing the subjects--and work by Bailensen and Yee, among others, shows how this can affect behavior even in economic settings. If you aren't taking advantage of the benefits of virtual worlds, and only suffering the costs, go back to the lab!


Is this paper this paper?


Virtual worlds work has tougher requirements for validity, but both the experimenters and the readers need to know if the work is intended to be mapping work, or not. Bailenson and Yee's work, for example, is largely not about mapping, but about the phenomena that occur in VWs simply because they are important in their own right.

If the work is intended to generalize elsewhere or offline (mapping), then there are a host of both internal and external validity issues to tackle. I list these and go into them in the mapping paper, which (sigh) hopefully will be in press one of these days.

But no, you can't blindly replicate a real-world lab and assume that the virtual version will always function as well (or as poorly, as Adam notes) as the real one. We don't have a lot of experience with VW Hawthorne effects, etc.

Still, the most obvious virtue here, even for small-scale lab work, is the unobtrusive observation part of it. You don't need the guy in the white labcoat (unless he's the confederate and there on purpose) because the engine is collecting the behavioral data. That potential makes these worth figuring out. It's not as if people invented and perfected real-world lab work in a few tries . . .


The paper does a good job of pointing out a few possible flaws in a particular economics experiment conducted in Second Life. The main one seems to be that filling out the form wasn't a requirement for getting paid. I noted that the experiment was set up by a Master's candidate in economics.

The short version I came away with: Dr. Duffy, as a professor of economics, found flaws in an economics Master's candidate's Second Life experiment and wrote about them. On the topic of real-world versus virtual, how many of Dr. Duffy's concerns are strictly about the experiment-as-virtual-worlds-experiment and how many are actually about the experiment-as-experiment?


Greg wrote:

Is this paper this paper?


Note to self: Before posting, search your own frickin blog for previous comment on topic.

Sorry Dr. Chesney.


Sorry for the miss-spelling, Prof. Castronova.

I applaud the goal of doing truly "macro experiments" in virtual worlds, and that this will require new norms for both implementation and evaluation.

That was the point of my "snarky" note.

I'm glad to hear that Top people are working on this problem and I look forward to seeing their output and (hopefully) joining in the effort myself.

-Prof. Dufey.


It seems to me that games with items that have an inherent cotextual utility and a cycle of expendablity would make for better markets for study.

I'd imagine that fashion based and luxury item type goods are harder to model in the real world too. I'd think it would be particularly hard to come to economic conclusions by examining cost differences between a store brand handbag from Mervins to the cost of a Prada bag, or the value of collectible baseball cards.

Eve Online's economy comes to mind with utility, production material sourcing, and replacement of item needs. Given its size and what I believe is still control by people with a techinical and perhaps intellectual bent, they might be cooperative in some test manipulations of pricing dynamics.

The comments to this entry are closed.