Duffy: How Not to Experiment in Virtual Worlds

Economist John Duffy of University of Pittsburgh went to an experimental lab in Second Life and came away unconvinced. Some thoughts:

• The virtual lab he visited was a replication of a real-world lab. We've learned, however, that using virtual world technology to simply replicate a real-world thing is usually counter-productive. The utility of virtual worlds does not lie in their ability to re-do, in virtual form, what we do in the real world.
• He carries with him some honored norms about what an economics experiment must be. Participants must not know what the experiment is about, for example, so you ought to ask every participant "Are you an experimental economist?" and exclude them if they are. Otherwise they'll recognize that they are in a test of backwards induction or whatever. Duffy did not face that kind of question, nor anything that would allow the screening of participants based on expertise. There were other issues, other ways in which an attempt to replicate a real-world econ experiment, in traditional form, simply breaks when placed in a virtual environment.
• He properly noted that the incentives (couple hundred Lindens) are very low in terms of cash value. Why would anyone think hard in return for a quarter? However, this raises a debate about the utility of the Linden dollar. Some people seem highly invested in the state of their virtual things, despite low real-world values. This in turn raises big questions in economics about how things, utility, and happiness relate. Punt.
• He was able to lie about his age, a problem for the internet at large.

I come away convinced that there's little point in trying to replicate the standard micro-level lab experiments in virtual worlds. While you gain in terms of distance communication, you lose in terms of body signals and eye contact and such. In that kind of hands-on experiment, you can't automate much.

I would note, however, that Duffy's criticisms are cast too widely when he concludes that these problems also condemn truly macro-level experiments such as those proposed by Bloomfield and Castronova.

Macro level experiments are different breeds of fish. In moving to a truly macro scale in virtual worlds, we're gaining not just distance communication but large increases in scale (from two dozen participants to hundreds of thousands), length (from two hours to thousands of hours), and emotional investment (from a couple of bucks to things over which people have commited murder). These advantages, it seems to me, far outweigh the costs in terms of less control over participants, selection effects, and errors in demographic data. Moreover, at such scales, there are ample ways of dealing with these issues: you can't do a Heckman selection correction in a lab experiment of 27 people, but you certainly can do it when you have 27,000.

My sense is that Duffy thinks the goal, my goal, is to place traditional economic experiments inside virtual worlds. I agree with him that that would rather not be worth the time. And that's why the goal, in my view, is much more expansive: I want to create a huge economy in a virtual world and then experiment on it. As this is a new and different beast, it cannot and should not conform to the older norms of micro-level experimental research. It needs new norms, and we have our top men working on them right now. Top. Men.

(When I saw that Professor Duffy had misspelled my name, I was tempted to refer to him as "Professor Dufey." But that would have been childish!)

Comments

[Adam Hyland]

1.

On a small but snarky note, I would say that most of his criticisms can be leveled at micro level experiments in the real world. Most of them are of limited value and are filled with contrivance. And this limitation doesn't impact large scale field effects research.

[Jenna Hoffstein]

2.

I completely agree with your first point (that replicating real-world things in virtual worlds don't usually work out well) and wanted to point out how true this has been particularly with advertising.
A lot of the backlash towards Second Life seemed to be centered around this fact - companies tried advertising in-game in ways that were analogous to real-world methods without realizing that, as an entirely new medium, new strategy around advertising was needed. The same concept - new media call for new tactics - is just as valid with academic experimentation.

[Robert Bloomfield]

3.

The number one point researchers need to remember is that every decision in an experiment needs to be justified, including the decision of where to conduct it. Choosing to leave the lab for a virtual world is a big decision that dramatically reduces the control the experimenter has over the setting, and reduces their ability to know who their subjects are.

This doesn't mean 'don't do it.' Instead, do it only if there is a compensating benefit. Ted's idea of macro experiments is one reason to go into the virtual world. Virtual worlds can have other benefits as well, such as allowing the experimenter to manipulate the appearance of the avatars representing the subjects--and work by Bailensen and Yee, among others, shows how this can affect behavior even in economic settings. If you aren't taking advantage of the benefits of virtual worlds, and only suffering the costs, go back to the lab!

[greglas]

4.

Is this paper this paper?

[Dmitri Williams]

5.

Virtual worlds work has tougher requirements for validity, but both the experimenters and the readers need to know if the work is intended to be mapping work, or not. Bailenson and Yee's work, for example, is largely not about mapping, but about the phenomena that occur in VWs simply because they are important in their own right.

If the work is intended to generalize elsewhere or offline (mapping), then there are a host of both internal and external validity issues to tackle. I list these and go into them in the mapping paper, which (sigh) hopefully will be in press one of these days.

But no, you can't blindly replicate a real-world lab and assume that the virtual version will always function as well (or as poorly, as Adam notes) as the real one. We don't have a lot of experience with VW Hawthorne effects, etc.

Still, the most obvious virtue here, even for small-scale lab work, is the unobtrusive observation part of it. You don't need the guy in the white labcoat (unless he's the confederate and there on purpose) because the engine is collecting the behavioral data. That potential makes these worth figuring out. It's not as if people invented and perfected real-world lab work in a few tries . . .

[John Beety]

6.

The paper does a good job of pointing out a few possible flaws in a particular economics experiment conducted in Second Life. The main one seems to be that filling out the form wasn't a requirement for getting paid. I noted that the experiment was set up by a Master's candidate in economics.

The short version I came away with: Dr. Duffy, as a professor of economics, found flaws in an economics Master's candidate's Second Life experiment and wrote about them. On the topic of real-world versus virtual, how many of Dr. Duffy's concerns are strictly about the experiment-as-virtual-worlds-experiment and how many are actually about the experiment-as-experiment?

[ecastronova]

7.

Greg wrote:

Is this paper this paper?

Yes.

Note to self: Before posting, search your own frickin blog for previous comment on topic.

Sorry Dr. Chesney.

[J. Duffy]

8.

Sorry for the miss-spelling, Prof. Castronova.

I applaud the goal of doing truly "macro experiments" in virtual worlds, and that this will require new norms for both implementation and evaluation.

That was the point of my "snarky" note.

I'm glad to hear that Top people are working on this problem and I look forward to seeing their output and (hopefully) joining in the effort myself.

-Prof. Dufey.

[Shander]

9.

It seems to me that games with items that have an inherent cotextual utility and a cycle of expendablity would make for better markets for study.

I'd imagine that fashion based and luxury item type goods are harder to model in the real world too. I'd think it would be particularly hard to come to economic conclusions by examining cost differences between a store brand handbag from Mervins to the cost of a Prada bag, or the value of collectible baseball cards.

Eve Online's economy comes to mind with utility, production material sourcing, and replacement of item needs. Given its size and what I believe is still control by people with a techinical and perhaps intellectual bent, they might be cooperative in some test manipulations of pricing dynamics.