Arden: Final Results

The results of the Arden project are available in a working paper here.  The working paper has also been submitted to a journal for peer review.

Summary: In a fantasy game setting, we made two equivalent worlds and set the price of potions to be higher in one than in the other. We found that people bought fewer potions when they were more expensive.

Discussion: The result suggests that people in fantasy games act in an economically normal way. Perhaps these game worlds can be used to study real economic behavior.

The results are based on an environment that was significantly more fun to play than the first one we made and represents the completion of the Arden project. My thanks go to the MacArthur Foundation for their early trust, their support through hard times, and of course the funding that made the study possible. I'd also like to thank the Bioware corporation for allowing us free use of Neverwinter Nights.

You can download the game environments we used for the study here.

As some of you might recall, two months ago I announced that the results were done, but I did not say what they were. I held back because I felt an obligation to the funder of the study, the MacArthur Foundation, to give them time to respond and comment before publicizing the results more widely. I am sorry to have given the impression of a 'tease' strategy. I won't do it this way again.


Comments on Arden: Final Results:

Tim says:

Hmm the DL site seems uninterested in actually mailing me my password in a timely manner. Or maybe it's this whacky free WIFI in DIA...

Posted Jul 24, 2008 8:18:23 PM | link

Adam Ruch says:

Congratulations Ed on finishing the project, looking forward to the finished paper.

Posted Jul 25, 2008 1:12:30 AM | link

dmyers says:

These virtual guys.

First they say it's about the loot.
Then they say it's about the porn.
Then they say it's about the loot again.

I'm so confused.

Posted Jul 26, 2008 12:10:41 AM | link

Adam Hyland says:

Good deal. An unambitious but clear start. this provides a way forward for a lot more work in the field without having to rely on explicit comparisons to time spent (where normal behavior may or may not apply) or reliance on repugnant transactions (RMT, twinking, etc). Very helpful and very clear. Thank you.

Posted Jul 28, 2008 1:36:42 PM | link

Isaac Knowles says:

As a player, I needed no convincing :)

Just how much data were you able to extract from this experiment (besides changes in quantity of potions sold)? For instance, did you observe substitution into related goods (i.e. better armor to decrease the need for potions, or better weapons to reduce time spent in danger, or other products which lead to healing)?

What about macro effects that might have resulted from adaptations / innovations - that is, technological change. Is there enough data to create a time series of elasticities of demand for potions, suggesting learning on the part of players as they innovate compensating strategies? Or was the elasticity constant, statistically speaking, over the course of the experiment?
Were advancement rates different?

Just a few of any number of questions I can think of. Thanks for writing this.

Posted Jul 29, 2008 3:24:38 PM | link

flashheart says:

It seems like one might need to run a kind of "burn-in", to test whether players in one world randomly happened to just be crappier than players in the other world, and therefore needed more potions. In a randomised experiment this shouldn't happen, but it can happen and if so one would need to adjust for skill level, as measured during the "burn-in".

Was it known that the potions were an essential part of the adventuring experience? In some games potions are a luxury or only a minor add-on, while in others one lives and dies by them. Presumably the elasticity of the good depends on its (presumably unmeasured) degree of importance? Like petrol vs. bottled water (not that one drinks either of these substances...)

Posted Jul 29, 2008 7:36:49 PM | link

Richard Bartle says:

There are existing virtual worlds out there that are (in terms of their code) identical in all respects: any virtual world with multiple shards is like this. If you were to select two shards (or servers, if you prefer that term) for the same game world that opened at the same time to the same basic audience, you should find that supply and demand of items is equivalent. They won't be identical, though, because of noise in the data (random numbers are random, after all).

Something it shouldn't be too hard to do is to check the auction house prices for popular, staple items over the course of a few weeks to find out average prices. This will give an idea of how much difference there is between systems that are (in the terms of the Arden experiment) identical. This difference can then be compared with the actual difference discovered between the two NWN servers, to see if that difference is significant or whether it could be dismissed as an artefact of the general randomness of the system.

Richard

Posted Jul 30, 2008 2:47:37 AM | link

Isaac Knowles says:

@Richard:

It seems to me that a similar goal can be met by altering the price of a different staple item in Arden, and then simply running the experiment again for several different items.

The difference between WoW servers (say), could potentially be huge - even if you're dealing with shards in the same battle group and type (i.e. pve, pvp, etc.) with the same population level on the same faction with the same sysadmins. The problem isn't the 'known knowns' when comparing the economies of servers of a game over which we do not have control - it's the 'unknown unknowns', which could create systematic patterns unbeknownst to the researcher. Arden eliminates that problem.

My sense is that the only big similarity you would find between (WoW) servers is the changes in the general price level over an extended period of time.

@flashheart:
I think that if individual purchase data were available, one could reasonably mitigate the need for a 'burn in' period. Random assignment to servers, especially over a number of iterations of the experiment, would ensure that average skill level was constant across shards. The significance (or not!) of the difference in quantity demanded between the two servers would generally account for skill imbalances (again, over a series of iterations of the experiment).

In the absence of individual data, however, you point is quite valid.

Posted Jul 30, 2008 9:46:09 AM | link

Adam Hyland says:

Richard,

A nice thing about the experiment conducted here is that it avoids the selection bias generated by shards with known differences in characteristics. That's a difficult bias to decompose for the purposes of analysis, so it is nice to see it avoided "outside of the lab".

But you are right that some changes may take advantages of natural experiments over time or across shards. I'm KICKING myself for not recording the prices of baskets of goods on the auction house in WoW before and after the introduction of daily quests (and the 2.3 introduction of more daily quests). That would have been a pretty simple baseline macro experiment for money supply in WoW. After the fact, that data is lost, but it would have made for a neat paper.

Posted Jul 30, 2008 12:03:24 PM | link

Edward Castronova says:

The reaction to this finding seems to be a lot like when I found that female avatars cost less on the open market. Some people think it's a really big deal, other people think it's ho-hum. If you play these games, you know that higher prices reduce sales, just like in reality. But people who don't play these games are skeptical. Even if you show them two identical servers where prices and sales vary, they are not convinced because there's no random assignment, and there's a host of other possible causal factors.

So here, we randomly assigned people and did only one thing, raised a potion price. People bought fewer potions. All that says is that being an elf doesn't make you turn off the rational economic calculator part of your brain. Not news to some people, big news to others.

Posted Jul 30, 2008 12:07:44 PM | link

Isaac Knowles says:

@ Adam

I've basically done what you suggest (click my name and find one of my entries on WoW research), except it was during the highly publicized account deletions during mid to late 2006. So instead of seeing price increases (which is what I think we're probably seeing now with daily quests), at the time we saw big price decreases. Interestingly, the change in price level was pretty consisten across shards and factions.

Exogenous monetary policy shocks ftw.

Posted Jul 30, 2008 2:58:28 PM | link

Tim says:

One question I think is, was the number of players involved (the sample) sufficient given thecomplexity of virtual world behaviors? Is there a level at which results are just potentially noise?

It's not economic in nature, but Valve Software has done some quite amazing stats generation for their games. Check out http://www.steampowered.com/status/ep2/ep2_stats.php for the single player HL2 Episode 2 stats, or http://www.dayofdefeat.com/stats/ for the multiplayer Day of Defeat game stats. These aren't of course persistent virtual worlds, but none the less a fascinating dataset about player behavior.

Posted Jul 30, 2008 3:01:37 PM | link

flashheart says:

Isaac, thanks for the reply, I see your point (didn't realise players were randomised on entry). There is still a risk that the randomisation won't work though, and you'll get a randomly distributed confounder you need to check.

Edward Castronova, I don't know if everyone who plays necessarily pays attention to the price of potions. Whether they do or not depends on whether a) potions are essential b) there is a wide range of other magic items available c) they're a fighter (potions are much more important to fighters than mages) and d) how fast they're accruing experience relative to gold. I understand that a),b) and d) are specific to the worlds and so presumably irrelevant in this experiment but c), for example isn't and d) does depend on how individuals play and the game balance. If your world is poorly designed to reward fighters, for example, and more people in one world choose thieves, then you may get a biassed result.

Essentially this means that to study this problem properly I think you either need a very well designed world or a randomised block experiment where players are randomised to both worlds within blocks of character types. I don't aim to impugn your world and I don't know if you ran a full randomised block model, but the judgement that potions are cost dependent is not necessarily as intuitively obvious as you make out. I often play mages, for example, and for mages potions are sometimes not even relevant - and for thieves potions of master thievery are so useful as to be very valuable, but so rare as to make their price almost irrelevant.

Posted Jul 30, 2008 6:42:25 PM | link

Richard Bartle says:

Ted>The reaction to this finding seems to be a lot like when I found that female avatars cost less on the open market.

The issue there was the interpretation of the data, not the data itself. The sensationalist reading got the headlines (and the paper is still regularly cited as if it proved that female players are regarded as inferior by male players). It wasn't even a full test of the open market, as you didn't include buy-out values.

I see what you're saying about the reaction to this NWN paper, though: its assertions may seem obvious, but they nevertheless had to be tested. "It's obvious" isn't part of the Scientific Method. If people are ever going to use virtual worlds for trials of economic or other social science theories, then the foundations have to be established as sound. That's what this paper does.

That, in my view, is what makes it important rather than ho hum.

Richard

Posted Jul 31, 2008 8:16:21 AM | link

Isaac Knowles says:

As a microeconomic test bed, I think game economies would excel in that role and therein lays the importance of Castronova's paper.

One thing that continues to bug me, however, is whether MMOs are worth pursuing as models of a macroeconomy. While I think its fair to say that virtual economies display price fluctuations due both to internal and "foreign exchange" dynamics, I'm less convinced that they provide utile analogies to business cycles and economic growth. UO's initial attempt at a self-regulating economy met a quick and merciless fate and was replaced by the faucet-drain system - a planned economy. If the institutions of a planned economy, in whatever form, are substantively different than those of a market economy, then the usefulness of a game economy to study changes in relative prices would be very limited. What innovation occurs will be highly restricted and/or mostly resistant to accurate measurement, and that makes even the testing of nominal shocks on real prices in a virtual economy a nearly impossible task. In that light, it seems that the value of the game macroeconomy stops and starts at its ability to exemplify the role of nominal shocks on nominal prices, which merely demonstrates what most economists already know: inflation/deflation is always and everywhere a monetary phenomenon.

So I don't know if this thread is dead yet, but if not does anyone have a comment on these matters?

Posted Jul 31, 2008 10:31:26 AM | link

flashheart says:

Isaac, I think that more complex economic models could exist in these worlds. For example, a Guild in world of warcraft could set itself up as a manufacturer of potions, selling them in the world and devoting itself to making money. A friend of mine tells me he may have seen something like this on his server, but he doesn't know. I think there is also a business model for gold farmers in such a system, and I have written a post about the model on my blog. These worlds have resources and manufacturing and money, so it could happen.

The problem is that, outside of the gold farming model, there is very little reason for anyone to do this. One doesn't enter these games to make money, but to make experience and kill monsters. It would be a wierd group of people indeed who chose to set up a guild which bought raw materials and sold finished goods - so it will probably never happen as an exclusively virtual economic force.

Unless an adventuring guild sets up some kind of manufacturing arm to support its raids. I watched video footage of a 40-player raid recently, which took 3 months to get from first stage to second; this must have involved a lot of training and would have really chewed through items. So I can imagine that a guild might be inclined to set up a sideline in manufacturing to support its training. But I can't see it becoming organised, because who wants to log into a fantasy world to work full-time on a production line?

Posted Jul 31, 2008 5:41:05 PM | link

Edward Castronova says:

The state of research is so early that almost any social scientific law is worth checking - does it work in the virtual space? The next project we're doing at SWI is precisely a test of whether some fundamental macroeconomic forces function normally in a multiplayer online game.

That will take a couple of years to do, though. We're applying production and management lessons from Arden...so don't hold your breath.

Posted Jul 31, 2008 7:05:57 PM | link

flashheart says:

I wonder how long it will be before researchers start asking for access to data from the companies, or cooperation on experiments. I would like to see survival data, for example. How entertaining! Or to see Blizzard do some experiments where they created a drought of a product, or manipulated its price.

Have you guys considered doing these things? Obviously Bioware cooperated with you, but cooperation in an extant virtual world would be a very powerful thing.

Posted Aug 1, 2008 3:21:41 AM | link