Mapping finalized

With many thanks to the comments here and via email from TNers (original thread), the Mapping project manuscript has moved from white paper to in-press manuscript. Yes, it took a long time.

Here is the final version.

I hope this is something researchers and thinkers find useful. From talks with funders and fellow researchers, I know that the issues of real/virtual comparisons and inferences are of keen interest, and it's my hope that this paper helps people think through the attendant issues. As I was contemplating the many uses I see in this space--disaster preparedness, military training--it occurred to me once again that the price of doing this wrong and drawing bad inferences is very high indeed.


Comments on Mapping finalized:

Richard A. Bartle says:

This is a very useful paper to have around; so many poor claims about virtual worlds are made that it's great finally to have something I point people at to tell them where they're going wrong. Of course, the greatest danger comes from non-academics who take mapping as a given (politicians, journalists, the general public); nevertheless, if we don't get our own house in order we can hardly expect them to, either.

I have some comments, all in the "future work" area:

1) You say that face validity is the simplest and most necessary condition. I agree, but note that it comes at different levels, some of which involve indirection. For example, if I want to find out people's political leanings I could have elections for NPC mayors in my MMO and see which way the players vote. This would be a syntax-level face validity. I could, however, be more subtle and offer multiple ways to solve a quest (give an NPC beggar money, fish, a fishing rod, an admonishment) and calculate political views from that. This would be a semantics-level face validity. I could even find something in real life that correlates with political views (eg. consumption of high culture) and use that as what I test, which can in turn be handled at the syntax, semantics or indirect level (the Game Studies equivalent terms being surface/dressing, mechanics and emergent). None of this is precluded by your paper, but I worry that it may suit people not to look beyond in-your-face validity when a lot of the strength of games in general and MMOs in particular comes from things that are incidental to its main thrust.

2) Mapping is something players may do in the absence of any structural similarity at all. Players can act as though there is a mapping in the code when there is none; this nevertheless establishes a possibly valid mapping. For example, players may anthropomorphise a pet to imbue it with qualities that it doesn't have at a a simulationist level; if you can establish that they do this, then you can run tests in the light of this that you couldn't run if you had to show what it was that pets did which made them (in the absence of player interpretation) behave like real pets.

3) Researchers need to be warned of the dangers of self-validation. For example, suppose an economist examined how the prices paid by NPC vendors for goods rose and fell and from that derived a formula that encapsulated what was going on. This would be a very useful piece of information to have if it applied to the real world, but for the MMO it could be that the designer had looked up that very formula and used it to determine how NPCs would adjust their prices. All the economist would have done in this case is examine empirical data to find the values that the designer was plugging in to the formula - which the designer could easily have just stated, had they been asked.

4) This is unidirectional. You talk about the mapping as if it were all "use virtual worlds as petri dishes to figure out things in the real world". Actually, the reverse also applies: sometimes, it's relatively easy to test ideas in the real world that would be very expensive to program into a virtual world. This is why designers will do pencil-and-paper mock-ups of gameplay, for example, or scour books looking for detail about the kind of creatures that live in mangrove swamps. Designers should be just as rigorous in their use of mapping from RW to VW as other people are in mapping VW to RW.

I also have one point of order: you say that Lessig first articulated the concept "Code is Law"; re-read footnote 7 to chapter 1 of Code and other Laws of Cyberspace.

>Here is the final version.

Too late to italicise those virtual world names you missed italicising on page 18, then?

My only complaint about your paper is that you say that practitioners should expect problems should their results challenge current orthodoxies. While I agree that they should, I fear that your words will be seized upon by anyone whose work produces contrary results for whatever reason. Young researchers in particular are apt to claim that orthodoxy is wrong when actually it's their research that isn't up to scratch. I probably wouldn't have mentioned this, but last week I reviewed two sets of papers (for a conference and a journal) and am so sick of seeing people using a single reference to Clifford Geertz's to give themselves carte blanche to repurpose their playing habits as research that right now I wish you hadn't said that in case you suffer the same fate. "Although this work runs counter to the commonly-held view that few angels play MMOs, such paradigm-changing results are to be expected in this very new field (Williams, 2010)".

Overall, though, I really like what I read here. It should be required reading for anyone wishing to draw parallels between the virtual and the real, for whatever reason.

Richard

Posted Mar 18, 2010 6:14:51 AM | link

Dmitri Williams says:

Great feedback, Richard. I very much appreciate it and wish you'd been one of the blind reviewers!

Points #1 and 2 I hadn't thought of and are quite insightful.

#3 presumes that the designer is pulling from "real world" economic insights, but it's often the case that these are theoretical as well, i.e. assumed to be correct, but often found later to be wrong. There's frequently less hard economic data than expected. Still, point well taken.

#4. I agree, and if you check that table in the paper you'll see a column about direction.

I'll be more precise about crediting Lessig on code, but I do think he's brought the ideas out and crystalized them best, even if drawn from other bits and pieces.

I hope you're wrong about the young bucks challenging the system and forgetting to do good work bit, but of course you're not. Still, if someone reads the whole paper and all of its demands on rigor and quality and then shoots off their mouth, they were probably prone to do it anyway . . .

Thanks again for the good read and comments. I do hope this thing gets used and leads to better work and thinking.

Posted Mar 18, 2010 12:00:24 PM | link

Isaac K. says:

I wonder if you could clarify a point you've made here and elsewhere: that the researcher must enter the world in order to research it. Now, my field is economics. Must I have attempted to merge one company with another in order to perform research on equity price responses for the merging firms? Must I have been a central banker to be an expert on monetary policy? Must I have been a legislator to research the role of bargaining in the lawmaking process?

From this economist's perspective, there are many features of virtual worlds I can study without having any prior experience in the world. RMT markets are an example. Price responses to monetary shocks where markets are centralized are another. I have a suspicion that any well-trained economist like Ted could have performed the same analysis as he did on your EQII data based solely on in-depth interviews of experienced players of that game (and, of course, the data). Clearly someone must enter the world to gather my data, but I need not be a player in order to analyze and understand that data, nor to draw solid conclusions from that analysis.

Indeed, many researchers will rely on others' accounts of a phenomenon because it would be too costly to acquire the experience themselves. What's more, being 'outside' the phenomena may make it easier for me to be objective in my analysis.

What is it about virtual worlds as objects of study that differs from other environments in which human interaction occurs?

Posted Mar 19, 2010 1:20:50 AM | link

Isaac K. says:

To clarify my last sentence, I mean what is different about them that requires the researcher to experience them firsthand in order to study them.

Posted Mar 19, 2010 1:36:26 AM | link

Dmitri Williams says:

Good questions, Isaac.

I'm not a trained anthropologist, so I'll have to channel one for a moment and give this my best shot. First, I think there is a real world answer to your question, and then I think there is an extra reason for virtual worlds work.

You say that you don't need to be a banker or legislator to understand the economic behavior of those people. I agree, and I can see the value of objectivity, but only up to a point. I think that one blind spot in traditional economics is to generally leave out the human and the social aspects of a system. If you assume rationality, or bounded rationality or doses of guile or whatever other parameters, that's only going to get you some, but not all of the way there.

For example, what if you studied stock market behavior and expected X, but found Y. Would your model be flawed? Surely you must allow for the fact that there may be some odd, irrational or unknown human element taking place. Let's say that there was a tradition among traders of heavy drinking on Thursday nights. The economists studying the system would wonder why Friday morning trading seemed so conservative, when the explanation might have been a high likelihood of hangovers. I am being cute, but I'm sure you get the idea: if you'd been a trader, you'd have had that hangover and would understand the dynamic instantly.

Studying any world--real or virtual--is better done if you are familiar with the customs and particularities.

Now for the virtual bit. Virtual worlds might be just like the real ones in the way people mix and behave. There's evidence of many things mapping, but there are also many cases when behaviors clearly don't. Sometimes the reason is the obvious one--the costs are different, changed by anonymity or artificial elements, e.g. resurrection. But other times the reasons why are much less obvious. They may be sociocultural as in my drinking example, but they may be systemic and due to some basic difference between the incentive schemes in the real world and the virtual one.

Here's an example. Members of a raiding guild behave much like members of a club or sport team. They pool resources, give emotional support, have leaders, etc. But in the real world, it's a lot harder to share resources and have trust because the world around you doesn't guarantee and protect trades. Banking isn't as easy. Conversely, if you scam someone in the real world we have cops and social networks that make the shadow of the future an element. In an MMO you could server and faction swap and be gone. Those are code-based differences that make RL and VWs different. You can account for them, but unless you've gone and played, you probably wouldn't intuit them, and you'd be expecting X but you'd observe Y. You might even attribute the difference to stochastic noise, but really you'd just be missing something systematic and knowable.

The assumption of perfect mapping is a dangerous one. I'd rather start with the assumption that there is none and then build it up one element at a time.

Posted Mar 19, 2010 2:05:49 AM | link

Richard Bartle says:

Isaac K.>Must I have attempted to merge one company with another in order to perform research on equity price responses for the merging firms?

Criminologists have a similar problem: must they have to commit crimes in order to be able to speak with authority about crime?

My own view is that MMOs have in the past had a particular problem with people who have written about them without understanding them. Such people will look at, say, RMT, write about it purely from an Economics or Law perspective, and then brilliantly conclude with an insight that they alone have had the perspicacity to see: MMO developers should themselves engage in RMT. If they'd actually played MMOs, or (not wanting to be criminals) interview people who play them, they would find that things are a good deal more complicated than that...

Playing MMOs is itself not without its problems, though: many people seem to think that a 2-4 hour nightly habit makes them uniquely qualified to be experts in all things MMO. They will speak with measured authority about what in practice they may know very little about. This is a trap that academics can fall into very easily: many is the time I have seen heavyweight theory dangerously misapplied because scant knowledge of the application domain (ie. MMOs) was arrogantly believed by the researcher involved to be complete.

Because MMOs have suffered in the past from know-nothings who have pontificated from on high in the complete absence of experience - or even a recognition that any is needed - players now happily mock anyone who writes about MMOs without having played them. The usual suspects are academics, journalists and politicians. This makes sense: if a politician is using a bad mapping (violence in games is isomorphic to violence in real life, say) but hasn't played the game, well of course they're talking rubbish and should be pilloried for doing so. To the politician, though, games are just like any other social problem: they can talk about the harm heroin addiction does to society without having taken heroin, so they can talk about the harm MMOs do to society without having played one...

Although it's great to see mad players talking trash in blogs about upstart academics, journalists and politicians who write ill-informed articles, unfortunately such players can also catch people in their firing line who make informed comments without having played. Designers, for example, who spend their working lives thinking about MMOs, do not need to play an MMO to be able to give a considered opinion about it: they can procedurally generate the play experience in their head from reading the design, watching people play or following comments in a forum. The more information they have then the more detail they will be able to go into, but even a small amount can be enough. I know I wasn't the only designer who knew Richard Garriott's Tabula Rasa was going to suck merely from reading the pre-release publicity; I didn't have to play it all the way through before I could confidently say that. I read several design documents a year: it's my job to envision how something will play before it's even made; it's ridiculous to assert that I, or any of the other design consultants around, actually need to play an MMO to the level cap and beyond before being allowed to comment on it. Nevertheless, because I got so many players moaning at me for not playing MMOs right they way through, I finally snapped did just that (in WoW and LotRO). I regarded this as obtaining the paper qualifications I needed for my job; it certainly wasn't a learning exercise for me.

Apart from designers, other people who study MMOs day in, day out, will eventually grok them. Academics are prime candidates here: Dmitri doesn't have to play EQ2 to study it (I don't know if he does play it, but he doesn't have to) because he's been there, done that. He has played MMOs extensively, he's surveyed thousands of people who play them, he's analysed terabytes of data; he's an expert. Contrast this with the postgraduate who has sat 16 volunteers (all second-year CS majors) in front of a computer, made them play Warhammer: Age of Reckoning for two hours, interviewed them afterwards, drawn conclusions about the effects of MMOs on people's predisposition to violence, and wants a PhD for it. I see this kind of thing all the time. Neither the postgrad nor the expert has necessarily played the MMO they're studying, but the difference is that the expert doesn't need to play it because, well, they're an expert!

Coming back to your problem as an Economist, it really depends on what you're studying. If you're purely attempting to make a mathematical model, for example, then by nature it will be abstract, therefore you could assume that MMO-specific details will be abstracted out; if the model works, you were right, and if it doesn't then you were wrong. You can study a system in depth without having to experience it in action (this is the "most gynaecologists are men" argument: if you fix tractor engines, you just need to know how tractor engines work - you don't have to be a tractor). The problem is that the moment you step outside your field of expertise, you do need domain-specific knowledge. That Economic model you made may be really useful for Economists, but if you then try to push it onto players or designers, that's when you really had better know about MMOs.

So my advice is that if you want to study MMOs and say anything about them as MMOs, you will save yourself a truckload of hurt if you play one first. Whether it's strictly necessary depends on what you intend to do with your results, but if you make even one obvious slip you will be torn apart by a baying mob of bloggers. Not the sort of thing you want your peers to discover when they Google your name..!

Richard

Posted Mar 19, 2010 5:55:23 AM | link

Richard Bartle says:

Dmitri>wish you'd been one of the blind reviewers!

You can send me stuff to read before it gets reviewed, you know.

>#3 presumes that the designer is pulling from "real world" economic insights, but it's often the case that these are theoretical as well, i.e. assumed to be correct, but often found later to be wrong.

Yes, that's right, but it's not what I meant.

Example: following the publication of my Player Types model, designers started to design MMOs that addressed the player types the model listed. If you wanted to validate the player types model today, you would look at MMOs, survey the players and conclude that yes, it's valid. However, it could be that the only reason it's valid is because the MMO was designed to support it. It's self-perpetuating. The reason you find Syd Field's movie structure applies to most movies is because most screenplays that get made today are deliberately written to fit his paradigm.

If you are studying, say, the group dynamics of a new MMO, you may find that beyond a certain point guilds become unmanageably large and break up. You spot that this point equates with Dunbar's number. Yay! You can establish a mapping between the social groupings in this MMO and the social groupings of the real world. Except, you can't: unbeknown to you, the designer is aware of Dunbar's number and deliberately structured the guild system so that it would start to disintegrate around that point, thereby splitting guilds fairly amicably rather than having them fall apart in acrimony or collapse to a vacuum. Your mapping is invalid, because what you think is natural is artificial. It may be that the kind of relationships people have in MMOs effectively add 50 to Dunbar's number, but that doesn't get picked up because the designer has effectively forced Dunbar's number on the groups.

This has happened in other industries. I recall talking to a TV Producer who related having been contacted by an academic regarding the length of the breaks between the end of a programme and the start of the ads. The academic had measured the length in milliseconds of the interstitial splash for hundreds of programmes across all channels, and discovered that they were almost constant. He used research on persistence of vision to calculate the "ideal" value, and it had come out very close to what the TV programmes had. The academic wondered if the producer would be interested in reading the full paper. The producer replied that the TV industry had known about the persistence of vision research for years and had used it to determine the ideal length of an interstitial splash.

>#4. I agree, and if you check that table in the paper you'll see a column about direction.

Oh, I didn't read it as that kind of direction, sorry. I thought it was to do with who was calling the shots!

>I do think he's brought the ideas out and crystalized them best, even if drawn from other bits and pieces.

He has, yes; I may have read City of Bits a few years earlier, but it was Lessig's work that threw the spark on the tinderbox. He wasn't the first to articulate the concept, but he was the first to think it through.

>I do hope this thing gets used and leads to better work and thinking.

Me too, but given the sorry overall state of academic research today I'm not optimistic.

Richard

Posted Mar 19, 2010 6:29:34 AM | link

Isaac Knowles says:

@Dimitri and Richard

I appreciate the responses. Obviously all the hype over VWs in recent years will motivate the uninitiated to have unreasonable expectations for the associated research. After some thought, I can see the value of experiencing the world first before researching it, but for a different reason. Rather than seeing something in real life and then heading into the VW to look for it, one needs to approach in-world research from the opposite direction - by finding interesting phenomena in VWs and then heading into the real world to find appropriate analogues. For instance, Dimitri, I think a scam-rampant VW is full of valid mapping prospects, but not for first-world contexts with strong third-party enforcement regimes. I know that when I first heard of RMT my mind went immediately to the possibilities for illuminating international economics and trade. For many reasons though, that doesn't map very well. But after looking at RMT, seeing what people use it for and the different ways that operators deal with it, I think it is very fertile ground for studying gray and black markets. That mapping has much greater force.

The headline excitement over virtual worlds seems to have originated with the phenomena that are extremely complex aggregates of micro-behaviors and institutions - macro-economies, cultures and societies, and epidemics. But it is the less-complex aspects of VWs that seem most ripe for exploitation for research - markets for single items or services,culture in starting areas, or even the corrupted blood as it was initially intended: a minor debilitating disease that can spread and, if not properly dealt with, able to severely disrupt life (raid-wipe on Hakkar = incentive for avoiding disease, healing disease, and not spreading disease). By not summing over inordinately large constituent phenomena, one can produce new, valuable knowledge, even if it is not as illuminating or awe-inspiring as was once hoped.

Posted Mar 20, 2010 7:18:23 PM | link

Edward Castronova says:

This discussion confirms that the concept of mapping was very much needed. It's going to be such a useful term as we go down the road. Nice job.

Posted Mar 22, 2010 9:29:20 AM | link

Gucci says:

I do hope this thing gets used and leads to better work and thinking.

Posted May 25, 2010 5:19:05 AM | link

josseflibna says:

Thermal paper rolls, ATM thermal paper rolls, colored thermal paper, printer ribbons, POS printer supplies and credit card supplies. POSPaper.com offers a full line of POS thermal paper rolls, printer ribbons, point-of-sale printers, credit card paper rolls, laser labels, barcode scanners, credit card sales slips, cash register ribbons, ATM paper, kiosk paper, cash drawers and other paper related supplies for your business including shopping bags and can liners.

Posted Jun 9, 2010 2:22:56 AM | link