Who plays, how much, and why? Answers.

As some on this list know, my research group has been working on a joint project with Sony Online Entertainment for the last two years. This collaboration has enabled our team to collect virtual world data on--as far as we know--an unprecedented scale. SOE has let us access the full data logs generated and collected by the world Everquest II.

This is one of those "be careful what you ask for" moments in science. We asked for everything, and many terabytes later, found ourselves hosting and analyzing massive data on supercomputers at NCSA. SOE also let us do a large-scale survey of their player base. Although there have been good surveys of virtual world populations done in the past, this is the first that took place within the game engine and with the help of the developer. As a result, it does not have the self-selection issues that the first such surveys have had, and the response rate was impressive.

This post will share the first of what we expect to be a dozen or more papers on virtual world behaviors. As the first, it's the broadest, but I suspect will be of interest and use to the wider virtual world community. You can find the full text of this first paper here, at the Journal of Computer Mediated Communication, with no access restrictions.

More below the fold . . .

There are several tables and graphs intended to be reference material for the community, and many findings on simple demographics. There were also a few unexpected gems in the data as well. My two favorite oddball highlights were finding that the older players played more than the younger ones, and that the game playing population appears to be more fit than the general US population.

Are these findings representative of all virtual worlds, or all MMOs, or all fantasy titles? I have my own speculations (pretty solid for fantasy diku games), and I welcome yours. Of course, until other developers open their doors in a similar fashion, it'll all remain speculation.

What can you expect from future reports? Our subsequent papers will involve research on gender differences, role players, economic modeling, social networks, group success and failure, raiding, detailed player behavior metrics, trust and community, and many others currently in the hopper. As we develop more and more metrics from the player behavior data, we will be merging these with the psychological, demographic, and attitude data from the survey. In other words, for the first time we will know who they are, what they think, and what they do on a truly systematic level.

The team working on these data hail from 7 different universities in the US and Canada, with four senior researchers and 15 Ph.D. students. The project leads are myself, Noshir Contractor at Northwestern, M. Scott Poole at Illinois, and Jaideep Srivastava at Minnesota (computer science).

Many thanks to Raph Koster for opening the door, and to the entire team at SOE for being engaged and helpful in the process. Immense thanks are also due to the National Science Foundation for sponsoring the work. We hope this project serves as a model for academic/industry collaborations in which we can move from begging for pro bono help to outright consulting-level research and development. By being the first-movers, SOE gets the benefit of a large team of experts on a wide range of social, economic and networking topics.


Comments on Who plays, how much, and why? Answers.:

Oliver Smith says:

I'd be interested to hear what tangible benefits Sony feel they have gained from this participation. Obviously, if Sony can show this has benefitted them in a more than merely academic "now we know" sense, that might help to open other doors...

I'm sure many developers will think "that would be really cool" but that's a hard sell past the suits.


Posted Sep 6, 2008 12:09:00 AM | link

Raph says:

I helped this get off the ground because

a) it behooves all businesses to understand their customers

b) it helps the whole industry by digging deeper into stuff none of us know, without really causing any competitive disadvantage

c) it helps establish SOE as a thought leader and presumably garners good press

At the time that I made the case, I WAS a suit. ;)

Posted Sep 6, 2008 2:13:04 AM | link

mirjam p.e. says:

I can't tell you how much I look forward to reading the papers that will come of all this. We have needed this for a long time, and you are doing it. Hurray for you!

Posted Sep 6, 2008 5:17:45 AM | link

Richard Bartle says:

Very well done on distilling this vast amount of information into meaningful chunks! Also, great kudos to SOE for giving you the data in the first place.

I haven't had time to read the paper yet, but just from scanning through it there are some amazing details (eg. the mean time spent in EQ2, at 25 hours, is 5 hours per week more than survey data usually reports, and 10 hours more than some industry pundits believed). I can't wait to delve into it more fully.

This work is destined to become the bedrock for research into MMORPGs for years to come. Congratulations to you, Nick, Scott and the rest of the team!

Richard

Posted Sep 6, 2008 7:43:08 AM | link

greglas says:

Ding!

Posted Sep 6, 2008 12:06:39 PM | link

Christopher Allen says:

One thing that I'd like to see is more information on the nature of different group sizes, along the lines of the Dunbar Number posts that I've made on my blog with data from UO and from Nick Yee's research on WoW.

Posted Sep 6, 2008 2:59:10 PM | link

Dmitri Williams says:

Thanks, all.

Oliver, we're working on the tangible benefits parts, but as Raph points out, some of what we learn will be unexpected. That's what good R&D should do: give actionable deliverables, but also be able to generate the out-of-left-field stuff.

On the tangible side, what I suspect will happen is that we'll develop models for customer retention based on behaviors and interactions. We will also be able to pinpoint stumbling points within the game, e.g. players who took 10 hours to get through zone X were twice as likely to drop their subscriptions. Some of this we'll be able to share with the world and some will stay proprietary (The first penguin to jump should get some extra fish). Regardless, it should establish the method, which we can then potentially do elsewhere.

We've proposed a joint academic/SOE session for GDC next year, so if it's accepted we'll roll out whatever the latest and greatest is there.

This first paper is really just the low-hanging fruit. The nitty gritty comes later as we start using the behavioral and group metrics. Those are probably all six months out and more. The next paper that should see the light will probably be the one on how male and female players differ online and off. There are some pretty cool findings there that I'm looking forward to sharing. After that, we'll have the role player study and a macroeconomic piece that Ted and I are currently writing up.

Posted Sep 6, 2008 2:59:47 PM | link

Dmitri Williams says:

Christopher, we'll have that group data in spades, but it's going to be a bit. One of the other PIs, Scott Poole, is a groups expert, and anther, Nosh Contractor, is a social network analysis expert. Between them, we'll be generating both actual group sizes for people undertaking joint actions, as well as the larger networks aspect of strong and weak ties. Both of these will be used in models as predictors of success/failure, satisfaction, group trust, retention, etc. Again, that's probably six months away.

Posted Sep 6, 2008 3:06:40 PM | link

Thomas Malaby says:

Congrats to all of you for a great contribution. That's a heck of a lot of data to manage and distill, and you all do a nice job.

My one concern is with the way you draw a straight line from users' reports of why they game, to a system of factors distilled from these representations (which, instead of being a useful model, seems to be put forth as somehow "real"), to Richard's model (which has always been useful, but itself has no deeper grounds for being treated as real in the same sense).

I have no objection to building such models and seeing how reliable the claims they generate are, but the two shortcomings in how that's done here are (a) the reliance on players' representations of their inner states without sufficient qualification, and (b) the extension of the conclusions from them beyond pragmatic modeling and toward a claim that such a system is out there driving what we see.

The first would be solved by broadening the methodology beyond the server data + interviews -- easily done, but until it is the claims here are overdrawn without that qualification. The second is in some ways the harder nut to crack, because while making humble, pragmatic claims seems to be something we all agree is a good idea, that quickly seems to get lost amid the market-like incentive that prompts academics to want to appear to have final answers.

Posted Sep 6, 2008 3:20:00 PM | link

Dmitri Williams says:

Can you clarify that a little? Is your objection that the motivations come from self-reports? Or that we would use them to test behavioral predictions?

The way we'll use these is to have some behavioral metric, say (I'm just making this up) how fast someone leveled. I'd predict, just off the cuff, that someone with a high achievement orientation would level faster than someone with a low one. So, we'd run a model on the 7k people we have motivations for and see how good a prediction that was. If we find that high achievement-oriented people do in fact level faster (either by correlation test or by simple variance explained), that'd be some validation for the measure, and a sense that the self report was accurate. If we found that the high achievement score wasn't a good predictor of leveling speed, that'd cast doubt on the measure, and suggest that the self reports weren't good. We'll be checking these things and reporting them as validity tests.

Posted Sep 6, 2008 3:44:33 PM | link

Thomas malaby says:

All of what you say is fine, and to the extent these predictions are borne out you would increase your confidence in the model as just that, a model. (Although it's worth keeping in mind that you are never dealing with replicated conditions - one can call this an experiment only in a rough sense.)

This would be very important support, but it would not do either of two things which if I understand you correctly you're asking about. First, this kind of support doesn't change the model into something else; that is, it doesn't give you any more reason to believe that the model you created is also a real mechanism going on inside people's heads. Second, and relatedly, it wouldn't be support that would extend to a claim that the users have reported their motivations truly or falsely. In other words, a "fit" between the model or scheme of motivations and the users' representations of why they do things, even buttressed by some predictive support, doesn't make the model something other than a model.

All of this doesn't take away from the contribution of the work, I hasten to add. I just think it's important to understand how much it helps us to understand what's going on.

Posted Sep 6, 2008 5:25:17 PM | link

Dmitri Williams says:

These are not experimental designs. It would take the cooperation of a developer to use that method with a PTR, beta, shard, etc. (and of course I'd love to do that).

As to whether the findings here apply elsewhere, i.e. in other games, other circumstances, etc., I welcome comments and feedback.

As to models, well, I like the ones that endure strong testing and predictions and hold up. I shun the ones that didn't work. And then I try to figure out why which was which. Going into mental mechanisms vs. models is beyond the scope of this post. For the devs, it'd be navel gazing. Write up a post on it and I'll be happy to engage.

Posted Sep 6, 2008 5:47:22 PM | link

Thomas Malaby says:

Forgive me, but I wouldn't think that proper qualifications of claims are ever beyond the scope of what we post.

Posted Sep 6, 2008 5:53:35 PM | link

Thomas Malaby says:

I realize that reply was too curt, now that I'm back from my son's soccer game. Another thread would be a good place to talk in that beardy way about what claims we can and cannot make about human motivation, but do hope that, to the extent this work is supposed to be useful for industry, that usefulness doesn't depend upon those claims being amplified beyond their scope. In fact, I think that would be the last thing that would be necessary for the work to gain the attention of devs. They, like most of us, are pragmatic, and interested in what is useful (like Richard's model has been for so long).

Posted Sep 6, 2008 6:42:01 PM | link

Szonja Odrovics says:

Congrats and many thanks:)

Posted Sep 7, 2008 3:23:20 PM | link

Matt MIhaly says:

This is really cool and has some great data points. Massive kudos to SOE for cooperating in this as well.

Having said that, from where I sit I have a couple of (hopefully constructive) criticisms:

1. I don't see any reason to assume this is representative for anything but the players of Everquest 2. There's no question that the data is not applicable to virtual worlds/MMOs generally nor is it applicable to fantasy MMOs in general, nor is it applicable to DIKU-type MMOs in general.

For instance, by selecting an MMO that requires a subscription to play you've already ensured the study is being performed on the select audience that has credit cards. That group skews a lot older than the general MMO-using population in America, for instance. (I'm assuming it's intended to study English-speaking people since EQ 2 is a minor game in many of the major MMO markets like China.)

2. When it comes to the fitness issue (aside from the fact that BMI is a relatively poor measure of fitness) my question is mainly around the disparity in methodology.

The Ogden, Fryar, Carroll, Flegal study that this study compares its results to involved a physical examination of its subjects to assist in determining fitness. This study just asked its participants for details like weight, which people tend to under-report.

Anyway, regardless of whether it tells us anything about player populations outside of Everquest 2 (and I'm not confident it does outside of perhaps DaoC), this is one of the most interesting things I've read in awhile. Congrats all around.

--matt

Posted Sep 8, 2008 12:20:23 AM | link

Richard Bartle says:

I note that you encouraged people to participate in your survey by offering them a "Greatstaff of the Sun Serpent" if they did.

I realise that it's probably a bit late to do anything about it now, but shouldn't you have offered a selection of rewards? Some players may not really care much about owning a rare item (which is kind of an achiever thing) and it wouldn't have attracted them; however, they might have been keen to participate if they got (say) guild status points.

Richard

Posted Sep 8, 2008 3:41:54 AM | link

James Bassett says:

Well done Dmitri and team! (and kudos to SOE)

It must have been a welcome change to have your hands on the real player data, instead of having to recreate it using chat or '/who' logging.

It would be interesting to see how all the past research (especially on WoW) would have differed if Blizzard was as forthcoming with its vaults of data.

Keep up the good work. Anticipating many more interesting papers to /follow

James

Posted Sep 8, 2008 5:25:33 AM | link

Vili Lehdonvirta says:

Wow, congratulations! Looking forward to reading this paper and the forthcoming ones.

Posted Sep 8, 2008 10:48:42 AM | link

Sara Jensen Schubert says:

This is excellent. Congratulations!

"Some players may not really care much about owning a rare item (which is kind of an achiever thing) and it wouldn't have attracted them"

I'm also concerned about the item reward potentially skewing the sample towards achievers.

Posted Sep 8, 2008 11:53:48 AM | link

Dmitri Williams says:

Matt-

Can you spell out exactly why and under what circumstances you don't think the results here would generalize elsewhere? The fantasy RPG seems to be about 85% of the MMO market (from Sir Bruce and Voig's data), and the basic elements from game to game are similar, if not exactly the same. I don't know what % are subscription based vs. transaction based or free. If these are large differences, is age the only difference you'd suspect, or would there be something else?

On the BMI measure, the question is always whether one measure can be trusted without corroboration. In our case, we'd of course love to get the players with a skin-fat caliper to be accurate, or even better, to use a water displacement tank. But, well, yeah, that's not going to happen. So what we did do is to ask them about their health and exercise, and here we have corroboration for the fitness finding, i.e. they appear to be healthier than average because they exercise more than average.

The second aspect you mentioned is inaccurate self-reporting about weight. I am sure it does happen. However, the issue is less about whether it happens and more about whether it would happen systematically more with gamers than with the general population. The questions were exact copies of those used in the national measures we were comparing with (simply asking weight and height as two different questions, and then developing BMI on our own). Is there a reason to think that game players would answer differently than those taking an NIH survey using the same question--both done with anonymity? I can't think of one, but I'm open to hearing possibilities. If anything, I expect the gamers to be more honest because their data are collected with greater anonymity.


Richard-
Yes, we had the same incentive for everyone, which opens up a range of possible problems in that some will find it more appealing than others. However, in our case, the item was made by SOE just for the study, and it was given attributes to be equally desirable to all classes and levels. That doesn't mean it would be valuable to, say, role players or socializer types probably. But the perfectly-tailored method also introduces problems in that it might over-compensate some groups, too. So, there's no perfect answer. What we normally do is to offer people cash, which has its own pros and cons.

What I can say is that our response rate was the best I personally ever seen in survey work. I can't divulge the numbers due to NDA, but it was *extremely* good, indicating that the incentive used was quite effective, and making it less likely to have under-reported a particular group. Can I know this with 100% certainty? No, but the response quality far exceeds the standards used by most of the polls we read in the papers. The only way to be 100% sure is to have a perfect census as a starting point, where you can then compare the sample against it post-hoc. Unfortunately, there is no such animal in our business.


Posted Sep 8, 2008 2:46:41 PM | link

JC says:

I participated in the survey. The reward of the staff was irrelevant to me. Sure it's nice looking in a screenshot, but it's only a level 4 item, as I recall it, so no one who's played more than about 10 hours would ever actually use it for anything *other* than a screenshot.

Frankly, it was so long ago, all I remember is that I took the survey, not what was in it. I've not yet read through the 1st paper, but I'm very interested to do so, since EQ2 is still my "game of choice" for the most part.

Posted Sep 9, 2008 1:43:13 AM | link

Neils Clark says:

Though according to this data EQ BMI is lower on average, given the high standard deviation (8.19) it would be interesting in future work to see graphical distributions for adult EQ BMI as they compare to the American adult population.

Especially with adolescent and child BMI, where the EQ average is lower, the SD higher (10.2 for EQ as compared to 1.33 National average), it would be interesting to see more specifically where the outliers lie.

Also, I apologize if I missed this somewhere (was kind of busy eating a big mac while I read it all), did you guys restrict any questions to US respondents?

Posted Sep 9, 2008 6:59:09 AM | link

Timothy Burke says:

Fascinating stuff, and a major accomplishment for you guys. Also, as others have said, SOE should feel proud for assisting, and not just as a public service: this really should help them and help the industry in practical terms, besides a lot of interesting intellectual questions that come out of this.

I see what Thomas was getting at, and I think it's just one of those points where two different epistemologies go for a walk in the woods together and their paths start to diverge in subtle but meaningful ways. Broadly speaking, I tend to come from where Thomas does on this: what people report about their own motivations is a very important piece of data, but it shouldn't be mistake for "what actually drives or causes particular practices in the world". From my point of view, it's not that there's some other metric that one should substitute instead, but that underlying motivations, causes, etc. are to my mind something we should hesitate to decompose and quantify one item at a time. As a metaphor (rather than exact analogy), try Heisenberg's Uncertainty Principle: you can't know a particle's trajectory and location at the same time, knowing one precludes knowing the other. Motivations strike me as always being composite, always containing contradictions, in some sense, always being a bit mysterious.

If someone asks me why I study games, my answers (even if confined to survey responses) depend a bit on who is asking and on the circumstances I'm in at the time I answer. Moreover, sometimes posing a question like that to me makes me have to think about something I rarely think about, and to invent or settle upon an answer which on later reflection I might decide is something else entirely. In other words, it's not as if I have a transparent understanding of my own motivations.

This is not usually a happy conversation when it goes on between more humanistic, muddy-thinking scholars from history, cultural anthropology or cultural studies and more systematic kinds of social science. Maybe a more satisfying version of this kind of discussion that bears on this particular point is the debate going on between social psychology and experimental economics on one hand and neclassical economics and cognate forms of political science & sociology on the other. A lot of modelling in the latter tradition depends on human motivation broadly conforming to the rat choice, utility-maximizing sketch; the former group of social scientists, using also quite rigorous quantitative and experimental instruments, argues that not only do human agents not work that way, but that human agents may understand their own motivations and decision-making processes very poorly.

None of this means that models aren't useful, and that people answering questions about their motivations don't provide useful data. I guess it's just that this seems to me to be information that is very different in its character from the rest of the study, and the claims which arise from it are necessarily fuzzier and messier.

Posted Sep 9, 2008 4:14:36 PM | link

Dmitri Williams says:

Keep in mind that social scientists also give an actual numerical qualification for the strength of the findings. We acknowledge that there is a proportion of the findings that we cannot account for, and sometimes this is small and sometimes large. So, we say precisely how fuzzy it is and let the reader make their own judgments about it. Meanwhile (when we are being careful) we limit the strength of our claims when we have highly fuzzy outcomes. Our assumption then is exactly what you were getting at: our model may be incomplete in terms of what we capture, when we capture it, or the stability of the thing that we were trying to capture. All true. You can all read the paper, see the "fuzziness" numbers we report (or indeed, ask me for more) and make an informed judgment. This is standard practice.

Neils:
Any ideas on why the SD would be wider for the players? I don't think it's an artifact of sample size because we had 7,000 respondents. So why would there be a greater distribution?

The servers we used were US-oriented, as EQ2 has servers zoned for countries and regions. We captured a small group of foreign players anyway of course, but removed them for the analyses that lead to comparisons with national US data.

Posted Sep 9, 2008 6:02:37 PM | link

Timothy Burke says:

I didn't really mean the conventional style of allowing that some measurements are fuzzy--I think there's a deeper epistemological problem here about what we understand motivation, etcetera to be.

Epistemological disagreements are ok! I think it's best that they be understood as such, though, rather than quibbling about whether such-and-such measurement has been correctly described or accounted for properly.

Posted Sep 9, 2008 8:20:35 PM | link

Neils Clark says:

Without seeing more specifically how the outliers plot, it's hard to venture a guess. Where the average is combined with such a high SD, though, could the data suggest that EQ players (child, adolescent or adult) are more likely to be further underweight or overweight - whereas those in the national averages sit more at a "healthy" (albeit higher) median?

I'm not sure, that's why I'm curious to see how, graphically, that data plots.

Posted Sep 9, 2008 10:12:40 PM | link

dmyers says:

Hmm, this is very interesting and all that. And I will be looking forward to the data.

But let me speak to the data for a moment.

It strikes me that this sort of relationship between SOE and the research team is, in fact, proprietary. That is, what access do others have to this reportedly mountain of data to draw conclusions and interpretations of their own?

If, for instance, some research team is planning to build their research careers (not just here but in some analogous instance of the future) on such data, to what extent do other researchers have the ability to tap this same data faucet?

Now I could well be missing the point entirely (my own interest, after all, remains in the phenomenological experience of play rather than in the psycho/demographics/sociometrics of the users/players), but one could say (well, I could say) that recommendations to pursue this sort of research agenda would necessarily include recommendations to establish a patronage-like relationship with your nearest good ole buddy game designer/producer/magnate. And such patronage-like relationships have not always, based on television research history, proven conducive to enlightenment.

What, I wonder, would others say?

Posted Sep 10, 2008 2:37:00 PM | link

Michael Hartman says:

First, I have to echo the general sentiment that this is very interesting research and I look forward to reading more as it comes available.

Second, I imagine Dmitri (and any reasonable social scientist) acknowledges that this kind of research is imperfect. As he mentions, that is why they try to estimate how "fuzzy" it is, and let people judge the findings as they will. This is not physics or chemistry, nor will it ever be. You cannot hold social science to the same standard as hard science, nor should you. The value comes from knowing SOMETHING rather than nothing, while always keeping in mind that the research and findings are never going to be perfect.

Third, I was happy to see so many of my general beliefs about the gaming population confirmed. Gamers are exceptional people!

Posted Sep 10, 2008 3:58:23 PM | link

Thomas Malaby says:

@Michael: The issue is not whether Dmitri acknowledges that his claims about his model are imperfect -- of course he does. As Tim said, the concern is not with the variable reliability of that model, but with the further claim that, whatever its degree of perfection, it is not just a model, but an (imperfect) account of the *actual* motivations that are *actually* in players' heads. Those are two different kinds of issues. The latter, as Tim said, is the epistemological issue -- not an objection about the degree of reliability of the claims, but rather the scope of the claims themselves, and specifically how the authors here take reported motivations as proof that their model is more than a model, but in fact an answer to the "why" question.

Posted Sep 10, 2008 5:50:50 PM | link

Michael Chui says:

It strikes me that this sort of relationship between SOE and the research team is, in fact, proprietary. That is, what access do others have to this reportedly mountain of data to draw conclusions and interpretations of their own?

Oddly enough, I seem to recall a mention of this during a talk I attended over a year ago (IIRC, it was on a new method of data modeling), and I remember a specific reference to the fact that the data is not available to people outside the research team.

It's up to Dimitri et. al. to actually answer the question, but based on that memory plus the fact that he said "I can't divulge numbers due to NDA", I think it's exclusive.

Props to Dimitri, Nick, and Scott. Great start.

Posted Sep 11, 2008 12:02:35 AM | link

steven says:

Well I think that taking a data is much temporary these days as its a thinking of people which generally changes from time to time within the people with thier surrounding condition,
and its pretty fimiliar that at every stage a person has its own capability.
==================
Steven

www.google.com

Posted Sep 11, 2008 5:57:45 PM | link

Dmitri Williams says:

We are indeed under NDA, but the intent is to protect internal business practices at SOE, not to limit the transparency of our work. In other words, things like server loads are not part of our reports and will never get shared, but the data analysis is intended to be as open as any other project.

So, if someone has a real interest in fact-checking our data or engaging in replications, they are welcome to get in touch with me. If they sign the same NDA, they can review the data. I agree that that's a key component of science.

Posted Sep 11, 2008 6:05:37 PM | link

Ben says:

I'm sure the authors have already thought of this but one relationship that seems obvious to me regarding the scientific paper and its comments about the physical health of gamers vs that of TV watchers is the effect of advertising on TV watchers.

Surely there there would be some relationship between TV watching and eating habits regarding healthy food. Therefore gamers watching less TV than average would result in them viewing fewer adverts encouraging them to eat poorly...

But i'm no expert.

Posted Sep 11, 2008 11:52:54 PM | link

Dmitri Williams says:

Ben, that sounds obvious now that you say it, but it hadn't occurred to me. TYVM for sharing the idea!

BTW, I neglected to mention the last EQ2 team that transitioned from Raph to the current excellent folks there. Without Scot Hartsman and Warren Bartolme, this probably all would have died an ignominious death.

Posted Sep 14, 2008 11:10:25 PM | link

Matt Mihaly says:

Dmitri Williams wrote:

Can you spell out exactly why and under what circumstances you don't think the results here would generalize elsewhere?

It's more that I don't see any evidence that they DO generalize elsewhere and in the absence of that evidence I don't think it's reasonable to assume they do.

It's like standing outside a particular Italian restaurant, surveying the people who eat there, and generalizing those results to all patrons of Italian restaurants or all patrons of all restaurants.

In this case, the study focused very specifically on mainly adult Westerners willing to subscribe to an online fantasy game. Runescape is far, far more popular than Everquest 2, for instance, and I don't see how you could generalize a single bit of that data from Everquest 2 into Runescape as the audiences are so different.



The fantasy RPG seems to be about 85% of the MMO market (from Sir Bruce and Voig's data), and the basic elements from game to game are similar, if not exactly the same.

Sure, and the basic elements of restaurants are similar from restaurant to restaurant (they have menus, waiters, kitchens, etc) but you'll find drastically different customer demographics from restaurant to restaurant.



I don't know what % are subscription based vs. transaction based or free. If these are large differences, is age the only difference you'd suspect, or would there be something else?

It's not just age. I'll bet there is almost no overlap between the audience for, say, Runescape and Everquest 2.

To use another analogy, does surveying people who watch one of the very popular CSI shows tell you much about people who prefer Masterpiece Theatre or Sesame Street?

--matt

Posted Sep 15, 2008 3:27:17 PM | link

Dmitri Williams says:

Matt, I'm with you on the analogy and I'm willing to agree that EQ2 might well not generalize to Runescape or many other virtual worlds. But I disagree on the analogy insofar as you would have us believe that Italian restaurants have nothing in common. They do have some basic similarities beyond having a kitchen. And surely they are more similar to one another than they are to French bistros or delis.

If I can torture your analogies a step further, I am suggested that EQ2 may well be CSI as you say, and I'm wondering about its viewers and how they compare to viewers of Numb3rs, NCIS, CSI: Miami, CSI: New York, Law and Order, etc., i.e. procedural law enforcement shows.

So let's say that we're talking cop shows, not all shows. I'm more than fine with that. In fact, it's the argument I often make to colleagues who don't know GTA from Bioshock.

Now, what I'm interested in are the variables and factors on which the players of one DIKU-style MMO might or might not generalize to another. In other words, in our worlds, what makes the cop show different from the medical show?

I would not expect, for example, that these results would likely apply to Runescape players, residents of There, or Club Penguin players. I would guess that they *might* apply to WoW, DaOC, Warhammer, etc., in part because the marketing of these is relatively similar (attracting a similar crowd) and the game mechanics are relatively similar (retaining that crowd in parallel). It's pure speculation, of course, until other companies release similar data.

What I'd love to hear is why EQ2 players are or aren't likely to match players of WoW, Lineage, Guild Wars, etc., etc. And if there are clear reasons why they would also differ from Runescape players, that'd be great to hear also. If it's a cultural difference, how would it play out? Would the payment model predict a different audience? I'm really just asking for wild speculation from a very informed readership.

Posted Sep 15, 2008 10:18:59 PM | link

Matt Mihaly says:

Dmitri wrote:

Matt, I'm with you on the analogy and I'm willing to agree that EQ2 might well not generalize to Runescape or many other virtual worlds. But I disagree on the analogy insofar as you would have us believe that Italian restaurants have nothing in common. They do have some basic similarities beyond having a kitchen. And surely they are more similar to one another than they are to French bistros or delis.

Are they more similar to one another than they are to French bistros or delis? Perhaps in general, but any specific Italian restaurant could easily have more in common with a French bistro than most other Italian restaurants. The food's general regional origins are just one factor of a bunch of factors that create the full picture.

Similarly, it's easy to see how a particular fantasy MMO game could be far more similar to a particular MMO set in 18th century India than to another fantasy game. Anarchy Online (sci-fi) has more in common with WoW (fantasy) than it does with EVE, for instance.

Anyway, again, I'm not saying that they don't have fundamentals in common. I'm just saying that there's no reason to assume that a given Italian restaurant's clientele translates to another Italian restaurant's clientele. A Sicilian restaurant attracts a different crowd from a Tuscan restaurant but they're both Italian restaurants.

More to the point, WoW attracts a different crowd from EQ2. How do I know? They have drastically different audience sizes. That, in and of itself, tells me that they're not going after the same audience, and that while we may find it easy to just whitewash them as the same, they're not. There is a reason that the people who play EQ2 in preference to WoW play EQ2 instead of WoW. It's not as if every single EQ2 player in existence doesn't know about WoW (though the reverse is likely not true). They choose to play EQ2 instead of WoW. That alone is significant and should tell you that WoW and EQ2 are not goods/services that may be substituted for each other.


Now, what I'm interested in are the variables and factors on which the players of one DIKU-style MMO might or might not generalize to another. In other words, in our worlds, what makes the cop show different from the medical show?

DIKU-style is a set of systems, not a genre, whereas cop show vs. medical show is genre rather than system. I'm really not sure what's more likely to attract a common set of players either: system or genre.

I guess what I'm saying is that there is more than one axis along which to measure content, and picking 'DIKU-style' or 'fantasy' or 'cop-show' or whatever is just way too broad.

Dmitri wrote:

I would not expect, for example, that these results would likely apply to Runescape players, residents of There, or Club Penguin players. I would guess that they *might* apply to WoW, DaOC, Warhammer, etc., in part because the marketing of these is relatively similar (attracting a similar crowd) and the game mechanics are relatively similar (retaining that crowd in parallel). It's pure speculation, of course, until other companies release similar data.

To pick one example out of those: There's no way this data can be reliably applied to WoW. World of Warcraft is immensely popular in Asia, and the other MMOs named are not, and not for lack of trying. Throw out WoW, and EQ2 and DaoC represent a very very small % of the virtual world playing population.

I hate myself for resorting to analogy so much, but what you're telling me sounds a lot like, "Well, her last love was a blonde surfer so it's a sure bet that she'll fall in love with my blonde surfer friend Brody." People are just more complicated than that, and even something like "DIKU fantasy" is SO much more complicated than the two words sound. They're extremely broad terms and even sticking a couple of them together doesn't really tell you much about what kind of audience the game is likely to attract. Imagine modifiers like 'text-based' or 'furry' or 'Vietnamese' thrown on before those two words, and think about how changed your perception of that DIKU fantasy game suddenly becomes.

Posted Sep 16, 2008 1:11:05 AM | link

Dmitri Williams says:

If I'm hearing this correctly, the major possible axes of difference are genre, marketing appeal and game mechanics, each of which might predict differences in the player base. Yes?

Posted Sep 16, 2008 2:24:54 PM | link

Matt Mihaly says:

Well, those are three of them but I'm sure we could come up with more. I also wonder how heavily a game's community factors in. For instance, two games could start with identical marketing and systems and end up with different communities. Those communities are, of course, part of the game and so the games themselves end up different and appealing to people that are different in some way.

Anyway, I really don't know how many different axes there might be, and how many you choose to use and how you weight them probably has a lot to do with how accurately you can find playerbases on games X and Y that truly do approximate each other.

I'm just trying to point out that each game and the resulting community is unique. Sampling one may or may not tell you about another and I don't think it's easy to determine whether two populations are similar by just picking a couple major game axes and seeing if the games are close to each other on those axes.

This isn't criticism of the study, btw. I realize only EQ2 let you have access to that kind of data. I think it's probably best not to build general theories about online game populations (or fantasy online game populations or whatever) based on a study of one community though.

--matt

Posted Sep 16, 2008 5:28:51 PM | link

gordon.calleja says:

Great work Dimitri! This probably falls under the epistemological differences discussion that Tim and Thomas have mentioned, but I would really appreciate an answer anyway.

In the statements you make about player motivations, immersion seems to function as a catch all category rather than a specific experiential phenomenon. In the factor loadings, for example, immersion covers role-playing, avatar customization, exploration (which seems to include both spatial/aesthetic exploration/appreciation and more generally other forms of unique knowledge not at all related to spatial exploration) and escapism.

How is customizing one's character more similar to spatial exploration than, say, socializing? Both are elements that can involve the player more in the virtual world, or if we want to be more precise with the use of immersion as a sense of habitation; both can contribute to feeling inside the environment. Immersion seems to be a higher order experiential phenomenon that includes achievement and sociality, not a catch all for other experiences outside of sociality and achievement.

It seems to me problematic and not incredibly useful (vis a vis the why question) to make decisive statements about motivations when two of the factors are definable while the third one isn't. How far does "Achievement
(M = 3.44, SD = .89) was rated as more important than immersion (M = 3.31,
SD = .87), which in turn was rated as more important than sociability (M = 3.16,
SD = .95)" really get us in understanding why players play when the immersion factor is so hazily defined?

Posted Sep 21, 2008 8:18:13 AM | link

Dmitri Williams says:

It'd be better if Nick jumped in since these are his scales and his validation work, but the idea is that there are several questions that all scale together and that we try to come up with a label that describes clusters of them. Factor analysis shows which questions go together, i.e. when question 3 goes up, so always do questions 5 and 8. So looking at 3, 5 and 8, we ask what about it do they have in common? If there is a common theme, we try to come up with a label that lets them hang together.

Of course, most of the time, we ask 3,5 and 8 purposefully, with the expectation that they will, in fact scale together.

So you've identified a cluster of questions that you don't think are properly labeled. I can see your point. Perhaps you can suggest a better label? Or maybe Nick can jump in and share more about the loadings--whether there were any shared between the immersion items and the others that suggest overlaps.

Posted Sep 22, 2008 12:50:48 PM | link

gordon.calleja says:

I don't see how useful it is to group together involvement related to spatial exploration, customization and role-playing since they address different experiential qualities.

I understand that immersion here is being used as a catch all label and one might say that a label is just a label. The problem is of course that when a label is already laden with meaning, as immersion is, it is not a neutral signifier (if any exist). Labels might start as such but they tend to start functioning as metaphors, which actively shape our understanding of a phenomenon.

Another concern I have with the approach is its reliance on questions which, as you say, have been devised purposefully. When it comes to the intense subjectivity of experiential domains (the whys) aren't you heavily influencing results by the way you choose and phrase questions? It seems to me that quantitative research can only take you so far into understanding the subjectivity of experience and answering those whys. I would argue that a stunning project such as this would be worth supplementing with qualitative work that can address experiential issues with a finer lens than self set questions with one direction answers allow.

Posted Sep 23, 2008 3:01:15 AM | link

Nick Yee says:

Gordon - The groupings of motivations were produced statistically and not combined in an ad-hoc manner. The reason why Role-Playing and Customization were grouped together is because they were more highly correlated with each other than either was to Socializing (for example). So when you ask:

How is customizing one's character more similar to spatial exploration than, say, socializing?

The answer is that the set of inter-correlations among the variables showed that pattern. And what the statistics show is that, contrary to your expectations, role-playing and customization are actually highly related.

Much of the statements used in the survey were derived from much earlier qualitative work. These include motivations suggested from earlier studies (such as Bartle's types) as well as from open-ended responses from my own surveys of online gamers.

You can get a sense of both the quantitative and qualitative work I've done in this area at: http://www.nickyee.com/daedalus/docs/tag.php?tagid=19&tag=motivations%20for%20play

So as you can see, I agree with you that different methods allow us to look at the same phenomena in different ways (with their inherent strengths and weaknesses) and have tried to use insights from qualitative data to inform how large problems might be constrained in ways such that they might be explored quantitatively as well.

Posted Sep 23, 2008 3:40:55 PM | link

gordon.calleja says:

That still does not answer my core concern. Why do you use an already meaning laden term like immersion to represent such varying phenomena? This suggests that immersion is made up of exploration, character customization, role-playing etc... and that seems to me a reduced version of the experiential phenomenon which the term has been deployed to signify in the context of virtual environments. If this is "just" a label I would urge you to reconsider using a label which has such strong connotations and an established meaning in the field(s) as it runs the risk of creating situations where researchers citing your invaluable work to simply state that the conclusions you make about immersion represents the experiential phenomenon in itself, not the label of a number of correlating factors.

I hope you don't think I m nitpicking here. The more humanistically oriented discussions of immersion are thorny for a very good reason. They bring in, at some point or other, a conversation touching upon the complexity of subjective consciousness and the question of being for which we have very few quantifiable answers, at least at this stage.

Posted Sep 24, 2008 6:23:11 PM | link

Isaac Knowles says:

Matt said: They choose to play EQ2 instead of WoW. That alone is significant and should tell you that WoW and EQ2 are not goods/services that may be substituted for each other.


I see what you're saying, but I think you run the risk of overstating the demographic differences between players of different games by looking only at game choice. Once you make a choice about which MMO to play, then the longer you play that game the more time and money (software and subscription costs) you invest into your toon(s) and relationships with others. The longer you play the game the higher costs of exit become.

So while I agree that there are probably a multitude of reasons that the population of one game is demographically different from that of another, I would put far less weight on the idea that the choice of one game rather than another (especially between two similar fantasy mmo's) reveals much of anything about player attributes.

Posted Sep 25, 2008 12:33:48 PM | link

Sandy Kearney says:

Dmitri

Congrats on the success and launch of this tremendous effort!

I truly hope we can get the world to integrate Social Science in this new medium, a goal I have on my list as you know from 2006.

Imagine the possibilities if one doesn't have to take this journey back in time although, traversing the Grand Canyon of MMORPG Data seems like it has a spectacular view!

Posted Sep 29, 2008 10:36:18 AM | link

jeims says:

very good

Posted Feb 5, 2009 5:26:36 AM | link