Role Player Study released

Given that they are the titular players, they don't get much attention, do they?

So, recently accepted for publication, and now cleared for posting on the Interwebs is our paper (no, sorry, not connected to the movie dotting this page) "Behind the Avatar: The Patterns, Practices and Functions of Role Playing in MMOs." You can find a pre-press copy here.

The paper combines the big trove of server-side data and quant analysis of our other EQII papers, with a full-on second step of participant observation and ethnographic interviewing. The result, we hope, is a pretty deep look into who role players are, why they play, and what makes them tick. The chocolate and peanut butter of combined qual-quant methods we think gives the paper good generalizability, but with depth to boot. As always, there were some obvious findings and some unexpected stuff, below the fold:

 

(These are all findings from EQII, so no, we don’t know about whether they generalize to other titles.)

Role players are a small fraction of the total player base. The most avid RPers are only about 5% of the players.

Role players role play regardless of the server’s label. They are basically playing another game altogether.

Role players come much more often from offline marginalized groups, suggesting that some may engage in the practice to find acceptance or a safe outlet for their identity.

Role players engage in the practice for a number of reasons, but the standout one tended to be for creativity. Escapism was present, but was rarely the main reason.

Full reference:
Williams, D., T. Kennedy & R. Moore (in press). Behind the Avatar: The Patterns, Practices and Functions of Role Playing in MMOs. Games & Culture.

 

Comments

[David Stocker]

1.

A very interesting paper. The hints of lower mental health give me pause as that confirms a negative stereotype that roleplayers have. It would be interesting to see how the mental health indicators compare with age within the roleplaying population. I'd expect to see older roleplayers being more balanced and confident.

I’d also be interested in seeing the differences between self identified dramists and simulationists; both of whom regard immersion as a motivator and would be drawn to roleplay, but from very different angles.

[Dooder]

2.

At a glance, I'd say the stats are generalizable. RPing has been increasingly diluted since MUD days, to the extent that they're often a persecuted group today. You may be playing an MMORPG, but it's hard to think of an MMO that was designed with RPing in mind. Whatever class, race, and in-game background you might be on the surface, everyone's still playing the make-friends-and-perform-rituals-of-accumulation-game. I'd wonder whether RPing communities are commonly more accepting of already discriminated groups within the game-world, but also if today these findings are less common as just about every meta-game practice is supported by a larger external community that carries its values and practices to a variety of games in little semi-autonomous enclaves.

For the former, I believe it is true. In the games I've been a part of that have had strong RP factions, they have been very open to players that had been rejected by most other players. I had assumed that was a recruitment practice to keep the small community alive, but it also was a logical fixture in their good vs. evil world-view. In any case, it was still 2002-05, and I've found the latter to be a more imposing phenomenon regarding ways of playing that aren't structurally supported in a virtual world. A large forum community can take their values, mores, and practices with them wherever they go.

[Jythri]

3.

Wow, I can't help shaking the feeling, as I skim this article, that we've been asked to provide comments on a finely painted portrait, and then spent an hour talking about the chemical compounds that make the paint pigments a particular sheen of green.

My simple observation on roleplaying in MMOs is this: players will do what you reward them to do. A few players (5% sounds about right) will step out of the box and run against the grain, having fun in their own way. But (and speaking as a frustrated old-school RPer) one of the main reasons folks don't muster the effort anymore is that it's hard to be a constant source of self reward.

RP is fun for those who enjoy it, yep. But it's hard, for example, in WoW to spend an evening on furthering your troll guilds' cultural causes when the same amount of time questing will get you a nice-looking robe, which actually contributes more tangibly to your sense of character identity. I've watched RPers in Wow slowly erode into the core gameplay that game advocates. In fact, despite strong efforts and good intentions, those RPers almost always shift back to other games which are more conducive to rewarding social interactions.

So, you can analyze the whys and wherefores. You can take a close look at personalities, sense of belonging, and psychological whatnot. We can even all recognize the astounding fact that there are creative people who will be creative in whatever they do.

But I believe the simple truth is this: the amount of RPing that is sustained year-to-year is directly proportional to the tangible rewards and support provided by the MMO for roleplaying. If it's fun, some people will do it. If it's fun AND rewarded, many people will do it.

The GAUNTLET is thrown: I dare a developer to be daring enough to design and produce a modern MMO that rewards social and story development. At some core, all of us are initially attracted to RPGs because we want to be in control of the main character from our favorite fantasy series.

I can't wait until we break out of WoWs spell, and actually advance the medium again.

Cheers from an old bard.

- Jythri

[Mandolin Bee]

4.

I have to agree with Jythri's assessment. I don't really think the study is particularly valid. To compare roleplayers, you'd have to look at venues where the reward is greater for social interaction.

SWG-PreCU had a strong roleplayer following. Skills were so wide open, and being a pure craftsman was possible, making an entire social aspect of the game not only satisfying, but necessary for the more combat-oriented players.

EVE-Online... even people who don't like roleplay end up roleplaying in EVE, whether they know it or not. How would this report treat that phenomenon? In my experience, people identify quite strongly with their persona in EVE... whether they are pirates, vigilantes or miners.

MUDs: OK, so MUDs are pretty much dead. However, they're not completely gone. A few of the best MUDs out there are produced by Iron Realms Entertainment. I would suggest exploring the social aspects of their games to discover the draw and rewards of roleplay. IMHO, if such a deep social aspect was somehow translated into an MMO experience, your 5% would rise to a far more competitive percentage with the rest of the player base.

Personally, I always considered myself a roleplayer, and had no problem finding other RPers in the early days of MMOs. These days, I have simply stopped looking. As much as I enjoy RP, I'm in the same boat as Jythri. There's no incentive, and I have been shuffled into the basic mechanic of "farm for lootz".

I get my roleplay in MUDs and in real life (pen n paper) now.

[Thomas]

5.

Also, you can't change the world, which renders most roleplay completely moot.

[faustusnotes]

6.

I'd like to comment on the papers' stats. It seems that there are some flaws:

* you claim the survey is a "stratified random sample" taken on 4 strata (4 different servers) but there is no evidence in the analysis that the stratified random sample has been taken into account
* I can't find a response rate for the overall survey or the servers.
* The differences in the groups are in some instances very small and only significant due to the large numbers in the survey. Consider the loneliness scale: high role-players differ from the low ones by 2 points on a scale of 4 to 80 (about 2.5%), which is not a big difference no matter how significant it might be. It appears that there was only 1 woman in the High RP group (out of 300 or so people!) but the gender difference between this group and the medium RP Group was statistically significant!
* There is no multiple regression analysis, so no adjustment for confounders. Given the supposedly significant demographic differences between groups, it might be wise to have done this. Particularly, adjustment for the 7 categories of education, and for social marginalisation, might have removed the mental health differences between groups
* Mental health appears to be estimated by a form of self-report. This is always a dubious measure.

Have I misread the paper?

[Dmitri Williams]

7.

I can't give the response rates because SOE doesn't want to disclose their server loads, and those numbers would let people back into them.

Not sure what the issue is with using 4 servers. That's what we did, so we could compare across them.

On significance vs. substance, we report a statistic in the paper known as Cohen's d. Because we have a large sample, everything is significant and we don't want that to be confused with substantive. So "d" reports the rough effects size, after allowing for sample size. That lets readers interpret and draw their own conclusions about what differences are small or large.

If there's a multiple regression model you have in mind, let me know and I can run it. Regression assumes that there is a dependent variable and that wasn't the test we were doing. We were doing simple comparisons between groups, so the ANOVAs are the most sensible tool to use. Also, as you probably know, regression isn't good for categorical variables like the education one.

Mental health was indeed drawn from a series of self-report items. There were a series of questions asking "Have you ever been diagnosed with ___?" On the one hand, these are sensitive issues and we might expect some deception. On the other hand, these were anonymous surveys and these are the exact items used by the national surveys done by NIMH. They were also consistent and scaled with each other and had the correlations we'd expect.

Have I misread your questions?

[faustusnotes]

8.

Thanks for the reply, the questions aren't misread at all. The 4 servers should be an issue because, as a stratified random sample, ideally the numbers of people selected in each server should be proportional to population, and then a probability sample survey adjustment would be required to adjust for the sample design. If you don't know the server load you can't do this, but presumably sony could have given you an estimate of a weight.

Do you know if the response rates were good or bad? Low response rates in a survey asking about mental health are a bad sign, particularly if one group in your final analysis is very unusual (like the high RP group).

I don't like Cohen's D as a measure of substance because it doesn't tell you anything that isn't obvious from the difference in means and the standard error. In this case, for example, those scales on loneliness differed between groups by 2.5% of the scale (and 5% of the mean value of the medium group) but were shown to be a "medium" effect by the d-statistic. Looking at the raw means, it's clear that the difference in loneliness is not interesting.

The differences in the three main measures of mental health (depression etc.) were bigger. But given the demographic differences a regression might adjust for these confounders - for example, a logistic regression of likelihood of mental health issue x vs. age, education level, role-playing level, sex, sexual orientation, etc.

My suspicion is that your role-playing group's mental health problems may be related to these confounders rather than the role-playing element of the individuals' personalities. And the confounders are a sensitive issue if there's a low response rate. I can't remember if the paper says whether the respondents were representative of the total server populations?

Anyway, thanks for answering the questions and I hope you haven't already been through all of them in peer review!!!

[Dmitri Williams]

9.

Response rates were very good--among the best I've seen. I can't go further, unfortunately.

I'll leave you to Cohen to dispute the usefulness or accuracy of "d." I like it, especially for large datasets, but to each his own. I think it's notable that the difference between the low and high PR groups shows the high RP players to be lonelier, but, no, it's certainly not a big difference.

Your comment on the various mental health measures is worth talking over. It's not my speculation that there is a causal relationship with RP creating mental health problems. I can't say anything about causality in these data, of course. I'd say that people with mental health issues are more likely to RP, and we spend some time in the discussion on that. Seeing the various measures correlate isn't surprising to anyone, I'm sure. We reported them more to give a sense that this was a multiple-measure indicator of generally lower mental health.

Still not sure what you want from the regression. If it's a bivariate regression of mental health vs. those various other measures, that's essentially the same as a correlation and those are reported in the paper in the first table.

[faustusnotes]

10.

a high response rate is certainly interesting! Though for social science papers "high" can still be undesirably low compared to, say, probability sample health surveys.

A bivariate regression of mental health vs. other measures isn't quite the same as a correlation for binary variables, I think, but I was more interested in a multiple regression (which is not the same as a correlation). The conclusion of the paper seems to be that people with mental health issues are more likely to rp; but this doesn't seem to have been adjusted for the confounding influence of sex, age, etc. Particularly, rp-ers were more likely to be from minority groups and non-heterosexual; we know these groups have higher rates of certain mental health problems (e.g. substance abuse), so before we conclude this about rp-ers we need to test if it is just their demographic background which explains the difference. A multiple logistic regression would help to show this.

[Dmitri Williams]

11.

DV role playing (scalar, not binary, so linear regression, not logit):
F = 23.163, Model sig .000
B's:
Age: .006, .013.
Gender (female): .132, .015
Education (1-5 scale): .020, .172
Non-straight: .584, .000
Minority: .300, .000

In other words, being older or female made a slight difference, while education did not. Being non-straight or a minority made a large difference. This is consistent with what we reported in the paper. See Figure B.

[faustusnotes]

12.

Ooh, you ran a regression, but it's linear on a scale with only 7 values, right?

I was thinking of testing the odds of having a mental health issue (e.g. one of the two with a significantly higher proportion amongst rp-ers) by role-playing, age, gender, etc.

Using role-playing as the DV when it has only 7 values probably won't produce a very statistically robust model, though some might argue that with 7000 observations you're doing okay. I expect the model in your comment to reproduce the results of the paper (give or take a few minor details). But I'm more interested in the finding of mental health problems being more prevalent amongst role-players, and think this needs to be adjusted for the confounders of minority, non-straight, etc.

Thanks for your attention to my comments! I appreciate you must be very busy though, so there's really no need to run new analyses for a random commenter!

[Dmitri Williams]

13.

Tell me exactly what model you want to see and I'll run it.

[SusanC]

14.

(For what it's worth, some of my collegues recently neglected to do a multiple regression in their paper, and got a lot of grief for this omission from the audience when they presented it. The subject matter wasn't anything to do with online games, but the statistical point seems to be much the same).

I, too, would like to see a multiple regression along the line of:

mental health problems = alpha + beta.(role playing) + gamma.(sexual orientation) + delta.(minority group)

Alternatively, you could use discriminant analysis: find the hyperplane in (role playing, sexual orientation, minority group) space that best separates those with mental health problems from those without.

[Dmitri Williams]

15.

I'm happy to run analyses for discussion, and it turns out that I am familiar with the pros and cons of multiple regression models. Please bear in mind that I am not a complete idiot (partial, perhaps) and that there are usually good reasons for running or not running particular flavors of analyses within the constraints of a journal article. Chief among there are the space taken when presenting two full studies and the clearest way to present answer to the paper's questions (which are different than the ones posed here).

Here are the models for predicting mental health outcomes, with betas given. All models have large F scores, but generally low r-squares and for statistical Achen-types, moderate to large standard errors. I think the simple correlations table is more illustrative of the inferences you can make.

For depression:
Role playing .045
Non-straight .11
Minority -.02 (ns)

For anxiety:
Role playing .018 (ns)
Non-straight .094
Minority -.001 (ns)

For substance addiction:
Role playing .007 (ns)
Non-straight .044
Minority .05

For behavioral addiction:
Role playing .023 (p < .10)
Non-straight .062
Minority .089

For ADD
Role playing .031
Non-straight .102
Minority .02 (ns)

[SusanC]

16.

Thanks, Dmitri! (And I apologize if my post came across as sounding critical; I didn't mean it to).

It's interesting that roleplaying is a risk factor for depression. (Of course, it's not clear what the direction of causation is).

[Dmitri Williams]

17.

Right, there's no way to tell from these data if depressed people are slightly more likely to role play, or if role play makes people slightly more depressed, or both. From the interview portion, I think our sense is that the first case is the most likely, but it would take a controlled experiment to prove it.

[faustusnotes]

18.

a controlled experiment! Let's force a bunch of kids to play tunnels and trolls till they go mad!

Are these linear or logistic regressions, Dmitri?

[Dmitri Williams]

19.

Linear.

[faustusnotes]

20.

so for example going from a role-playing score of 1 to 7 increases your ADD scale score by about 0.25? Is this a standardised scale or a numerical scale? If the latter the effect is very small, yes? Or is the DV binary?

[Dmitri Williams]

21.

Those are standardized Betas so you can see the relative impacts. They are not b's, which let you see the precise relationship from scale to DV.

The DV's are all binary, as they are answers to the question, "have you ever been diagnosed with ___?"

I would call these relationships small to medium sized. If you check the paper, you'll see the differences across the board between low and high RPers on these various conditions. It's all in a table.

[faustusnotes]

22.

A linear regression on binary dependent variables? When some of those variables have probabilities close to 0, and subgroup sizes less than a couple of thousand? I think a logistic regression giving the odds of having the condition would be statistically more valid.

[Dmitri Williams]

23.

It would be, but it's not like these coefficients are backwards or something. If anyone cares a lot, I'll re-run the models. I still suggest reading the paper and seeing the table that has all of this already laid out.

[oomu]

24.

Jythri is right. It's exactly what I experienced for years and years.

People needs to be rewarded. RP can be fun for _many_ people, but it needs rewards, it needs to result in something in the very game (object, new settings, cloth, specials quests, whatever)

without rewards, slowly, people are pushed to the Real Game. For example in wow : instances and raids (and quests or pvp, okay).