Review of "The Effect of Videogame Violence on Physiological Desensitization to Real-World Violence," by Nicholas L. Carnagey, Craig A. Anderson, and Brad J. Bushman, Journal of Experimental Social Psychology, 2006. Many thanks to Andy Havens for sending us this link.
The authors present the result of an experiment in which 257 college students played video games and then were exposed to films of real-world violence. Heart rate (HR) and skin conductance (sweat; SC) were measured. These unconscious measures are accepted as valid indicators of arousal. In the main experimental condition, subjects were separated into two groups, one that played a violent video game and one that player a non-violent video game. HRs and SCs were measured before the experiment, after video game play, and after films of real-world violence. The data indicate that for those who played nonviolent video games, arousal rose during video game play, and again during the violent films. Those who played violent video games experienced an increase in arousal during the video games, and a decrease during the violent films. The authors argue that these patterns are significant, and indicate that those who play violent video games become desensitized to violence.
While the experiment appears to have been competently conducted, the statistical treatments that lead the authors to conclusions of significant desensitization collectively constitute a veritable handbook of quantitative error and deception. This review can only select some of the more egregious failures. Unfortunately, the abysmal quantitative skills evidenced here are in fact all too common in this literature and most of its close relatives. Entire disciplines seem to believe that they are discovering things, when it seems to this reviewer that they are making up their discoveries as they go along.
In particular, the primary failings one sees again and again in this literature are:
1. Failure to honestly and fully report data
2. Failure to distinguish statistical from substantive significance
3. Failure to develop statistics relevant to prior theory
4. Failure to draw careful and appropriate policy inferences
The Carnagey et. al heart rate data will be used as examples of each failing.
1. The authors do not fully and honestly report their heart rate data. Six small numbers are all that's necessary: the authors should simply tell the reader what they found. What were the average heart rates for people who played violent or non-violent games, at the three times in the experiment? They bury four of these numbers in parenthesized text on page 5, but some digging reveals: Heart rate before game play, 66.4 beats per minute (bpm) and 65.5 bpm for violent and non-violent games; and heart rate after game play but before film, 69.3 and 68.4. The authors do not report heart rate after the film, which of course is the most important set of numbers. Instead, the reader is referred to a figure (p. 5). As best one can tell, heart rate after the film looks like about 70.5 for the non-violents, and about 68.5 for the violents. Thus, the data are
Before experiment, after game, after film:
Violent: 66.4 69.3 68.5
Non-Violent: 65.5 68.4 70.5
It would also be necessary for the reader to see the measured standard deviations of the respondent heart rates across the respondent sample. Otherwise, it is impossible to tell whether these variations in heart rate averages are substantively significant (see below). Absent these data from Carnagey et al., we have to rely on outside data. It turns out that average heart rates at rest for human beings run between 65 and 70 beats per minute. A range of 50-100 is normal (source; source).
Instead of these numbers, the authors report their HR data using a single figure, indeed one that makes use of one of the most frequently denounced practices of statistical charlatans: the vertical axis is not grounded at zero. The range of HR values in the figure runs from 60 to 75 bpm - the entire range falls within the normal range for a human heart. On such a graph, of course, the numbers reported above do look like significantly different numbers. But it is a deception of significant magnitude, well worthy of an 'F' in an introductory econometrics course.
2. The authors cannot distinguish (or do not understand) the difference between statistical and substantive significance. An overview of similar follies can be found here. The idea is simply this: a long time ago, statisticians invented a certain kind of test for the relationship between the mean of a variable and its variance. They called the test a "test of statistical significance." If the mean turned out to be bigger than its standard deviation, they said that the mean was "statistically significant." In one of the most unfortunate linguistic twists imagineable, generations of quantitatively clumsy followers have morphed this notion into a general test of whether a variable is large or not. It's unfortunate becuase statistical significance is just an artifact of the data, it can never tell us whether a number matters or not. In fact, any statistic will become statistically significant if the sample is large enough; as the sample gets larger, the variance around the mean gets smaller, and presto! statistical significance. But theoretical questions don't change on the basis of whether the data set is big or small. The classic story is of an animal husbandry scholar who found that the length of hair on the left side of a sheep's back is statistically significantly longer than on the right side. Of course, he had data from thousands and thousands of sheep. Even a tiny difference in average length - 0.0001 inches - would become statistically significant if you measured hair on a million sheep. Yet the test of substantive significance - "is this a meaningful difference in hair length or not?" is absolutely independent of sample size. And substantive significance is all we should care about (see item 4 below).
Nonetheless, the bugbear of statistical significance is loose among poorly-grounded fields, among which one must now, on the basis of their acceptance of this paper, sorrowfully include experimental social psychology. It is common among bad statisticians to
a) Drop the word 'statistical' when referring to the significance of a finding
b) Report only statistical significance tests, not substantive significance tests
c) Report all statistically significant findings and ignore all statisticall insignificant ones
Carnagey et al. commit at least one of these three of these errors in every paragraph of their results discussion, and they frequently commit all three. Almost all of their statistical discussion is devoted to F-tests, which are statistical significance tests. On page 5 they assert, for example, that the difference in HR from game to film was "large" for both groups, inserting a parenthetical F-test result as their only support for that assertion. Large. The reader may judge: is the difference in HR of 69.3 to 68.5 bpm "large" by any standard of substantive significance? This reviewer does not think so, especially given the understanding that individuals may have HRs between 50 and 100 bpm and still be considered normal.
To report statistical significance tests as tests of substantive significance is more shamefully deceptive than the simple graph cheat identified in (1), but it is of the same color.
3. The authors fail to develop these statistics within the context of a sound theory. Carnagey et al. do spend a long time talking about theory, but interesting things start to happen once they apply their theories to the data. In their "Preliminary Analyses" on page 4, the authors describe what they call "significant" and "insignificant" modifiers to the study's results. One might theorize, for example, that prior exposure to video games might affect how HR responds. Or, perhaps being male or female might matter. Family background might make a difference. Since this is a random-assignment study, it's unlikely that these effects will be important. Still, the authors were careful to do a post-hoc assessment of the data along these kinds of lines. Where their practices turn shady, however, is when they conclude from "insignificance" of difference that a given variable can be completely dropped from the study. It probably does not need saying that the standard of significance here is statistical significance, and therefore this practice is to use the old statistical significance bugbear to substantively alter what is considered theoretically important, prior to the construction of the study's primary statistics. The proper procedure is to complete all theoretical reasoning prior to data manipulation. If theory suggests that a variable such as sex matters, it should be included in the entire analysis. The correct protocol for studying any effect is to embed it in a regression analysis so that its effect can be isolated while holding the effects of other variables constant. Again random-assignment is one way to do what regression is supposed to be doing, namely, to hold other factors at bay. But it is even better to do what Carnagey et al. apparently do, which is to approach the post-experimental data using regression as well. The bad practice comes in dropping entire variables from the analysis simply because some aspect of them was statistically insignificant at a prior step. If theory dictates that they matter, they should be in the final regression. To exclude them for some reason related to an ad hoc statistical significance test is another terribly bad practice; very likely, the inclusion of all theoretically-relevant variables in the analysis would make the HR differences reported even smaller.
4. The authors do not draw careful and appropriate policy inferences. The policy issue in this line of research is whether violent video games are so bad for us that our use of them should be controlled, either by governments, our loved ones, or ourselves. Carnagey et al. reveal themselves to be utterly insensate to this question. Rather, they conclude that any measurement of desensitization, so long as it passes a statistical significance test, is worthy of public notice. Returning to the heart rate data: playing a violent video game reduces heart rate when viewing subsequent violent content by two bpms. This indicates something about arousal. One can debate whether it is "significant". Let us assume it is. Does this arousal effect indicate a significant amount of desensitization to violence in the real world? Carnagey et al. apparently believe so, judging from the title of their paper. Should individuals therefore decrease their exposure to this content? That is indeed the implicit message running through this paper. That conclusion is far from being warranted, however. The true policy issue is this: would a significant decrease in exposure to violent video games lead to a significant decrease in real-world violence? The statistics in this paper do not support such a conclusion.
Indeed, because of its ham-handed and deceptive treatment of data, this paper probably should not have been published.
Hey Ted,
Ouch. I hope you are sending this in to the journal of experimental social psychology. A somewhat more sober version of this critique absolutely warrants publication and debate. This stuff has been getting a free ride for too long.
Posted by: Bart Simon | Jul 27, 2006 at 11:52
Sad to see this kind of sloppiness in the Journal of Experimental Social Psychology of all places. Makes me wonder about the analytical skill of not just the investigators but the primary reviewers too. The experimental design here seems pretty straightforward though; I'd say it's a great advanced undergraduate project.
I'd also add the conditions of having people from both groups watch non-violent videos or do some other form of passive viewing or active cognition tasks, and see how their arousal measures fare: is there a (substantive, significant) connection between interactive violence and viewed violence, or merely between game and video, game and cognition, some combination, or nothing at all?
This is an important area and thus one that merits extra experimental and analytical care, not dismissiveness or dissembling from any POV.
Posted by: Mike Sellers | Jul 27, 2006 at 11:59
Yet more proof that "statistics" can be bent to "prove" whatever you want. The problem is that, even if these people are ripped apart in academic circles, none of us in the GP will know about it. The media will not pick up on the correction because the correction will not be exciting. They've done their damage and that's good enough for them.
Posted by: Chip Hinshaw | Jul 27, 2006 at 13:55
Ted: Nice spanking. Thanks.
Here was my only question after reading this (to put it much less erruditely than you did) truly stupid piece of work: does this mean that any action that raises heart-rate and skin conductance is "bad" because it [insert one of several hoity-toity Latin debate terms here] would lead to "desensitization" towards violence?
Now... correct me if I'm wrong (not as harshly as you did them, please, though; this is a mere comment on a blog, not a published article, for garsh sake), but heart rate and sweat are tied to... uh... excitement, aren't they? Don't those things go up whenever we do anything that makes us excited?
So... the issue they're trying to raise seems (again, to lil' ol me) to be that violence in a video game is exciting (by these two measures). And that once I'm excited by a violent video game, I won't be as relatively excited by another kind of violence.
OK. I propose the following experiment. Put some folks in a tent-revival, old-timey setting for a good hour of heavy-handed, fahr-n-brimstone preachifying in place of the violent video games. And put another group into a Buddhist meditation class. Compare and contrast. I think we'll find that "Yea Bo!" style preachin' leads to desensitization towards violence, whereas meditation does not, and leads towards hyper-sensitization" towards violence; i.e., the time spent doing something extra-mellow would give you a real difference in your readings. Which starts to show how stupid this methodology is, before you even get into the stats.
I bet a very exciting, non-violent game (racing, snow boarding, Tony Hawk, sex related stuff, etc.) would register pretty high on the desensitometer, and a bad, boring, yet really, really violent game (lots of gore, but nothing to do) would score low.
Dumb, dumb, dumb. But then again... After a life of reading about how bad video games are for me, I'm highly desensitized to dumb.
Posted by: Andy Havens | Jul 27, 2006 at 18:20
/applaud
While the process of peer-review may have failed at the pre-publication stage here, Ted, your systematic dismantling recuperates it, showing just how important it is to judge academic merit via old-fashioned fine-tooth combs. No amount of juggling of "citation rankings" can replace that. Bravo.
Posted by: Thomas Malaby | Jul 27, 2006 at 20:14
Ted said: "Nonetheless, the bugbear of statistical significance is loose among poorly-grounded fields, among which one must now, on the basis of their acceptance of this paper, sorrowfully include experimental social psychology."
I'm just as frustrated with this paper as many of you, but Ted, please be careful about bashing entire academic disciplines based on one paper.
I'm also going to push back on some of your statistical critiques. You mention 3 problems that you perceive are common in this literature and which you argue are prevalent in this paper:
a) Drop the word 'statistical' when referring to the significance of a finding
In the field of psychology, the word "significant" when referring to data is specifically reserved to mean "statistically significant". You are in fact not allowed to use the word "significant" in any other way when referring to data in Psych journals. This is a linguistic convention that preserves its meaning while making it less cumbersome to put inferential stats into English sentences. Every discipline has shorthands, and it is unfair to criticize academic language because it doesn't conform to everyday English usage.
b) Report only statistical significance tests, not substantive significance tests
The authors of the paper in fact report an effect size value (an estimate of substantive significance) *every* time they report a statistical significance test. This is the Cohen's d value at the end of every test result. I also want to point out that this practice is now necessary in most Psych journals to get published.
c) Report all statistically significant findings and ignore all statistically insignificant ones
The authors in fact report most of the relevant non-significant results. For example, in the first two paragraphs of the Results section alone, they report 8 non-significant results. This practice is also now necessary in most Psych journals to get published.
Ted, I agree with you that there is something very deceptive about the paper, but I can't help but feel that your statistical critiques are also partly deceptive, confuse the lay reader more, and cause antagonism towards an entire academic discipline.
I would argue that the main deceptions are theoretical rather than statistical, and the stats in and of themselves are fine:
- Is there any stimulus X that people do not habituate to in a short time frame? And how is this different from the well-known concept of habituation in general?
- Arousal is known to increase the likelihood of action. Decreased arousal would in fact mean a lower likelihood to act.
- Can we learn anything about elective media use from *forced* media use studies in experimental settings?
The stats are this paper's least worrisome problems.
Posted by: Nick Yee | Jul 27, 2006 at 22:14
Another control case that should be present: Subjects that are *already* desensitized to real violence. National guardsmen recently returned from Iraq, contestants from the Ultimate Fighting Championship, inmates from a maximum security prison. If "violent" media shows desensitization by these methods in those already accustomed to violence, the whole approach is crap.
--Dave
Posted by: Dave Rickey | Jul 28, 2006 at 01:51
Ted > The true policy issue is this: would a significant decrease in exposure to violent video games lead to a significant decrease in real-world violence?
I know I bang on about this, but one also had to add: ‘,,,and what would be the costs of a significant decrease in exposure to violent video games’.
I add this as policy questions of this nature tend to be treated in a broadly utilitarian fashion i.e. we count the costs and benefits. Cars for example have the effect of leading to huge amounts of death through road traffic accidents and pollute thought their creation and use. Indeed I would suggest that Cars cause more harm than video games – but we don’t band them. This is because we want the benefits that they bring.
Video games have the benefit of bringing countless hours of enjoyment to millions and millions. We should throw this onto the scales of any policy debate.
I know the policy position is far more complex than this, but the pleasures of gaming seems entirely absent from the policy debate - which is staggering as that’s the point of the artifacts we are discussing.
Posted by: ren reynolds | Jul 28, 2006 at 03:33
If it feels good, it must be a sin. Pointing out that games are fun and entertaining doesn't help in debates of public policy, you need "redeeming social value."
Personally, I think videogames reduce youth violence and MMO's reduce suicide rates. But I'm hardly in a position to do the studies to prove it.
--Dave
Posted by: Dave Rickey | Jul 28, 2006 at 03:37
Ted - I just wrote a very similar review, only to stumble upon yours via reddit!
As it turns out, our criticisms are very different. As Nick was suggesting, I did not emphasize statistical problems so much as ignored trends in the data.
Posted by: DevInt | Jul 28, 2006 at 10:15
I would agree with Nick that the stats are likely less problematic than claims the paper makes that go beyond its data. Chief among these is the fact that their experiment is solely a comparison of physiological reactions, yet the authors' conclusions involve unwarranted and specious inferences from physiological data to attribution of aggressive mental states and predicted behavior.
It's funny that they use the Kitty Genovese example to open the shoddy argument connecting their data to helping behavior: The Kitty Genovese episode is very much mythologized.
I have written about a several of the physiological experiments coming out of the Anderson circle of research (particularly the ERP experiments), and their greatest flaw is that they tend to completely overlook rival hypothesis and routinely expand their analysis well beyond what their data justifies.
Posted by: monkeysan | Jul 28, 2006 at 10:25
Monkeysan can you link us to your writings on the topic? I wasn't familiar with Anderson's work before this article, and would be eager to read more.
Posted by: DevInt | Jul 28, 2006 at 10:27
@Devint:
They are a part of my next book, The New Model Society: How The Science of Videogames Is Transforming Our Culture, Our World and Ourselves.
I'll also be examining the ERP stuff in an upcoming column for Next-Gen.
Posted by: monkeysan | Jul 28, 2006 at 11:01
Nick Yee:
"In the field of psychology, the word "significant" when referring to data is specifically reserved to mean 'statistically significant'."
That's news to me. It seems an odd convention, though. It seems to me the word 'significant' should be reserved for things that are actually significant, and that some other term, such as 'booscient' be created to as a shortening of 'statistically significant.' Look at it this way: suppose someone was always writing 'The World Champion Boston Red Sox' and a critic pointed out that the Red Sox are not always World Champions. One convention might be to say "OK, to keep it clear, we will use the term 'World Champions' and that will always refer to the Red Sox.' It does clarify the author's intent to those who know the convention, but it's not a very apt convention.
Yee:
"The authors of the paper in fact report an effect size value (an estimate of substantive significance) *every* time they report a statistical significance test. This is the Cohen's d value at the end of every test result."
Another bit of learning for me: there's a thing called "Cohen's d" that is an effect-size statistic. However: a straight reading of the paper reveals no discussion of the effect sizes, and no attempt assess them in light of the real world. There are no sentences that say "The Cohen's d indicates that this is a very large effect, one that has distinct and meaningful real-world implications of the following kind..." Those sentences would have tipped off a reader from another discipline that Cohen's d is a measure of substantive impact. Maybe I trust myself too much, but I wasn't tipped off. That suggests that the authors aren't actually devoting any white space at all to substantive significance. So, OK, they "report" substantive significance, by including an effects size, buried after the F-test. That doesn't solve the problem I'm describing. It's "reporting" but not reporting, not "talking to the reader about."
Yee:
"The authors in fact report most of the relevant non-significant results. For example, in the first two paragraphs of the Results section alone, they report 8 non-significant results."
No. What they say is "effect X was not significant, and was dropped." Doing that is not
"reporting non-significant results."
It is
"not reporting non-significant results."
Right? To say "this thing was not blue so I am not going to talk about it" is to confirm that, in fact, blue things are not going to be talked about here. Or? So, no, they do not report non-significant results.
Let me finish and exit by saying that this debate doesn't matter too much. So, these approaches are norms in psych journals. Wow. So, the field of psychology insists on this use of the word 'significant'; it reports but ignores "Cohen's d" and similar substance indicators; it drops stat-insig findings. OK, but the world of actual substance is going to evolve however it wants, whether or not psychologists choose to study it or not.
Posted by: Edward Castronova | Jul 28, 2006 at 12:22
Oh no, just when I was starting to like this blog, you insult psychology! :) I understand your frustration with psych terminology, but bear in mind that psychologists are primarily concerned with the world of cognitive functions, which (frustratingly to some) is not necessarily the same as the world of actual substance. haha.
Posted by: DevInt | Jul 28, 2006 at 12:38
Dave Rickey wrote:
National guardsmen recently returned from Iraq, contestants from the Ultimate Fighting Championship, inmates from a maximum security prison.
Oi. I do UFC-type fighting (as an amateur) and I don't believe it would affect the level of psychological scarring I could undergo while watching, say, civilians blown apart by bombs. All violence isn't the same, and one has to wonder, as well, whether becoming desensitized to one kind of violence affects your level of sensitivity to another.
(I also think there's something profoundly different between consensual violence - UFC, boxing, American football, etc - and non-consensual violence.)
--matt
Posted by: Matt Mihaly | Jul 28, 2006 at 13:33
Disclaimer: economics is just as bad. This is a general problem across social science, not just in psych. It's one of the reasons why the scientific approach to social policy of the 1960s and 1970s failed. It also explains why game developers, it could be argued, often do a better job of designing societies than social scientists. (Speaking of next books, this is fodder for mine.)
Broadly speaking, social scientists focus on verification of effects, not on whether those effects matter in some sense. In game design, you only care about an effect if it substantively changes player experience.
Posted by: Edward Castronova | Jul 28, 2006 at 13:48
>Put some folks in a tent-revival, old-timey setting for a good hour of heavy-handed, fahr-n-brimstone preachifying in place of the violent video games. And put another group into a Buddhist meditation class. >Compare and contrast.
*Rolls eyes*. I guess that's why we see all that sectarian violence between Buddhists and Muslims in India, then, eh? Is this due to their religions and their practices? Or, failure to practice religion? Or?
>I think we'll find that "Yea Bo!" style preachin' leads to desensitization towards violence, whereas meditation does not, and leads towards hyper-sensitization" towards violence; i.e., the time spent doing something extra-mellow would give you a real difference in your readings.
So...that's why mass murder um..occurs so often in America, due to this sort of religious service being common? *cough* So that's why...every time we see a fire-and-brimstone tent revival meeting, we also *cough* see a rise in crime everywhere?
Uh..you're going to show a hydraulic connection, then, between heart rates and blood oxygenation rates increased by Holy Rollers, as contrasted to meditators?
Ok, well, junk those examples -- you yourself seem to indicate they may not be so valid?
Let's take something not religious!
How about if you take 100 young men, and let 50 percent have sex, and 50 percent abstain? Then see, which of their football teams win the next day? Which commit violent crimes? Which, I dunno, win at video games?
Do you believe that there is a relationship between sex, which, like religious revivals, raises the heart rates and endorphins, and their performance on various tasks or their being prone to crime? That it creates a hydraulic-like relationship between winning games or crime, i.e. if you have sex, you won't win the game, or won't commit the crime? Would you be willing to test that theory?
Why is it that hydraulic concepts of how human beings work are so widespread? The idea that the energy in a human being is like water. Pour it out -- it turns a windmill and makes electricity. Don't pour it out -- it doesn't turn a windmill.
What if gaming is like sex or Holy Roller meetings for some people, but like meditation for others? And how will you determine the difference?
Posted by: Prokofy Neva | Jul 28, 2006 at 16:33
Matt Mihaly wrote:
But one of the assumptions of the original topic, and of that whole methodology of experiment, is that all violence is equivalent, whether observed or participated in, and whether real or simulated, and that desensitization to violence makes you more willing to engage in violence.
Blowing apart those assumptions (which are barely plausible bullshit based on outdated theories of neurocognitive functionality long since abandoned by serious researchers in that field) invalidates all the studies based on them.
So they sould be attacked from both ends: Establish whether all violence and representations of violence are equivalent, and whether those who engage in violence under various conditions respond differently to violent media than random others, and equivalently to each other. According to the assumptions of the theory, they should stand up.
The reason it annoys me to see this kind of work in that particular journal is that experimental social psyhchology should be as rigorously controlled and vetted as experiments in physics, or we might as well roll the clock back 50 years and start talking about the id and the super-ego.
--Dave
Posted by: Dave Rickey | Jul 28, 2006 at 19:21
http://www.euwowgold.com
wow gold the need
do you play the wow gmae
do you like the wow gold
Posted by: wowgold | Jul 29, 2006 at 02:48
I'm in no way versed on how the body or mind works, but the whole thing seems a little dodgy to me.
In my mind, if you're getting someone into a more alert state, it seems quite a likely byproduct that they'll show less of a change when you later give them more stimulus - so I imagine if you had someone run a mile, they'd show less response to the video, and if you showed them a scary horror movie with lots of stuff jumping out at them, they'd show less comparative response. I would have expected that a study like this should have added other comparative tests.
I'm also a little confused as to why (it looked at a glance like) they're testing the short term response, which I really feel is more likely to show to effects of adrenaline. It seemed to me like the definition of physical response was based off holding other things equal, rather than stacking the initial conditions.
I was also wondering how other factors, like being used to being shocked, would affect response without really effecting desensitisation in all of the other senses of the definition - like perhaps fire fighters who may be used to high levels of adrenaline.
Maybe someone can explain if they covered those things in there already.
Posted by: Daniel Speed | Jul 29, 2006 at 12:50
I'm not equipped to get into the statistical math, but I agree with both Ted and Nick on objection #4. Something smells funny about how these data lead to that conclusion.
The impression I have with this study, as with much research with policy implications, is that it seems to put the conclusion before the science. Violent video games increase one's propensity to commit violent acts in real life (the theory goes)... now let's go find some data to support that belief.
Consider: What if the exact opposite results had been recorded? Who here thinks a higher heartrate after both a violent game and a violent film wouldn't generated a paper like "The Effects of Videogame Violence on Physiological Provocation to Real-World Violence"?
The optimal solution to this kind of thing is for every researcher to be perfectly objective in all things -- just gather data and let someone else decide what it means. Since that's never going to happen, the next best thing is to insist that everyone follow a transparent process. When each step along the path from data to conclusion is shown, at least then we have a shot at detecting when someone's following the data wherever it leads, and when someone's trying to push an agenda.
In this case, let's hear it for process.
Ted: game developers, it could be argued, often do a better job of designing societies than social scientists. (Speaking of next books, this is fodder for mine.)
Yeep. I'm definitely going to have to read that book, then, as the premise behind this thesis runs smack into my belief that neither game designers nor other social engineers have a good enough record to actually be allowed to design societies for real.
I'm on record here at TN (in the Shades of Collusion thread) as saying that, but to reiterate: humans are terrible at trying to tell each other how to live in society with each other. The few times and places we've done a half-decent job of it haven't come from giving a few people the power to impose their visions on others -- they've come from people negotiating freely how to live with each other and from others copying social systems that work better than what they've currently got.
Design by consensus is messy and imperfect, but top-down central planning models are a lot less perfect. And the more utopian the vision, the less perfect such societies seem to be.
Given the severe social pathologies of most online game worlds, I look forward to reading how game designers are more worthy of a shot at designing physical societies than anyone else (psychology researchers, for example). I'm open to being persuaded... but it's going to take a lot of persuading!
--Bart
Posted by: Bart Stewart | Jul 29, 2006 at 19:16
--Dave
Our experiments are a lot less messy. And we don't have a choice, we create societies regardless, and will have to take a pro-active stance in doing so. To *not* attempt to direct our societies now that we know they can be influenced by our game design would be even less responsible.Posted by: Dave Rickey | Jul 29, 2006 at 20:02
I think most of the severe pathologies of online worlds have their root cause at the anonymity and the difficulties in applying long term negative effects, where someone can quit or create a new avatar. Perhaps you can blame the world design, but I think we're just forced to work against human nature more than normal, with fewer tools.
Posted by: Daniel Speed | Jul 30, 2006 at 08:25
Dave, if you're just talking about gameworlds, I'm with you completely. I've said elsewhere exactly what you said: Game developers must engage in "social engineering." There's no guarantee that doing so will produce a game that's fun, but not to do so will certainly guarantee an unsatisfying directed entertainment experience.
No (or few) developer-imposed social rules = social world. There's nothing wrong with that for what it is, but it isn't a game.
And Daniel, I agree with you about the social perils of anonymity -- I'm a big fan of Axelrod's work -- but isn't anonymity a design decision?
It might not be something that can be designed into the real world (at least, not with current technology... which is probably a Good Thing). But game designers have the power. The question is whether there's a good way to use that power to better encourage positive social behaviors in gameworlds.
You'd still be working somewhat against human nature, but as Dave implies, that's not a sufficient justification for not trying.
--Bart
Posted by: Bart Stewart | Aug 01, 2006 at 18:49
If only I had the budget, I would like to demonstrate that bungee jumping has a far greater effect in 'desensitising people to violence', by the criteria presented in this paper. :) Thanks for this.
Posted by: ChrisBateman | Aug 02, 2006 at 04:16
The annonymity is usually a conflict between the attempt to make a game that makes money, and the attempt to make a society. If we were out to make the best online society we could, we'd probably want to ensure that you had a one-to-one correspondance between online avatars and real people. We could perhaps sacrifice all those people who have no credit cards (as a form of identity verification) and wish to pay through other means - without that, all you do is restrict the behaviour to those who can afford the cost of the extra accounts.
In the meantime, I think that we continue to look for ways of dealing with the issues within the scope of our goals, which is a harder problem. In general, we are probably attacking a harder problem than we need to, because of that, and therefore seen to fail to address the (imo) simpler issue.
Posted by: Daniel Speed | Aug 02, 2006 at 09:56
Some excerpts from an email I sent to Mr. Carnagey:
I for one can see some major flaws.
One of them was in the choice of games used. In particular, the non-violent games were, in a word, boring. They produce very little excitement in the players. Success in those games is more often based on maintaining a calm attitude, leading to steady reactions to patterns, rather than excitement and quick thinking. Tetris can be likened to meditation, and the other non-violent games listed in the article can be likened to Chinese water torture.
Obviously, this could have a significant effect on the responses of your test subjects. It is also critical to your interpretation of those responses. One group was performing an exciting activity, the other was performing a calming activity. It doesn't take a PhD to know that someone who has been doing something exciting will react less to further excitement than someone who has been doing something calming, even boring.
Another flaw is that the subjects were not exposed to any real-world violence. People who had been watching simulations of violent actions on a video screen were less aroused by additional violent actions on a video screen. But how would they have reacted to something in the real world, such as a dramatized attack on the investigators, or even formalized violence such as a couple of amateur boxers putting on a brief demonstration bout for their benefit? All you have actually proved is that people who have been watching something on the screen are less excited by seeing more of the same on a different screen than people who haven't been. Nobody needs to write a paper to prove that!
I'm also curious as to what you would have concluded if you had found that the violent game players' responses to videotaped violence had increased rather than decreased. I suspect you would have said that playing the games increases their excitement at seeing "real" violence, and hence they would be driven to seek it out or participate in it. It appears to me as though you had your conclusion in hand; you were simply seeking data to support it.
Imagine a study where half of your group performed calisthenics for 20 minutes while the other half read magazines. Then, once their heart rates were back to normal levels, both groups were timed on a kilometer run. You could come up with equal "proof" that exercise reduces athletic performance, and it would be just as meaningful.
I'm put in mind of an episode of the Hogan's Heroes TV show from long ago where a character says something like "The purpose of research is not to find out new facts; it is to provide scientific proof of what you already know." That was supposed to be so ludicrous that it was funny ... yet you appear to be conducting research on exactly that basis.
And I'm not a scientist. If I can see gaping holes in the design of the study, there's something very badly wrong in there. I do like the study that I proposed, though. I'm sure it would prove that my dedication to hardcore slacking is in fact the true way to health!
Posted by: Wanderer | Aug 03, 2006 at 05:23
One last note on the side topic of whether game designers would be better than others at designing physical-world social systems.
In one of those funny moments of synchronicity, I recently found the following observation from Adam Smith:
The man of system ... is apt to be very wise in his own conceit; and is often so enamoured with the supposed beauty of his own ideal plan of government, that he cannot suffer the smallest deviation from any part of it. He goes on to establish it completely and in all its parts, without any regard either to the great interests, or to the strong prejudices which may oppose it. He seems to imagine that he can arrange the different members of a great society with as much ease as the hand arranges the different pieces upon a chess-board. He does not consider that the pieces upon the chess-board have no other principle of motion besides that which the hand impresses upon them; but that, in the great chess-board of human society, every single piece has a principle of motion of its own, altogether different from that which the legislature might chuse to impress upon it. If those two principles coincide and act in the same direction, the game of human society will go on easily and harmoniously, and is very likely to be happy and successful. If they are opposite or different, the game will go on miserably, and the society must be at all times in the highest degree of disorder.
-- Adam Smith, The Theory of Moral Sentiments, Part VI, section 2, paragraph 42
I wonder what Smith would have thought about how virtual worlds are being designed.
--Bart
Posted by: Bart Stewart | Aug 07, 2006 at 16:23