« Synthetic Worlds Initiative | Main | Infinite Mind Moving to SL »

Jul 27, 2006

Comments

1.

Hey Ted,

Ouch. I hope you are sending this in to the journal of experimental social psychology. A somewhat more sober version of this critique absolutely warrants publication and debate. This stuff has been getting a free ride for too long.

2.

Sad to see this kind of sloppiness in the Journal of Experimental Social Psychology of all places. Makes me wonder about the analytical skill of not just the investigators but the primary reviewers too. The experimental design here seems pretty straightforward though; I'd say it's a great advanced undergraduate project.

I'd also add the conditions of having people from both groups watch non-violent videos or do some other form of passive viewing or active cognition tasks, and see how their arousal measures fare: is there a (substantive, significant) connection between interactive violence and viewed violence, or merely between game and video, game and cognition, some combination, or nothing at all?

This is an important area and thus one that merits extra experimental and analytical care, not dismissiveness or dissembling from any POV.

3.

Yet more proof that "statistics" can be bent to "prove" whatever you want. The problem is that, even if these people are ripped apart in academic circles, none of us in the GP will know about it. The media will not pick up on the correction because the correction will not be exciting. They've done their damage and that's good enough for them.

4.

Ted: Nice spanking. Thanks.

Here was my only question after reading this (to put it much less erruditely than you did) truly stupid piece of work: does this mean that any action that raises heart-rate and skin conductance is "bad" because it [insert one of several hoity-toity Latin debate terms here] would lead to "desensitization" towards violence?

Now... correct me if I'm wrong (not as harshly as you did them, please, though; this is a mere comment on a blog, not a published article, for garsh sake), but heart rate and sweat are tied to... uh... excitement, aren't they? Don't those things go up whenever we do anything that makes us excited?

So... the issue they're trying to raise seems (again, to lil' ol me) to be that violence in a video game is exciting (by these two measures). And that once I'm excited by a violent video game, I won't be as relatively excited by another kind of violence.

OK. I propose the following experiment. Put some folks in a tent-revival, old-timey setting for a good hour of heavy-handed, fahr-n-brimstone preachifying in place of the violent video games. And put another group into a Buddhist meditation class. Compare and contrast. I think we'll find that "Yea Bo!" style preachin' leads to desensitization towards violence, whereas meditation does not, and leads towards hyper-sensitization" towards violence; i.e., the time spent doing something extra-mellow would give you a real difference in your readings. Which starts to show how stupid this methodology is, before you even get into the stats.

I bet a very exciting, non-violent game (racing, snow boarding, Tony Hawk, sex related stuff, etc.) would register pretty high on the desensitometer, and a bad, boring, yet really, really violent game (lots of gore, but nothing to do) would score low.

Dumb, dumb, dumb. But then again... After a life of reading about how bad video games are for me, I'm highly desensitized to dumb.

5.

/applaud

While the process of peer-review may have failed at the pre-publication stage here, Ted, your systematic dismantling recuperates it, showing just how important it is to judge academic merit via old-fashioned fine-tooth combs. No amount of juggling of "citation rankings" can replace that. Bravo.

6.

Ted said: "Nonetheless, the bugbear of statistical significance is loose among poorly-grounded fields, among which one must now, on the basis of their acceptance of this paper, sorrowfully include experimental social psychology."

I'm just as frustrated with this paper as many of you, but Ted, please be careful about bashing entire academic disciplines based on one paper.

I'm also going to push back on some of your statistical critiques. You mention 3 problems that you perceive are common in this literature and which you argue are prevalent in this paper:

a) Drop the word 'statistical' when referring to the significance of a finding

In the field of psychology, the word "significant" when referring to data is specifically reserved to mean "statistically significant". You are in fact not allowed to use the word "significant" in any other way when referring to data in Psych journals. This is a linguistic convention that preserves its meaning while making it less cumbersome to put inferential stats into English sentences. Every discipline has shorthands, and it is unfair to criticize academic language because it doesn't conform to everyday English usage.

b) Report only statistical significance tests, not substantive significance tests

The authors of the paper in fact report an effect size value (an estimate of substantive significance) *every* time they report a statistical significance test. This is the Cohen's d value at the end of every test result. I also want to point out that this practice is now necessary in most Psych journals to get published.

c) Report all statistically significant findings and ignore all statistically insignificant ones

The authors in fact report most of the relevant non-significant results. For example, in the first two paragraphs of the Results section alone, they report 8 non-significant results. This practice is also now necessary in most Psych journals to get published.

Ted, I agree with you that there is something very deceptive about the paper, but I can't help but feel that your statistical critiques are also partly deceptive, confuse the lay reader more, and cause antagonism towards an entire academic discipline.

I would argue that the main deceptions are theoretical rather than statistical, and the stats in and of themselves are fine:

- Is there any stimulus X that people do not habituate to in a short time frame? And how is this different from the well-known concept of habituation in general?
- Arousal is known to increase the likelihood of action. Decreased arousal would in fact mean a lower likelihood to act.
- Can we learn anything about elective media use from *forced* media use studies in experimental settings?

The stats are this paper's least worrisome problems.

7.

Another control case that should be present: Subjects that are *already* desensitized to real violence. National guardsmen recently returned from Iraq, contestants from the Ultimate Fighting Championship, inmates from a maximum security prison. If "violent" media shows desensitization by these methods in those already accustomed to violence, the whole approach is crap.

--Dave

8.

Ted > The true policy issue is this: would a significant decrease in exposure to violent video games lead to a significant decrease in real-world violence?

I know I bang on about this, but one also had to add: ‘,,,and what would be the costs of a significant decrease in exposure to violent video games’.

I add this as policy questions of this nature tend to be treated in a broadly utilitarian fashion i.e. we count the costs and benefits. Cars for example have the effect of leading to huge amounts of death through road traffic accidents and pollute thought their creation and use. Indeed I would suggest that Cars cause more harm than video games – but we don’t band them. This is because we want the benefits that they bring.

Video games have the benefit of bringing countless hours of enjoyment to millions and millions. We should throw this onto the scales of any policy debate.

I know the policy position is far more complex than this, but the pleasures of gaming seems entirely absent from the policy debate - which is staggering as that’s the point of the artifacts we are discussing.

9.

If it feels good, it must be a sin. Pointing out that games are fun and entertaining doesn't help in debates of public policy, you need "redeeming social value."

Personally, I think videogames reduce youth violence and MMO's reduce suicide rates. But I'm hardly in a position to do the studies to prove it.

--Dave

10.

Ted - I just wrote a very similar review, only to stumble upon yours via reddit!

As it turns out, our criticisms are very different. As Nick was suggesting, I did not emphasize statistical problems so much as ignored trends in the data.

11.

I would agree with Nick that the stats are likely less problematic than claims the paper makes that go beyond its data. Chief among these is the fact that their experiment is solely a comparison of physiological reactions, yet the authors' conclusions involve unwarranted and specious inferences from physiological data to attribution of aggressive mental states and predicted behavior.

It's funny that they use the Kitty Genovese example to open the shoddy argument connecting their data to helping behavior: The Kitty Genovese episode is very much mythologized.

I have written about a several of the physiological experiments coming out of the Anderson circle of research (particularly the ERP experiments), and their greatest flaw is that they tend to completely overlook rival hypothesis and routinely expand their analysis well beyond what their data justifies.

12.

Monkeysan can you link us to your writings on the topic? I wasn't familiar with Anderson's work before this article, and would be eager to read more.

13.

@Devint:

They are a part of my next book, The New Model Society: How The Science of Videogames Is Transforming Our Culture, Our World and Ourselves.

I'll also be examining the ERP stuff in an upcoming column for Next-Gen.

14.

Nick Yee:
"In the field of psychology, the word "significant" when referring to data is specifically reserved to mean 'statistically significant'."

That's news to me. It seems an odd convention, though. It seems to me the word 'significant' should be reserved for things that are actually significant, and that some other term, such as 'booscient' be created to as a shortening of 'statistically significant.' Look at it this way: suppose someone was always writing 'The World Champion Boston Red Sox' and a critic pointed out that the Red Sox are not always World Champions. One convention might be to say "OK, to keep it clear, we will use the term 'World Champions' and that will always refer to the Red Sox.' It does clarify the author's intent to those who know the convention, but it's not a very apt convention.

Yee:
"The authors of the paper in fact report an effect size value (an estimate of substantive significance) *every* time they report a statistical significance test. This is the Cohen's d value at the end of every test result."

Another bit of learning for me: there's a thing called "Cohen's d" that is an effect-size statistic. However: a straight reading of the paper reveals no discussion of the effect sizes, and no attempt assess them in light of the real world. There are no sentences that say "The Cohen's d indicates that this is a very large effect, one that has distinct and meaningful real-world implications of the following kind..." Those sentences would have tipped off a reader from another discipline that Cohen's d is a measure of substantive impact. Maybe I trust myself too much, but I wasn't tipped off. That suggests that the authors aren't actually devoting any white space at all to substantive significance. So, OK, they "report" substantive significance, by including an effects size, buried after the F-test. That doesn't solve the problem I'm describing. It's "reporting" but not reporting, not "talking to the reader about."

Yee:
"The authors in fact report most of the relevant non-significant results. For example, in the first two paragraphs of the Results section alone, they report 8 non-significant results."

No. What they say is "effect X was not significant, and was dropped." Doing that is not

"reporting non-significant results."

It is

"not reporting non-significant results."

Right? To say "this thing was not blue so I am not going to talk about it" is to confirm that, in fact, blue things are not going to be talked about here. Or? So, no, they do not report non-significant results.

Let me finish and exit by saying that this debate doesn't matter too much. So, these approaches are norms in psych journals. Wow. So, the field of psychology insists on this use of the word 'significant'; it reports but ignores "Cohen's d" and similar substance indicators; it drops stat-insig findings. OK, but the world of actual substance is going to evolve however it wants, whether or not psychologists choose to study it or not.

15.

Oh no, just when I was starting to like this blog, you insult psychology! :) I understand your frustration with psych terminology, but bear in mind that psychologists are primarily concerned with the world of cognitive functions, which (frustratingly to some) is not necessarily the same as the world of actual substance. haha.

16.

Dave Rickey wrote:

National guardsmen recently returned from Iraq, contestants from the Ultimate Fighting Championship, inmates from a maximum security prison.

Oi. I do UFC-type fighting (as an amateur) and I don't believe it would affect the level of psychological scarring I could undergo while watching, say, civilians blown apart by bombs. All violence isn't the same, and one has to wonder, as well, whether becoming desensitized to one kind of violence affects your level of sensitivity to another.

(I also think there's something profoundly different between consensual violence - UFC, boxing, American football, etc - and non-consensual violence.)

--matt

17.

Disclaimer: economics is just as bad. This is a general problem across social science, not just in psych. It's one of the reasons why the scientific approach to social policy of the 1960s and 1970s failed. It also explains why game developers, it could be argued, often do a better job of designing societies than social scientists. (Speaking of next books, this is fodder for mine.)

Broadly speaking, social scientists focus on verification of effects, not on whether those effects matter in some sense. In game design, you only care about an effect if it substantively changes player experience.

18.

>Put some folks in a tent-revival, old-timey setting for a good hour of heavy-handed, fahr-n-brimstone preachifying in place of the violent video games. And put another group into a Buddhist meditation class. >Compare and contrast.

*Rolls eyes*. I guess that's why we see all that sectarian violence between Buddhists and Muslims in India, then, eh? Is this due to their religions and their practices? Or, failure to practice religion? Or?

>I think we'll find that "Yea Bo!" style preachin' leads to desensitization towards violence, whereas meditation does not, and leads towards hyper-sensitization" towards violence; i.e., the time spent doing something extra-mellow would give you a real difference in your readings.

So...that's why mass murder um..occurs so often in America, due to this sort of religious service being common? *cough* So that's why...every time we see a fire-and-brimstone tent revival meeting, we also *cough* see a rise in crime everywhere?

Uh..you're going to show a hydraulic connection, then, between heart rates and blood oxygenation rates increased by Holy Rollers, as contrasted to meditators?

Ok, well, junk those examples -- you yourself seem to indicate they may not be so valid?

Let's take something not religious!

How about if you take 100 young men, and let 50 percent have sex, and 50 percent abstain? Then see, which of their football teams win the next day? Which commit violent crimes? Which, I dunno, win at video games?

Do you believe that there is a relationship between sex, which, like religious revivals, raises the heart rates and endorphins, and their performance on various tasks or their being prone to crime? That it creates a hydraulic-like relationship between winning games or crime, i.e. if you have sex, you won't win the game, or won't commit the crime? Would you be willing to test that theory?

Why is it that hydraulic concepts of how human beings work are so widespread? The idea that the energy in a human being is like water. Pour it out -- it turns a windmill and makes electricity. Don't pour it out -- it doesn't turn a windmill.

What if gaming is like sex or Holy Roller meetings for some people, but like meditation for others? And how will you determine the difference?

19.

Matt Mihaly wrote:

Oi. I do UFC-type fighting (as an amateur) and I don't believe it would affect the level of psychological scarring I could undergo while watching, say, civilians blown apart by bombs. All violence isn't the same, and one has to wonder, as well, whether becoming desensitized to one kind of violence affects your level of sensitivity to another.

(I also think there's something profoundly different between consensual violence - UFC, boxing, American football, etc - and non-consensual violence.)

But one of the assumptions of the original topic, and of that whole methodology of experiment, is that all violence is equivalent, whether observed or participated in, and whether real or simulated, and that desensitization to violence makes you more willing to engage in violence.

Blowing apart those assumptions (which are barely plausible bullshit based on outdated theories of neurocognitive functionality long since abandoned by serious researchers in that field) invalidates all the studies based on them.

So they sould be attacked from both ends: Establish whether all violence and representations of violence are equivalent, and whether those who engage in violence under various conditions respond differently to violent media than random others, and equivalently to each other. According to the assumptions of the theory, they should stand up.

The reason it annoys me to see this kind of work in that particular journal is that experimental social psyhchology should be as rigorously controlled and vetted as experiments in physics, or we might as well roll the clock back 50 years and start talking about the id and the super-ego.

--Dave

20.

http://www.euwowgold.com
wow gold the need
do you play the wow gmae
do you like the wow gold

21.

I'm in no way versed on how the body or mind works, but the whole thing seems a little dodgy to me.

In my mind, if you're getting someone into a more alert state, it seems quite a likely byproduct that they'll show less of a change when you later give them more stimulus - so I imagine if you had someone run a mile, they'd show less response to the video, and if you showed them a scary horror movie with lots of stuff jumping out at them, they'd show less comparative response. I would have expected that a study like this should have added other comparative tests.
I'm also a little confused as to why (it looked at a glance like) they're testing the short term response, which I really feel is more likely to show to effects of adrenaline. It seemed to me like the definition of physical response was based off holding other things equal, rather than stacking the initial conditions.
I was also wondering how other factors, like being used to being shocked, would affect response without really effecting desensitisation in all of the other senses of the definition - like perhaps fire fighters who may be used to high levels of adrenaline.

Maybe someone can explain if they covered those things in there already.

22.

I'm not equipped to get into the statistical math, but I agree with both Ted and Nick on objection #4. Something smells funny about how these data lead to that conclusion.

The impression I have with this study, as with much research with policy implications, is that it seems to put the conclusion before the science. Violent video games increase one's propensity to commit violent acts in real life (the theory goes)... now let's go find some data to support that belief.

Consider: What if the exact opposite results had been recorded? Who here thinks a higher heartrate after both a violent game and a violent film wouldn't generated a paper like "The Effects of Videogame Violence on Physiological Provocation to Real-World Violence"?

The optimal solution to this kind of thing is for every researcher to be perfectly objective in all things -- just gather data and let someone else decide what it means. Since that's never going to happen, the next best thing is to insist that everyone follow a transparent process. When each step along the path from data to conclusion is shown, at least then we have a shot at detecting when someone's following the data wherever it leads, and when someone's trying to push an agenda.

In this case, let's hear it for process.

Ted: game developers, it could be argued, often do a better job of designing societies than social scientists. (Speaking of next books, this is fodder for mine.)

Yeep. I'm definitely going to have to read that book, then, as the premise behind this thesis runs smack into my belief that neither game designers nor other social engineers have a good enough record to actually be allowed to design societies for real.

I'm on record here at TN (in the Shades of Collusion thread) as saying that, but to reiterate: humans are terrible at trying to tell each other how to live in society with each other. The few times and places we've done a half-decent job of it haven't come from giving a few people the power to impose their visions on others -- they've come from people negotiating freely how to live with each other and from others copying social systems that work better than what they've currently got.

Design by consensus is messy and imperfect, but top-down central planning models are a lot less perfect. And the more utopian the vision, the less perfect such societies seem to be.

Given the severe social pathologies of most online game worlds, I look forward to reading how game designers are more worthy of a shot at designing physical societies than anyone else (psychology researchers, for example). I'm open to being persuaded... but it's going to take a lot of persuading!

--Bart

23.
Yeep. I'm definitely going to have to read that book, then, as the premise behind this thesis runs smack into my belief that neither game designers nor other social engineers have a good enough record to actually be allowed to design societies for real.
Our experiments are a lot less messy. And we don't have a choice, we create societies regardless, and will have to take a pro-active stance in doing so. To *not* attempt to direct our societies now that we know they can be influenced by our game design would be even less responsible.

--Dave

24.

I think most of the severe pathologies of online worlds have their root cause at the anonymity and the difficulties in applying long term negative effects, where someone can quit or create a new avatar. Perhaps you can blame the world design, but I think we're just forced to work against human nature more than normal, with fewer tools.

25.

Dave, if you're just talking about gameworlds, I'm with you completely. I've said elsewhere exactly what you said: Game developers must engage in "social engineering." There's no guarantee that doing so will produce a game that's fun, but not to do so will certainly guarantee an unsatisfying directed entertainment experience.

No (or few) developer-imposed social rules = social world. There's nothing wrong with that for what it is, but it isn't a game.

And Daniel, I agree with you about the social perils of anonymity -- I'm a big fan of Axelrod's work -- but isn't anonymity a design decision?

It might not be something that can be designed into the real world (at least, not with current technology... which is probably a Good Thing). But game designers have the power. The question is whether there's a good way to use that power to better encourage positive social behaviors in gameworlds.

You'd still be working somewhat against human nature, but as Dave implies, that's not a sufficient justification for not trying.

--Bart

26.

If only I had the budget, I would like to demonstrate that bungee jumping has a far greater effect in 'desensitising people to violence', by the criteria presented in this paper. :) Thanks for this.

27.

The annonymity is usually a conflict between the attempt to make a game that makes money, and the attempt to make a society. If we were out to make the best online society we could, we'd probably want to ensure that you had a one-to-one correspondance between online avatars and real people. We could perhaps sacrifice all those people who have no credit cards (as a form of identity verification) and wish to pay through other means - without that, all you do is restrict the behaviour to those who can afford the cost of the extra accounts.
In the meantime, I think that we continue to look for ways of dealing with the issues within the scope of our goals, which is a harder problem. In general, we are probably attacking a harder problem than we need to, because of that, and therefore seen to fail to address the (imo) simpler issue.

28.

Some excerpts from an email I sent to Mr. Carnagey:

I for one can see some major flaws.

One of them was in the choice of games used. In particular, the non-violent games were, in a word, boring. They produce very little excitement in the players. Success in those games is more often based on maintaining a calm attitude, leading to steady reactions to patterns, rather than excitement and quick thinking. Tetris can be likened to meditation, and the other non-violent games listed in the article can be likened to Chinese water torture.

Obviously, this could have a significant effect on the responses of your test subjects. It is also critical to your interpretation of those responses. One group was performing an exciting activity, the other was performing a calming activity. It doesn't take a PhD to know that someone who has been doing something exciting will react less to further excitement than someone who has been doing something calming, even boring.

Another flaw is that the subjects were not exposed to any real-world violence. People who had been watching simulations of violent actions on a video screen were less aroused by additional violent actions on a video screen. But how would they have reacted to something in the real world, such as a dramatized attack on the investigators, or even formalized violence such as a couple of amateur boxers putting on a brief demonstration bout for their benefit? All you have actually proved is that people who have been watching something on the screen are less excited by seeing more of the same on a different screen than people who haven't been. Nobody needs to write a paper to prove that!

I'm also curious as to what you would have concluded if you had found that the violent game players' responses to videotaped violence had increased rather than decreased. I suspect you would have said that playing the games increases their excitement at seeing "real" violence, and hence they would be driven to seek it out or participate in it. It appears to me as though you had your conclusion in hand; you were simply seeking data to support it.

Imagine a study where half of your group performed calisthenics for 20 minutes while the other half read magazines. Then, once their heart rates were back to normal levels, both groups were timed on a kilometer run. You could come up with equal "proof" that exercise reduces athletic performance, and it would be just as meaningful.

I'm put in mind of an episode of the Hogan's Heroes TV show from long ago where a character says something like "The purpose of research is not to find out new facts; it is to provide scientific proof of what you already know." That was supposed to be so ludicrous that it was funny ... yet you appear to be conducting research on exactly that basis.

And I'm not a scientist. If I can see gaping holes in the design of the study, there's something very badly wrong in there. I do like the study that I proposed, though. I'm sure it would prove that my dedication to hardcore slacking is in fact the true way to health!

29.

One last note on the side topic of whether game designers would be better than others at designing physical-world social systems.

In one of those funny moments of synchronicity, I recently found the following observation from Adam Smith:

The man of system ... is apt to be very wise in his own conceit; and is often so enamoured with the supposed beauty of his own ideal plan of government, that he cannot suffer the smallest deviation from any part of it. He goes on to establish it completely and in all its parts, without any regard either to the great interests, or to the strong prejudices which may oppose it. He seems to imagine that he can arrange the different members of a great society with as much ease as the hand arranges the different pieces upon a chess-board. He does not consider that the pieces upon the chess-board have no other principle of motion besides that which the hand impresses upon them; but that, in the great chess-board of human society, every single piece has a principle of motion of its own, altogether different from that which the legislature might chuse to impress upon it. If those two principles coincide and act in the same direction, the game of human society will go on easily and harmoniously, and is very likely to be happy and successful. If they are opposite or different, the game will go on miserably, and the society must be at all times in the highest degree of disorder.
-- Adam Smith, The Theory of Moral Sentiments, Part VI, section 2, paragraph 42

I wonder what Smith would have thought about how virtual worlds are being designed.

--Bart

The comments to this entry are closed.