A new study from Craig Anderson and colleagues at Iowa State University claims to be definitive proof that violent video games cause aggression. The claim is based on a meta analysis of studies in the media effects paradigm, which has been discussed here before. There will debate about the validity of the study's conclusions, just as there has been about the individual studies on which this meta-analysis is based: Were samples properly chosen? Can you learn anything from a study of correlations? Do short-term effects matter? And so on.
The debate misses the point. For the sake of argument, let's grant every scientific claim that Anderson and colleagues make. According to Professor Anderson, exposing a person to a violent video game makes the person feel more aggressive. As to the size of this effect, Professor Anderson expresses it thus: "These are not huge effects--not on the order of joining a gang vs. not joining a gang. But these effects are also not trivial in size."
Very well. Let's say we have 10,000 perfect studies showing positive effects of videogame violence on individual aggression, and these effects are big enough to show up on the radar consistently without being the biggest blip anyone's ever seen. If this were the state of the world (I don't think it is, but again, for the sake of argument), what would we make of it?
Professor Anderson: "From a public-policy standpoint, it's time to get off the question of, 'Are there real and serious effects?' That's been answered and answered repeatedly. It's now time to move on to a more constructive question like, 'How do we make it easier for parents--within the limits of culture, society and law--to provide a healthier childhood for their kids?'"
Hear hear! I say. And yet - what an odd time to be saying this. Why would a professor ever decide to study a question that, by his own admission, is not very "constructive?"
Indeed, why has there ever been MediaFX research? I ask this because it's fairly obvious to me that nothing in the current MediaFX paradigm is relevant to policy issues. The questions asked in this paradigm are not relevant to decisions we make in the real world. This problem is revealed most clearly, as I have said many times before, in the paradigm's obsession with statistical significance. Researchers in this paradigm are looking for an "effect" without having any idea what an "effect" might mean. Put another way, there is a difference between statistical and substantive significance. Policy assertions - such as the assertion that we must do something so that parents can help their kids make healthy choices - are substantive assertions. There are many substantive proposals on the table. Regulate game sales. Intervene in parent-child relationships. Have a videogame safety program in the schools. Restrict and codify content. Restrict advertising. Massage social norms through PSAs. There are many things that can be done. We should only do them if there is substantive evidence in their favor. But the MediaFX paradigm does not talk about any of these things. It only talks about "statistically significant effects of Medium A on Individual Feeling B."
Results like this, by themselves are devoid of meaning. What evidence do they provide? Consider a thought experiment. Suppose you have N studies showing that exposure to videogame violence causes an X% increase in aggressive thoughts. It is mathematically true - as part of the mechanics of statistics - that for any value of X, there exists a value of N such that X is "statistically significant." In other words, no matter what X is - no matter how substantively small or big the aggression effect may be - you can cook it into a "statistically significant" effect simply by adding N. If your meta-analysis of 50 studies does not result in a 2% average effect appearing to be "statistically significant," a meta-analysis of 500 or 500 million studies will. Just keep adding studies until you get the magic asterisk of statistical significance. The asterisk is sufficient to satisfy professors and editors. But real people in the real world balk, and properly so, at the effect size of X. "Two percent?!?!?!" they say. "That's tiny!" And they are right. It is tiny. It may be statistically significant, but it is not interesting from a policy perspective. (And with due respect to those trained to believe so, Cohen's d does not solve this problem. No statistic spit out by a program or algorithm can replace the human mind in assessing whether a number matters.)
Thus, the evidence provided by the MediaFX paradigm is not ab initio substantive evidence. We have to take the evidence that's been produced and inquire further: Is this substantively meaningful?
Let's consider: Is it substantively meaningful that someone exposed to a violent videogame feels X% more aggressive in the aftermath, where X is a small but reliably non-zero number? Well, let's consider the result in light of policies that might be enacted in the real world. Suppose we had a PSA campaign against violent videogames. Suppose this campaign cost $20m. Suppose it reduced exposure to violent videogames by 15%. I am being generous here in assuming that a low-cost government policy could actually have a large effect on behavior, an assumption not warranted by any evidence I am aware of. But let's just say. Fine, we've spent $20m and the number of people exposed to violent videogames has fallen by 15%. Is the world a better place? Sure, er, maybe. According to the evidence, these 15% of game players are now X% less aggressive. I guess that's a good thing in and of itself. Or is it? What does it mean? How important is that change? Does this change make our world $20m better off? Is it worth $20m to have reduced post-gaming aggressive feelings of 15% of gamers by X%? If you want to say yes, you have to explain to me how that X% reduction in aggression among those people has made the world a better place.
Now, that's the tough sell, isn't it? Is there anything in the research showing that a policy of reducing violent videogame playing has an effect on the social level of violence? No. Is there evidence in the paradigm that people who choose to play fewer violent videogames gain substantive benefits in other areas of their lives as a causal result (and not just a correlation)? No. Is there evidence that an effects size of X would be sufficient to make a policy of cost Y pass a benefit-cost test? No. Not only is there no evidence; there can be no evidence, not the way the paradigm is constructed. The paradigm is constructed in such a way that these issues never even appear. There is no evidence on substantive questions in the MediaFX paradigm not because of sample size problems or failures to perform random assignment or lack of longitudinal analysis, the things these debates always run around. There is no evidence on substantive questions because substantive questions are not asked. Period.
The MediaFX paradigm generally does not provide evidence on substantive questions. It simply asks "Did watching this TV show make you feel different today?" Why does that matter? Who knows.
How ironic, then, that Professor Anderson now says that it's time to "get off" the effects question and move on to research on the policy question, which he seems to view as "What's the best way to force parents to save their children from this evil thing?" I am eagerly awaiting his work here. But as far as I'm concerned, he should have started working on that eons ago; that's the only relevant question to begin with; one wonders why researchers ever "got on" the MediaFX question in the first place. What has it produced? The studies so far are not tales told by idiots, but they are certainly full of sound and fury, and even more certainly, they signify nothing.
