The choice of games
One of the most basic requirements of good experimentation is that the various conditions be as similar as possible except for the variable of interest. As long as the conditions differ only in terms of that variable, any differences in the dependent measure can be attributed to that variable; but if the conditions differ in other ways, any differences in the dependent measure could be due to any of the ways in the which the conditions differ. That is why great effort is usually made to equate the various experiment groups on every factor except the one of concern.
For example, imagine some researchers wanted to test the effectiveness of a new drug designed to reduce flu symptoms. The researchers design a study in which some people who have the flu get the drug three times a day while others get a placebo three times a day. Because the drug does not taste very good, it is mixed with a vitamin-rich fruit drink to make it more palatable, while the placebo is taken with water. The study shows that those who are given the drug report that their symptoms were less severe that those who were given the placebo. The researchers conclude that the drug works.
This conclusion is clearly not justified because the two conditions differ in more ways than just the presence or absence of the drug. Those who got the drug also drank the fruit drink. The difference between the conditions on the dependent variable could be due to the drug or to the fruit drink. This is so obvious that no serious scientist would make such a silly mistake. In good drug research the conditions are identical except for the drug - everyone gets the same instructions, everyone takes an identical-looking pill, takes the pill under identical circumstances, and every effort is made to have the drug-pill and the placebo-pill taste the same. In fact, as I’ll discuss later, both patient and physician do not even know what condition the patients are in until after the experiment is completed. Only when all of this is done is it legitimate to attribute differences to the drug. Similarly, in any kind of experiment, only if the experimental and control conditions are identical in every respect except the variable of interest can one conclude that differences on the dependent variable are due to that variable.
This ideal level of comparability between conditions has never been realized in the research on video games, even when the experimenters tried to equate conditions. For example, Anderson and Dill (2000) compared Wolfenstein, a violent video game, with Myst. They selected these two programs with considerable care. They conducted a pilot study on several video programs and found that these two did not differ in the ratings they received on various important dimensions. In particular, there were no differences on physiological measures or on ratings of action. In other words, the authors tried to find games that were equivalent.
However, it seems obvious that their attempt was not entirely successful. In the first place, players rated Wolfenstein more exciting than Myst. But perhaps equally important, anyone familiar with the two knows that they are entirely different programs. Myst was a very popular program that sold millions - but it differs in many ways from the violent game. To begin, it is not really a game but a puzzle. The players find themselves on a strange island and must figure out what is going on. This involves finding esoteric clues, interpreting them, and then using them correctly. It is extremely difficult and requires great ingenuity to solve. Those who like the game find this interesting and fascinating. But there is no action (it is hard to imagine why it was rated similar to Wolfenstein on this dimension) and nothing that makes it similar to a game. So it is not really an appropriate comparison, since presumably the question is whether games that involve violence differ from games that do not involve violence in their effects on aggression - not whether playing games differs from engaging in problem solving. Or, to put it another way, as in any experiment, you do not compare apples and oranges, but rather two kinds of apples.
Ballard and Lindeberger, 1999 did a better job. They compared NBA Jam, an exciting sports game with three versions of Mortal Kombat that differed in the amount of violence (or at least the graphic nature of the violence). It is possible that Mortal Kombat was more exciting and had more action than the basketball game, but at least all of the programs were exciting, all were games, and it was possible to compare the three level of Mortal Kombat to look for differences due to varying amounts of violence. Graybill et al (1987) probably did still better. They used six games that did not differ in ratings of excitement, difficulty or enjoyment but only in ratings of violence. All of them were lively, action-packed games in which players compete against the computer. Presumably it is better to have six games than to have only two since this reduces the chance that any effects are due to the specific games.
My point here is not to say that all of the studies failed to equate the games or that those that did equate them found no effect on aggression. Rather, the point is that in considering all of the research, it is important to understand that few of the experiments came close to solving and none solved perfectly the tricky problem of making the violent and non-violent games comparable on all variables other than the amount of violence. It may be that this cannot be done, but we must recognize that it is a limitation of the work and a serious limitation in some of the experiments.
The lack of comparability of the video games is not a subtle or picky criticism - it is absolutely basic to the design and interpretation of the research because it leaves open the interpretation of any difference that is found between conditions. And as in other media violence research, one obvious interpretation is that any effect is due to differences in arousal. Indeed, in their review of the research Anderson and Bushman found that exposure to violent video games increased physiological arousal. If the violent video game is more arousing than the non-violent comparison program, one would expect more aggression (or almost anything else) in the condition with higher arousal. If so, there is no reason to attribute the effect to the violence - it might be just the arousal. Since all of this research compared games with violence to games (or programs) without violence, and since the two types of games differed in many ways, it is possible any effects on aggression could be due to arousal or other factors rather than to the presence of violence. Because of this problem, one must be extremely cautious in interpreting the results of this research and especially cautious in deciding that the effects are due to the amount of violence in the games.
From Evaluating the Research on Violent Video Games by Jonathan L. Freedman