Reviews play a crucial role in the success or failure of a game. Customers pay attention to reviews; a good review can significantly boost sales, while a bad review can seriously hurt sales.
Thus, we have an entire industry, with thousands of people working to create games, earning several hundred million dollars each year, whose fate depends largely on the whims of a few hundred reviewers. It’s frightening to contemplate the power these people wield, but it’s even more frightening to realize just how incompetent some of them are.
General Computer Magazines
There are three basic types of reviewers. The worst are at the major general computer magazines. These magazines review dozens of new products, hardware and software, in every issue. They view games as junk software. Perhaps it’s because games are the cheapest products they see; perhaps it’s because game publishers seldom advertise in their publications. Whatever the reason, they approach games with callous insouciance. Rhett Anderson, writing in the last issue of the Journal, referred to magazine staff evaluating games "with a joystick in one hand and a sandwich in the other."
The result is an uneven body of reviews. If you’re lucky, they’ll be impressed with a few simple tricks and wax rhapsodic over the transcending brilliance of your work. In this case, you can be embarrassed for them, collect the kudos, and hope to make a clean getaway. If you’re not so lucky, they’ll roast you for imaginary defects and bogus bugs.
There is one common element to their tastes in games: they seem to prefer "lite" games. The deep, complex, lengthy games never attract their attention. An Ultima 6 will not catch their fancy, nor will a Sierra graphic adventure, nor an SSI wargame. Give them a game with lots of flash and no gameplay and they’re happy.
(In the interests of intellectual honesty, I must point out that this editorial was inspired by a hatchet job of my game Balance of the Planet appearing in MacWorld. To give you an idea of just how little care and attention went into the review, consider this: the reviewer thought that the game was a HyperCard stack, when in truth it is a standalone application. Anybody who has a Mac can immediately grasp from this just how inattentive the reviewer was.)
Beginners at Game-Specific Magazines
The game-specific magazines do a better job. Their editors are very knowledgeable about games; indeed, the editors at gaming magazines probably have a broader appreciation of the body of games out there than professional developers.
However, the editors don’t write all the reviews. Many of their reviews come from first-timers. As Rhett Anderson pointed out in his article, the pay for game reviewers is lousy. This insures a constant turnover of reviewers. Turnover may be healthy in the some situations, but breaking in beginners can be a painful process, especially when YOU are the victim of the beginner’s mistakes.
The principal weakness of the beginners is that they haven’t developed enough experience to appreciate the complexities of game design. In their eagerness to demonstrate their acumen, they tend to write preachy reviews that pontificate on the design errors of the game. It is tempting to benignly patronize their bright-eyed foolishness, but we must remember that these amateurs do a lot of damage through their ignorance.
Old Pros
Lastly, there are the old pros: people who have been reviewing games for years. We have a great set of people here: Arnie Katz, Bill Kunkel, Joyce Worley, Ken St. Andre, Shay Addams, Scorpia, Scott Mace, Alan Emrich, R. Bradley Andrews, and many more. These reviewers have been around the block a few times, and they know how to evaluate a game. First, they give it the time it deserves, knowing that the proof of the game is in the playing.
Second, they have established a context for evaluating games. They know what is realistically possible given the economic and technical realities, and they weigh a game’s strengths and weaknesses with these in mind.
Third, they have long since shed any illusions about being closet game designers. They know that their forte is criticism, not design, and they appreciate the difference.
Fourth, they recognize the narrowness of any individual’s tastes, including their own. "This game didn’t run to my taste because of X, but people who like games with X will enjoy this game." If only all reviewers were so honest!
Lastly, they care. They care about good games and the advancement of the art. They want to see better games and they see their criticisms as a way to improve the industry.
What to do?
Obviously, we need more old pros and fewer beginners. The problem is, how to do we motivate a thoughtful person to stay in the game-reviewing business long enough to develop some polish and insight?
The central problem here is money. The magazines cannot afford to pay their reviewers enough money to motivate them to take their work seriously. Most reasonable people move on to other endeavors after a few reviews. We need to do something about this.
Look at it this way: here we have a product that cost hundreds of thousands of dollars to develop being reviewed by somebody getting a hundred bucks for his time. This is a mismatch. If you view reviews as a subset of marketing, then surely the reviews would merit a lot more money than that.
Of course, this thinking runs afoul of the reviewer’s need to maintain editorial integrity. We can’t pay the reviewers; they must preserve their independence if they are to have any credibility. So how can we get money into their hands without compromising their integrity?
A Sketch Proposal
I suggest that, if we act as a community rather than a set of individual business interests, we might be able to solve this problem. What if we set up a fund and award cash prizes on an annual basis to those reviewers who most impress the community? Here’s one way that it could work:
First we solicit contributions from a variety of sources. The publishers are an obvious source; after all, a major publisher spends tens of thousands of dollars each year just on public relations a few thousand bucks to materially improve the quality of game reviews should be justifiable. The Software Publishers Association. might also be willing to contribute, as well as the Computer Game Developers’ Conference. And of course, if ever a professional association of game developers gets off the ground, it could probably kick in some money, too.
It should be possible to raise, say, $25K this way. This money would then be awarded to the best game reviewers on an annual basis, perhaps with the award presentations announced at the conference. I think that the money should be spread among a group of reviewers rather than concentrated on a single one. That way, every reviewer out there knows that he has a real shot at getting an award. If five reviewers split $25K on a rank-weighted basis, there should be plenty of money to motivate the reviewing community.
The judges for the awards must be chosen with care. If they are too close to the products, the awards will be perceived as bribes for favorable reviews; the taint would ruin the awards. Ideally, the judges would be knowledgeable consumers whose objectivity would be beyond question. But how do you find such a person? It’s almost as if "knowledgeable, objective consumer" were an oxymoron.
The second best scheme would have a large group of developers casting votes for the best reviewer. While their individual objectivity may be suspect, their aggregate objectivity deserves more respect, and they certainly qualify as knowledgeable.
It’s obvious that we do not have any organizations in place to implement any such scheme. But if we ever build an association, this would be a likely task to start with. Remember, though, that the ultimate goal of any such system must be to motivate good reviews, not favorable ones, and the best way to get good reviews is to have good reviewers and good people of any stripe cost money.