Brad Wardell's views about technology, politics, religion, world affairs, and all sorts of politically incorrect topics.
What constitutes a fair review?
Published on October 4, 2006 By Draginol In GalCiv Journals

I don't like negative reviews of things I make. And no matter how far into the fetal position they put me, there's really nothing that can be done except hope that your product or service is good enough that, on the balance, users and reviewers will respond favorably.

The reviews on Galactic Civilizations II have been very positive overall. But we also got our share of reviews that I either wasn't happy with or felt were unfair.  But what can one really do?  In my case, if I felt a review was unfair and was able to document specific examples in the review or put together some other coherent argument as to why the revie was unfair, I'd contact the review site/magazine/whatever.

But one thing that must be done with great care -- don't attack the integrity of the reviewer or magazine publicly, because that's a lose-lose situation.

The average GalCiv II review was roughly a 4.5 out of 5.  But at the outter edges there were two reviews that were under 3.5 out of 5.  Conspiracy? No.  Were they fair reviews? Impossible to say.

Recently there was a hub-bub between the company that makes Sword of the Stars and Tom Chick.  Back in 2005, Tom was contracted for a lump sum to help write the Galactic Civilizations II manual since Tom had spent the last 4,000 years ribbing me for how craptacular our user manuals were.  Roughly a year later, Chick, as one of the best known reviewers in the industry, gave a negative review to Sword of the Stars. Kerebos, the game's developer felt there was a conflict of interest.

Maybe I'm jaded but I just don't see it.  That would be like me saying that because we spent hundreds of thousands of dollars with PC Gamer (or CGW or whatever) in advertising that PC Gamer that they should knock down similar games.

From a purely business point of view, games don't compete in the sense they do in other markets. That is, a person who likes GalCiv II is likely to give Sword of the Stars or Space Empires V a serious look at. Buying GalCiv doesn't preclude a purchase of other games that are in the same remotely similar genre. 

More to the point - when it's time to put a product on retail, the # of units they'll take is largely defined by how well other games that are similar did. 

Ever sell a house? It's a lot like that.  You WANT your neighbors who sell their houses to get as much money as possible because that sets up how much you are likely to make selling yours.  If other houses do poorly on the market, it affects you. 

It's not commonly known but one of the beta testers for Galactic Civilizations II was Soren Johnson (designer of Civilization IV).  Some of the features GalCiv II has in common with Civ IV came form Soren's suggestions.  The point being, game developers usually see themselves on the same team. We want Civ IV to succeed and they wanted GalCiv II to succeed.  Similarly, we want Sword of the Stars to succeed and we want Space Empires V to succeed.

So while it is frustrating when your baby gets a negative review, it's never a good idea to impugne the character or reputation of the person doing the review.  I've written my share of frustrated emails but they've always focused on either the review or that the reviewer's perspective came across as someone who just doesn't normally like the type of game they were reviewing (i.e. a first person shooter guy getting assigned to write a 4X strategy game and saying it's "boring" and giving it a low score).

Sometimes it may not seem this way but we developers are on the same side.  We want other games to succeed.  Moreover, the best known reviewers may not always agree with us on our games but they do have a high degree of integrity in my experience.


Comments (Page 2)
3 Pages1 2 3 
on Oct 05, 2006
You know...quite frankly this doesn't surprise me at all. I didn't buy SOTS for one main reason: the devs were really, really unprofessional. They were mean and nasty to anyone who voiced any kind of negative opinion about their game on their forums. It disgusted me and I simply turned away. So to see this very visible "hub bub" to me isn't surprising--Kerberos lacks social skills and it's just hilarious that they screwed with Tom Chick.
on Oct 05, 2006
I BSed my way through enough papers in school on material I didn't read to recognize a faker. If you look hard at the reviewers in this industry you'll find that a good portion of them only haphazardly play the game before reviewing it, and most openly admit they don't play the game extensively first. Such folks are as of much value as movie reviewers that only watch a few minutes and then just rely on a synopsis of the plot to review it. It's a shame when games don't get the lauding they deserve because reviewers refuse to put in the 40 hours it takes to complete many of them.

Maybe that's why the graphics-fetishist ideal of game fandom has not been defeated. It's easy to review graphics. The meat of the game, the playability, the story, the game mechanics, all have to be played through to really understand. I would think someone who wrote the manual would be MORE qualified to write a review, because God knows most of the reviewers don't bother even reading the manual.

on Oct 06, 2006
To me the only reviews that count are ign and gamespot. Sometimes I've looked into a game solely because of their good grade and that I sort of might like the game.
on Oct 06, 2006
i can see the potential for a conflict of interest there. some people don't have the cash to buy multiple games. maybe someone can only buy 1 game a year or so,,,and they will probably go with the better reviewed game. to some hobbyists, or those with high levels of disposable income and time, maybe "trying out" a bunch of sim. games may be in the cards, but i don't think most people are that rich or "into it.." the real estate analogy really doesn't work for me. you may root for another house selling for a high dollar if you are not selling your house at the time, but if you are, that other house is competition. in your scenario, i see it as more of the same or sim. time, thus , it is competition. and every good marketer knows that most people's entertainment budgets are limited, so they want to position their entertainment product as "the purchase that has to be made 1st" because the consumer might not have the scratch to make it their second purchase.
on Oct 06, 2006
Agree with Sean's point that people tend to purchase the perceived #1 on the list, IMHO.

I worked part-time with a software developer focusing on a niche market, and we face a similar situation - even though we believe we have made the best product available on the planet, most consumers still go for the biggest brand's perceived #1, much inferior product.

Having said that, I think kerberos' reponse is pathetic. It makes them look immature and unprofessional. OTOH, I feel for them as I can imagine what kind of pressure (both financial and emotional) they must be facing with their not-so-good product.
on Oct 06, 2006
That would be like me saying that because we spent hundreds of thousands of dollars with PC Gamer (or CGW or whatever) in advertising that PC Gamer that they should knock down similar games.

by the way,,,this is the reason "consumer reports" magazine exists. because people see that conflict of interest between an advertiser and a product. i'm not saying that's right, wrong, good or bad, it just is.
on Oct 07, 2006
I have been on the SOTS forums a lot. I didn't find the Devs to be mean and nasty at all. Basically you got what you gave. I have seen TONS of posts that constructively criticized the game and the Devs answered with either a) Why they did what they did, or Said it was an interesting idea.

On the other hand if you went on and ranted about how the game was crap, and how the devs didn't know what they were doing etc, etc - then yeah - you got it with both barrels from the devs - and deservedly so. Same thing should happen on this board.

I have found the devs on the SOTS forums to be as active and honest in the forums as the devs here.

Dano

PS - I really like SOTS, and I agree with Frogboy that there is plenty of room for both games (they each approach the genre differently)
on Oct 07, 2006
was about to say the same dano, but you've put it in better words, so i'll just second you here   
on Oct 08, 2006
beg your pardon?
on Oct 08, 2006
Please ignore the spambot behind the curtain
on Oct 08, 2006
Weatehr there was a conflict or not is not up to me. I cannot know a person's true motives. However, I do feel the review was rubbish. I could not agree with it and reading it made me angry. I got a feeling he didn't play the game enough. But whatever, that's old news now.
on Oct 10, 2006
Maybe I'm late to the game on this topic, but I just read all the reviews, the comments on the CGW boards, etc., and came away with the following impressions:

1. Sure it's a potential conflict. Games in general -- but strategy games in particular demand a great deal of our time, and we make choices about which ones to play. I'm playing GalCiv2 and not SotS, in part because of reviews.

I work in the publishing industry, and we deal with the appearance of conflict through the use of a disclaimer, disclosing the potential conflict, which must have been anticipated given the degree of discussion that Chick implies went on prior to the review. In this case, a disclaimer might have followed the review, and been worded as follows:

"Reviewer Tom Chick is a freelance writer credited with authorship of the manual for Galactic Civilizations II, a competing title. There is no link between his compensation and the success or failure of any reveiwed title."

2. Even given the potential conflict, I trust Tom Chick's reviews as much as ever, and when I'm making decisions about which games to play, I always look for his reviews for their insight and the quality of his writing.

I'll go so far as to say that I'd love to read Tom Chick review of GalCiv2 -- I still believe he could offer an honest opinion on what they got right and what they got wrong. Just lay it all on the table, and your integrity will never be questioned, particularly when you're as good at what you do as Tom is.
on Oct 10, 2006
Just lay it all on the table, and your integrity will never be questioned

unless he posts on joeuser, then that's all people will do, lol

on Oct 11, 2006
I agree with Dano. the Kerberos crew are extremely nice. I have read several of the SotS reviews and i find most of them to be total crap. Most of the "flaws" they found came from not spending time to learn the game. I admit the first time i played the game I was rather put off. But over a little time it did grow on me and now I love it. also the few things that are wrong with it should be fixed in the upcoming patch, and future expansion. Frogboy, I truly appreciate how civil you are about this.

The lead designer for SotS is nice, if a bit..... outspoken. Yes the first few posts of his i read he came off as a jerk, i almost didnt buy the game because of that. After sticking around the forums for a bit I learned that he didnt mean to come off as a jerk he just didnt put up with a bunch of people telling him that his game was total crap. he may not be as polite as Frogboy, but he is just as devoted to the game and its community.
on Nov 07, 2006
Ever sell a house? It's a lot like that. You WANT your neighbors who sell their houses to get as much money as possible because that sets up how much you are likely to make selling yours. If other houses do poorly on the market, it affects you.


I suppose that a lot of the time the industry works that way, but there are certainly exceptions. For example, Galciv was definitely able to capitalize on MoO3's shortcomings.
3 Pages1 2 3