Serenity is one of those games that people have some pretty polarized opinions on. People either like it, or they @Q#$%@#$ hate it.
Here's a review of it:
http://www.rpg.net/reviews/archive/11/11650.phtml
It got Ennied for Best Production Values in 2006, and was also nominated for Roleplaying game of the year, but was beat by Artesia:
http://www.gamingreport.com/article.php?sid=20532
http://www.originsgamefair.com/awards/2005
The first printing of the game had a couple of issues, one of them being no character sheet. From the hue and cry that was raised, you'd think that was a capital sin. *shrug*
Some people have some problems with the system. They don't particularly care for the ship being statted out like a person, and there's some that don't like the math of the dice. It uses different size dice for attributes, like Savage Worlds.
The game system itself has actually been around for a while, and has been refined as it's gone along. The same system is what's powering the Battlestar Galactica rpg.
If you're coming at things strictly from a d20/D&D perspective, yeah it's probably going to be an...uncomfortable... fit. If you've played around with some other systems and like more than one game system, it'll be easier for you to make some adjustments to get what you want out of it.
One thing that some people bump up against is a difference in expectations when it comes to the game. It's kinda like saying, "Star Wars". People have different thoughts about what really makes the setting. In the case of Firefly, some people want to emphasize the whole cruising from planet to planet and barely scraping by. Others want to do some sort of "Big Damn Hero" kind of thing, and maybe focus more on struggling against the Alliance. A few are more intrigued with the idea of rogue psychics floating around trying to avoid getting nabbed.
Into that whole mix you have tossed the retro-tech of the setting. Anti-grav and six-shooter style pistols, next to horsedrawn wagons. Scanners and hypo sprays alongside a blend of western/southern etiquette rules.
From what I've seen some people "stumble" on the whole "consistency" thing. D&D has focused on very precise balance and consistency, and Serenity is more on the handwavey side it seems to me. The rules are a bit different, the setting mixes outmoded styles and ways of thought with future tech and considerations and it's just too jarring.
That's not a criticism of folks by the way. I mean the show couldn't appeal to a large enough group consistenly enough to stay on the air. Much though I like the show, it and the rpg aren't going to be to everyone's taste.
If it's any help in judging things:
I happen to like Palladium settings (I don't think the rules are going to destroy the universe like some people make out but I tend to avoid them), I thought C&C was fine as far as the rules were concerned (although they seem to need to be "fine tuned" to match folk's tastes), I'm a big fan of Unisystem (Cinematic), I dislike default D&D rules (I'm neutral on the whole 4E fight people seem to be engaged in), I think BESMd20 has the right idea but needs to be straightend up, I really want to run a Secret of Zir'An game, and I'm currently enjoying Monsters and Other Childish Things (it uses a simplified version of the rules for Wild Talents/Reign), and I have some _very_ mixed opinions on Fudge (the rpg. the desert rocks).
Me? I don't have a problem with the game, and I don't regret having bought it when it first game out.