Yeah, the official CR rules are voodoo. Basically, what they did is they made the creatures, then playtested them against a bunch of different parties. They took the data from playtesting, and assigned CRs. The problem with this is that it isn't always perfect, and sometimes the playtesters might have been using superior tactics.
Fortunately, our own dear Upper_Krust has spent a CONSIDERABLE amount of time trying to come up with a consistant method of generating REAL CRs, based on the following assumptions:
Upper Krust's Revised Challenge Ratings v5 said:
There are two general principles regarding CR: firstly that it signifies a moderately challenging encounter for a party of 4-5 characters of the same power....
eg. CR 15 suggests the opponent would be a moderate challenge for a party of four (or five) 15th-level characters.
...secondly, that CR parallels character level.
eg. A 10th-level character is CR 10 (at least before Ability Scores).
Under this system, a babau is CR 10, not 6. Heck, Balors become CR 33. It's been extensively playested.
Upper Krust goes a step further though, and not only recreates CRs, but figures out what a given party should be able to handle in a much more accurate and quantified manner. As such, he redevoled "Encounter Levels"... which are a measure of
relative power. I mean, when you're level 1, fighting a level 3 foe is incredibly difficult. When you're level 20, fighting a level 23 foe isn't quite as daunting. He makes the assumption that if you double the challenge, the EL increases by 2 (an EL 14 is twice as difficult as an EL 12). Then, by comparing ELs, you can determing how much of the party resources should be consumed in that encounter, and also the best way to award XP.
For example, fighting an encounter of +4 EL is incredibly difficult, whether you're level 2 or level 200. (If you're level 200, your EL is only 31, as EL is logrithmic) Fighting an EL +5 is for all intents and purposes, impossible to win.
The EL rules are pretty simple to follow, and have nice easy to read charts. It factors in multiple creatures being fought (so if you pit the party against 5 quasits and 2 babaus, you'd total the CRs (the revised, real CRs) and get 5*3.5+2*10=37.5, which translates (logrithmically) to EL 21, but then you subtract 5 because there are so many creatures, for an EL of 16, which covers a CR range of 14-15. As such, this is an encounter valid for parties of levels 10-20.
(the reason for a penalty to EL for having more creatures in the encounter is for things like tossing 500 rats at the party. A rat is CR 1/8. 500/8=62, which is EL 21. But 500 rats could be easily slaughtered by a couple well placed fireballs, and those rats will be hitting higher level parties only 5% of the time, and doing very little damage with each bite. True, 500 rats is more challenging than just 2, since they simply keep comming, provide flanking bonuses, and the sheer numbers might eventually mean some bites get through before great cleave and area attacks take them out, but that's why having 500 creatures lowers the EL by only 17, giving 500 rats a true EL of 7, equivelent to a CR of 3.)
Okay, so it sounds complicated, but I've been using it in my games for a while now, and it really IS easy and fast and convenient and above all... ACCURATE. And you can't beat that.
I've attached the full document below. At least give it a look over. Use the bookmarks and jump to "Revised Challenge Ratings" if you just want to see the more accurate values for creatures, and not how those numbers were generated.