The one area (ha ha) I can see ambiguous area of effect spells causing an issue is with sleep, which explicitly can affect both allies and enemies. Adjudicating which creatures are affected by a sleep spell is always tricky.
I might end up using the Rules Cyclopedia spells.
Easy answer there is to have it always go for the weakest (i.e. lowest HD/level) creatures in the AoE first, and work up from there. If there's a tie, creatures closest to the AoE's center get affected first.The one area (ha ha) I can see ambiguous area of effect spells causing an issue is with sleep, which explicitly can affect both allies and enemies. Adjudicating which creatures are affected by a sleep spell is always tricky.