I'd say Intelligence. If you have a +2 INT as opposed to a +2 Cha, then you can drop the skill points into those charisma based skills. Sure your max won't be the same, but you also have lots of other skills including some non-obvious ones. The druid can benefit from Herbalist and I especially found Sculpture to be good for using the shape wood/stone spells.
I also found that the animal handling/animal empathy skills did not need to be maxed out over much, though that may be more campaign specific.