It really depends on what sort of world you want to build. If it bears even the slightest resemblance to medieval Europe, then Magical Medieval Society: Western Europe is probably the most important book for you to own.
There are lots of other sourcebooks that might or might not fit in with your ideas. For example, if you need help with a pantheon, Book of the Righteous might be helpful. If, on the other hand, you need some cities and towns, ready-made, Seven Cities might be just the ticket.
Can you give us any details on the feel you are hoping to acheive? High magic, low magic, hordes of illiterate barbarians roaming an untamed wilderness or lots of effete aristocrats manipulating each other in a string of large, decadent cities?