A villain called Archimagus from Spenser's The Faerie Queene dwelled in a tower, IIRC. I can't remember for the life of me if there are any sort of tower/wizard references in Chaucer or not.
I don't think it's too strange for towers to have been used as dwellings--not only due to their physical impressiveness, much like an obelisk or the like--but also for any sort of defensive advantage (able to observe the surrounding area, having a high defensive point, etc.). Also, any symbolic elements of towers can be seen in the Tower card of the Major Arcana of the Tarot (for example, the Hanged Man represents Odin hanging from the World-Tree).
And, possibly for Saruman (maybe Sauron), the tower could have been seen as a structure going along with the idea of "metal & wheels" that Treebeard says the wizard's mind is focused on--sort of a Middle-Earth skyscraper, if you will. In the movie, it stood as an axis to the wheel-like structure of the mines/forges/campgrounds for his forces.