.....
My Temple campaign started with 5 players, but then was wittled down to three. I think the 4 PCs would do fine with the module up until the Temple of All-Consumption, but they definetely will have to find some allies or some very creative strategies. No kick in the door style IMO.
The alternative, and this is what I did once my group shrank, was to plop down an intervening quest between Hommlet and Rastor. I wasn't playing the campaign in Greyhawk, so this was a bit easier, since I moved the ToAC (all the PCs gained from Hommlet was the name Rastor, not that village's location or distance) to a much farther distance then Monte wrote. The added quest in question was a actually a fight against an outpost of the Water Temple that allowed the party to use a teleportation circle to reach the continent the ToAC was on. This allowed me to move the campaign from a forested European feeling area to an area that sort of felt like China's Xinjiang region. The players in my group were playing the campaign in Beijing, and most of us had been to that province.....and were all fascinated by it.
IMC the four elemental temples each had a somewhat greater influence outside the ToAC itself. The elemental temples were actually the main arm of Tharizdun's cult, since the doomdreamers were few and often having bouts of madness after communion with their dark god.
The Water Temple was the one that happened to control the bridge to the continent with Hommlet on it. The process of the PCs defeating the Water Temple outposts and also a mercenary orcish army that the Water Temple had funded allowed the three PCs and a NPC azer fighter (basically a cohort) to reach the ToAC with the PCs' average level being 8th.
Hope that helps,
C.I.D.