Galaxy Guide #9: Fragments from the Rim
This is probably one of the best background/fluff sourcebooks that WEG ever put out, and it covers a wide gamut of material, from Rebels to Imperials to Bounty Hunters to slang to drinks to media and even some Force-stuff thrown into the mix.
Some other WEG books that are good for fluff (but are a bit more focused) are:
Galaxy Guide #10: Bounty Hunters
Pirates and Privateers
Platt's Guide to Smuggling
Rebel Alliance Sourcebook
Imperial Sourcebook
Galaxy Guide #11: Criminal Organizations
Galaxy Guide #8: Scouts
Galaxy Guide #6: Tramp Freighters (not only smugglers, but also an entire sector to base a campaign around)
Stock Ships (has a quite a few freighter designs not yet redone for SWSE, and includes floorplans for all the ships in the book)
Lords of the Expanse (covers the Tapani Region, which is a great setting for groups that prefer intrigue over combat)
Bear in mind that most of the WEG stuff is going to only cover the Rebellion Era and the EU stuff that happens between then and the New Jedi Order.