KB, your question is fairly straightforward. I'm using the PHB, not the SRD, but I'm sure they match in this part.
Imagine that you were fighting with a double weapon instead of two weapons. If you only made one attack instead of two, would you incur the two-weapon penalty? Of course not. Why should it be any different when wielding two weapons?
Refer to page 143 of the PHB. It plainly says that if you wish to make an extra attack (only allowed in the first place because you are wielding two weapons), then you have to make a full-round attack. No full-round attack, no attack with the off weapon (or the prime weapon, if for some reason you choose to make your only attack with the off weapon).
Page 143 also allows for converting a full attack into a standard attack + move sequence. My take on that is that's the "oh, he dropped after the first attack option". So, if your character declared a standard attack followed by a move at the beginning of the round, then I'd assume that was a flat-out attack with the primary (or off, if so declared) weapon with none of the distraction from trying to fit in a second attack by the other weapon. I wouldn't assess the two-weapon fighting penalty to that attack. On the other hand, if your character declared a full attack and then conditionally converted it to a standard attack + move sequence ("I'll do a full attack on the orc in the green armor, but if the troll flanks me, then I'll stop after the first attack and move 30' to under the bridge"), then I'd assess the two attack penalty to the first attack, then allow the character a move action. Clear?
So, if you want to use the Two Weapon Defense feat, you have to be actually wielding two weapons, but you don't have to be attacking with two weapons. If that were true, you'd have to make a full attack every time you used the Two Weapon Defense feat, and that doesn't make much sense to me. For example, what if you wanted to stack Two Weapon Defense with Total Defense (a standard action)? Page 142 says that Total Defense doesn't stack with fighting defensively or with the Combat Expertise feat because they both require an attack. Are you required to attack to use the Two Weapon Defense feat? No, you aren't. In fact, it specifically says on page 102 that you can combine Two Weapon Defense with either fighting defensively or with Total Defense (although you only bump your defensive bonus from +1 to +2 when doing so). Since you can't use Total Defense and a Full Attack in the same round, I think that makes it pretty clear.