In my experience casters often were either in the back row and did not need to move or were caught in melee and could only do a five foot step when casting. The times you wanted to move and cast were fairly rare.
The perception that it is a detriment is pretty big though.
Conceptually the big detriment for sorcerers for me was the being behind a level on top level spells so most every other level they were limited to lower level spells. It made them a definite step down from cleric, druid, and wizard options in power potential.
I recall it being a pretty common house rule to bring sorcerer progression in line and remove the casting time penalty. It was generally agreed by the end that, if anything, the sorcerer deserved compensation in the form of their extra slots per level for the decreased flexibility of a known spell list.