Yeah this is one of those things that I've run into. Like going from the Gygaxian definition of hit points, characters who mostly evade damage would best be modeled as having high hit dice- like Monks or Rogues. But instead the game chooses to model this in other ways, like having abilities to reduce damage taken, because a lot of people cross their eyes because they see that Fighters and Barbarians have lots of hit points.
Fighters and Barbarians tend to be big, tough guys, so obviously that's where their hit points come from.
And of course, Constitution is the big offender here. If hit points aren't meat, then why do you get a Con bonus to hit points instead of say, a Dexterity bonus?