When viewed through this lens, the only true balance you can hope for is a soft balance wherein no one feels particularly outshined by another player at the table.
How horrible would be a game that had some sort of hard character balance! Soft balance is perfect for me. A type of hard class/character balance would indicate a rigid set of mechanics with fixed power comparisons and exact equivalencies.
A Rakshasa will be a very different combat if the party is composed of a Warlock, Wizard, Cleric, and Druid as opposed to a Fighter, Ranger, Barbarian, and Monk.
Awesome! That sounds good to me. How boring would a game be where party composition did not affect how the encounters played out. Sounds to me like player agency actually matters when building characters in 5E.
The second, greater point is that a focus on encounter balance hinders 5E's promise of uniting mechanics with the fictional world.
Then why do you focus on encounter balance? The rules certainly don't. Sure, there are rules/guidelines for encounter building and daily XP budgets. But we all know that these are crude and inaccurate at best. So yea, the rules don't have hard lines around encounter balance, so why do you?
By arranging the world in a way that adheres to strict power level, you bring to the forefront the combat simulation of the games more so then it already it.
But it doesn't adhere to strict power levels. But it does have power levels. And this is good. It's core to the D&D experience. But it doesn't mean that it emphasizes the combat pillar.
Since the DM is encouraged in a way of thinking that models the game as a combat simulator
What? Hmm... I think 5E does a much better job of not doing this than any other edition. My experience is that first editions had almost no useful rules or tools for roleplaying. Only combat and exploration / trap & hazard resolution. And those were... save or die. 3.X had rules for roleplaying, through explicit mechanics that turned role playing into roll playing with all sort of idiosyncrasies (too many overlapping and non-specific skills with mutual bonuses).
The scope of what I can create is hemmed in, which is fine in certain ways, but the stories that I want to tell are partially rendered incompatible with the game itself.
Yep, and this is true of any system that has rules. Especially resolution systems that rely upon mechancs and random number generators.
Playing the game purely in this way leads to a more rote experience, reducing the scope of the game in effect.
Then don't? Really, all of your problem seem to stem from your assumptions about how the game must be run. Nothing is stopping you from running and playing it differently than you self imposed restrictions.
Instead of creating new tools for helping DMs come up with creative ways that a level 5 party could beat an Ancient Red Dragon (such as with a Bard-esque arrow to a weak spot over its heart, or by finding a special gem that steals the dragon's vitality, or by giving ways a legion led by the PCs could potentially trap, restrain, and butcher the dragon), we instead get a bunch of stat blocks that show the dragon in various power stages, limiting the stories that are being told to "Can you kill this thing in a straight up fight now or later?"
Who says you can't do all these things? The DMG (I'm pretty sure!) actually suggest these types of "solutions" or encounters. I agree that the books don't tell a DM or player how to be creative, because there are just too many ways to do this. You could write thousands of pages laying out possibilities for just resolving an encounter with a dragon without resorting to combat, and you would still be incomplete.
How many discussions does this board have, how many memes get posted around the web, all about the 'party going off the rails'? This is exactly what they should be doing, finding unique and new ways to solve problems that the DM or adventure designer did not outline. Not because they couldn't, but modules are just outlines, they should never be taken as the single most interesting or best way to solve a problem. Combat is just the easiest way to document it, and if combat stats are not included, then the designer/DM has had one of the 3 pillars removed. Not true if the other pillars are not detailed to the same level. (because they can't be, because they are much more complex).
It really sounds to me like you have hemmed yourself into assuming the game is written to only be played one-way. IMO, it's not.
