It's difficult to enter a conversation 20 pages in. Not sure how 'on-topic' it is at this point as I've only read the first two pages but, to comment on the OP:
I think this isn't unique to 5e. As far as I can remember, this has been an issue with every version of D&D I've played - speaking specifically about how appropriate CR to PC level can affect the narrative.
A few things I've always noticed in d20 games:
1. The PC abilities interact primarily with the combat mechanics. Often abilities that interact with other pillars are considered by many as 'too weak' (the Battle Master's "Know Your Enemy") or trivializes an entire pillar (some of the Ranger abilities or, arguably, the Outlander background on the exploration pillar.) There are almost no abilities besides spells and, maybe, one or two feats that interact with the social pillar.
Because of this, a DM is more likely to use combat to provide PCs an opportunity to use their abilities. As they level, combat gets more difficult as a way to test and challenge those abilities which has strange consequences on the World.
2. The Campaign World has to reconcile the fact that there are Dragons and other nearly unbeatable creatures wandering the countryside. Why has the wilderness become so dangerous just because the PCs are higher level? At low levels, an Ogre is rare, but now they just randomly leap out of the bushes when traveling down the road!
And if the DM splits their world into 'civilization and wilderness' Where the civilized lands have less 'danger' (which enables 1st level PCs to travel and explore without encountering an Ogre or Ancient Dragon), then the assumption is the Ruling Power must have a standing army with NPC Wizards or Fighters that have enough power to stave off such threats. This has repercussions on the structure of the world itself.
As far as alternating the challenges for PCs go, one of the better DMs I've had did a few things to mix it up and made the campaign world make sense:
1. The fiction was well-established. Most people didn't travel the Ogre Hills...because it was reputedly full of Giants. We ventured there at second level and it was a TPK as we encountered 3 wandering Hill Giants. We learned our lesson and rolled up another set of characters. Sure, there's an Ancient Dragon in the Dragon Peaks but it's so far away and difficult to access that, even if you wanted to get there, it would be near impossible to access. As you levelled, you were able to venture further into more dangerous territory and your reputation for being able to survive those challenges allowed NPCs to send you on adventures because you were some of the few who would choose to brave those areas. Sure, there were other powerful NPCs but they were rival adventuring groups or those working for Powerful People who liked to keep their Body Guards close.
2. At higher levels, we would often encounter challenges that were super easy. Of course we would! We were some of the most powerful adventurers in the country! The problem is that we had too much to do! Some jobs were a nuisance. We eventually hired lower-level NPCs to do all the adventures that had some urgency attached to them but which we didn't have time to do. Sure, there was Ogres attacking our home village but we really needed to go to the Shadowfell to take care of that Necromancer that was summoning an Undead Dragon. We sent Team B to the village. And, eventually, Team C. We had more than enough secondary magical items to kit them out.
I think many Campaign worlds get structured this way because of how the narrative must interact with the mechanics