You're treating magic and spellcasting as entirely homogenous, it's like an extension of that point "well why doesn't everyone in a fantasy setting learn magic", not everyone can. Not everyone has learned the right lessons, not everyone can apply the right lens... And even if something does click into place for you, that's only one approach amongst many. A Wizard is different to a Cleric is different to a Bard. And so for the half casters and beneath. "oh well these classes flatly know more about/practice more magic so they should have access to every practice" doesn't scan.
Like I was just reading some of Berserk this week, and there's a part where the gang go to the Elf homeland that's also home to a small conclave of mages. And they're dumbfounded by the two magic users in the group's ability to practice magic without drawing a magic circle first, nevermind the magic they actually demonstrate. That's what it reminds me of (obviously extremely different language and details for the magic, but nevertheless).