I have never liked the subclass system in 5E, but it has taken me a while to really understand why. Specifically, it has taken me running Pathfinder 2E to help me to understand why. So there is going to be some comparison in this post. But I am not also just saying "play PF2!" There are a lot of things I do like about 5E, and so it is an interesting design challenge to look for an alternative to subclasses that fits into the overall design ethos of 5E. And that is true even if I never go farther with any design other than this post.
A Note: This is not a + thread because I don't want to give the impression that I want to shut down discussion. However, if your point of view is that the existing subclass system is not only good but the right way to do things, I am not sure there is much for us to discuss. You might be happier in another thread.
The first, biggest problem with subclass in 5E is that you are forced to choose it and have it define your character ever forward, but you explicitly don't get to do that at 1st level. So, what? You just build plan to 3rd instead? It is a very strange choice that does not seem to have much of an upside. It simply locks you out of your defining features until you have bypassed the "squishy" phase of the game. Folks who paid attention during the Next playtest might be able to shed some light on the design intent here, but I haven't read anything that makes it make sense.
One solution to this is to have players pick subclass at 1st level and provide something -- a class feature or whatever -- at first level. And that's fine, but it only solves one aspect of the problem.
Another problem with subclass in 5E is that they are generally pretty rigid. For most of them, you pick it at 2nd or 3rd and them make few choices going forward besides your ASIs or spells. Some, like the Warlock, are better than others, with lots of choices in the form of invocations. But the Warlock is a design outlier (and actually one of the best designed classes in the game; but that's another discussion) and most players won't have too many chances to make development choices over the next year or 18 months of play. That feels bad to me.
So what do we do about that? This is where my comparison to Pathfinder 2E comes to the fore. Classes should have lots of options in a few different archetypical paths. PF2 does this through Class Feats, but that isn't the only solution. I think something like Diablo style talent trees is another viable system. Some players will diversify and pick up a lot of different "initial" talents, while others will be happy to hyper focus and follow the tree all the way to the fruit (as it were). As long as there are no traps or taxes, this works fine.
Another idea is to pair class feats/talent trees with a reintroduction of prestige classes (and whatever they were called in 4E). Let players who want to decide on a very focused thing with in-fiction ramifications. Prestige classes are kind of a fraught subject, of course, but to be clear I am mostly talking about those that tie the PC to the world with some cool mechanical bits and bobs. I am not talking about combat monster hyper-specialization prestige classes like the chain fighter or whatever.
In the end, the goal is to allow players to both create the character they want and to let the character develop throughout the campaign informed by the campaign. I don't think subclasses are a good way to achieve that goal.
What do you think?