Typically, there is no limit.
If you don't feel that is realistic, I'd suggest using a progression similar to 3e:
Large (base 2x2): 2 Medium or 4 Small creatures
Huge (base 3x3): 4 Medium or 8 Small creatures
Gargantuan (base 4x4): 8 Medium or 16 Small creatures
Gargantuan (base 5x5): 16 Medium or 32 Small creatures
etc.
For fantastic creatures like purple worms, I'd no longer count any swallowed creatures once they've been reduced to 0 hp (or negative bloodied value for characters), since they've effectively been dissolved by the potent digestive acid.
I'd also like to point out that real-life creatures that swallow their prey whole (e.g. constrictor snakes) have no problem swallowing creatures that are larger than they themselves are. Their bodies are flexible enough to expand accordingly.
Hehe, yeah, I have also seen some crazy pictures of snakes devouring animals much larger than they are... however, they won't put up much of a fight while feasting like that.
What we did when this came up, was settle with one size category lower as the capacity and the 2:1 ratio for every size level for smaller portions.
But as I said, I was curious, whether there is an actual limit.