Well, not many votes, but I'm in the minority so far LOL!.
As DM, I ruled no at the time and here is why:
Repelling blast affects a (singular) creature. A swarm is not a singular beast. Other SA rulings, involving things like Wild Shape of example, state a swarm is several creatures--hence why you cannot Wild Shape or Polymoprh into a Swarm of Rats.
Also, giving the conditions immunities: grappled, prone, restrained, the ability to "shove" (a form of a special attack akin to grapple and knock prone) or push the swarm as a creature narratively didn't make sense in the moment.
Something like this could work IMO.
I'll have to discuss this in our next session with the group to decide how we want to proceed in the future. I'm still inclined to rule no in the future, but we'll see.
FWIW, the swarm surprised our rogue, hit for 7 damage, and had initiative so hit again the second round for another 7 damage before the rogue has his turn. So, the rogue went from 17 hp to 3 hp. The repelling blast would have pushed the swarm back so the rogue could move back without provoking an OA. It was a non-issue, since the rogue used cunning action to disengage before retreating behind the tank.