"Him or me" is a terrible way to persuade someone.
Depends on whether you mean it. One should not throw down the gauntlet like this without being prepared to deliver - that is, if you're going to say "Either he goes or I go," you need to be ready to walk away (and not be resentful afterward) if the answer is "I'm not willing to kick him out."
However, if it's gotten to the point where it really is him or you, and less drastic efforts to resolve the problem have failed, I see no reason not to make your position clear.