If he fails his Saving throw, he will let me in without any problems. (Will he feel that he was enchanted?)
That's right. Will he feel it? If he makes his save, he'll know someone just tried to cast a spell on him. If he fails... I'd be inclined to say no, but there might be a ruling I'm not aware of.
The example in the spell description is suggesting to a knight that he give his horse to the next beggar he meets. I think if the knight fulfilled the suggestion and then realised he'd been enchanted, he'd just go take his horse back... sort of defeats the purpose

So I'd say he'd not automatically know.
He might be able to reason it out later... "Why would I do that? ... I must be bewitched!" ... I'm not sure.
I know that Hypnotism specifically says that on a failed save, they don't remember the spell, but Suggestion doesn't mention it.
I can also stand before him and Hypnotize him (better no other people watch this). Then speak the same words as above to him. Perhaps he will let me in, if he had been close to letting me in without the spell. But he will not let me in, if he had strict orders to let noone pass through.
Correct?
Sounds about right. Check out DMG p149 : NPC Attitudes. That's what Hypnotism is referring to when it talks about "two steps friendlier".
I would say an Indifferent guard would laugh at you, a Friendly guard would be very apologetic about how his orders wouldn't allow it, and a Helpful guard would find some way to let you through.
Remember that Charisma can also influence attitudes. So an Unfriendly guard could be made Indifferent with a successful DC 15 Charisma check, and then made Helpful with a Hypnotism.
(while I pass through and the hypno is over, won't he try to stop me again?)
Read Hypnotism again. Even after the spell ends, the subject retains the new attitude towards you, but only with regard to that particular suggestion.
So if you came out and asked for directions to the moneychangers, the Indifferent guard might just grunt at you. But if you asked to be let back past, he'd suddenly act Helpful, since that's the trigger the Hypnotism left.
-Hyp.