I think all three options are good calls, depending on how you want the situation to work.
1: The rope grows as it is technically "worn", now there's a tied-up giant in the room. Much lulz ensues.
2: The rope breaks, because cinematically that's awesome. A cool fight ensues.
3: The creature is unable to grow and the spell fails, because the rope counts as limiting space.
And an unmentioned, and possibly disturbing 4: the creature gets sliced in half.