Hmmm. The spell is pretty clear that it is any manufactured metal object, with weapons and armor in a 'such as' clause. I then proceeds to use the term 'object' repeatedly, so I think it does a pretty good job of making the intent clear. It runs into trouble, I suppose, where there might be an object that isn't really worn or held --maybe a metal icon that is decidedly a distinct separate object (so inarguably targetable) that you have tied to the top of your quarterstaff or something. That speaks to me as a sufficiently obscure enough situation that it would fall under 'DM makes call' rulings.
Edit: re-reading the spell, it states, "If a creature is holding or wearing the object and takes the damage from it, the creature must succeed on a Constitution saving throw or drop the object if it can." -- indicating that you must 'drop' something you are wearing. Somewhat confusing wording, but also hard to see how it could be abused. So just an odd novelty of the writing.