While there are no "designated campsites" in Quetico, I might have checked my map to see what it showed on the island. If it showed 2 sites, there's an argument you might use. If it showed the site on the site of the island you were standing on, there's another argument you might use.  Finally, if your camp was set-up and their camp was not yet set up, there's one more argument.

On the other hand, ST makes a good observation.  You were not alone and had another's well-being to consider.  With all the nut jobs around these days, it's most likely I wouldn't have argued at all & moved on.  Fortunately, there are a lot of campsites down there.

Ignorant folks like that don't deserve much of your time. Better part of valor & safety probably means that I would choose to move on... but he would earn my death stare before I left.  I'm not terribly noble that way when it comes down to it.  I would probably try to make him squirm as I left....

Jimbo   8-)