You said you found a unoccupied campsite. They said they were at a campsite on the opposite side of the island. There are many campsites on islands areas that are close to each other, sometimes you can see the other campers. Unless the island was so small you could literally see each other I wouldn't consider moving or ask someone else to move.
   
I camped on a island in Burntside lake that had 2 campers on the other side. I had paddled to that site first, found it was occupied. They offered to share, I said I'd go to the one across the island.

  I hope you didn't leave but since you had another to consider, I understand if you did.