Depends on how attractive a fishing spot it is.   :D
Never toss your lure in another angler's pool.

Once I paddled up to a couple swimming, nowhere near a site.  They were pleasant and enjoyed chatting.  Suddenly I realised they were skinny dipping & wanting to get out of the water.  

Anchor off a campsite?  Depends on how close and the topography.  If someone is camped on a barren rock in the middle of the lake, hard to not be in view of their site.  One certainly does not linger & gawk at an occupied site, not without chit-chat.  Smile, nod, keep moving.


Folks who get indignant at paddlers passing by shouldn't set up camp on main street.  That's the usual dirty-look situation I've seen.  People who think they're entitled to behave as though they're the only ones on the lake.  If you want solitude, go find the biggest unmaintained portage without any back-doors.  Anywhere within 1 day of travel from the car isn't solitude, it's nature, it's nice, it's not solitude.