I responded as a birthday suit swimmer but in reality my swimming is a mix of birthday suit, shorts, and/or PFD depending on the situation.

I sometimes do as IC does at the end of hot, sticky portages.  I barely slow down as I shed the pack and just keep right on walking into the water.  Usually I remember that I have a knife and other things in my pockets that I don't want to get wet before I get too far, though.  If the need is dire, I walk back to dry land, shed the shorts (but not the BVDs since this is a portage) and head back in for a full immersion.  Other times I will stop short and just stick my head into the water...an amazingly effective cool-down without the total immersion.

In camp, if I'm swimming to get clean, it's b-day suit time.  If I'm swimming to just escape the heat, I usually wear shorts and will at least have the PFD handy in case I start feeling lazy.

Cliff-jumping?  Been there, done that, and will do it again if the opportunity arises.  It's not a must-do activity or anything, but if it's convenient and safe (and yes, safe is a matter of personal opinion) then it's just another way to enjoy the trip.