kypaddler,
A few years back, maybe before your "time" we had a lengthy and spirited discussions about wearing PFD'S and other safety issues related to taking a swim.
I've only been over once, 2nd year solo, and I made a mistake and payed for it. It was May, and the water was damn cold.
I was close to shore, and my ONLY concern was getting to shore and getting warm. I keep everything lashed in canoe, except the paddle I'm using, so when I went over, I let the wind/waves push me to shore, I knew the canoe would he following.
I have a Cabelas comfort mesh vest, no excuses for not wearing on hot days. It has two big zippered pockets with my "self rescue" gear.
I have zippo lighters in waterproof container, plus a magnesium firestarter bar. Also some tinder. I also have a small 4oz space blanket, and IMHO, a potential lifesaver, a half dozen chemical heat packs, the kind oxygen activates.
Dependiong on YOUR physical shape and water temp and exposure time, you MAY or MAY NOT, have the dexterity to start a fire with the tools you have?
When I hit shore, the first thing I did was use my teeth, and tore open 2 heat packets, and placed in each armpit. Then 2 more OVER my t-shirt, tucked in my underwear OVER my kidneys. I wear ALL fleece/synthetic on trips, regardless of season, so I just removed fleece pullover, "spun" dry overhead, and put back on. The heat pads soon had my CORE temp up to normal, and my canoe had washed into shore by then.
I never even changed clothes. I bailed out canoe, got some granola in me, and made a hot cup of coffee, while waiting for wind to die down.
I would consider staying with a overturned canoe , IF it was close enough to shore, I thought I could hang on and "guide" it in with the wind and waves.
In group situation, I would hang on to canoe, let your buddies paddle you in (less chance of dumping them), and let them retrieve the canoe/gear, IF they could without to much risk to themselves.
Packs/paddles/canoes float, anything that wasn't tied in and can sink, already has

If you a seasoned tripper and know to line your packs with plastic liner, they'll float high and dry for a long time.
NO piece of gear is worth dying for, take care of the paddlers first, and regroup when you can.
I agree sitting on your PFD, is like sitting on a seatbelt. It's a lot easier to pull a person with PFD into a canoe, than a person that is potential in a panicked state and not thinking of YOUR safety. It's also easier to hang on and let the canoe take you in, being buoyant in a PFD saves you energy, you need to keep your core temp up.
Whatever you decide to do, have a PLAN, that ALL know about and go along with. IMHO it's better even in a group, if EACH canoe has the right safety gear, or even better you have some heat packs in EVERYBODY'S PFD.