I agree, with cold water temperatures, your priority is getting to shore as quickly as possible.  During the shoulder seasons, you should carry waterproof fire making supplies in your pfd.  You mentioned Bailey Bay swims and how long you would be in the water.  My dad had such an experience last summer, he and his canoeing partner capsized in the middle of Bailey Bay and spent over an hour in the water before they reached shore.  Luckily it was late August and the water was warm.  If it had been early May, they would have been in big trouble.