I'm not sure there is an ideal body size for canoe tripping.  I've seen big, tough football-player types that were wimps on portages.  My wife and other women I have traveled with in the far north, on trips of over six weeks in length, are only in the 5-2 to 5-4 range and have done portages of around three miles in length.

I'm 5-10 and 155 pounds, 49 years old, and have yet to meet someone my age that can carry an 80-pound Old Town Tripper like I do.  Not bragging, but just stating a fact.  

I throw a ball like a girl, though :).

Like anything physical, it's more a matter of skill and conditioning that comes from spending a LOT of time on the trail.