My Dad was 85 on his last trip with us.  We all paddle our own solo kayaks.  He did just fine but on the last few years of traveling together I was the one to carry both the kayaks and make the extra trip across the portage for the balance of the gear.  There were also a few times I ended up towing him against strong winds that were too tiring.  His problem more than fatigue was that his kayak was very light, he himself was under 160 pounds by then, and he packed the bare minimum.  So against the wind he had no ballast and got blown back more than the rest of us.  He still paddles his kayak at 94 but no longer stays overnight out in the woods.

The last few years I have been going on 10 day trips with two 80 year old women who paddle their own boats.  They don't like portaging so we pick a route on one of the bigger lakes or a route which only takes one or two short portages to get somewhere else.  Last year son Dan and a friend of his also were along and they carried the ladies boats and most of their gear across the portages for them (Big Knife from Carp to Knife).

I think the physicality is the big issue.  I have "younger" lady friends in their late 50's and early 60's who would like to come along but because of cantankerous backs or weak wrists they can't do the work or sleep on the ground.  I am fortunate at almost 61 to have been built to a sturdy model and not to have any joint pain or stiffness to aggravate the camping.  :)