I'm on the side of mental toughness being at least half of the equation, being in reasonable shape aerobically/anaerobically the other half.

Body type as someone else mentioned has it's pluses/minuses depending on what your attempting to do.

I used to backpack with a guy who had a 38" inseam, even though he was only 6' tall. On a relatively flat grade or stepping over logs, he could out-walk me any day. He rarely led the route though, becasue he suffered on the uphills/downhills. He also was a very moody person, which translated into others never knowing who was going to show up that particular day. So to me, being mentally tough, not a whiner/complainer means a lot.

You also need to know your limits and not let your ego rule the day.
Go to any gym and look around, you have any number of people, usually male, who are sacrificing form, to keep up with the other guy(s).

I double portage, even triple if the conditions warrant. If my life depended on me risking personal injury to make it out or save someone else, you could load me like a mule and I'd go until I couldn't go anymore.

That said, unless it's an emergency, you don't need to impress anyone, you just need to carry your share of the load, and enjoy the trip.
Common sense should rule the day especially on any kind of wilderness trip. If 3x trips across a slippery/wet granite slope is quicker/safer than staggering under  huge 1x trip load, then that's what you should be doing, solo or in a group.

Solo, your only limited by your personal strengths and weakness's, physical/mental. In a group situation, it's the weakest link in the chain principal. The trip route/gear should be based on what the " average" abilities of the members are, not the above avergae one's, unless they/you are able and willing to do what the others can't :-/

You can always increase the difficulty/range of your route if you find you've underestimated everyones abilities,  but sometimes it's hard to scale back and keep everyone happy if you overreach ;)