I've gone with 3 adults twice. Rented a Minn3, then next year rented a Seneca. The Minn3  was better, but neither was fun. Anyway you go, finding room for packs and people is tough. I tried to find a Minn4 to rent, but couldn't find one. I thought it would work best for 3 people after removing one seat. "Never again" is my best answer.
Renting is expensive, but I wouldn't want to own one either. I see the need, but hopefully the kids could handle their own boat by their early teens. I've seen several scout groups traveling with 13-14 year olds handling their own boat. You might have to be careful about your travel days, i.e. weather & distances, but I'll bet the everyone would have more fun.
Personally I wouldn't mind carrying both boats with light packs, if everyone could handle the rest of the gear.