My wife's rule was that we had to wait until the girls could squat in the woods on their own.  We were not going to haul diapers in/out.  They were 5 and 6 their first trip.  We planned a route to base camp and made the decision when they said they were done portaging that was the lake we would camp on.  They made it 3 portages in 90+ degree weather.  We base camped for 3 days and came out.  We had 1 storm and weather over 90 degrees every day.  They still loved it and last summer was their 5th year in a row.  They are determined to beat my current run of about 30 years in a row and counting.