I'm thinking the kid(s) will adjust faster than you and your wife will too a different approach to tripping ;)
 After all, native people all over the world travel and live off the land with their young children.
 I think the proper age is when YOU can deal with what it takes for a safe and fun trip. Kids just want to have fun, they don't care where their having it ;D
 Starting off them off young, making short trips before YOUR patience wears out or their bored will pay dividends down the road.
 My married friends who started their kids off as babies/toddlers on family car camping/backpacking trips, have kids who now take their kids as soon as possible.
 The ones I know who waited until their kids were 'older" found that they had developed other interests, like video games and other couch potato pursuits. They made such a fuss, it wasn't worth the hassle dragging them along  :(