I would echo starwatcher about time of year.  We were up in July one time. My wife and my first trip after our oldest was born.  We felt we needed alone mom and dad time without an infant and did a 4 day trip while the baby was at Grandma's.  Unfortunately, being July the weather was over 90 degrees every day and one day we were on a lake with an infant in one group.  In 90+ heat the shade is not very cool.  The baby cried a lot of the day and I don't think the parents' skills would have helped at all.

I have taken my children from a young age (my wife made me wait until they could pee in the woods on their own).  They started at 5 & 6 y/o for their first trip and have gone every summer since.  The biggest thing with travelling with young ones and I would say the same for infants as well, is that when they are ready to be out of the canoe you have chosen the lake you are going to camp on.  No sense pushing them if they are not going to enjoy it.  The whole point of taking kids young is to let them experience a place we enjoy, learn some life lessons and have fun.  If it becomes work they will not enjoy it.

One other thing I have done with my kids when they were young is stay to small lakes where wind is not as big of an issue.  We had at least one day of storms the first 3 years my kids went up with my wife and me.  We can all plan the perfect trip and prepare our group as much as possible, but it is the unexpected things that are the challenges.