We kept canoeing "just for the two of us" when our children were small.  We usually did a canoe trip every other year and our children stayed with our friends in Minneapolis who had children the same age.  I was a full-time mom and I justified this because I was raised to believe that time alone as spouses without the children was good for the health of the marriage relationship.  I still believe that, and I think, for us, it was true.

On the "off year" we tried to do a family vacation of some other sort, sometimes camping, sometimes not.  It would depend upon where we wanted to go and what we could afford.  

Then when our daughter was 8 we offered her a canoe trip of her own with the two of us, and she enjoyed a four-day trip in Algonquin PP.  When our son turned 8 we offered him a canoe trip of his own and he said he would prefer the time in Minneapolis with his friends.

One family trip when they were older teenagers was fun, and I thought perhaps we had planted a seed in both of them for canoe-tripping someday on their own.

Result?  Our daughter and her husband did a canoe trip in the BWCA once a few years ago.  It was a challenging trip, they enjoyed it, and I somehow don't think they will ever do another.  It was sort of "been there, done that."

Our son married a young lady from Delaware who rides horses and isn't an indoor person, but camping is not her "thing".  I sincerely doubt he will ever get her out in a canoe for more than a quick paddle on a small lake or stream.

You can always find reasons to feel guilty.  Why bother?  Your children will find their own interests and they may not parallel yours.  Or they may "discover" canoeing and camping on their own, and realize that they have that in common with you.  Take your trips while you can and share your experiences with them.  Enjoy.

Just two cents from an old lady.   ;)