Travel to Canada requires a government photo id and  proof of citizenship (a birth certificate etc). It is the return to the US that requires a passport, passport card or enhanced driver's license. So if you're flying into Canada and paddling into the US a passport card is sufficient. Paddling to Canada and flying to the US requires a passport.

From the Canada Border Services Agency;
"If you are a U.S. citizen or permanent resident, you must carry proof of citizenship such as a passport, birth certificate, a certificate of citizenship or naturalization, a U.S. Permanent Resident Card, or a Certificate of Indian Status along with photo identification."