I have never had a "never again" moment but can visualize them quite clearly. For me they don't entail hardship or bugs or long days.

For me it would be doing something dangerous or stupid. Like battling big waves because I need to get out on time or feel peer pressure to perform beyond my comfort level.

The other situation I could foresee would be going with the wrong person or group. IMO that could ruin a trip far quicker than almost anything else. So far I have been very fortunate in that regard as well and have only had positive results.