I don't think being tough, difficult, or requiring fitness necessarily qualifies it as being a sport.
Personally, I run distance races, I go cycling, go on hikes up and down mountains that make my muscles ache for a week. This weekend I went on a 28 mile kayak trip. All of those I would consider hobbies or activities. I wouldn't get insulted if somebody told me that.
A high school athlete in track and field, who is going against others in a meet. I would call that a sport.