I played football until I realized that everyone else was bigger than 5'6" and I realized that was out, I enjoy golf, but on Long Island there is way too much waiting around to play 9 holes for 4 hours, it's just not worth taking that much time out of the day.
Now my biggest sport is skiing, I am a professional ski instructor (I'm certified level II), Although I am in semi retirement as I havent been working at my resort (Mount Snow VT) in 5 years. I still maintain my cert level and hope to go back to teaching when the kids get older and less dependant.
As Exhiled In Illinois stated, it's a great sport for families, It can get expensive, but what other sport can the whole family do together as a group? I find even when the kids go off in one direction and explore, my wife and I will take our daughter and go another way, we will meet up at the end of the day and swap stories. My sons 18 and 16 are still trying to finally topple the old man (44), but I still have an edge on the race course and on the all mountain tour. Although they may on occasion get to the bottom first, It's usually reckless behavior that gets em there.