I certainly wouldn't want to put out a wide spread indictment on the entire stadiums of fans - and at the same time I don't think it's fair to say fans like me that would prefer to not go to games are the type who want you to sit down, want no banging of their chair and that you don't want in the stadium.
In full disclosure I have a relatively small amount of game day experience. I'm 39 years old and I've been to only about 15 games - of those I'd say only 5 were pleasurable experiences. One rough experience in particular was when I was somewhere around 15-16 years old my brother took me to a game (birthday gift) and 2 Jets fans were in the stands wearing Jets jerseys. Of course it started out with the entire section pointing at them saying A-hole - then these guys finally stood up for themselves and yelled something back because they were being heckled unmercifully - several (10-15) guys from this section commenced to beat the snot out of these two guys, in fact it was the worst live beating I've seen in my life. At the end they weren't moving and no one did anything about it because the mob would have turned on you (or at least it felt that way).
I've got 8-9 more terribly miserable experiences albeit not this bad from being at the stadium. All the things you describe are awesome - celebrating a TD with your fellow fans is an incredible experience, but I've got to tell you there's no way I'd take my boys into that stadium based on my small sample of experiences. I get it most people that go say hey it's no place for kids so what, but for me that ruins some of why I'd want to go.