I've been to the Derby a couple times. Was stationed at Ft. Knox, was just up the road.
Maybe there is a lot of money there, but it's also a really poor handicap scenario. Go for the experience.
I seriously recommend buying an infield pass, and go for the hell of it. If you really like horse racing - the Kentucky Derby is pretty cool, and really doesn't cost that much, compared to other sports.
Wonderful time, many layers of flavors. As far as ambience, America doesn't have too many traditions left, unless they are in Chinese or Spanish.