I went in 99 or so, was on the infield and it was a great time. Not nearly as wild as preakness (you can't bring in your own booze,have to buy it) but there was still a little bit of wildness to it. Just as good as the derby is the atmosphere in the city. The city has a great vibe on Thurs-Fri-Saturday night, bars are packed, people having fun.
Definitely go if you can.
(don't think infield tickets are that hard to come by)