I canceled mine before last season for very similar reasons as you. I'm still a huge Bills fan, and I'll be looking to make 1-2 trips a year (last year, I went to Jacksonville, St Louis, and 1 game in Buffalo), but I just could't justify that level of commitment when there hasn't been much return.
My seat is better filled by someone who can party pre-game, go to the game, then have a short ride home. The 400 mile drive home, loss after loss, really starts to make you wonder why you continue to put yourself through the torment.