Did the same thing as the OP. Seasons for 5 years but because of life and current state of the Bills, we decided to opt out. I stand with my choice and am excited for the change of pace. I think there exists a concept that by not attending the games or holding season tickets, you are less of a fan than someone who does. This is not true and is just someone's way of making their nearly inexistent influence on the organization seem a little more important to them.
I wouldn't let that thought of guilt cross your mind anymore. Enjoy your family and friends. Don't allow FOMO to influence how you feel. In the end, you're just a Bills fan supporting his team so who cares how often you can go or where you cheer them on from