Absolutely it would matter. Even though I can't make it back for games anymore, there's something about your hometown team and the associated memories that make a difference. Times spent at Rich/RWS - the sights, smells, the rabid love of the Bills by the fans - are all things that help you identify with the team. Them moving would really make a difference as to how ardently i would follow the team. For me I don't think it would be any different than a Bills fan living in Buffalo. Psychologically they become "someone else's - not ours anymore." I have to concede that i'd probably root for them for a few years because of their origin and familiarity with players, but it would fade over time.