Technically, LA is the original home of the Chargers and the long-time home of the Rams. San Diego "stole" the Chargers in 1961 and St. Louis "stole" the Rams in 1995 if you want to go down that path. Conversely, Toronto has never been the home of the Buffalo Bills and aren't even in the same country.
More importantly, there's a big difference between moving upstate 120 miles than moving out of the country. San Diego has history, don't get me wrong, but that history could transfer to LA pretty seamlessly. Regarding the Chargers, also remember none of their fans are south of San Diego...that's Mexico. It's a move, yes, but still very much remaining in the Southern California family.