The difference between say Manning or Luck having an awful game and Peterman is that you stick with Manning and Luck because they were top of the draft talent that were projected to be franchise QBs in the NFL. They get a much longer rope then a 5th round draft pick who's scouting report had him pegged as a potential future backup QB. Sure he could surprise all the pro scouts and end up even a fringe starter, but that isnt likely to happen.
It's very rare we will ever see another Brady, a franchise guy drafted in the 5th or later, or someone from a minor league make it big like Kurt Warner.
Think he has high end backup/fringe starter ability is a bit of wishful thinking just because of his play in the preseason and training camp. And theres very few here that truly believe he shouldn't be in the league, most just dont think he is going to be anything more then mediocre at best and isnt anything special that cant be found on waivers during the season.