Because we haven't won the SB. It's that simple. Fans would still complain if they had won it. But it wouldn't be a large majority.
The biggest fear is that the defense is going to get obliterated by a great team in the postseason. Which has happened more than a few times. Thus far, I've seen nothing to show that will not occur again.