I agree with this. Sometimes I am amazed at how accurately they spot the ball given the chaos and speed of the game. Or when they notice the slightest twitch for a false start.
It is holding and pass interference that are the real issues. Those two penalties are so nuanced and subjective, and every single instance is unique, so it is tough to have a baseline to compare. Roughing the passer and defenseless receiver are two others that are so nuanced and subjective.
The catch rule used to be an issue, but I think they get that right nearly all the time now.