The only thing left to determine with your solution is how many times can you challenge the calls you've described. If it's unlimited the games would be incredibly long and if you limit them, you may not have a challenge left. For example say the Saints didn't have a challenge left when that pass interference wasn't called?
It's a tough problem to fix.