No one has ever called Irsay smart. PFT has an interesting view on this though:
"Just as Manning’s career seems to be winding down, Irsay would have had to worry about Pagano doing so well elsewhere that it would have made Irsay look stupid for letting him go — especially if the Colts had struggled under a new, high-priced, big-name coach."
http://profootballtalk.nbcsports.com/2016/01/04/irsay-may-have-feared-pagano-thriving-elsewhere/