One perspective to look at is from Fraziers.
Hes obviously not the colts first choice. On top of that he probably isn't their second or third or fourth. If Frazier struggles at all (which he most likely will) I can see them firing him early in his second season. If he finishes 3-13 or below I could see them firing him after one season (kinda like the niners did to chip Kelly).
Frazier is smart and probably realizes he won't get his first choices on staffing. He also has to change their defense over to a 4-3. He may not have Luck. That colts team also has hardly any talent. To me, I would refuse the job if I were him. Stay in a stable coaching position and wait for a better opportunity.
But to to be honest, I'm surprised he's even taking the interview