Josh being Josh got them to the AFC championship game and divisional, which they should have won. The problem with taking that element out of his game is when a defense has the offense shut down, he doesn't have that tool in his tool box anymore to create a spark.
I'm not saying he can't evolve into a more traditional passer. I'm saying don't take away what makes him, him. Do you see Andy Reid trying to make Mahomes a traditional passer? No. He lets Pat be Pat and takes advantage of his unique play set.