For now, at least, the smart move is to keep him in the pocket. Although Josh's arm is just as dangerous as his legs, our receivers aren't all that reliable. If you let him run, he'll hurt you. If you let him throw, there's a decent chance that the pass gets dropped, even if it was a good toss.
Yeah, if I'm the opposing DC, I put a spy on him.