There was note of his father being Dr. Charles Rosen, famous spine surgeon who went to U Penn and his mother who went to Dartmouth and played lacrosse. Josh was UCLA of course and his political commentary was, to some, admirable.
I think the negative blowback you mention was something continued to grow with time and coverage. Maybe I'd say his cocky draft night statement was the tipping point for his reputation.