Even though you're well meaning, that phrase was used in the past as, "You're so well spoken for a black man (or woman)."
Like, the shock and awe that a person of color isn't using ebonics and choosing to enunciate.
It's real. Blame racists for ruining, what could have been, a simple compliment.