Hello Taka I take it to mean that some people appear to be "perfect", in manner and character; their perfection makes us feel uncomfortable, because we are conscious of our own contrasting imperfection. A "rough diamond" on the other hand may have unusual qualities; but perhaps he is not particularly urbane, or is ill at ease socially. This makes us feel less inferior.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.