I am not very good at grammar, but wouldn't be easier to say: I don't know anyone except (I'd say "but") the man in a black hat, whose language is familiar to me. I'd write it this way, though it seems to me a odd sentence...
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.