Of course if you replace "ya" with "you," it's perfectly neutral. Sometimes "ya" is used casually to create a sense of colloquialism, or to make you sound like a native. If perchance he views you as an outsider, he may have considered it to be an affectation.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.