This is the way it is expressed. Raising one's voice to means "to talk in a loud voice (to)" It implies that the person is agitated, upset or angry with you. " I do not like somone who raises their voice to me.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.