I generally use manipulate to mean the actual changing of something. I think what you want here is "toyed with" or "played with". The meaning then would be that the person tried to manipulate your feelings.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.