"To me" means "in my opinion"; moreover, you say "attractive to ". You can also say "I'm quite attracted to you". The meaning is the same, as far as I can see.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.