Your parallel structure is correct. I think a more emphatic rendering would be neither to you nor to me . You could also go with: It isn't beneficial to either one of us.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.