I prefer to keep "me". It makes the sentence easier to parse, and the repetition of "me" does not bother me. However, I would be quite happy with something like "my father neither likes nor respects me" (here the reader is less likely to stumble over the construction).
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.