I'd rather you didn't means that the speaker does not wish the other person to do something. I'd rather not means that the speaker does not wish to do something. Only the first is appropriate in your situation.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.