Use your when you are speaking of the saying or the telling, which is almost always the case: I don't remember his sending me an email (= I don't remember getting his email) . In fact, I cannot think offhand of a sentence in this form (postmodifying nonfinite clause) that would refer to the person rather than the action when using you . But then, it is getting late.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.