The tenses are technically correct, but, in the kind of context that I think you are envisaging, it seems a bit odd for someone else to say what the speaker "would" do -- almost as if the speaker has no mind of his own, or is not able to direct his own actions. Your sentence may be possible in certain contexts, such as counterfactual, but my guess is that you probably mean something like this: My mistress said that if you came back, I was to tell you she doesn't want to see you.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.
gambolerHas here 'I was to tell you' the same meaning as 'she wanted me to tell you'More like "she instructed me to tell you".
gambolerWould it be better to say 'I had to tell you'?No, not better.