Greetings,
if you mean 'You said to me', then yes, it is correct. 'You told me' means pretty much the same thing. Mind, though, that the sentence is incomplete: it lacks the direct object ('You said that to me') or a direct-speech clause "Go home", you said to me, (remember?).
Respectfully, Gleb Chebrikoff