"say ... to you" is OK in a case such as "He wants me to say this to you". The problem in your sentence is that the intervening "that" clause disrupts the connection, so that "to you" gets lost or mixed up with the other clause.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.