You left the message with his secretary. The message wasn't for the secretary, it was for Tom - Tom was the intended recipient. To normally implies movement - you could say "I gave a message to his secretary", (but the "left with" construction it is better).
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.