Yes, I agree with your statement about what the two might mean. Although they would be interchangeable without being incorrect (you could say "I had an interview with him" when you are talking about a journalistic interview), it would be much more common for them to have the meanings you indicate in your post. So I guess, although they could mean the same thing, they generally don't.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.