"I am afraid you have the advantage of me" is usually used when the speaker feels the addressee knows something that the speaker doesn't, especially when the addressee knows the speaker's name but not vice versa. I don't perceive any sense of "rejection to invitation". This is a formal and perhaps slightly old-fashioned expression.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.