The only thing "wrong" with participant of the conference (I would use participant in the conference, in the same way that someone is in a meeting) is that the conference, in a metaphysical sense, doesn't exist independant of the participants. You can be a member of a gym because a gym is a permanent fixed structure which has equipment, etc. In principle, a conference
I would still prefer "Conference participants will attend the banquet." Why use extra words when you don't have to. But if you really wanted to use your structure then I'd probably use "of" or perhaps "at" but not "in."
Note that IBm extends and invitation TO you, not FOR you. (And it's a 5-star hotel.)