I think the meaning of sentence "If I had gone to the party, I would have had to invite him to my house." is:
I didn't go to the party, but if I I had, I would have been obliged to / I would have been under an obligation to / it would have been an obligation for me to invite him to my house. (but since I didn't go to the party, there wa