"Thank you for asking me...won't be able to come."

Hi all

I am wondering if this is a natural way to express thanks when someone let's you know about an function. I know you can say "thank you for inviting me" but I am wondering if I can say it this way as well since inviting sounds too formal sometimes.

A: Dinner this Sunday at Paul's?

B: Thank you for asking me. I'm sorry but I won't be able to make it.

Thank you.

Cheers

PBF