Usage of to infinitives after nouns

1. I had a chance to talk with her.

2. A : God gave me three chances and I used the first chance to buy a house.

I have learned that to infinitives modifies the word chance like a chance to talk with her, but I was wondering if the to infinitive to buy a house modifies the first chance or the verb used?

I feel like to buy a house functions as an adverb and modifies the verb used, not the noun chance.

What do you native English speakers think? Thank you so much and Happy new year.