You are thinking that "to eat" describes the mango itself and now why it was brought along?
The infinitive of purpose, I think it's called. It says why you did something, so it's an adverb modifying the verb "to bring" and not an adjective modifying the noun "mango."
(PS: I'd like to point out that a number of us are not "sir.")