. Sorry, moli, but your point is wrong. Both forms are fine, though we often omit 'to'. Please notice that the original poster was interested specifically in 'this situation', and in this situation, both of these are correct English:
Here is the pen I gave to / gave your brother.
There is another situation in which 'to' is not natural: I gave your