I dunno the "rule", exactly, but this is how I use these: Using "had" suggests a time further back in the sequence of events than the normal past "gave". In your example the use of "yesterday" is not enough reason to NEED the "had" bit. You could just say "I gave him two dollars y'st" we use "had" to show sequencing - so - "I had given him 2 dollars to buy a toy but he spent it on candy, so I got mad at him".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.