In this situation always look to separate out the two parts - 'I' / 'me' and 'my brother' in this case - and deal with one at a time. Which would be correct?
I ...(did / saw / ate etc.)
or
Me ...(did / saw / ate / etc.)
???????
That should give you the answer. However, you would sound more natural saying 'My brother and I...'