"that" and "down"

Junkie. Pothead. That’s where I’d been headed: the final, fatal role of the young would-be black man. Except the highs hadn’t been about "that", me trying to prove what a "down" brother I was. Not by then, anyway. I got high for just the opposite effect, something that could push questions of who I was out of my mind, something that could flatten out the landscape of my heart, blur the edges of my memory.
(Dreams from My Father by Barack Obama )
Please answer the following questions.
1. What does "that" refer to in the above sentences?
Is it "the final, fatal role of the young would-be black man" or "me trying to prove what a "down" brother I was"?
2. What does "down" mean?

Does "down" mean or poor?