Can we use the noun, demon in this case, to describe another noun, brother?-- Yes. Which one is better? 'A demon of a brother' is general, whereas 'his demon of a brother' is specific.-- Therefore, neither is 'better'. It depends on whether you are talking of a specific brother.
I am talking about a specific brother. Let's say Jim is afraid of his brother. To dramatize it, would it be right to say: Jim feared his demon of a brother. Or, would that be odd?