yet : either way is fine. Your sentence has indirect speech, so it should read: I asked him whether his brother had come back (yet). Personally, I prefer to use or not when using whether .
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.