I think it has to do with recognizing the antecedent as a person . I make that distinction between 'who' and 'that'. I'd like to add, here, that I would have written the sentence as: I was the only one in my family who couldn't hear.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.