Take attention away from

Is it natural to say “take attention away from”? I couldn’t find it in the dictionary. Is there another verbal phrase I can use? Would you say this differently?

Your long caveman-like beard had taken my attention away from your unibrow. I hadn’t noticed it until you decided to shave off that beard of yours. I can’t believe I hadn’t noticed that unibrow before.

Thanks