Distract by or from?

Do I get distracted from or by something? (the reason why I am being distracted).

I think it's "by" when we talk about the reason why I am distracted (e.g a person is distracting me) and "from" when we talk about what I am distracted from (e.g studies)? Or am I wrong?