Hello As you know I'm an English learner but allow me to throw my two cents. I think you are right. "Vote someone out from office" is an idiomatic phrase.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.