It's grammatically correct, but it's unusual to say, in my opinion - especially in the first person. Most uses of "hold sway" (have dominant influence) that I have seen were in writing and in the third person. It's also frequent with "over".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.