"I helped her with her studying" would probably be more common. "With" and "in" are about the same. "Study" as a simple noun could refer to a specific "research study" which I helped her with, or it could refer to a long-term association, helping her over a long period of time, rather than on a single occasion.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.