You could use either. Use the simple past if it was one event. If it was an ongoing abuse you might consider the past imperfect, "You wouldn't be on his side if you knew what he used to do to me".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.