flagellation usually refers to whipping or beating. I would describe what you are talking about as mortification (of the flesh) . These terms can of course be used figuratively as well as literally.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.