It doesn't really work without 'the fact'-- yet 'the fact that' is one of the most anathematized phrases in style manuals (and I agree). I try my utmost to recast the sentence: Aside from your hunger....
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.