I'd say "due me" is more natural. " "Due to" is one of those phrases like, "because of" which evokes another meaning, although perhaps not in this particular sentence. My loss of apetite was due to a stomach ache.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.