There is no universal rule. I have a sort of rule of thumb I use myself: In formal writing, retain "that" when you can, and in informal writing, omit it when you can. The version with "that" is almost always clearer, but you start to sound like a machine if you always keep it.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.