It refers back to some earlier subject. So "that" means "that topic" or "that subject" or "that issue."
For example: A. The killer set the boy on fire. B. What sort of lighter did he use? A. I find it odd that you're making that your main concern. (that now refers to "what sort of lighter.")