I was taught to capitalize Figure 4 or Table 2 , so I would say yes. That's primarily from science; I don't know the advanced mathematical standards.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.