If of the is used, the reference is to specific students or things which were previously referenced in the conversation. Few students (among all students, we don't care which students) are going to the lecture. When you get to the lecture you won't see many students there.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.