If you use the comma, it means you are probably trying to draw attention to the fact that you need to include the person you are talking to in order to reach five. If you don't use it, you would be drawing more attention to the fact that five people are needed. The comma acts as a breath gap, which could imply that whatever comes after it is the main point.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.