I think 'we are family of five' mean we are five members in a family whereas the other sentence would mean that five members from a family(family may have more members) are at some specific place among others.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.