It should be past tense even if grandfather is still alive. "Have" has different meaning. My grandfather had three sons and two daughters. I am the oldest son of his youngest daughter. "Had" in that context means "gave birth to, father." Obviously, that happened in the past. My neighbors have a large family. They have six kids. (In this context, it states a familial relationship.