When it is your grandfather, do not use "the". When is it someone else's grandfather, use "the".
Grandfather and Billy are fishing. (Your grandfather) The grandfather and his grandson are fishing. (Not your grandfather. Grandfather is not capitalized, unless it is used as a proper name - We took Grandfather Wiliiams shopping.)