Both are OK. I would use the one without house because probably it was the granparents being visited and the house is incidental. Also, it should be grandparents' because it is plural.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.