It would always be "remains of a castle." rather than "castle remains," as in the sentence: "On the hilltop there are the remains of a castle." This would never be said as: "On the hilltop there are castle remains."
The pattern, "_____ remains," is only used for humans and animals, as in the sentence: "On the hilltop we found human/animal remains."