No. These are possible, with different meanings: It has (got) to be destroyed. It had got to be destroyed.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.
tenjingIt had to be destroyedIt was necessary to destroy it. (Yes, your interpretation is correct.)
tenjingIt had to have been destroyedHere I interpret "had to" as "must":