It happened long ago. = "ago" is adverb that modifies the verb; It happened a week ago. = "ago" is an adjective that modifies the noun "week." "Week" is a noun used to modify the verb = It happened to the extent of a week past.
Some authorities explain it this way: "ago" modifies "happened." When did it happen? Answer: ago. (You could not say: It happened long.) Then they explain that "long" (an adverb) modifies "ago." (How much ago? Long ago.)