Verb tense question

Hi: could you justify the use of "had said" in the following quotation, please? there seems no apparant reference time in the context. thank you.

It is appropriately, in the chapter gently entitled “The Village” that the writer of Walden declares himself to be the author of “Civil Disobedience”, the same man who had said that “I simply wish to refuse allegiance to the state, to withdraw and stand aloof from it effectually”