No, adverbs can modify all sorts of things, including pronouns as your example proves [to be honest I sometimes think the adverb class is a bit of an untidy ragbag].....
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.