"to have moved away" indicates that the action of "move away" had potentially already happened (but in this case in fact hadn't happened) by the time of "I was pretty sure". "to move away" would indicate that the action "move away" potentially happened at or around the time of "I was pretty sure".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.
"to have moved away" indicates that the action of "move away" had potentially already happened (but in this case in fact hadn't happened) by the time of "I was pretty sure". "to move away" would indicate that the action "move away" potentially happened at or around the time of "I was pretty sure".