Yes. I refused his request and he asked me a second time. "He asked me the second time" would work, but "He asked me for the second time" is more idiomatic.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.