As you say, it's all to do with context. If he was in the middle of the job, both are possible. If he was no longer working, then the past perfect is the more appropriate form.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.