That's fine and common. Past perfect refers to a previous past event that is finished : By the time we arrived at the zoo the monkeys were having their bath (and they are still washing themselves). By the time we arrived at the zoo the monkeys had had their bath (so we missed that sight).
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.