It's not incorrect; you can argue that it's not necessary, since you can only recount a conversation that has taken place already, and therefore there is no need to mark the order of events with the past perfect.
However, I'd probably use it here, but contracted: ... a conversation I'd had with Barbara.