I would say "I enjoy working with X". "fond of" could take a person as complement. " I other cases, such as activities, it can be used, but "enjoy" is my preference.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.