I think it's OK with only one "had"; simple past is sufficient to show it's past (unless a context requires "had had"). "He had a though t for me": he thought of me, maybe remembered to call me or write to me; whose "most difficult days" are these? His or mine?
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.