There's not much difference, but I'd interpret "I saw him climb the tree" as meaning "I watched him continuously, or nearly so, throughout the entire tree-climbing process". I'd take "I saw him climbing the tree" to mean "I saw him while he was in the process of climbing the tree, but I did not necessarily see the whole process".
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.