Your assesement is correct. The person answering the question is stating that they do not think that "it" would make a difference. Expressing dobut is not an absolute negation; rather, he is stating that it is unlikely (probability is low) that a difference would be made.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.