"more likely" is a comparison with something else. It means more likely than previously, more likely than in some other circumstance, more likely than some other person, etc., depending on the context.
"more than likely" means "very likely". It is not a comparison with something else.
Both are possible in your sentence about passing the exam, depending on what you want to say.