In my opinion, and within the context you have given, "How do you mean" is colloquial, borderline-poor English. "What do you mean" is better and can be used in all situations. " is not formal or literary.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.