His position is that white supremacy must be preserved.
Etiquette requires a "gentleman" to lift his hat out of respect to a lady. As a southern gentleman, he must observe this courtesy.
Yet to show respect to a Negro would be a breach of behavior for someone who practices white supremacy.
So to avoid this conflict, he simply avoids wearing a hat.