I think this must be an Americanism, so I'll try to answer. I believe that adding the "by" (with the object of the preposition understood but omitted) makes the visit less formal. "By" is also used with "stop," which implies that instead of planning a visit, the visitor has just happened to pass, or was in the neighborhood anyway: Is it okay if I stop by your house this afternoon and drop off those decorations?
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.