Usually per se means that it does (or does not) strictly follow the definition of the thing referred to. So this would mean he has never been to the military as such, not corresponding to the definition we would usually give. It doesn't make much sense to me out of context, but perhaps this could be said: I've never served in the army per se, but I was an independent consultant there and spend a lot of time with the troops.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.