I believe that "The few times that we do do it, it isn't great" is, indeed, "natural." You are emphasizing the fact that you
seldom do it, and on those few occasions on which you actually do it, it is not great.
*****
I usually do not eat ice cream, but when I do eat ice cream, I really, really love it!