In my view, both would be acceptable. Googling produces 67,300 pages using approximable , but only 161 pages using approximatable . The vast majority of all these pages seem to come from mathematics-related websites, so the prime users seem to prefer the former.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.
Actually, though I am not myself a native speaker, I would instinctively think of "approximatable" as the correct form. Which, incidentally, is supported by the upper explanation, since the verb "to approximate" does exist (at least according to the Merriam-Webster, I didn't check further).