It depends on the context. The use of "of" would imply an inadequacy throughout the system. In other words, the system itself is inadequate.
The use of "in" would imply that the inadequacy affects something less than the whole of the system. In other words, only part of the system is inadequate.