In my usage, "all of" can -- confusingly -- mean easily that amount ("all of that and probably more"), or it can mean only that amount, implying at the same time that the amount is not very great. To me, the first meaning seems the core one, and the second feels slightly ironic.
For example:
"That building is all of 100 ft tall" -- It's easily 100 ft tall, and probably more.