Come down

Here is a sentence from a tv show: One guy, and the whole thing comes down.
The character wants to say, that because of one guy's mistake, the entire hacker group was caught and sent to jail.
But what does exactly 'come down' mean in this sentence? Fall apart? Fail? Go down, like a drowning ship?
Can you please define 'come down' in this sentence and give more examples with this verb? I ask these questions, because I have a feeling, that this is pretty much the only way, you can use it. 'Our plan came down, their marriage is coming down etc.' doesn't sound naturally. Thank you very much for your time!