How small do things need to be, to be dangerous?

An exchange of opinions is happening on twitter, between J. Craig Venter and Nassim Nicholas Taleb, about whether genetically modified crops are good or not. I tried to summarize the case for why GMOs are dangerous like this:

Ok, this is how I understand why GMOs are dangerous:

1. Nature has a certain tolerance (hysteres, non linearity) for shenanigans, if you go outside that envelope you may end up in a wasteland. This is potentially lethal if you do it on a global scale

2. GMOs are modification of small self-replicating things, hence have the potential to go global, and out of control. Even if that was not your intention.

3. It's not about the *probability* of this happening but about the *pay-off* (consequences) if it happens

So self-replicating things can be dangerous, but I started thinking, how small does something need to be, to be out of control?

If you look at land based organisms: If we should get, say genetically modified evil elephants, it would really not be a a huge problem. We can simply locate all those elephants and kill them. With smaller things it gets harder.