"Something that will cause a delay" or "Something that will slow the speed of progress" would be good definitions.
"Slight harm" wouldn't really be a great way to describe a setback. A setback may involve a small defeat, however that isn't really a direct definition.