There is no rule.  There is neither a hard answer. If you have few blocks your accuracy will be worse, but not null. With just one block, you accuracy will be higher than random. The more blocks you add, the higher accuracy you will have (exponentially saturaring).
You may find interesting the ResNet paper, as it shows how accuracy increases by adding more blocks.