I would try to start with a known good baseline and try to adapt the architecture to your use case.
Also, the number of neurons defines e.g. if your model has a bottleneck, which might be useful for certain use cases.
That being said, I think a lot of architectures are developed empirically and the reasoning about why the model is working find is created afterwards.