So the hard rule is input channels = output channels of your last layer (unless you pool across channels which would be less common).
Beyond that, I’d recommend looking at e.g. the vgg models for inspiration. Usually you increase sharply from 3 (RGB) to 50 or so and then, say, double that near the pooling. But there are many philosophies about that, and you should use what best suits your needs.
Best regards
Thomas