For the criterion I settled on log(eval loss + 1) * log(training wall clock time + 1) to compare between networks, even trained for different amounts of time. I select for the lowest 2/3rds of the population according to that metric each generation. Essentially that measures how good was the trade-off between accuracy and model complexity: if the model takes longer to run each epoch, it had better converge faster per epoch.
Architectures with superior inductive bias for your problem will converge faster at the beginning of training, at least that’s my intuition.
It’s important to do just a couple epochs when first running a new architecture, test that criterion, and see if it’s hopeless (say, twice as bad as your nth best) ASAP so you don’t waste time.
On your scaling approach, that sounds alright, but beware has a greedy aspect to it; i.e. with that policy you’re only allowing architectures that perform well on their first few layers to be candidates for enlarging, where it could be that there are larger architectures which perform well as a unit but do not perform well just with their first layers, which therefore you’d never test. You can really only tell whether this is a viable assumption by experiment, but I think it’s a good start.
You could, similarly, set the calculation ops budget to 1m, get the best N networks and seed them as the initial population of the next phase where you increase the ops budget to 2m, and so on. In my code, a op budget allows a network to only add so many layers, kernel sizes, etc, so by increasing it, mutations could build upon well-performing networks adding new layers or wider kernel windows and such.
I also had arbitrary residual connections randomly sprinkled in, which was a huge complication. But at the very least you should have a residual connection spanning the input to the output of the last conv block before the fully connected layers.