Hello @ptrblck ,
Thought you might be curious about what I was working on. I have implemented an algorithm that dynamically splits neurons between training epochs. It works and the results look very promising. On my research, the fixed time results have shown around an 11% increased accuracy and 18% lower loss on CIFAR-10 over a control group starting with the same number of parameters. I’ve uploaded the python files and experiment results to Github here: GitHub - CerebralSeed/Neural-Splitting-with-Dice-Roll-in-Pytorch: This demonstrates a dynamic neural splitting module implemented in Pytorch and combined with a "best of n" starting parameters. 1
I’m working on unlimited time experiments next, and so far they also look very promising with time to best validation accuracy cut in nearly half and achieving 2-3% higher accuracy. Will hopefully post that next week.
Cheers.