Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer
Designing a neural network is a thousand times harder than I imagined.<p>After AlphaGo, I tasked myself with creating a neural network that would use Q-Learning to play Reversi (aka Othello).<p>At that point, I had already utilized Q-Learning (the tabular version, not using a neural network) for some very simple and mostly proof-of-concept projects, so I understood how it worked.  I read up only perceptrons, relu, the benefits&#x2F;disadvantages of having more&#x2F;fewer layers, etc.<p>Then I actually started on the project, thinking &quot;I know about Q-Learning, I know about neural networks, now I just need to use Keras and I&#x27;ll have a network ready to learn in about twenty lines of python.&quot;<p>Boy was that naive.  Regardless of how much you understand the CONCEPTS of neural networks, actually putting together an effective one that matches the problem state perfectly is so, so difficult (especially if there are no examples to build off of).  How many layers?  Dropout or no, and if so, how much?  Do you flatten this layer, do you use relu, do you need a SECOND neural network to approximate one part of the q-function and another to approximate a different part?<p>I spent MONTHS messing with the hyperparameters, and got nowhere because I&#x27;m doing this on a desktop pc without CUDA, so it takes days to train a new configuration only to find out it hardly &quot;learned&quot; anything.<p>At one point after days of training, my agent actually had a 90% LOSE rate against an opponent that played totally randomly.  To this day I am baffled by this.<p>I went into the project thinking &quot;I have this working with a table, the q-learning part is in place -- just need to drop in a neural net in place of the table and I&#x27;m good to go!&quot;  It&#x27;s been almost a year and I still haven&#x27;t figured this thing out.