Speed up TF tests by reducing hidden layer counts
A lot of our slow TF tests are caused by TF compilation. TF compilation isn't really affected by layer width at all - the main thing is just the number of operations it has to build a graph for. By reducing the number of hidden layers, compilation gets much faster, (hopefully) without interfering with test coverage at all.
Would be nice if you can show the timing for one model (before v.s. after) 🙏 . Thanks.<|||||>_The documentation is not available anymore as the PR was closed or merged._<|||||>@ydshieh testing locally BERT went from 510 seconds -> 220 seconds<|||||>I don't know if it was @sgugger either - a lot of this code is really old! I see `tf.tuple()` in there, and even I had to look up the TF 1.x docs to remember what that was supposed to do, lol<|||||>I know he is probably not the one to decide use `5`, but he might know the history :-)