What I wish someone had told me about tensor computation libraries
I was not aware that the PyMC developers have forked and continued Theano: <a href="https:&#x2F;&#x2F;github.com&#x2F;pymc-devs&#x2F;Theano-PyMC" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pymc-devs&#x2F;Theano-PyMC</a><p>It seems very active right now.<p>Here some further information: <a href="https:&#x2F;&#x2F;pymc-devs.medium.com&#x2F;the-future-of-pymc3-or-theano-is-dead-long-live-theano-d8005f8a0e9b" rel="nofollow">https:&#x2F;&#x2F;pymc-devs.medium.com&#x2F;the-future-of-pymc3-or-theano-i...</a><p>I haven&#x27;t really found references to its new name &quot;Aesara&quot;.<p>Apparently, the main new feature for Theano will be the JAX backend.<p>I wonder though, my experience when working with Theano, and also deep with the internals (trying to get further graph optimizations on theano.scan):<p>- Some parts of the code are not really clean.<p>- The code is extremely complex and hard to follow. See this: <a href="https:&#x2F;&#x2F;github.com&#x2F;pymc-devs&#x2F;Theano-PyMC&#x2F;blob&#x2F;master&#x2F;theano&#x2F;scan&#x2F;op.py" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pymc-devs&#x2F;Theano-PyMC&#x2F;blob&#x2F;master&#x2F;theano&#x2F;...</a><p>- This also made it very complicated to perform optimizations on the graph. See this: <a href="https:&#x2F;&#x2F;github.com&#x2F;pymc-devs&#x2F;Theano-PyMC&#x2F;blob&#x2F;master&#x2F;theano&#x2F;scan&#x2F;opt.py" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pymc-devs&#x2F;Theano-PyMC&#x2F;blob&#x2F;master&#x2F;theano&#x2F;...</a><p>- In this specific case, it&#x27;s also a problem of the API: theano.scan would return the whole sequence. But if you only need the last entry, i.e. y[-1], there is a very complicated optimization rule which checks for that. Basically many optimizations around theano.scan are very complicated because of that.<p>- Here is one attempt for some optimization on theano.scan: <a href="https:&#x2F;&#x2F;github.com&#x2F;Theano&#x2F;Theano&#x2F;pull&#x2F;3640" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Theano&#x2F;Theano&#x2F;pull&#x2F;3640</a><p>- The graph building and esp the graph optimizations are very slow. This is because all the logic is done in pure Python. But if you have big graphs, even just building up the graph can take time, and the optimization passes will take much longer. This was one of the most annoying problems when working with Theano. The startup time to build the graph could easily take up some minutes. I also doubt that you can optimize this very much in pure Python -- I think you would need to reimplement that in C++ or so. When switching to TensorFlow, building the graph felt almost instant in comparison. I wonder if they have any plans on this in this fork.<p>- On the other side, the optimizations on the graph are quite nice. You don&#x27;t really have to care too much when writing code like log(softmax(z)) -- it will optimize it also to be numerically stable.<p>- The optimizations also went so far to check if some op can work inplace on its input. Which made writing ops more complicated, because if you want to have nice performance, you would write two versions, one which works inplace on the tensor, and another one not. And then again 2 further versions if you want CUDA as well.