How to Tame Your Jax: Navigating TPU VMs with TFRC in 2021
From the bug log describing the TFRC team:<p>&quot;&quot;&quot;For a Google project, this was somewhere between &quot;space aliens&quot; and &quot;narnia&quot; on the Scale of Surprising Things.&quot;&quot;&quot;<p>What you need to understand is that the people who are doing Jax and TFRC are a group of excited brilliant engineers on the cusp of junior&#x2F;senior status who have a complete understanding of the current state of ML hardware, and have built a collection of powerful tools that are far simpler than TensorFlow.  It&#x27;s still in the rapid expansion phase, new ideas can be tried out quickly, and engaging with the developer community enables a virtuous cycle that improves the system rapidly.<p>TensorFlow on the other hand has turned into a complex software behemoth where nearly everything requires extraordinarily long software development efforts to make minor changes, and any changes in the core impact thousands of users.<p>What I learned from interacting with TFRC and the Jax community is that the real secret value is in XLA and the runtimes that wrap around it.  Once you&#x27;ve mastered those, you&#x27;re well on the way to using large resources like a TPU with a CPU (I think that&#x27;s just the host CPU and RAM, BTW) and harnessing all the compute elements simultaneously to get peak performance.<p>I expect that as more people see it&#x27;s easier to program a simple layer on top of TFRC and XLA, tensorflow will not be as popular.