Differentiable Finite State Machines
For anyone coming in a bit confused by this, the magic here is in the concept of "differentiable". I'm fairly frequently touting the benefits of thinking in terms of differentiable programming and the most common response is "what does that even mean?"<p>The core idea is that any program that is "differentiable" means you can take the derivative of that program the same way you can take the derivative of a simple function in calculus.<p>Why does that matter? Well being able to take the derivative of any problem in turn means you can then optimize that problem with gradient descent. Chris Olah recently had a great discussion on Twitter [0] about what's great about gradient descent, in short:<p>> Simple gradient descent creates mind-boggling structure and behavior, just as evolution creates the awe inspiring complexity of nature.<p>Differentiable programming is, in a strange way, similar to logic programming in the sense that you can start solving problems by defining what the solution looks like. In this example you define the input and desired output and then learn the FSM that maps one to the other. Recall from Theory of Computation classes that a FSM the simplest, least powerful programming language (FSM an example of a regular language).<p>What's amazing here is that you're essentially seeing gradient descent writing a program. IMHO, that is insanely cool.<p>What makes differentiable programming so interesting compared to something like logic programming is that you have a potentially much larger space of problems that can be solved in this manner <i>and</i> compared to the backtracking algorithm used by logic programming languages, gradient descent allows for potentially faster (though also approximate) solutions at scale.<p>Currently the vast majority of differentiable programming examples are essentially just building neural nets, but examples like this one show that we can think much more broadly about problems in terms of differentiable programming as a proper programming paradigm.<p>0. <a href="https://twitter.com/ch402/status/1533164918886703104?cxt=HHwWgICpxdzX8sYqAAAA" rel="nofollow">https://twitter.com/ch402/status/1533164918886703104?cxt=HHw...</a>