An idea from physics helps AI see in higher dimensions
One of the strangest things in AI, to me, is that you can average the weights of multiple different models to create a single model that’s better than any of the individuals.<p>This is how distributed training often works, for example. Data parallelism.<p>I don’t understand why it still works in higher dimensions, but it seems to.<p>The intuition is that the multiple models are “spinning” around the true solution, so averaging gives the final result more quickly. But it works even early in the training process.