Gradient Descent Models Are Kernel Machines
If you're surprised by this result because you've used kernel machines and didn't find them very good at generalization, keep in mind that this assumes a kernel function that accurately reflects similarity of input samples. Most work with kernel machines just uses Euclidean distance. For instance, in an image recognition model it would have to identify 2 images with dogs as more similar than an image with a dog and an image with a cat.With a sufficiently magical kernel function, indeed you can get great results with a kernel machine. But it's not so easy to write a kernel function for a domain like image processing, where shifts, scales, and small rotations shouldn't affect similarity much. Let alone for text processing, where it should recognize 2 sentences with similar meaning as similar.