How does gradient descent help SVM learn a linearly separable hyperplane? So I see the Perceptron Algorithm applied to learning an SVM, where $\theta$ is the normal vector to the linearly separating hyperplane. How does the update
$$\theta^{t+1}\leftarrow\theta^t+\alpha y_ix_i$$

There's this visual example, but I'm not sure how updating $\theta$, causes it to adjust in the correct direction.
Help push the hyperplane to learn a linear separation between the points $x_n$, where $y_i$ and $x_i$ are misclassified points?

What you've described is not gradient descent. It's the Perceptron learning algorithm. Additionally, the perceptron algorithm doesn't learn the separating line of maximum margin as is the case with SVM. The two are related but not the same. Here's a visual explanation of the algorithm:

Why the learning algorithm for a perceptron actually works is a bit mysterious. It seems plausible that the hyperplane might just bounce around forever as the weights get updated. However, there's a clever convergence proof for the algorithm (and if the algorithm converges, we've necessarily found a separating hyperplane).
The trick is to monitor the length of the weight vector as the number of weight updates increases. We can show that, if the data is linearly separable, the algorithm converges after at most $\frac{R^2}{\gamma^2}$ updates where $R$ is the length of the longest $x$ vector and $\gamma$ is the longest distance between a potential separating hyperplane and the nearest $x$. The proof is kind of tricky and long so I won't post it here, but I like Shivaram Kalyanakrishnan's paper on this.
Shameless plug
I have a free course on neural networks and the first half of my course is devoted to understanding Perceptrons. The visual above, I plucked from my course.
