There might be a substitution for SVM, that is, a SVM-like loss. This paper holds the opinion that minimizing their loss at the last layer is analogous to minimizing the margin in an SVM classifier. (See this paper 152 sec.3.1 above, equation (3)). It looks like a hinge loss.
Sorry that I don’t know much about it. And the author did not give a theoretical analysis. You can try the loss if you want to.