Hi,
Using an SVM as the loss function for a cnn is not necessarily good as it will give sparse gradients and won’t improve training.
You can also consider all other layers fixed, extract the feature for the last layer and use a regular svm solver to train the last layer as postprocessing of your training.
Finally you can check this 40 paper that tries to extend the use of svms for cnn.