Is there a Good Illustrative Example where the Hinge Loss (SVM) Gives a Higher Accuracy than the Logistic Loss Vladimir Vapnik wrote:

“When solving a problem of interest, do not solve a more general
problem as an intermediate step. Try to get the answer that you really
need but not a more general one.”

Vapnik wrote this in the context of transductive learning, but the principle could also be applied to statistical pattern recognition tasks.  If we just need to perform a hard classification, then we should aim to determine the decision boundary directly, rather than estimating the posterior probability of class membership and then threshold that at 0.5 (or some other threshold depending on the misclassification costs).  In principle, we may be able to get better generalisation in terms of accuracy by following this principle as it focusses on the needs of the application, and avoids modelling compromises that benefit modelling of aspects of the data that are not relevant to the application at the expense of other aspects that are.
The performance of the SVM on real-world problems suggest that there may be some merit in this, but it is not easy to find a clear example where this is demonstrably the case.  Can anyone suggest an example where a SVM will give better performance than an equivalent logistic regression model (i.e. kernels and regularisation should be available for the logistic regression model, so we are just comparing the loss used, rather than the other aspects of the models)?  Preferably the example should demonstrate why this is the case.
Obviously the no-free lunch theorem suggests that no classifier system is going to be superior for all datasets, and the theoretical justification for the SVM is largely a worst-case analysis, so the SVM may be better in pathologically difficult problems, but not necessarily the average case.
Personally I prefer kernel logistic regression over the support vector machine, largely because the hyper-parameter tuning is much easier, and I think they are likely to give broadly similar results.  However, I have an open mind where the needs of the application mean that the limitations of the SVM (e.g. having to know misclassification costs a-priori) are not a problem.
Please can we avoid discussions of whether accuracy is a good performance metric.  It has issues, but there are some applications where accuracy is the relevant statistic of interest, and there have already been good discussions of this topic.
Note the answer here is the sort of thing I am looking for, but unfortunately it isn't correct (see my comment on the answer).

The results you shared are fascinating. Here's some more exploration in a similar direction.
The idea behind the Bousquet example is to create a situation where optimizing the logistic loss prioritizes fitting the underlying probability distribution at the expense of accuracy. But, it's not clear to me that accuracy and fit to the distribution would have to be opposed here. For example, it seems like a model that exactly matches the underlying distribution should yield both optimal accuracy and optimal logistic loss (at least in expectation).
I'll build a similar example that tries to simplify things, with only two models in the hypothesis space. One gives better accuracy but worse fit to the underlying distribution, and the  other does the opposite. The tension between these two objectives is explicitly baked into the problem. I'll work directly with expected losses, so issues related to finite samples and/or optimization won't play any role.
True distribution
Suppose each point $x$ is drawn i.i.d. from the uniform distribution on $[-1, 1]$ and its class label $y \in \{-1, +1\}$ is drawn from a Bernoulli distribution $p(y \mid x)$. Similar to the Bousquet example, the conditional probability of the positive class is a 'wavy step function' (see plot below):
$$p(y=1 \mid x) =
.598 \ \sigma(100 x) - .201 + .2 \sin(20 x)$$
where $\sigma$ is the logistic sigmoid function.
Models
Suppose our hypothesis space contains only two models (with no free parameters so 'fitting' means choosing one or the other):

The first model is a step function:
$$\hat{p}_1(y=1 \mid x) = \begin{cases}
  \sigma(1) & x \ge 0 \\
  \sigma(-1) & x < 0 \\
\end{cases}$$
The second is a wavy step function, similar to the true distribution, but with slightly different parameters:
$$\hat{p}_2(y=1 \mid x) =
.398 \ \sigma(100 x) - .301 + .3 \sin(20 x)$$
Where needed (e.g. for computing the hinge loss), 'raw' classifier outputs are computed as $f(x) = \operatorname{logit}(\hat{p}(y=1 \mid x))$. Since we're interested in accuracy, point predictions are computed as the mode of the predicted distribution over class labels (equivalent to the sign of the 'raw' output), which is the optimal decision under the 0-1 loss.

Note that the hypothesis space doesn't contain the true distribution. We're forced to choose between two approximations that make different tradeoffs. The first model (step function) is designed to make more accurate point predictions, at the expense of fit to the underlying distribution. In contrast, the second model (wavy step) is designed to match the underlying distribution better, at the expense of accuracy. To confirm that these tradeoffs are indeed present, the table below shows the expected 0-1 loss (better for model 1) and expected KL divergence from the true distribution to the model (better for model 2).
Hinge vs. logistic loss
Here are the expected losses for each model, where the expectation is taken w.r.t. the true data generating process (calculated by numerical integration). The best model according to each loss function is shown in bold+parentheses:
$$\begin{array}{rc}
  & \text{Model 1 (step)} & \text{Model 2 (wavy step)} \\
  \text{0-1} & \mathbf{(.208)} & .269 \\
  \text{KL} & .113 & \mathbf{(.044)} \\
  \text{Hinge} & \mathbf{(.416)} & .580 \\
  \text{Logistic} & .521 & \mathbf{(.474)} \\
\end{array}$$
Suppose we choose a model from our hypothesis space by minimizing the expected loss (i.e. what empirical risk minimization tries to do by proxy). As the table shows, minimizing the hinge loss gives the first model (prioritizing accuracy), whereas minimizing the logistic loss gives the second model (prioritizing fit to the underlying distribution).
Notes
Contrary to this example, accuracy and fit to the underlying distribution aren't always opposed. Even when such a conflict exists, the hinge and logistic losses may not necessarily behave as shown above.
For example, the amplitude of model 1's step function matters. It doesn't affect the accuracy (which only depends on the sign), but it does affect the hinge loss. Increasing the amplitude too far (overconfidence) incurs increasing penalties for misclassified points. And, shrinking it too far (underconfidence) penalizes correct predictions, which increasingly fall inside the margin. In both cases, the hinge loss will eventually favor the second model, thereby accepting a decrease in accuracy. This emphasizes that: 1) the hinge loss doesn't always agree with the 0-1 loss (it's only a convex surrogate) and 2) the effects in question depend on the hypothesis space.
In practice, I'd bet that regularization plays an important role too, together with the model selection algorithm. For example, regularization strength is often chosen to maximize validation set accuracy. Even if the logistic loss is used to fit the parameters, using the 0-1 loss for model selection might sacrifice fit to the underlying distribution in favor of accuracy.
