A follow-up . I did try on other datasets (MNIST, FashionMNIST, EMNIST, SVHN), and below are generated samples on these datasets.
generated1339×424 205 KB
I would say that all the samples seem reasonably ok. And they all show similar behaviors in the metrics which I still find very surprising.
For example, if I plot the accuracy of the discriminator  (I would have expected it to be at 50% when the fakes look real):

And if I plot the loss of the generator (I would have expected it to be low when the generated samples look real):

And finally, because I really find it funny (even impressive actually), if I take the model trained on FashionMNIST and SVHN and let me generate samples by interpolating in the latent space using 3 seeds (for the top-left, top-right and bottom-right).
generated2602×602 126 KB
generated2682×682 183 KB