Hello, @Darktex!
Sorry for the late reply. Somehow this got buried in my notifications before the new year.
With regards to using smaller models — since the privacy budget can depend on the parameter count, it does stand to reason that a smaller model might give you more “bang for buck.” I actually tested this out a little bit by comparing MobileNetV2 to ResNet18 on a few runs but I wasn’t able to get a major improvement (granted, I could’ve tried much harder).
I’m looking forward to seeing your results on this! I’ll keep you posted on anything from my end as well. I think it would be useful/cool to maintain a sort of “state-of-the-art” leaderboard for provably DP models at a given (eps,delta). I could help kick this off with some of my experiments, although 68% is already significantly better than anything I’ve obtained. My best runs don’t even make it to 60%.