About losing privacy the longer you train, maybe an example can help internalize this concept.
Let’s leave ML aside for now and let’s just imagine we are building something like Google Maps’s feature that tells you how busy a store is depending on the time of day. To make this privacy-safe, you can use differential privacy and instead of submitting the actual position of a user down to the right centimeter, you instead add some noise and randomize each user’s position within a 3 meter radius. This will not alter your aggregate significantly because if you have many users, you’ll still be able to count correctly as the noise will tend to cancel out over many many users.
Now the problem is that the noise will also cancel out if we keep on asking for your position over and over. This is what happens with DP-SGD too the longer you keep training