The code is running fine, no error raising.
EDIT : Looks like the solution is to provide one optimizer to each worker in PySyft. An example is given here, in the PySyft tutorials. 15
Thanks for your help !