So when exactly is it a problem? How are you implementing A3C? Are you sharing the main model parameters or does your training loop body look like this:
loss = fn(input)
loss.backward()
copy_grads_to_shared_model(model, shared_model)
shared_model_optimizer.step()
copy_params_to_local_model(model, shared_model)