I figure it out myself: I was using adam to optimize which accumulates gradients form previous steps and therefore setting gradients to zero by hand has no effect on the momentum term in adam. Using RMSprop solves this problem.