Sorry for the interruption, I recently retry running the code and notice the clustering weights are no trained, and the gradients are zero. My original aim is to used set_weight to initialize the parameters of the clustering layer, and then train it with gradient.
Not sure it worked previously, or maybe I was just too careless.