The error indicates that the variable and its corresponding slot variable (such as momentum) in optimizer are created in different tf.distribute strategies’ scope. In your case, it will error out if your model is not created under strategy.scope but your optimizer is. To solve it, you can use CREATE_MODEL_WITH_SCOPE to control whether you want to create optimizer under the scope or not.