Yuefeng_Zhou:

The error indicates that the variable and its corresponding slot variable (such as momentum) in optimizer are created in different tf.distribute strategies’ scope. In your case, it will error out if your model is not created under strategy.scope but your optimizer is. To solve it, you can use CREATE_MODEL_WITH_SCOPE to control whether you want to create optimizer under the scope or not.


Thank you for the response. As I mentioned in the post, I expect that I should get the correct behaviour from this script when CREATE_MODEL_WITH_SCOPE=True, but not necessarily when CREATE_MODEL_WITH_SCOPE=False for the reasons you outlined.
In the case of CREATE_MODEL_WITH_SCOPE=True, both the model and the optimizer are created under the same scope. In this case, under Tensorflow 2.4, I get the correct behaviour if I use a Sequential model, but not a model using the functional API. Under Tensorflow 2.5, I get the correct behaviour when using the functional API to build the model, but not when using a Sequential model.
So, I would still like to have an explanation for why variables are getting created under different scopes depending on the Keras API that I use to build the model, and also why different versions of Tensorflow have different behaviour.