I am getting the same warning. For me, this warning occurs frequently, but not on every training step.
Using the code below, I fit the model and print the shape of input data and target data.
    print(states.shape, targets_full.shape)
    self.model.fit(states, targets_full, epochs=1, verbose=1)

The output shows that the shapes are always the same, yet sometimes I get the warning and sometimes I do not. A random snippet of the output is pasted below.
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2445
2021-06-15 13:02:19.379526: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1214
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1480
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2558
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1036
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1048
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2647
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1280
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1074
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2401
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1160
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1348
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2424
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1197
2021-06-15 13:02:20.594590: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1415
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2358
2021-06-15 13:02:20.790803: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.1212
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1649
(64, 9) (64, 3)
2/2 [==============================] - 0s 2ms/step - loss: 0.2220
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1064
(64, 9) (64, 3)
2/2 [==============================] - 0s 3ms/step - loss: 0.1488
2021-06-15 13:02:21.299211: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled

Any help is understanding or resolving this warning would be appreciated.