Fine-tuning TF models on Colab TPU
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi, 

I am trying to fine-tune TF BERT on Imdb dataset on Colab TPU. Here is the full notebook:

https://colab.research.google.com/drive/16ZaJaXXd2R1gRHrmdWDkFh6U_EB0ln0z

Can anyone help me what I am doing wrong?
Thanks


I've read your code, and I don't see anything strange in it (I hope). It seems to be an error due to training a (whatever) model on TPUs rather than Transformers.

Do you see [this](https://github.com/tensorflow/tensorflow/issues/29896) issue reported in TensorFlow's GitHub? It seems to be the same error, and someone gives indications about how to resolve.

> ##  Questions & Help
> Hi,
> 
> I am trying to fine-tune TF BERT on Imdb dataset on Colab TPU. Here is the full notebook:
> 
> https://colab.research.google.com/drive/16ZaJaXXd2R1gRHrmdWDkFh6U_EB0ln0z
> 
> Can anyone help me what I am doing wrong?
> Thanks<|||||>Thanks for the help. Actually my code was inspired from [this colab notebook. This](https://colab.research.google.com/github/CyberZHG/keras-bert/blob/master/demo/tune/keras_bert_classification_tpu.ipynb) notebook works perfectly but there is one major difference might be causing the problem:

- I am force installing tensorflow 2.x for transformer notebook because transformer works only for TF>=2.0 but colab uses TF 1.15 on colab for TPUs

So I went to GCP and used a TPU for TF 2.x. The error changed to this:
![image](https://user-images.githubusercontent.com/11090613/71145292-e138bf80-2242-11ea-9ce8-31a53a0f9a1d.png)

But yes, you are right this issue might be related to TF in general.<|||||>In the official docs of GCP [here](https://cloud.google.com/tpu/docs/supported-versions), they show the current list of supported TensorFlow and Cloud TPU versions; _only_ **TensorFlow 1.13, 1.14 and 1.15 are supported** (they don't mention TensorFlow 2.0).
I think it's why you have the problem you've highlighted.

> Thanks for the help. Actually my code was inspired from [this colab notebook. This](https://colab.research.google.com/github/CyberZHG/keras-bert/blob/master/demo/tune/keras_bert_classification_tpu.ipynb) notebook works perfectly but there is one major difference might be causing the problem:
> 
> * I am force installing tensorflow 2.x for transformer notebook because transformer works only for TF>=2.0 but colab uses TF 1.15 on colab for TPUs
> 
> So I went to GCP and used a TPU for TF 2.x. The error changed to this:
> ![image](https://user-images.githubusercontent.com/11090613/71145292-e138bf80-2242-11ea-9ce8-31a53a0f9a1d.png)
> 
> But yes, you are right this issue might be related to TF in general.<|||||>tpu supports . tf2.1<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
