I found out that the problem was that my GPU memory is about 2GB while the recommended memory size is 16 GB.
Upon this I subscribed to Colab Pro  as it provides 16 GB of GPU memory and enough disk space to run the script.
Here 17 is the link to the issue on github which also contain some tips were mentioned by one of the contributors that might help others0