Hey @philschmid thanks for the solution,
It ended up being a CUDA memory issue, I bumped down the learning rate, upped the instance number to 2 and put batch_size at 2 and it worked after a while.
Thanks