Answering my own question just in case it helps anyone else.
The problem turned out to be my computer running out of GPU memory. I managed to replicate the same issue in a windows machine with a Radeon RX 570 graphics card.
Not enough memory explains why removing the cropping layer or reducing the number of pixels cropped was triggering the issue, it looks like I was right at the limit of how much memory I had available and just a few pixels were making the difference.
Interestingly enough, monitoring the output of tf.memory() did not help since the problem is NOT the tensors created by the application but by TFjs internally, so printing out these stats at every batchEnd or epochEnd did not show any abnormality.
A couple of ways I managed to get around the problem was to reduce the batchSize which limited the number of tensors allocated by my application and also to reduce the inputShape to ingest a smaller image.
Hope that helps someone! It took me weeks to track it down.
Cheers.