The solution was to copy the image
img = img.copy()

it seems that Tensor.numpy() actually gives you an immutable view of the underlying data structure so you can display it but not modify it. To modify it I had to create my own copy