I figured out the problem and explained it on YOLO github 11, but I will duplicate it here for web-search.
Indeed, the error was in the mismatch of tensor size. I was feeding the input tensor (ToTensor()(image_like_nparray)). And if you enter a tensor, it should be by size (imgsz, imgsz) (512 for me), and my tensor was (504, 378), which led to the error.
And to make it all right, you need to load numpy.array, and then model will bring everything to the tensor and the required sizes by itself.
P.S. @ptrblck Thanks, you’re cool.