pbontrager:

Thank you for your quick reply. I am well aware that tutorials are not meant to be benchmark implementations but I was surprised that it did not train usable representations at all.


Wanted to reiterate one of the statements regarding these examples again:

NOTE THAT THIS COMMAND WILL ERROR OUT IF ANY CELLS TAKES TOO LONG TO EXECUTE. In that case, make your code lighter/faster. Remember that examples are meant to demonstrate workflows, not train state-of-the-art models. They should stay very lightweight.

So, in the case of this example, we get to 35% which is better than random chance (10%). But I do understand your point and I will try my best to suggest from my experience (that’s the whole purpose of having a discussion forum, isn’t it :)).



 pbontrager:

By true random cropping I mean a random crop and resize as mentioned in the paper. This crops a random percentage of the original image, given as hyperparameters, and then resizes the cropped image to the original size. “tf.image.random_crop” as used in the tutorial, randomly crops an image at the size given, since we’re just cropping a 32x32 image to 32x32 it is effectively doing nothing. I thought that the lack of cropping was causing the tutorial code to not work, but fixing this did not improve the performance.


Random-resized crops. You are right. I should have included a note about this in the tutorial itself. I was a bit worried about the lines of code. But refer to this implementation 6 and see if this works. It’s a bit different from PyTorch’s RandomResizedCrop() but it has resulted well in the other experiments I have performed in this area.



 pbontrager:

I did not match the backbone from the paper and their hyperparameters. I used what you used in the tutorial as it is very close. I’ll try changing the backbone and maybe it’ll help but I’d be very surprised if a slightly different backbone changes the linear transfer score from 35% to the 90% reported in the paper. It would also be pretty damning for SimSiam if it turned out the backbone was the primary contributor to benchmark success.


I agree here. But what happens is the relation between your dataset’s complexity and architectural complexity matters. In some methods like SimCLR 1, this effect is very pronounced. But in methods like Barlow Twins, it’s lesser. You can find an implementation of Barlow Twins here 3 and within just 100 epochs of pre-training, I was able to get to 62.61% in terms of linear evaluation on CIFAR-10.



 pbontrager:

Thanks for any help. I’m just trying to get a clean, simple, implementation that performs roughly similar to the paper and hopefully find a few tweaks that could allow a person to get a usable trained model from the tutorial.


I appreciate this so much. The field is relatively new, so as contributors we wanted to get the workflow out as soon as possible without missing out on the important bits. So, I am here to suggest anything I can to better your results.
What you are currently running into (poor performance) is likely because of a phenomenon called representation collapse. This is when your backbone predicts the same output for a given image and self-supervised methods for vision easily latch into this problem. More on this later.
One thing to note this, for linear evaluation, often a separate configuration of hyperparameters and augmentation pipeline are incorporated. For SimSiam’s case, they are specified under the A. Implementation Details section. Unfortunately, they did not state these for CIFAR-10. So, this leaves us a fairground for further experimentation.
At this point, you’d be totally correct to think that self-supervision (for computer vision) is often sensitive to hyperparameters. But methods like Barlow Twins, VICReg 1 help to eliminate that to some extent. They also gracefully mitigate the problem of representation collapse (DINO 1 as well).
I hope this helps.