Thank you for this reply. I’ll have time to experiment with this more this week. For context, I am running the code outside of a notebook on my own machine so I can train 100 epochs pretty quick. I train the model, save it, and then load into an eval script that uses the code from the tutorial to train on linear evaluation but allows me to use different hyperparameters. I have also made my own random crop and resize using TensorFlow’s sampled_distorted_bounding_box.
def random_crop_and_resize(image, size, area_range):
    # Crop randomly smaller and resize to size
    begin, shape, _ = tf.image.sample_distorted_bounding_box(
        tf.shape(image),
        [[[0.0, 0.0, 1.0, 1.0]]],
        area_range=area_range)
    image = tf.image.crop_to_bounding_box(image, begin[0], begin[1], shape[0], shape[1])
    image = tf.image.resize(image, (size, size))
    return image

Thanks for the Barlow Twins recommendation, the paper seemed neat but I didn’t realize that it is empirically less prone to representation collapse. Currently, I need SimSiam for some work I want to do extending it, so getting my own working version of it that I understand is important. I also believe that this can shed some light on how important different aspects of the core algorithm are vs training tricks.
I hadn’t realized from any of the papers that any augmentations were used during the linear evaluation training step. I will experiment with this a bit more. I’ll also inspect the output representations from training to see if they’ve collapsed to a fixed output. I’ll post here if the tuning is successful or if I get stuck again.
Thank you.