Thanks for the information.
I would assume training from scratch might fail due to the small size of the dataset.
Per-layer learning rates are probably not necessary to create a solid baseline model.
If playing around with some hyperparameters doesn’t yield any performance increase, you could try to overfit a tiny subset (e.g. 10 images). If that still doesn’t work, your code or training procedure might have a bug which we’ve missed so far.