Thank you @jmaronas for your response. I get that transferring pretrained weights generally improves performance. However, in my experience with training FCNs, I have had little luck actually getting any learning from scratch. The paper also emphasizes that you do “need” pretrained weights. Even the original Github by the authors of the FCN paper states:
Why are all the outputs/gradients/parameters zero? : This is almost universally due to not initializing the weights as needed. To reproduce our FCN training, or train your own FCNs, it is crucial to transplant the weights from the corresponding ILSVRC net such as VGG16.
This makes me think it’s probably not just me and that in some cases one might not get any results without pretrained weight, for FCNs. My question is what makes fully convolutional networks so dependent on pretrained weights?