@Liang


A quick debug step would be to first freeze all the network parameters (model to eval and using volatile Variables), extract the ReLU_5 features and perform the bilinear pooling. Even without training, the outer-product and pooling of the relu_5 features gives a performance boost over the regular CNN features. If this works as expected, then you would have a sanity-check that at least your forward pass is working.


Fix the batch-normalization parameters of the CNN.


Train the classifier layers first, before fine-tuning the whole network.


The optimizer should be SGD following the original paper.