Xavier is an initialization technique; with a better random initialization, it’s easier to train a network.
Since training accuracy is not good, you might still under fit. Well, can be a lot of things; are dropout the same, optimizer the same, weight decay (it seems caffe usually don’t apply weight decay to biases), batch_size, etc.