Are you sure that you are loading a pretrained model ? And that you are properly loading its weights ?
Another side question ? Why are you using the C++ API ? If you want to do inference with deep CNN models, the python overhead may not be significant, since the bottleneck will probably be the convolutions, which are done by cuDNN in both API’s (assuming that you use a GPU for inference).