There is clearly something wrong, I performed the same experiment with CNTK and there is a big difference when freezing:
Using CNTK with simpsons dataset:
freeze:
Test accuracy is 0.763938618926
Process time 621.0402624607086 seconds
finetune:
Test accuracy is 0.959335038363
Process time 1635.1398813724518 seconds