In that case you should create separate layers, i.e. one conv layer containing the pretrained filters and another one with the random weights, and concatenate the results of these layers after they are applied.
This would make it possible to just optimize the random weights and freeze the old ones.