Thanks. That seems to be a straightforward idea. I will have to save the whole model and not only the state dict though (otherwise I cannot load the pertained model).
I would be nice to be able to load the state dict and by pass the layers whose dimensions do not matched. A solution is to change this line of the code 68:
                if input_param.shape != param.shape:
                    # local shape should match the one in checkpoint
                    error_msgs.append('size mismatch for {}: copying a param of {} from checkpoint, '
                                      'where the shape is {} in current model.'
                                      .format(key, input_param.shape, param.shape))
                    continue

to
import warnings
...
                if input_param.shape != param.shape:
                    # local shape should match the one in checkpoint
                    message = 'size mismatch for {}: copying a param of {} from checkpoint, '
                                      'where the shape is {} in current model.'
                                      .format(key, input_param.shape, param.shape)
                    if strict:
                        error_msgs.append(message)
                    else:
                        warnings.warn(message)
                    continue