You could manually delete the unwanted ref_model parameters and buffers directly as seen here:
class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(3,4)  # trainable parameters
        self.ref_m = nn.Linear(10, 10)    # fixed parameters
        for m in self.ref_m.parameters():
            m.requires_grad=False

model = MyModel()
checkpoint = model.state_dict()

keys_to_delete = []
for key in checkpoint:
    if 'ref_m' in key:
        keys_to_delete.append(key)
for key in keys_to_delete:
    del checkpoint[key]
print(checkpoint)

Note however, that you won’t be able to use model.load_state_dict(checkpoint) with the default setting of strict=True anymore since obviously keys are missing. In this case you would have to use strict=False and make sure that only the expected ref_m keys are missing.
I assume ref_model is an entire model and not just buffers, as you could use self.register_buffer with persistent=False in the latter case and could avoid manipulating the state_dict.