It really depends. Even if 4 replicas of your model can fit into the memory of one GPU, they still need to compete for the same set of streaming multiprocessors and other shared resources on that GPU. You can try if using multiple streams would help, e.g.:
s0 = torch.cuda.Stream()
s1 = torch.cuda.Stream()
with torch.cuda.stream(s0):
    output0 = model_replica0(input0)

with torch.cuda.stream(s1):
    output1 = model_replica1(input1)

s0.synchronize()
s1.synchronize()