I cannot reproduce this issue using your model definition and this code:
model = PSPNet()
x = torch.randn(2, 3, 224, 224)

process = psutil.Process(os.getpid())
for idx in range(100):
    print(idx, process.memory_full_info().rss / 1024**2)
    out = model(x)

The memory info results in a usage between 202 to 513MB for 120 iterations:
0 202.50390625
1 341.28515625
2 427.9609375
3 477.71484375
4 429.0390625
5 449.1484375
6 449.1484375
7 449.1484375
8 473.1796875
9 447.20703125
10 454.21484375
11 425.265625
12 438.8125
13 462.953125
14 438.78125
15 450.0078125
16 470.640625
17 495.65234375
18 495.65234375
19 466.71484375
20 490.94921875
21 425.29296875
22 449.8359375
23 474.328125
24 498.0078125
25 474.984375
26 427.91796875
27 498.296875
28 498.296875
29 435.6875
30 459.5703125
31 485.3515625
32 426.7265625
33 449.15234375
34 449.15234375
35 426.125
36 450.09765625
37 425.78125
38 451.04296875
39 474.50390625
40 498.73828125
41 436.09375
42 451.3046875
43 489.19921875
44 489.19921875
45 489.19921875
46 460.77734375
47 485.265625
48 426.62890625
49 499.5859375
50 499.5859375
51 430.8125
52 455.55859375
53 431.5078125
54 493.12109375
55 444.44921875
56 466.10546875
57 434.92578125
58 513.296875
59 464.46875
60 464.46875
61 479.6796875
62 479.6796875
63 479.6796875
64 430.80859375
65 454.78125
66 504.28125
67 406.48046875
68 479.4375
69 479.4375
70 479.4375
71 479.4375
72 430.640625
73 503.85546875
74 503.85546875
75 454.6015625
76 479.60546875
77 479.60546875
78 479.60546875
79 484.24609375
80 437.57421875
81 510.7890625
82 462.0859375
83 481.9375
84 505.13671875
85 443.75
86 468.23828125
87 468.23828125
88 468.23828125
89 443.578125
90 516.53515625
91 467.83203125
92 516.296875
93 467.4765625
94 467.4765625
95 479.59375
96 430.65234375
97 455.140625
98 455.140625
99 492.5234375
100 492.5234375
101 418.58203125
102 491.796875
103 491.796875
104 423.875
105 471.82421875
106 423.0234375
107 449.31640625
108 423.9765625
109 448.20703125
110 473.47265625
111 448.171875
112 454.1015625
113 502.82421875
114 454.14453125
115 454.14453125
116 428.69921875
117 491.859375
118 443.1875
119 497.58203125