I experimented with a few learning rates generally but starting from the forth epoc (3/9 where 0/9 is the first epoc) the loss diverges and gets bigger after each iteration. When I exceed 10 epocs the program terminates due to not having enough memory (my laptop runs on a CPU and has 8GB RAM). I thought if I added a line to delete the model output “loss_dict”  (5 tensors with I am assuming 5 separate computational graphs) after assigning the sum of the losses of “loss_dict” to “losses” this would solve the issue but it did not. I also delete “losses” after the optimizations and parameter updates are done but I’m still running out of memory. I modified the code to show where I am inserting the del statements.
Here are a few sample experiments I did:
EXP1:
Pretrained:True
momentum:0.9
weight_decay:0.0001
LR decay gamma:0.1
LR decay step size:2
Epoch 0/99
lr:0.001
loss_classifier:tensor(0.9254, grad_fn=)
loss_box_reg:tensor(0.0304, grad_fn=)
loss_mask:tensor(2.1443, grad_fn=)
loss_objectness:tensor(0.1386)
loss_rpn_box_reg:tensor(0.0056)
Total Train Loss: 3.2443
Epoch 1/99
lr:0.001
loss_classifier:tensor(0.7347, grad_fn=)
loss_box_reg:tensor(0.0005, grad_fn=)
loss_mask:tensor(2.7981, grad_fn=)
loss_objectness:tensor(0.3345)
loss_rpn_box_reg:tensor(0.0202)
Total Train Loss: 3.8879
Epoch 2/99
lr:0.0001
loss_classifier:tensor(0.4770, grad_fn=)
loss_box_reg:tensor(0.0003, grad_fn=)
loss_mask:tensor(1.5643, grad_fn=)
loss_objectness:tensor(0.0863)
loss_rpn_box_reg:tensor(0.0060)
Total Train Loss: 2.1340
Epoch 3/99
lr:0.0001
loss_classifier:tensor(0.4392, grad_fn=)
loss_box_reg:tensor(3.1149e-06, grad_fn=)
loss_mask:tensor(0.8555, grad_fn=)
loss_objectness:tensor(16.7120)
loss_rpn_box_reg:tensor(43.9244)
Total Train Loss: 61.9311
Epoch 4/99
lr:1.0000000000000003e-05
loss_classifier:tensor(0.4223, grad_fn=)
loss_box_reg:tensor(3.1140e-06, grad_fn=)
loss_mask:tensor(0.8396, grad_fn=)
loss_objectness:tensor(17.1041)
loss_rpn_box_reg:tensor(81.0183)
Total Train Loss: 99.3843
Epoch 5/99
lr:1.0000000000000003e-05
loss_classifier:tensor(0.4154, grad_fn=)
loss_box_reg:tensor(3.1139e-06, grad_fn=)
loss_mask:tensor(0.8382, grad_fn=)
loss_objectness:tensor(17.0591)
loss_rpn_box_reg:tensor(132.7647)
Total Train Loss: 151.0775
Epoch 6/99
lr:1.0000000000000002e-06
loss_classifier:tensor(0.4108, grad_fn=)
loss_box_reg:tensor(3.1138e-06, grad_fn=)
loss_mask:tensor(0.8367, grad_fn=)
loss_objectness:tensor(17.5224)
loss_rpn_box_reg:tensor(223.1389)
Total Train Loss: 241.9087
Epoch 7/99
lr:1.0000000000000002e-06
loss_classifier:tensor(0.4144, grad_fn=)
loss_box_reg:tensor(3.1138e-06, grad_fn=)
loss_mask:tensor(0.8365, grad_fn=)
loss_objectness:tensor(18.0877)
loss_rpn_box_reg:tensor(382.8543)
Total Train Loss: 402.1930
Epoch 8/99
lr:1.0000000000000002e-07
loss_classifier:tensor(0.4104, grad_fn=)
loss_box_reg:tensor(3.1138e-06, grad_fn=)
loss_mask:tensor(0.8364, grad_fn=)
loss_objectness:tensor(17.2382)
loss_rpn_box_reg:tensor(582.1258)
Total Train Loss: 600.6108
Epoch 9/99
lr:1.0000000000000002e-07
loss_classifier:tensor(0.4049, grad_fn=)
loss_box_reg:tensor(3.1138e-06, grad_fn=)
loss_mask:tensor(0.8364, grad_fn=)
loss_objectness:tensor(17.4174)
loss_rpn_box_reg:tensor(895.7090)
Total Train Loss: 914.3677
Epoch 10/99
lr:1.0000000000000004e-08
Traceback (most recent call last):
File “C:/Users/farbo/Documents/GitHub/Springboard/Capstone/toyexample/toylearner.py”, line 175, in 
loss_dict = model(inputs, targets)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in call
result = self.forward(*input, **kwargs)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\generalized_rcnn.py”, line 47, in forward
images, targets = self.transform(images, targets)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in call
result = self.forward(*input, **kwargs)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\transform.py”, line 41, in forward
image, target = self.resize(image, target)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\transform.py”, line 81, in resize
mask = misc_nn_ops.interpolate(mask[None].float(), scale_factor=scale_factor)[0].byte()
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\ops\misc.py”, line 101, in interpolate
input, size, scale_factor, mode, align_corners
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\functional.py”, line 2485, in interpolate
return torch._C._nn.upsample_nearest2d(input, _output_size(2))
RuntimeError: [enforce fail at …\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 19241573796 bytes. Buy new RAM!
Process finished with exit code 1
EXP 2 :
Pretrained:False
momentum:0.9
weight_decay:0.0001
LR decay gamma:0.1
LR decay step size:2
Epoch 0/99
lr:0.001
loss_classifier:tensor(0.6455, grad_fn=)
loss_box_reg:tensor(1.5951e-05, grad_fn=)
loss_mask:tensor(0.9080, grad_fn=)
loss_objectness:tensor(0.6955, grad_fn=)
loss_rpn_box_reg:tensor(0.0125, grad_fn=)
Total Train Loss: 2.2615
Epoch 1/99
lr:0.001
loss_classifier:tensor(0.6415, grad_fn=)
loss_box_reg:tensor(9.1854e-06, grad_fn=)
loss_mask:tensor(0.7481, grad_fn=)
loss_objectness:tensor(0.6944, grad_fn=)
loss_rpn_box_reg:tensor(0.0104, grad_fn=)
Total Train Loss: 2.0944
Epoch 2/99
lr:0.0001
loss_classifier:tensor(0.6334, grad_fn=)
loss_box_reg:tensor(1.4011e-06, grad_fn=)
loss_mask:tensor(0.6792, grad_fn=)
loss_objectness:tensor(0.6945, grad_fn=)
loss_rpn_box_reg:tensor(0.0034, grad_fn=)
Total Train Loss: 2.0105
Epoch 3/99
lr:0.0001
loss_classifier:tensor(0.6323, grad_fn=)
loss_box_reg:tensor(2.0635e-06, grad_fn=)
loss_mask:tensor(0.6925, grad_fn=)
loss_objectness:tensor(0.6919, grad_fn=)
loss_rpn_box_reg:tensor(44.4427, grad_fn=)
Total Train Loss: 46.4594
Epoch 4/99
lr:1.0000000000000003e-05
loss_classifier:tensor(0.6304, grad_fn=)
loss_box_reg:tensor(2.0636e-06, grad_fn=)
loss_mask:tensor(0.6914, grad_fn=)
loss_objectness:tensor(0.6912, grad_fn=)
loss_rpn_box_reg:tensor(77.8039, grad_fn=)
Total Train Loss: 79.8169
Epoch 5/99
lr:1.0000000000000003e-05
loss_classifier:tensor(0.6314, grad_fn=)
loss_box_reg:tensor(2.0636e-06, grad_fn=)
loss_mask:tensor(0.6912, grad_fn=)
loss_objectness:tensor(0.6912, grad_fn=)
loss_rpn_box_reg:tensor(138.1149, grad_fn=)
Total Train Loss: 140.1288
Epoch 6/99
lr:1.0000000000000002e-06
loss_classifier:tensor(0.6308, grad_fn=)
loss_box_reg:tensor(2.0636e-06, grad_fn=)
loss_mask:tensor(0.6911, grad_fn=)
loss_objectness:tensor(0.6919, grad_fn=)
loss_rpn_box_reg:tensor(225.4656, grad_fn=)
Total Train Loss: 227.4794
Epoch 7/99
lr:1.0000000000000002e-06
loss_classifier:tensor(0.6300, grad_fn=)
loss_box_reg:tensor(2.0636e-06, grad_fn=)
loss_mask:tensor(0.6911, grad_fn=)
loss_objectness:tensor(0.6910, grad_fn=)
loss_rpn_box_reg:tensor(360.3577, grad_fn=)
Total Train Loss: 362.3698
Epoch 8/99
lr:1.0000000000000002e-07
loss_classifier:tensor(0.6301, grad_fn=)
loss_box_reg:tensor(2.0636e-06, grad_fn=)
loss_mask:tensor(0.6911, grad_fn=)
loss_objectness:tensor(0.6912, grad_fn=)
loss_rpn_box_reg:tensor(544.0331, grad_fn=)
Total Train Loss: 546.0455
Epoch 9/99
lr:1.0000000000000002e-07
loss_classifier:tensor(0.6296, grad_fn=)
loss_box_reg:tensor(2.0636e-06, grad_fn=)
loss_mask:tensor(0.6911, grad_fn=)
loss_objectness:tensor(0.6913, grad_fn=)
loss_rpn_box_reg:tensor(903.4316, grad_fn=)
Total Train Loss: 905.4436
Epoch 10/99
lr:1.0000000000000004e-08
Traceback (most recent call last):
File “C:/Users/farbo/Documents/GitHub/Springboard/Capstone/toyexample/toylearner.py”, line 175, in 
loss_dict = model(inputs, targets)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in call
result = self.forward(*input, **kwargs)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\generalized_rcnn.py”, line 47, in forward
images, targets = self.transform(images, targets)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in call
result = self.forward(*input, **kwargs)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\transform.py”, line 41, in forward
image, target = self.resize(image, target)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\transform.py”, line 81, in resize
mask = misc_nn_ops.interpolate(mask[None].float(), scale_factor=scale_factor)[0].byte()
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\ops\misc.py”, line 101, in interpolate
input, size, scale_factor, mode, align_corners
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\functional.py”, line 2485, in interpolate
return torch._C._nn.upsample_nearest2d(input, _output_size(2))
RuntimeError: [enforce fail at …\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 19241573796 bytes. Buy new RAM!
Process finished with exit code 1
EXP 3:
Pretrained:True
momentum:0.9
weight_decay:0.0001
LR decay gamma:0.1
LR decay step size:2
Epoch 0/99
lr:1e-06
loss_classifier:tensor(1.0575, grad_fn=)
loss_box_reg:tensor(0.0339, grad_fn=)
loss_mask:tensor(1.5133, grad_fn=)
loss_objectness:tensor(0.1397)
loss_rpn_box_reg:tensor(0.0056)
Total Train Loss: 2.7500
Epoch 1/99
lr:1e-06
loss_classifier:tensor(1.0740, grad_fn=)
loss_box_reg:tensor(0.0004, grad_fn=)
loss_mask:tensor(1.8356, grad_fn=)
loss_objectness:tensor(0.3324)
loss_rpn_box_reg:tensor(0.0202)
Total Train Loss: 3.2626
Epoch 2/99
lr:1e-07
loss_classifier:tensor(1.0606, grad_fn=)
loss_box_reg:tensor(0.0001, grad_fn=)
loss_mask:tensor(1.3172, grad_fn=)
loss_objectness:tensor(0.1014)
loss_rpn_box_reg:tensor(0.0060)
Total Train Loss: 2.4854
Epoch 3/99
lr:1e-07
loss_classifier:tensor(1.0685, grad_fn=)
loss_box_reg:tensor(2.1282e-06, grad_fn=)
loss_mask:tensor(0.8204, grad_fn=)
loss_objectness:tensor(18.2750)
loss_rpn_box_reg:tensor(44.1334)
Total Train Loss: 64.2974
Epoch 4/99
lr:1.0000000000000002e-08
loss_classifier:tensor(1.0588, grad_fn=)
loss_box_reg:tensor(2.1282e-06, grad_fn=)
loss_mask:tensor(0.8204, grad_fn=)
loss_objectness:tensor(17.2795)
loss_rpn_box_reg:tensor(78.6653)
Total Train Loss: 97.8240
Epoch 5/99
lr:1.0000000000000002e-08
loss_classifier:tensor(1.0760, grad_fn=)
loss_box_reg:tensor(2.1282e-06, grad_fn=)
loss_mask:tensor(0.8208, grad_fn=)
loss_objectness:tensor(17.1733)
loss_rpn_box_reg:tensor(140.2065)
Total Train Loss: 159.2766
Epoch 6/99
lr:1.0000000000000003e-09
loss_classifier:tensor(1.0520, grad_fn=)
loss_box_reg:tensor(2.1282e-06, grad_fn=)
loss_mask:tensor(0.8211, grad_fn=)
loss_objectness:tensor(17.3521)
loss_rpn_box_reg:tensor(219.8562)
Total Train Loss: 239.0814
Epoch 7/99
lr:1.0000000000000003e-09
loss_classifier:tensor(1.0626, grad_fn=)
loss_box_reg:tensor(2.1282e-06, grad_fn=)
loss_mask:tensor(0.8212, grad_fn=)
loss_objectness:tensor(16.9220)
loss_rpn_box_reg:tensor(360.1990)
Total Train Loss: 379.0047
Epoch 8/99
lr:1.0000000000000002e-10
loss_classifier:tensor(1.0660, grad_fn=)
loss_box_reg:tensor(2.1282e-06, grad_fn=)
loss_mask:tensor(0.8213, grad_fn=)
loss_objectness:tensor(17.2279)
loss_rpn_box_reg:tensor(582.5305)
Total Train Loss: 601.6457
Epoch 9/99
lr:1.0000000000000002e-10
loss_classifier:tensor(1.0705, grad_fn=)
loss_box_reg:tensor(2.1282e-06, grad_fn=)
loss_mask:tensor(0.8214, grad_fn=)
loss_objectness:tensor(16.8634)
loss_rpn_box_reg:tensor(893.6630)
Total Train Loss: 912.4183
Epoch 10/99
lr:1.0000000000000003e-11
Traceback (most recent call last):
File “C:/Users/farbo/Documents/GitHub/Springboard/Capstone/toyexample/toylearner.py”, line 175, in 
loss_dict = model(inputs, targets)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in call
result = self.forward(*input, **kwargs)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\generalized_rcnn.py”, line 47, in forward
images, targets = self.transform(images, targets)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in call
result = self.forward(*input, **kwargs)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\transform.py”, line 41, in forward
image, target = self.resize(image, target)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\transform.py”, line 81, in resize
mask = misc_nn_ops.interpolate(mask[None].float(), scale_factor=scale_factor)[0].byte()
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\ops\misc.py”, line 101, in interpolate
input, size, scale_factor, mode, align_corners
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\functional.py”, line 2485, in interpolate
return torch._C._nn.upsample_nearest2d(input, _output_size(2))
RuntimeError: [enforce fail at …\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 19241573796 bytes. Buy new RAM!
Process finished with exit code 1

EXP 4:
Pretrained:False
momentum:0.9
weight_decay:0.0001
LR decay gamma:0.1
LR decay step size:2
Epoch 0/99
lr:1e-06
loss_classifier:tensor(0.6587, grad_fn=)
loss_box_reg:tensor(0.0081, grad_fn=)
loss_mask:tensor(1.0298, grad_fn=)
loss_objectness:tensor(0.6957, grad_fn=)
loss_rpn_box_reg:tensor(0.0133, grad_fn=)
Total Train Loss: 2.4057
Epoch 1/99
lr:1e-06
loss_classifier:tensor(0.6585, grad_fn=)
loss_box_reg:tensor(5.9168e-07, grad_fn=)
loss_mask:tensor(0.9116, grad_fn=)
loss_objectness:tensor(0.6954, grad_fn=)
loss_rpn_box_reg:tensor(0.0105, grad_fn=)
Total Train Loss: 2.2759
Epoch 2/99
lr:1e-07
loss_classifier:tensor(0.6585, grad_fn=)
loss_box_reg:tensor(1.3808e-06, grad_fn=)
loss_mask:tensor(0.8032, grad_fn=)
loss_objectness:tensor(0.6952, grad_fn=)
loss_rpn_box_reg:tensor(0.0035, grad_fn=)
Total Train Loss: 2.1605
Epoch 3/99
lr:1e-07
loss_classifier:tensor(0.6584, grad_fn=)
loss_box_reg:tensor(1.5408e-06, grad_fn=)
loss_mask:tensor(0.7098, grad_fn=)
loss_objectness:tensor(0.6902, grad_fn=)
loss_rpn_box_reg:tensor(42.8156, grad_fn=)
Total Train Loss: 44.8740
Epoch 4/99
lr:1.0000000000000002e-08
loss_classifier:tensor(0.6580, grad_fn=)
loss_box_reg:tensor(1.5408e-06, grad_fn=)
loss_mask:tensor(0.7098, grad_fn=)
loss_objectness:tensor(0.6905, grad_fn=)
loss_rpn_box_reg:tensor(83.0204, grad_fn=)
Total Train Loss: 85.0788
Epoch 5/99
lr:1.0000000000000002e-08
loss_classifier:tensor(0.6591, grad_fn=)
loss_box_reg:tensor(1.5408e-06, grad_fn=)
loss_mask:tensor(0.7099, grad_fn=)
loss_objectness:tensor(0.6908, grad_fn=)
loss_rpn_box_reg:tensor(138.3920, grad_fn=)
Total Train Loss: 140.4518
Epoch 6/99
lr:1.0000000000000003e-09
loss_classifier:tensor(0.6594, grad_fn=)
loss_box_reg:tensor(1.5408e-06, grad_fn=)
loss_mask:tensor(0.7099, grad_fn=)
loss_objectness:tensor(0.6906, grad_fn=)
loss_rpn_box_reg:tensor(214.6539, grad_fn=)
Total Train Loss: 216.7137
Epoch 7/99
lr:1.0000000000000003e-09
loss_classifier:tensor(0.6589, grad_fn=)
loss_box_reg:tensor(1.5408e-06, grad_fn=)
loss_mask:tensor(0.7099, grad_fn=)
loss_objectness:tensor(0.6908, grad_fn=)
loss_rpn_box_reg:tensor(371.0978, grad_fn=)
Total Train Loss: 373.1573
Epoch 8/99
lr:1.0000000000000002e-10
loss_classifier:tensor(0.6588, grad_fn=)
loss_box_reg:tensor(1.5408e-06, grad_fn=)
loss_mask:tensor(0.7099, grad_fn=)
loss_objectness:tensor(0.6902, grad_fn=)
loss_rpn_box_reg:tensor(565.0930, grad_fn=)
Total Train Loss: 567.1518
Epoch 9/99
lr:1.0000000000000002e-10
loss_classifier:tensor(0.6583, grad_fn=)
loss_box_reg:tensor(1.5408e-06, grad_fn=)
loss_mask:tensor(0.7099, grad_fn=)
loss_objectness:tensor(0.6898, grad_fn=)
loss_rpn_box_reg:tensor(877.6884, grad_fn=)
Total Train Loss: 879.7464
Epoch 10/99
lr:1.0000000000000003e-11
Traceback (most recent call last):
File “C:/Users/farbo/Documents/GitHub/Springboard/Capstone/toyexample/toylearner.py”, line 175, in 
loss_dict = model(inputs, targets)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in call
result = self.forward(*input, **kwargs)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\generalized_rcnn.py”, line 47, in forward
images, targets = self.transform(images, targets)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in call
result = self.forward(*input, **kwargs)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\transform.py”, line 41, in forward
image, target = self.resize(image, target)
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\models\detection\transform.py”, line 81, in resize
mask = misc_nn_ops.interpolate(mask[None].float(), scale_factor=scale_factor)[0].byte()
File “C:\Users\farbo\Anaconda3\lib\site-packages\torchvision\ops\misc.py”, line 101, in interpolate
input, size, scale_factor, mode, align_corners
File “C:\Users\farbo\Anaconda3\lib\site-packages\torch\nn\functional.py”, line 2485, in interpolate
return torch._C._nn.upsample_nearest2d(input, _output_size(2))
RuntimeError: [enforce fail at …\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 19241573796 bytes. Buy new RAM!
Process finished with exit code 1