Hey @ptrblck! 
What I mean with Residual Block is illustrated in the image below:
The Residual Block will get as input the tensor from conv2, apply some convolutions and the output of the Residual Block will be added to the tensor of conv8(I will change my implementation to sum ehehehehe).
In my understanding, the Residual Block and the operation of sum would be the “Residual Connection”…I didn´t understood why the sum operation is performed inside the Residual Block as you showed on the link is the “Residual connection”. Could you explain?
Abaout the loss values are not decreasing, do you think it would just be an adjustment to the larning rate or could this Residual Block be interfering with that?
Obs: the tensor’s dimensions in the code are a little bit differente from the image.