I dive into the code, I find the error occurs in the SkipNet_Encoder. Due to the number of the output is 5 instead of 1. So the output_shape for the SkipNet_Encoder exist 5 list, which is not available to compute np.prod. This is exactly why summary throws out the error.
My debug information can be seen below:
ps: the breakpoint is set to the 93 line in the torchsummary.py.
'SkipNet_Encoder-15',
              OrderedDict([('input_shape', [-1, 3, 128, 128]),
                           ('output_shape',
                            [[-1, 256],
                             [-1, 64, 64, 64],
                             [-1, 128, 32, 32],
                             [-1, 256, 16, 16],
                             [-1, 256, 8, 8]]),
                           ('nb_params', 0)])),

Solution:
You can try to build encoder with only one output, not multiple outputs. Or you can use summary separately in the model.
Hoping my analysis make sense to you ~