I did basically a mixture of 4 things


Reduce number of channels per Convolutional Layer. I kept the depth of the network as it is since I thought going deep was the key here. But I thought I could reduce the number of channels per layer and still get the same performance.

Remove the Cropping part when combining decoder and encoder. I made sure the sizes matched.
Do Global Average pooling of the feature map for the classifier part.

Reduce the number of Linear Layers in classification part.

I am not sure how much each point affects the time performance, but I was able to reduce inference time on CPU with fixed 1 thread from 1.2-1.6 seconds to 0.2-0.3 seconds