Solved by ptrblck in post #8 
              
              
                Some layer might decrease the spatial size of the activation, such that it results in an empty tensor. 
As @charan_Vjy suggested, print the shape of the input directly before passing it to the adaptive pooling layer.