Some layer might decrease the spatial size of the activation, such that it results in an empty tensor.
As @charan_Vjy suggested, print the shape of the input directly before passing it to the adaptive pooling layer.