The drop in performance sounds reasonable, since your inputs would only contain:
(112*40*40) / (128*128*128) * 100
~ 8.54%

valid data while the rest of the input values doesn’t contain any information.