So on my laptop (4x i7-5600U CPU @ 2.60GHz in cpuinfo), the test seems to run in ~95ms (with the most lazy, inaccurate measurement and running it once or 11x to get some handle on the invocation overhead).
tv@aComp:~/python/pytorch/pytorch$ python3 test/test_quantized.py TestQuantizedOps.test_adaptive_avg_pool2d_nhwc
.
----------------------------------------------------------------------
Ran 1 test in 0.115s

OK
$ python3 test/test_quantized.py TestQuantizedOps.test_adaptive_avg_pool2d_nhwc{,,,,,,,,,,}
...........
----------------------------------------------------------------------
Ran 11 tests in 1.050s

OK

I don’t know what factor you would suppose to exist between my laptop and your computer.
We would likely want to know if the test suddenly regressed 2x in performance, so turning off the deadline might not be the best thing, but maybe you could have a decorator turning it off or making it more generous for your specific architectures. (These are just some thoughts, I don’t know about any “official policy” or anything.)
Best regards
Thomas