Thank you for sharing this snippet of code. It is really helpful.
I profiled the code and it seems that the issue is not due to the hooks, but maybe the function I am replacing. Regardless of whether the batch size is 256 or 10,000, the first layer that I replace takes 12 seconds on GPU and the second one takes about 6 seconds. As a result, the configuration that has a large number of small batches takes a lot longer.
Most of my functions include a single line, which is very long and includes bit-wise operations on bool tensors, e.g.
out[:, 0] = (x[:, 0] & ~x[:, 1]) | (x[:, 124] & (x[:, 118] & ~x[:, 254]))...
In both cases, the GPU utilization is around 20%.