It’s in the code: 2^12=4096. The model we’re using has a fairly small memory footprint and we want to use large batches to maximize GPU memory utilization for bulk scoring.
I get this behavior on 2-8 GPUs, not just 3, so the odd number of GPUs shouldn’t be a factor. Do you think I should make batch size a multiple of the number of GPUs?