Solved:  the discrepancy was due to our development environment running on torch, and our delivery environment using pytorch.  While pytorch is faster than torch, the inference batch is no faster using pytorch than one by one inference, whereas in torch there is a significant speed up.