The concept is called https://en.wikipedia.org/wiki/Data_parallelism 2. Batch dimension of tensors is assumed to be independent, meaning that operations that transform other dimensions can be executed in parallel . And threading to do this is “implementation detail”, i.e. it is backend specific, and also operation specific (e.g. vector/matrix operations treat some dimensions as special)