If the code runs on single GPU, all the data is in a big batch. All the batch data is fed to 8 head attention model to calculate the relation between geometric and appearance separately. Thus all the dimensions are perfect aligned. If code got launched on multiple GPUs, each batch has it own data. This results of course the different dimensions corresponding to the inputs. That’s the problem, as I understand,if it’s correct. Still looking for a solution.