mojulian:

I went through the forums and had a look at the synchronization issue between cpu and gpu which I also think is the real bottleneck of my training… However, can it really be, that synchronization takes up most time…


Synchronizations themselves are not taking time, but are synchronizing with another process and would thus accumulating time. E.g. if your GPU is busy executing the forward pass of the model the CPU would have to synchronize and thus wait for the GPU if you are trying to print the output. E.g. here:
output = model(input) # executed on the GPU
print(output) # CPU synchronizes since the values of output are needed to be able to print them

the print statement would need to synchronize with the GPU as it would otherwise just print uninitialized memory. In a profile the print operation could thus accumulate the time of the forward pass and could look expensive, while in fact the forward pass itself takes the majority of the time.
Manual synchronizations via torch.cuda.synchronize() are often used to properly profile the code.
Alternatively, also the PyTorch profiler or Nsight Systems can be used as they would show the timeline and you would be able to see how long each operation takes.



 mojulian:

Additionally, If I set BatchSize to 32 I already run out of Cuda memory, which seems a bit strange since I one Batch would roughly take (since 3234484484) ~ 77MB, or am I missing something?


Usually the input size is small compared to the intermediate activations created during the forward pass and/or the model parameters. Take a look at this post which explains it in more details with some examples.