to sum it up I create batchs of size 5,4,3 or 2 depending on the number of target values I can batch. For  one input I need to forward it through the network, compare it to a bunch of targets and backpropagate  the loss.
Instead of doing it for one target at a time I am doing it for 5 at a time ( 4,3,2 if there are not enough left to do a 5 target batch). So if I have 5 targets I use 5 inputs (seems to be the maximum i can use in  my gpu memory.
It worked well as it sped up my training, but after some epochs my GPU memory utilization drops from 90% to 10% and then stays around this value, slowing the training a lot.
Sorry for the explanation which is a bit confusing.