Solution is to use Masking() layers available in keras with mask_value=0 . This is because when using empty vectors they are calculated into the loss, by using Masking() , as outlined by keras the padding vectors are skipped and not included.