Since your model is training with a small dataset and is failing with the entire set, you could check how the gradients (and their magnitudes etc.) behave in both approaches. Since the loss is apparently not changing after a while I would guess that the gradients might converge towards zero.