It’s a bit hard to tell what might be wrong, but since you are not able to overfit a small data sample, I would probably take a look at the model architecture.
Were you using a “known good” model or did you create this architecture as an experiment yourself?