Hi @ptrblck,
Thanks for responding. This is not an overfitting issue for sure, but to provide you some insight in to the model performance I added some of the tensorboard graphs at the end. Yes, the issue occurs when I load the checkpoint.
To provide some more context, this is an open-domain chatbot model. I trained these kinds of models using Pytorch many times before and I know the difference between when such a model is not trained well in some way, and when there is really something technically wrong. When it is not trained well (under-fitted or over-fitted), you can still make a conversation with it, but it makes not much sense, and/or you might see glitches in the text generation. In the case of the ‘faulty’ checkpoint the model simply outputs garbage, without any structure at all.
Further, it is very strange that all parameter Tensors, over all layers(!), start with the same kind of values (zero, or very close to zero). This can’t be right.
Some more details; chat_gru and conversation_gru are actually not GRUs, but Simple Recurring Units (SRUs), also see https://github.com/asappresearch/sru.
Because I started my experiment using the SRUs for the first time, I thought maybe the issue is related to using the SRUs. But if that is the case, why would other layers (e.g. feed-forward layer out) also have the first two parameters close to zero? Could it be that on the C level, memory gets overwritten? (SRU has a C/CUDA implementation)
When I started the experiment, I also started using Pytorch 1.6.0, so maybe it is related to that, I can’t really tell.
Below are some screenshots of my Tensorboard for this experiment:
Moving average of cross entropy loss for training and validation set at earlier “good”/“functioning” checkpoint:
good-checkpoint-performance-241120202888×1346 525 KB
Moving average of cross entropy loss for training and validation set at later “faulty”/“garbage-out” checkpoint:
faulty-checkpoint-performance-241120202932×1438 550 KB
Cross entropy over whole training set, calculated once per epoch:
epoch-training-cross-entropy-before-faulty-checkpoint-241120201206×688 44.1 KB
Cross entropy over whole validation set, calculated once per epoch:
epoch-validation-cross-entropy-before-faulty-checkpoint-241120201160×654 46.1 KB
Last note : I use a dropout of 0.2, this is why the training loss is higher then the validation loss.