@ptrblck thanks! Your examples for Variant A and Variant B work for me as well. When I use my “core model” (with embedding layer, RNN layer, linear layers) it still only works with A. All I did was to put the code from CoreModel directly into the SiameseNetwork class. I simply cannot see the difference at the moment.
I probably will need to slowly extend the basic work Variant B to see where it begins to break down.
EDIT: Yeah, slowly building up the basic network of Variant B did the trick. I cannot really tell where I made an error in the first place. I assume(!) that I didn’t correctly re-initialize the hidden state of the RNN layer at the right time(s).