It’s not very orthodox but you can send them to the gpu inside the forward function. Note that this won’t be compatible with dataparallel.
Anyway it seems complicated to generate a dictionary, if you have 2 NN why don’t you code the dataloader to load 2 set of inputs?