Hello,
I have been trying to use either pytorch 1.6.0 rc or pytorch nightly (Currently torch-1.7.0.dev20200702+cu101) in order to get access to native 16 bit precision support.
On the implementation side I am using pytorch-lightning (0.8.4) to do all the training boiler plate code.
Whenever I try to run any training loop, as I go through the validation_epoch_end (code below) I get the following error:
 File "/mnt/shared/home/mvera/.virtualenvs/VMQE/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 979, in fit
    self.single_gpu_train(model)
  File "/mnt/shared/home/mvera/.virtualenvs/VMQE/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 185, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/mnt/shared/home/mvera/.virtualenvs/VMQE/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1139, in run_pretrain_routine
    False)
  File "/mnt/shared/home/mvera/.virtualenvs/VMQE/lib/python3.7/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 342, in _evaluate
    eval_results = model.validation_epoch_end(outputs)
  File "/mnt/shared/home/mvera/redacted.py", line 365, in validation_epoch_end
    losses[loss] += value
RuntimeError: Trying to pass too many CPU scalars to CUDA kernel!

I can find absolutely no threads/issues on this error anywhere. Could you provide me with some pointers on how to solve this?
Extra information:
I am basically running a HugginFace Transformers model with a FF on top for sentence classification. Which HuggingFace model doesn’t seem to matter.
Code for validation_epoch_end is:
        losses = defaultdict(lambda: torch.tensor(0.0))
        for output in outputs:
            for loss, value in output['val_losses'].items():
                losses[loss] += value
        for loss in losses:
            losses[loss] /= len(outputs)


Any pointer for places/things to look for would be great.
Thanks in advance!