Thanks for the help, well about alphas I don’t see any further usage…but maybe I am wrong. The full code is available here (the softmax call occurs at line 93:

  
      github.com
  
  
    caolingyu/Purr/blob/master/models.py 3
# coding = utf-8
"""
    Holds PyTorch models
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn.init import xavier_uniform
from torch.autograd import Variable

import numpy as np

import math
import random
import sys
import time

sys.path.append('../')
from constants import *
from preprocess import extract_wvs


  This file has been truncated. show original

  
  
    
    
  
  


and even after setting dim=1 I still have OOM with big training text files on 4 GB GPU. There are actually two “memory black-holes” in the code, one in models.py at the level of the forward function and another one that is killing me (and apparently I will have to buy a bigger GPU board)  is a “classical” memory-hole at the level of loss.backward() in this file, at line 174:

  
      github.com
  
  
    caolingyu/Purr/blob/master/train.py
# coding = utf-8
"""
    Main training code. Loads data, builds the model, trains, tests, evaluates, writes outputs, etc.
"""
import sys
# sys.path.insert(0, '..')
import torch
import torch.optim as optim
from torch.autograd import Variable

import csv
import argparse
import os 
import numpy as np
import operator
import random
import sys
import time
from tqdm import tqdm
from collections import defaultdict


  This file has been truncated. show original

  
  
    
    
  
  


I am training the model with a 1,7 Mio lines text file, having an average of 500 chars per row and I can’t succeed even with a batch_size=1 (OOM after several Ks of trained lines). If the training file drops to about 200K rows then I can finish the training with batch_size = 4. I tried everything possible on earth (except accumulating losses…which I don’t know how to implement yet in this code) to get out of the never-ending:
RuntimeError: CUDA out of memory. Tried to allocate 438.63 MiB (GPU 0; 4.00 GiB total capacity; 2.64 GiB already allocated; 389.80 MiB free; 5.39 MiB cached)  but it seems impossible.