Word Language Model training question
Given the sequence of characters: “a” “b” “c” “d” … “z” and bptt equal to 3 and ignoring batching for simplicity: first sequence: src=“a”,“b”,“c”; trg=“d” second sequence: src=“d”, “e”, “f”; trg=“g” Perhaps, I am wrong but doesn’t it mean that an amount of data proportional to the value of bptt isn’t being used during training (in the example above sequences src=“b” “c” “d”, trg=“e” and src=“c” “d” “e”, trg=“f” aren’t in the training set)?
emanjavacas
b,c,d target=“e” is covered by carrying the hidden state forward between sequences.
smth
emanjavacas
In other words, the parameter bptt seems to be tweaking two things: how many steps back to include in the rnn computational graph. how many examples to take into account for estimate perplexity on. but perhaps I am understanding things in a wrong way…
emanjavacas