&gt; 3. Lack of training data<p>&gt; [...]<p>&gt; There&#x27;s not an incredibly rich structure to extract, and with short outputs the weirdness doesn&#x27;t compound and cycles aren&#x27;t as likely. A common small dataset for playing with RNNs is all of Shakespeare which is somewhere in the region of 1M words.<p>He does state that the network is trained with 2M headlines, meaning ~5-20M words. That <i>should</i> be enough.<p>I would have thought that RNN would somehow work better. It would be interesting to see direct comparison of fake hacker news headlines generated with Markov chains versus RNN.