Talking to myself: how I trained GPT2-1.5b for rubber ducking using my chat data
From anecdotal testing, using the 774M/1.5B GPT-2 models for anything less than hundreds of megabytes of input data will result in worse generation quality than using the smaller 124M/355M models.The addiction to the larger GPT-2 models is IMO a trap.