readme: ---
  language: 
    - "en"
  thumbnail: "https://styles.redditmedia.com/t5_2to41/styles/communityIcon_qedoavxzocr61.png?width=256&s=9c7c19b81474c3788279b8d6d6823e791d0524fc"
  datasets:
    - "reddit_tifu (subset: short)"
  widget:
  - text: "I told my friend"
  license: mit
  ---
  
  # mgfrantz/distilgpt2-finetuned-reddit-tifu
  
  This model was trained to as practice for fine-tuning a causal language model.
  There was no intended use case for this model besides having some fun seeing how different things might be screwed up.
  
  ## Data
  
  This model was trained on "short" subset of [`reddit_tifu`](https://huggingface.co/datasets/reddit_tifu) dataset.
  The data was split into 90% train and 10% validation using `dataset.train_test_split`, with a seed of 0.
  
  To prepare the data for training, the `"tldr"` and `"documents"` fields were joined by `"\n\n"`.
  When multiple items were in the `"tldr"` or `"documents"` fields, only the first item was selected for joining.
  These joined documents were tokenized using the `"distilgpt2"` tokenizer.
  
  Finally, tokenized texts were concatenated end-to-end and split into blocks of 128 tokens.
  
  **TODO:** Add a different separation token between documents that can be used to stop generation.
  
  ## Training
  
  This model was trained in Colab by fine-tuning [`distilgpt2`](https://huggingface.co/distilgpt2) for 174390 steps (3 epochs).
  Default training arguments were used, except for `learning_rate=2e-5` and `weight_decay=0.01`.
  At the conclusion of training, a training loss of 3.52 and a validation loss of 3.44 were observed.