readme: \n---
  tags:
  - Causal Language modeling
  - text-generation
  - CLM
  model_index:
  - name: MechDistilGPT2
    results:
    - task:
        name: Causal Language modeling
        type: Causal Language modeling
  ---
  ## MechDistilGPT2
  This model is fine-tuned on text scraped from 100+ Mechanical/Automotive pdf books.
  
  Base model is DistilGPT2(https://huggingface.co/gpt2) (the smallest version of GPT2)
  
  ## Fine-Tuning
  * Default Training Args
  * Epochs = 3
  * Training set = 200k sentences
  * Validation set = 40k sentences
  
  ## Framework versions
  * Transformers 4.7.0.dev0
  * Pytorch 1.8.1+cu111
  * Datasets 1.6.2
  * Tokenizers 0.10.2
  
  ## References
  https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb