readme: ---
  language: "en"
  thumbnail: "https://raw.githubusercontent.com/stevhliu/satsuma/master/images/astroGPT-thumbnail.png"
  widget:
  - text: "Jan 18, 2020"
  - text: "Feb 14, 2020"
  - text: "Jul 04, 2020"
  ---
  
  # astroGPT 🪐
  
  ## Model description
  
  This is a GPT-2 model fine-tuned on Western zodiac signs. For more information about GPT-2, take a look at 🤗 Hugging Face's GPT-2 [model card](https://huggingface.co/gpt2). You can use astroGPT to generate a daily horoscope by entering the current date.
  
  ## How to use
  
  To use this model, simply enter the current date like so `Mon DD, YEAR`:
  
  ```python
  from transformers import AutoTokenizer, AutoModelWithLMHead
  
  tokenizer = AutoTokenizer.from_pretrained("stevhliu/astroGPT")
  model = AutoModelWithLMHead.from_pretrained("stevhliu/astroGPT")
  
  input_ids = tokenizer.encode('Sep 03, 2020', return_tensors='pt').to('cuda')
  
  sample_output = model.generate(input_ids,
                                 do_sample=True, 
                                 max_length=75,
                                 top_k=20, 
                                 top_p=0.97)
                                  
  print(sample_output)
  ```
  
  ## Limitations and bias
  
  astroGPT inherits the same biases that affect GPT-2 as a result of training on a lot of non-neutral content on the internet. The model does not currently support zodiac sign-specific generation and only returns a general horoscope. While the generated text may occasionally mention a specific zodiac sign, this is  due to how the horoscopes were originally written by it's human authors.
  
  ## Data
  
  The data was scraped from [Horoscope.com](https://www.horoscope.com/us/index.aspx) and trained on 4.7MB of text. The text was collected from four categories (daily, love, wellness, career) and span from 09/01/19 to 08/01/2020. The archives only store horoscopes dating a year back from the current date.
  
  ## Training and results
  
  The text was tokenized using the fast GPT-2 BPE [tokenizer](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizerfast). It has a vocabulary size of 50,257 and sequence length of 1024 tokens. The model was trained with on one of Google Colaboratory's GPU's for approximately 2.5 hrs with [fastai's](https://docs.fast.ai/) learning rate finder, discriminative learning rates and 1cycle policy. See table below for a quick summary of the training procedure and results.
  
  | dataset size  | epochs | lr                | training time | train_loss | valid_loss | perplexity | 
  |:-------------:|:------:|:-----------------:|:-------------:|:----------:|:----------:|:----------:|
  | 5.9MB         |32      | slice(1e-7,1e-5)  | 2.5 hrs       | 2.657170   | 2.642387   | 14.046692	|
  