readme: # Exo-Machina
  A deep language model, GPT-2, is trained on scientific manuscripts from NASA's Astrophysical Data System pertaining to extrasolar planets and the references therein. This pilot study uses the abstracts of each article as training data in order to explore correlations in scientific literature from a language perspective. A language model is a mathematical representation for an algorithm used to generate sequences in the same way a human would to form sentances. Each word or letter in a sentance is encoded to a numerical value (e.g. using word2vec) and is appended to a list forming sequences that represent up to a paragraph worth of text. The sequences are fed into the [GPT-2](https://openai.com/blog/better-language-models/) 117M model and trained for 500,000 steps with fine tuning. After training, the language model is used to generate new text from scratch and from user input. 
  
  - ### [Browse samples](https://pearsonkyle.github.io/Exo-Machina/)
  
  - ### [Train a model on Google Colab](https://colab.research.google.com/drive/1Pur0rFi5YVdn7axYRacXWFMic4NxRexV?usp=sharing)
  
  ### Get started fast:
  
  ```python
  from transformers import pipeline
  
  exo = pipeline('text-generation',model='pearsonkyle/gpt2-exomachina', tokenizer='gpt2', config={'max_length':1600})
  machina = lambda text: exo(text)[0]['generated_text']
  
  print(machina("Transiting exoplanets are"))
  ```
  
  ## Training Samples
  ~40,000 Abstracts from NASA's Astrophysical data system (ADS) and ArXiv. 
  
  ![](https://huggingface.co/pearsonkyle/gpt2-exomachina/raw/main/exoplanet_keywords.png)
  
  A few generated samples are below: 
  
  - *We can remotely sense an atmosphere by observing its reflected, transmitted, or emitted light in varying geometries. This light will contain information on the planetary conditions including* `temperature, pressure, composition, and cloud optical thickness. One such property that is important is...`
  - *The reflectance of Earth's vegetation suggests*
  `that large, deciduous forest fires are composed of mostly dry, unprocessed material that is distributed in a nearly patchy fashion. The distributions of these fires are correlated with temperature, and also with vegetation...`
  - *Directly imaged exoplanets probe* `key aspects of planet formation and evolution theory, as well as atmospheric and interior physics. These insights have led to numerous direct imaging instruments for exoplanets, many using polarimetry. However, current instruments take`
  
  Letting the scrape run for ~2 hours found articles from these publications:
  ```
   5364 - The Astrophysical Journal
   3365 - Astronomy and Astrophysics
   2704 - Monthly Notices of the Royal Astronomical Society
   1355 - The Astronomical Journal
   617 - arXiv e-prints
   498 - Icarus
   388 - Publications of the Astronomical Society of the Pacific
   324 - The Astrophysical Journal Supplement Series
   245 - Nature
   187 - Journal of Geophysical Research
   167 - Science
   145 - Astronomische Nachrichten
   129 - Planetary and Space Science
   114 - Space Science Reviews
   109 - Geophysical Research Letters
  ```