readme: ---
  license: bigscience-openrail-m
  datasets:
  - c4
  - CarperAI/pile-v2-small-filtered
  language:
  - en
  pipeline_tag: text-generation
  ---
  
  # crumb/exalted-frog-25
  
  My first attempt home-brewing a pretrained transformer >`100m` params on English text on my own local machine, trained on a combination of both the *c4* and *pile-v2-small-filtered* datasets. Named after the first run that didn't crash on wandb, exalted-frog-25 is a `124M` param GPT2 architecture trained from scratch over the course of two four-hour sessions on an RTX-2060.
  
  It ran until it crashed, seeing in total `36,651` samples of `1,024` tokens each (`37,530,624` tokens in total, it's under-trained. `44.5%` came from C4 and `55.5%` came from the pile) It ended in a training loss of `5.45` and an evaluation loss of `4.202`. For the first run, it was trained with a batch size of `32`, and the second run finetuned with a batch size of `64`.
  
  ## Uses
  
  uh dont use it it's super undertrained I just wanted to see if I could do it, I am eventually going to create a suite of ultra-small encoding and decoding transformers, hopefully taking into account the best and latest research for efficiency.