readme: ---
  license: bigscience-bloom-rail-1.0
  datasets:
  - c4
  language:
  - en
  library_name: transformers
  tags:
  - causal-lm
  - gpt-j
  ---
  6.7m (6,700,128) param GPT-J model.
  ```
  n_positions - 128
  n_embd - 64
  n_layer - 4
  n_head - 8
  rotary_dim - 64
  tokenizer - gpt-j
  ```
  
  First, trained on 4,194,304 samples from the [c4](https://hf.co/datasets/c4) dataset, at a length of 128 tokens each, that comes out to 536,870,912 (0.53B) tokens seen during training. A batch size of 16 with 128 gradient accumulation steps was used, making the effective batch size 2048. A cosine learning rate schedule was used starting at 1e-3.