A collection of all the models trained for the Jaxgmg project. Architecture is a standard IMPALA conv. net
David Quarel
davidquarel
·
AI & ML interests
None yet
Recent Activity
updated
a Space
about 6 hours ago
davidquarel/jaxgmg-hf
published
a Space
1 day ago
davidquarel/jaxgmg-hf
updated
a model
11 days ago
davidquarel/jaxgmg_al_1e0
Organizations
Jacobian SAEs with all combo of jacobian_coeff - Layer0 in (2e-4, 4e-4, 6e-4, 8e-4)
Layer1 = 3e-3
Layer2 = 3e-3
Layer3 in (2e-2, 4e-2, 6e-2, 8e-2)
-
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_8.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_6.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_4.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_2.0e-02
216k • Updated
A collection of SAEs, trained on the input and output of each MLP layer of Peter's shakespeare_64x4 GPT model
jln.shk_64x4: JSAE (resid_mid -> resid_out) 7.5k steps
j.mlp_layer.shk_64x4: JSAE (mlp_in -> mlp_out) 7.5k steps
Pairs of Jacobian SAEs, 8 in total.
Each pair is placed across the MLP blocks,
one on resid_mid, one on resid_post. Uses DyT
A sweep of JSAE training with jacobian penalty from 1e-3 to 1e-2, trained for 20k steps. (added zero sparsity for baseline)
Depricated. Just a bunch of bad SAEs. Probably don't use. Trained between transformer blocks
A collection of all the models trained for the Jaxgmg project. Architecture is a standard IMPALA conv. net
jln.shk_64x4: JSAE (resid_mid -> resid_out) 7.5k steps
j.mlp_layer.shk_64x4: JSAE (mlp_in -> mlp_out) 7.5k steps
Pairs of Jacobian SAEs, 8 in total.
Each pair is placed across the MLP blocks,
one on resid_mid, one on resid_post. Uses DyT
Jacobian SAEs with all combo of jacobian_coeff - Layer0 in (2e-4, 4e-4, 6e-4, 8e-4)
Layer1 = 3e-3
Layer2 = 3e-3
Layer3 in (2e-2, 4e-2, 6e-2, 8e-2)
-
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_8.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_6.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_4.0e-02
216k • Updated -
davidquarel/jsae.shk_64x4-sparse-8.0e-04_3.0e-03_3.0e-03_2.0e-02
216k • Updated
A sweep of JSAE training with jacobian penalty from 1e-3 to 1e-2, trained for 20k steps. (added zero sparsity for baseline)
A collection of SAEs, trained on the input and output of each MLP layer of Peter's shakespeare_64x4 GPT model
Depricated. Just a bunch of bad SAEs. Probably don't use. Trained between transformer blocks