Llm, Transformers, Fine Tuning, Gpu, Scaling Law.
A of the paper. n_trainable_params = num_parameters(model, requires_grad=True) trainable_flops = flops_per_param( model.max_seq_length, model.config.n_layer, model.config.n_embd, n_trainable_params ) # forward + backward + gradients (assumes no gradient accumulation) ops_per_step = 3 if training