Machine Learning, Fine Tuning, Kids And Tech, Language Model, Kids Education.
requirements. mlp Multi-Layer Perceptron (MLP) After the self-attention mechanism, there is a multi-layer perceptron (PhiMLP) consisting of two linear layers (fc1 and fc2). These layers with 4-bit quantization capture non-linear patterns in the data. The activation function used is