Update README.md

README.md

@@ -116,7 +116,15 @@ For hyperparameters, we explored the following ranges:
 - Learning rate: `{5e-6, 1e-5, 2e-5, 3e-5, 5e-5, 1e-4}`
 - Number of epochs:  
   - Tasks with a large number of instances: `{1, 2}`  
-  - Tasks with fewer instances: `{3, 5, 10}`  
+  - Tasks with fewer instances: `{3, 5, 10}`
+
+In the experiments, we loaded several Japanese models that are publicly available on HuggingFace using `AutoModel` and constructed classification models by appending a classification head consisting of a linear layer, a GELU activation function, and another linear layer.
+This was done because HuggingFace's `AutoModelForSequenceClassification` comes with different implementations for each model, and using them directly would result in classification heads that differ from one model to another.
+
+For the embeddings fed into the classification layer, we used the embedding of the special token at the beginning of the sentence.
+That is, `[CLS]` in BERT and `<s>` in RoBERTa. 
+Note that our model does not perform the next sentence prediction (NSP) task during pretraining, so `<s>` is added at the beginning of the sentence, not `<cls>`.
+Therefore, we used the `<s>` token for classification.
 
 We conducted evaluations using 5-fold cross-validation.
 That is, we trained the model on the `train` set and evaluated it on the `validation` set.