<h1 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h1>
<h2 dir="auto">Information</h2>
<p dir="auto">One more question did run_bertology also support albert model?<br>
Model I am using (Bert, XLNet ...): Bert</p>
<p dir="auto">Language I am using the model on (English, Chinese ...):English</p>
<p dir="auto">The problem arises when using:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> the official example scripts: (give details below)<br>
run_bertology.py</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own modified scripts: (give details below)<br>
python3.7/site-packages/transformers/data/processors/glue.py</li>
</ul>
<h1 dir="auto">Because the dataset SciEntsBank-3way labeled as "correct", "incorrect", "contradictory".</h1>
<p dir="auto">-223 return ["contradiction", "entailment", "neutral"]<br>
+223 return ["correct", "incorrect", "contradictory"]</p>
<h1 dir="auto">Because the dataset SciEntsBank-3way structure. label at first position, text_a at second position and text_b at third position.</h1>
<p dir="auto">-232 text_a = line[8]<br>
-233 text_b = line[9]<br>
-234 label = line[-1]<br>
+232 text_a = line[1]<br>
+233 text_b = line[2]<br>
+234 label = line[0]</p>
<p dir="auto">The tasks I am working on is:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> an official GLUE/SQUaD task: (give the name)<br>
mnli</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> my own task or dataset: (give details below)<br>
dataset: SciEntsBank-3way(<a href="https://www.cs.york.ac.uk/semeval-2013/task7.html" rel="nofollow">https://www.cs.york.ac.uk/semeval-2013/task7.html</a>)</li>
</ul>
<h2 dir="auto">To reproduce</h2>
<p dir="auto">Steps to reproduce the behavior:</p>
<p dir="auto">python ./run_bertology.py     --data_dir SciEntsBank-3way<br>
--model_name bert-base-uncased<br>
--task_name mnli<br>
--max_seq_length 128<br>
--output_dir ./tmp/$TASK_NAME/<br>
--try_masking</p>

<p dir="auto">Iteration: 100%|██████████| 4561/4561 [02:15&lt;00:00, 33.57it/s]<br>
INFO:<strong>main</strong>:Attention entropies<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	3.03255	2.82196	1.77876	1.64802	3.27255	2.91101	3.34266	3.03600	2.73255	3.09043	1.35738	2.52412<br>
INFO:<strong>main</strong>:layer 2:	2.73629	1.11241	2.86221	2.44852	0.95509	2.39331	0.45580	2.82749	2.93869	2.88269	2.19532	2.48865<br>
INFO:<strong>main</strong>:layer 3:	0.05847	1.66529	1.91624	2.79214	2.31408	2.67645	2.18180	2.62745	2.48442	0.05168	2.52636	2.49648<br>
INFO:<strong>main</strong>:layer 4:	1.54150	2.90387	2.40694	2.06858	2.77907	0.80181	2.69664	2.88957	2.70095	1.19583	2.33666	1.83265<br>
INFO:<strong>main</strong>:layer 5:	2.34246	2.64519	2.03515	1.37404	2.88754	1.67422	2.14421	1.41457	2.03571	2.69347	1.98139	1.44582<br>
INFO:<strong>main</strong>:layer 6:	1.71052	1.10676	2.28401	1.87228	2.55920	1.75916	1.22450	1.35704	1.92916	1.02535	1.67920	1.60766<br>
INFO:<strong>main</strong>:layer 7:	1.63887	1.93625	1.83002	1.20811	1.58296	1.65662	1.55572	2.38742	2.09030	1.69326	1.42275	1.08153<br>
INFO:<strong>main</strong>:layer 8:	1.95536	1.73146	1.59791	1.17307	1.12128	1.95980	1.11606	1.11680	1.97816	1.64787	1.53183	1.28007<br>
INFO:<strong>main</strong>:layer 9:	1.54698	1.96436	1.45466	2.03807	1.60202	1.44075	1.36014	2.32559	2.59592	2.09076	1.75704	1.85274<br>
INFO:<strong>main</strong>:layer 10:	2.00444	1.91784	2.12478	1.99289	1.58305	2.48627	2.08822	1.69971	2.70500	1.71860	2.03850	2.38604<br>
INFO:<strong>main</strong>:layer 11:	2.76158	1.53031	1.99278	2.26007	1.97855	1.66471	1.90139	2.13217	2.45516	1.83803	1.99372	2.15438<br>
INFO:<strong>main</strong>:layer 12:	1.73656	2.10304	2.72498	1.85723	2.04607	2.20456	2.16210	1.82173	2.18728	2.71702	1.84256	1.83663<br>
INFO:<strong>main</strong>:Head importance scores<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	0.30328	0.05899	0.06971	0.03727	0.13938	1.00000	0.04436	0.03679	0.22807	0.07911	0.19918	0.05241<br>
INFO:<strong>main</strong>:layer 2:	0.04867	0.02256	0.21194	0.04069	0.23058	0.15942	0.65188	0.38251	0.47535	0.40172	0.10869	0.34316<br>
INFO:<strong>main</strong>:layer 3:	0.06349	0.08003	0.56604	0.41141	0.38410	0.16264	0.29070	0.37301	0.28161	0.18325	0.45048	0.02401<br>
INFO:<strong>main</strong>:layer 4:	0.74869	0.15986	0.29754	0.02072	0.20961	0.06570	0.35717	0.44580	0.01144	0.11113	0.26962	0.28707<br>
INFO:<strong>main</strong>:layer 5:	0.05413	0.58029	0.29859	0.64154	0.25539	0.11611	0.36774	0.05591	0.19390	0.34493	0.04906	0.02742<br>
INFO:<strong>main</strong>:layer 6:	0.24067	0.06599	0.45376	0.22384	0.40461	0.53808	0.06806	0.21937	0.04209	0.13334	0.19226	0.57838<br>
INFO:<strong>main</strong>:layer 7:	0.33972	0.12576	0.31489	0.10031	0.29630	0.19341	0.28052	0.29937	0.78337	0.09395	0.23640	0.05812<br>
INFO:<strong>main</strong>:layer 8:	0.23342	0.27415	0.27682	0.22111	0.23234	0.79778	0.03235	0.09092	0.40418	0.01651	0.21795	0.22528<br>
INFO:<strong>main</strong>:layer 9:	0.01306	0.88878	0.08858	0.45180	0.04019	0.08035	0.13417	0.15899	0.39753	0.01761	0.10785	0.01428<br>
INFO:<strong>main</strong>:layer 10:	0.01597	0.01365	0.08691	0.04718	0.01268	0.32052	0.00453	0.05614	0.81534	0.00000	0.02659	0.66734<br>
INFO:<strong>main</strong>:layer 11:	0.86446	0.00818	0.05306	0.12751	0.13587	0.00293	0.06480	0.22173	0.21643	0.04838	0.48050	0.32190<br>
INFO:<strong>main</strong>:layer 12:	0.08048	0.32489	0.56753	0.28201	0.37204	0.09334	0.26549	0.07130	0.00372	0.53481	0.24909	0.36108<br>
INFO:<strong>main</strong>:Head ranked by importance scores<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	41	108	102	123	81	0	119	124	62	100	72	114<br>
INFO:<strong>main</strong>:layer 2:	116	129	70	121	61	79	8	28	17	25	89	35<br>
INFO:<strong>main</strong>:layer 3:	107	99	13	22	27	77	46	29	49	76	20	128<br>
INFO:<strong>main</strong>:layer 4:	6	78	44	130	71	105	33	21	138	88	53	47<br>
INFO:<strong>main</strong>:layer 5:	112	10	43	9	55	87	31	111	73	34	115	126<br>
INFO:<strong>main</strong>:layer 6:	57	104	18	64	23	14	103	67	120	84	75	11<br>
INFO:<strong>main</strong>:layer 7:	36	86	40	91	45	74	50	42	5	92	58	109<br>
INFO:<strong>main</strong>:layer 8:	59	52	51	66	60	4	125	94	24	132	68	63<br>
INFO:<strong>main</strong>:layer 9:	136	1	95	19	122	98	83	80	26	131	90	134<br>
INFO:<strong>main</strong>:layer 10:	133	135	96	118	137	39	140	110	3	143	127	7<br>
INFO:<strong>main</strong>:layer 11:	2	139	113	85	82	142	106	65	69	117	16	38<br>
INFO:<strong>main</strong>:layer 12:	97	37	12	48	30	93	54	101	141	15	56	32<br>
Iteration: 100%|██████████| 4561/4561 [01:54&lt;00:00, 39.80it/s]<br>
INFO:<strong>main</strong>:Attention entropies<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 2:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 3:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 4:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 5:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 6:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 7:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 8:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 9:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 10:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 11:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 12:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:Head importance scores<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	0.30328	0.05899	0.06971	0.03727	0.13938	1.00000	0.04436	0.03679	0.22807	0.07911	0.19918	0.05241<br>
INFO:<strong>main</strong>:layer 2:	0.04867	0.02256	0.21194	0.04069	0.23058	0.15942	0.65188	0.38251	0.47535	0.40172	0.10869	0.34316<br>
INFO:<strong>main</strong>:layer 3:	0.06349	0.08003	0.56604	0.41141	0.38410	0.16264	0.29070	0.37301	0.28161	0.18325	0.45048	0.02401<br>
INFO:<strong>main</strong>:layer 4:	0.74869	0.15986	0.29754	0.02072	0.20961	0.06570	0.35717	0.44580	0.01144	0.11113	0.26962	0.28707<br>
INFO:<strong>main</strong>:layer 5:	0.05413	0.58029	0.29859	0.64154	0.25539	0.11611	0.36774	0.05591	0.19390	0.34493	0.04906	0.02742<br>
INFO:<strong>main</strong>:layer 6:	0.24067	0.06599	0.45376	0.22384	0.40461	0.53808	0.06806	0.21937	0.04209	0.13334	0.19226	0.57838<br>
INFO:<strong>main</strong>:layer 7:	0.33972	0.12576	0.31489	0.10031	0.29630	0.19341	0.28052	0.29937	0.78337	0.09395	0.23640	0.05812<br>
INFO:<strong>main</strong>:layer 8:	0.23342	0.27415	0.27682	0.22111	0.23234	0.79778	0.03235	0.09092	0.40418	0.01651	0.21795	0.22528<br>
INFO:<strong>main</strong>:layer 9:	0.01306	0.88878	0.08858	0.45180	0.04019	0.08035	0.13417	0.15899	0.39753	0.01761	0.10785	0.01428<br>
INFO:<strong>main</strong>:layer 10:	0.01597	0.01365	0.08691	0.04718	0.01268	0.32052	0.00453	0.05614	0.81534	0.00000	0.02659	0.66734<br>
INFO:<strong>main</strong>:layer 11:	0.86446	0.00818	0.05306	0.12751	0.13587	0.00293	0.06480	0.22173	0.21643	0.04838	0.48050	0.32190<br>
INFO:<strong>main</strong>:layer 12:	0.08048	0.32489	0.56753	0.28201	0.37204	0.09334	0.26549	0.07130	0.00372	0.53481	0.24909	0.36108<br>
INFO:<strong>main</strong>:Head ranked by importance scores<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	41	108	102	123	81	0	119	124	62	100	72	114<br>
INFO:<strong>main</strong>:layer 2:	116	129	70	121	61	79	8	28	17	25	89	35<br>
INFO:<strong>main</strong>:layer 3:	107	99	13	22	27	77	46	29	49	76	20	128<br>
INFO:<strong>main</strong>:layer 4:	6	78	44	130	71	105	33	21	138	88	53	47<br>
INFO:<strong>main</strong>:layer 5:	112	10	43	9	55	87	31	111	73	34	115	126<br>
INFO:<strong>main</strong>:layer 6:	57	104	18	64	23	14	103	67	120	84	75	11<br>
INFO:<strong>main</strong>:layer 7:	36	86	40	91	45	74	50	42	5	92	58	109<br>
INFO:<strong>main</strong>:layer 8:	59	52	51	66	60	4	125	94	24	132	68	63<br>
INFO:<strong>main</strong>:layer 9:	136	1	95	19	122	98	83	80	26	131	90	134<br>
INFO:<strong>main</strong>:layer 10:	133	135	96	118	137	39	140	110	3	143	127	7<br>
INFO:<strong>main</strong>:layer 11:	2	139	113	85	82	142	106	65	69	117	16	38<br>
INFO:<strong>main</strong>:layer 12:	97	37	12	48	30	93	54	101	141	15	56	32<br>
INFO:<strong>main</strong>:Pruning: original score: 0.091866, threshold: 0.082679<br>
INFO:<strong>main</strong>:Heads to mask: [117, 125, 140, 114, 121, 44, 112, 96, 109, 107, 108, 93, 105, 39]<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 2:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 3:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 4:	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 5:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 6:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 7:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 8:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 9:	0.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000	1.00000	0.00000<br>
INFO:<strong>main</strong>:layer 10:	0.00000	0.00000	1.00000	1.00000	0.00000	1.00000	0.00000	1.00000	1.00000	0.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 11:	1.00000	0.00000	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 12:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000	1.00000<br>
Iteration: 100%|██████████| 4561/4561 [01:54&lt;00:00, 39.68it/s]<br>
INFO:<strong>main</strong>:Attention entropies<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 2:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 3:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 4:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 5:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 6:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 7:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 8:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 9:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 10:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 11:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 12:	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:Head importance scores<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	0.39574	0.02468	0.08140	0.12466	0.12766	1.00000	0.09364	0.03733	0.21125	0.05515	0.27669	0.01294<br>
INFO:<strong>main</strong>:layer 2:	0.04871	0.01492	0.02147	0.00766	0.25008	0.16705	0.73248	0.48019	0.40388	0.39327	0.06609	0.38033<br>
INFO:<strong>main</strong>:layer 3:	0.14803	0.02220	0.64758	0.29125	0.45867	0.02242	0.34411	0.33109	0.30959	0.29897	0.41782	0.01806<br>
INFO:<strong>main</strong>:layer 4:	0.82395	0.20768	0.33463	0.05595	0.30457	0.01353	0.38665	0.33563	0.02096	0.12274	0.15990	0.30594<br>
INFO:<strong>main</strong>:layer 5:	0.24467	0.77639	0.26634	0.43066	0.28580	0.15136	0.29888	0.09479	0.20161	0.38652	0.07106	0.09292<br>
INFO:<strong>main</strong>:layer 6:	0.24192	0.04294	0.19242	0.18251	0.64465	0.64657	0.03439	0.17273	0.05866	0.20935	0.14715	0.51240<br>
INFO:<strong>main</strong>:layer 7:	0.24221	0.11722	0.54783	0.09908	0.30887	0.33625	0.18271	0.09798	0.76243	0.19917	0.26639	0.02415<br>
INFO:<strong>main</strong>:layer 8:	0.34639	0.10483	0.42852	0.23310	0.20756	0.85146	0.05960	0.06187	0.25805	0.12922	0.14193	0.28091<br>
INFO:<strong>main</strong>:layer 9:	0.02696	0.97099	0.08023	0.36748	0.05116	0.06451	0.07015	0.23535	0.39404	0.14999	0.01570	0.01164<br>
INFO:<strong>main</strong>:layer 10:	0.02307	0.01918	0.09727	0.05241	0.03105	0.32034	0.02875	0.08710	0.92011	0.00000	0.05011	0.59763<br>
INFO:<strong>main</strong>:layer 11:	0.81794	0.00753	0.05141	0.14622	0.09715	0.00008	0.03071	0.09413	0.17420	0.05189	0.69652	0.31542<br>
INFO:<strong>main</strong>:layer 12:	0.09312	0.36467	0.56134	0.25867	0.37935	0.06446	0.34758	0.09796	0.03789	0.56186	0.25654	0.37430<br>
INFO:<strong>main</strong>:Head ranked by importance scores<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	24	126	100	85	84	0	96	120	64	111	52	138<br>
INFO:<strong>main</strong>:layer 2:	117	136	131	140	58	75	8	18	23	26	104	29<br>
INFO:<strong>main</strong>:layer 3:	79	130	10	49	19	129	36	40	43	47	22	134<br>
INFO:<strong>main</strong>:layer 4:	4	66	39	110	46	137	27	38	132	86	76	45<br>
INFO:<strong>main</strong>:layer 5:	59	6	54	20	50	77	48	94	68	28	102	98<br>
INFO:<strong>main</strong>:layer 6:	61	118	70	72	12	11	121	74	109	65	80	17<br>
INFO:<strong>main</strong>:layer 7:	60	87	16	89	44	37	71	90	7	69	53	127<br>
INFO:<strong>main</strong>:layer 8:	35	88	21	63	67	3	108	107	56	83	82	51<br>
INFO:<strong>main</strong>:layer 9:	125	1	101	32	115	105	103	62	25	78	135	139<br>
INFO:<strong>main</strong>:layer 10:	128	133	92	112	122	41	124	99	2	143	116	13<br>
INFO:<strong>main</strong>:layer 11:	5	141	114	81	93	142	123	95	73	113	9	42<br>
INFO:<strong>main</strong>:layer 12:	97	33	15	55	30	106	34	91	119	14	57	31<br>
INFO:<strong>main</strong>:Masking: current score: 0.092085, remaning heads 130 (90.3 percents)<br>
INFO:<strong>main</strong>:Heads to mask: [15, 11, 41, 13, 106, 35, 14, 25, 29, 83, 1, 126, 66, 7]<br>
INFO:<strong>main</strong>:lv, h &gt;	1	2	3	4	5	6	7	8	9	10	11	12<br>
INFO:<strong>main</strong>:layer 1:	1.00000	0.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000	1.00000	0.00000<br>
INFO:<strong>main</strong>:layer 2:	1.00000	0.00000	0.00000	0.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 3:	1.00000	0.00000	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000<br>
INFO:<strong>main</strong>:layer 4:	1.00000	1.00000	1.00000	0.00000	1.00000	0.00000	1.00000	1.00000	0.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 5:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 6:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 7:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000<br>
INFO:<strong>main</strong>:layer 8:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 9:	0.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000	0.00000	0.00000<br>
INFO:<strong>main</strong>:layer 10:	0.00000	0.00000	1.00000	1.00000	0.00000	1.00000	0.00000	1.00000	1.00000	0.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 11:	1.00000	0.00000	1.00000	1.00000	1.00000	0.00000	0.00000	1.00000	1.00000	1.00000	1.00000	1.00000<br>
INFO:<strong>main</strong>:layer 12:	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	1.00000	0.00000	1.00000	1.00000	1.00000<br>
Iteration:   0%|          | 0/4561 [00:00&lt;?, ?it/s]<br>
Traceback (most recent call last):<br>
File "run_bertology.py", line 426, in <br>
main()<br>
File "run_bertology.py", line 421, in main<br>
head_mask = mask_heads(args, model, eval_dataloader)<br>
File "run_bertology.py", line 179, in mask_heads<br>
args, model, eval_dataloader, compute_entropy=False, head_mask=new_head_mask<br>
File "run_bertology.py", line 104, in compute_heads_importance<br>
head_importance += head_mask.grad.abs().detach()<br>
AttributeError: 'NoneType' object has no attribute 'abs'</p>
<h2 dir="auto">Environment info</h2>

<ul dir="auto">
<li><code class="notranslate">transformers</code> version:2.8.0</li>
<li>Platform:</li>
<li>Python version:3.7</li>
<li>PyTorch version (GPU?):</li>
<li>Tensorflow version (GPU?):</li>
<li>Using GPU in script?:yes</li>
<li>Using distributed or parallel set-up in script?:</li>
</ul>
<h3 dir="auto">System Info</h3>
<p dir="auto">Transformers: 4.25.1<br>
Python: 3.10.8<br>
OS: Manajro-KDE</p>
<h3 dir="auto">Who can help?</h3>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/sgugger/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/sgugger">@sgugger</a></p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<ol dir="auto">
<li>Create the following compute metrics method:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    accuracy_score = accuracy.compute(predictions=predictions, references=labels)
    auc_score = auc.compute(prediction_scores=logits[:, 1], references=labels)
    f1_score = f1.compute(predictions=predictions, references=labels)
    aupr = average_precision_score(y_score=logits[:, 1], y_true=labels)
    return {**f1_score , **{'PR-AUC': aupr}, **accuracy_score, **auc_score}"><pre lang="def" class="notranslate"><code class="notranslate">    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    accuracy_score = accuracy.compute(predictions=predictions, references=labels)
    auc_score = auc.compute(prediction_scores=logits[:, 1], references=labels)
    f1_score = f1.compute(predictions=predictions, references=labels)
    aupr = average_precision_score(y_score=logits[:, 1], y_true=labels)
    return {**f1_score , **{'PR-AUC': aupr}, **accuracy_score, **auc_score}
</code></pre></div>
<ol start="2" dir="auto">
<li>Create a trainer object with multiple eval datasets:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="training_args = TrainingArguments(
    output_dir=&quot;./results&quot;,
    learning_rate=2e-5,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    weight_decay=0.01,
    evaluation_strategy=IntervalStrategy.STEPS,
    report_to='wandb',
    run_name='ChemBERTa Test',
    log_level='critical',
    logging_steps=1, 
)"><pre class="notranslate"><code class="notranslate">training_args = TrainingArguments(
    output_dir="./results",
    learning_rate=2e-5,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    weight_decay=0.01,
    evaluation_strategy=IntervalStrategy.STEPS,
    report_to='wandb',
    run_name='ChemBERTa Test',
    log_level='critical',
    logging_steps=1, 
)
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    model=model,
    args=training_args,
    train_dataset=dataset[&quot;train&quot;],
    eval_dataset={'Validation': dataset[&quot;validation&quot;], 'Test': dataset[&quot;test&quot;]},
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)"><pre lang="trainer" class="notranslate"><code class="notranslate">    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset={'Validation': dataset["validation"], 'Test': dataset["test"]},
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
</code></pre></div>
<ol start="4" dir="auto">
<li>Run the trainer<br>
<code class="notranslate">trainer.train()</code></li>
</ol>
<p dir="auto">Result:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/34383384/205439136-0364b8fe-3e71-488b-b1a4-acdcdd9bef3e.png"><img src="https://user-images.githubusercontent.com/34383384/205439136-0364b8fe-3e71-488b-b1a4-acdcdd9bef3e.png" alt="image" style="max-width: 100%;"></a><br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/34383384/205439150-34a5fab9-9775-4771-8a6e-1d33ebed36fd.png"><img src="https://user-images.githubusercontent.com/34383384/205439150-34a5fab9-9775-4771-8a6e-1d33ebed36fd.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">the problem arises in <code class="notranslate">transformers/utils/notebook.py/</code><br>
in line 244, it raises <code class="notranslate">KeyError: 'Validation F1'</code><br>
The reason for that is that only on the first iteration it populates the names of the columns for the table it prints in the notebook (lines 237-243).<br>
But the trainer runs each dataset in a standalone <code class="notranslate">self.evaluate</code> run<br>
So when the 2nd <code class="notranslate">self.evaluate</code> with the test dataset calls<br>
<code class="notranslate">self.callback_handler.on_evaluate(self.args, self.state, self.control, output.metrics)</code><br>
the table already has the names of the columns from the validation set, so lines 237-243 don't run again<br>
and then on line 244, since it's the test dataset, all the metrics are called "Test " so it raises an error since it iterates over the table columns and doesn't find the validation metrics.</p>
<p dir="auto">A workaround is removing the if statement in line 238 and having it update columns each time it's called<br>
and changing line 244 to: <code class="notranslate">self.inner_table.append([values[c] if c in values else 'NaN' for c in columns])</code></p>
<p dir="auto">i.e.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    def write_line(self, values):
        &quot;&quot;&quot;
        Write the values in the inner table.

        Args:
            values (`Dict[str, float]`): The values to display.
        &quot;&quot;&quot;
        if self.inner_table is None:
            self.inner_table = [list(values.keys()), list(values.values())]
        else:
            columns = self.inner_table[0]
            # if len(self.inner_table) == 1:
            #     # We give a chance to update the column names at the first iteration
            for key in values.keys():
                if key not in columns:
                    columns.append(key)
            self.inner_table[0] = columns
            print(f'{columns=}, {values=}')
            self.inner_table.append([values[c] if c in values else 'NaN' for c in columns])"><pre class="notranslate"><code class="notranslate">    def write_line(self, values):
        """
        Write the values in the inner table.

        Args:
            values (`Dict[str, float]`): The values to display.
        """
        if self.inner_table is None:
            self.inner_table = [list(values.keys()), list(values.values())]
        else:
            columns = self.inner_table[0]
            # if len(self.inner_table) == 1:
            #     # We give a chance to update the column names at the first iteration
            for key in values.keys():
                if key not in columns:
                    columns.append(key)
            self.inner_table[0] = columns
            print(f'{columns=}, {values=}')
            self.inner_table.append([values[c] if c in values else 'NaN' for c in columns])
</code></pre></div>
<p dir="auto">(the print is only for debugging)</p>
<p dir="auto">but then it creates a weird table where the rows are duplicated at the logging step<br>
and half of the columns in the first row are NaN<br>
and the other half is NaN in the 2nd row.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/34383384/205439041-587335cb-0004-4b8e-8e7f-2f1d76c549be.png"><img src="https://user-images.githubusercontent.com/34383384/205439041-587335cb-0004-4b8e-8e7f-2f1d76c549be.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">edit: Another bug is on <code class="notranslate">NotebookProgressCallback.on_evaluate</code> lines 342-345</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="for k, v in metrics.items():
                if k == f&quot;{metric_key_prefix}_loss&quot;:
                    values[&quot;Validation Loss&quot;] = v"><pre class="notranslate"><code class="notranslate">for k, v in metrics.items():
                if k == f"{metric_key_prefix}_loss":
                    values["Validation Loss"] = v
</code></pre></div>
<p dir="auto">This forces the output to consistently be named validation loss, even if I use the test dataset; this means that in the table created I don't get 2 different losses printed, cause the test loss overrides the validation loss</p>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">A single row containing both the validation and test metrics</p>