6

Model

InstructionBLIP [77]
mPLUG-OwlLLaMA2 [12]
Otter [78]
CogVLM-17B [29]
CogVLM-17B+AIT (Ours)
CogVLM-17B+CLoT (Ours)

Size

13B
7B
7B
7B
7B
7B

2T1
19.8
22.3
15.8
37.6
57.4
66.9

3T1
13.7
12.7
9.9
26.4
37.4
47.6

Image to Text (I2T)
4T1
15.5
15.0
8.5
18.3
33.5
43.4

5T2
1.1
4.2
7.1
2.5
21.8
30.7

Rank
65.5
59.9
61.3
64.6
64.6
69.4

Avg.
23.1
22.8
20.5
29.9
42.9+13.1
51.6+21.7

2T1
22.3
24.2
3.8
35.1
55.4
64.8

3T1
16.0
13.7
3.3
27.8
46.5
52.9

Text to Text (T2T)
4T1
17.0
12.6
4.8
24.8
26.4
33.6

5T2
0.7
3.1
5.4
7.5
18.2
21.8

Rank
59.5
59.2
58.5
64.1
64.4
68.6

Avg.
23.1
22.6
15.1
31.9
42.2+10.3
48.3+16.4

Table 3. The accuracy (%) of choice questions and the NDCG (%) of ranking questions on various mutilmodal non-multilingual models
(English). See notations in Table 2. We only consider I2T and T2T since English IT2T is not available due to cultural preference.

Model
GPT-3.5 [71]
GPT-4 [71]

LLAMA2 [4]

Baichuan2 [79]

Qwen [5]

ChatGLM3 [74]

Vicuna-v1.5 [6]
Let’sThinkOutsidetheBox