1
00:00:00,095 --> 00:00:01,582
（屏幕呼啸）
(screen whooshing)

2
00:00:01,582 --> 00:00:02,659
（贴纸弹出）
(sticker popping)

3
00:00:02,659 --> 00:00:05,379
（屏幕呼啸）
(screen whooshing)

4
00:00:05,379 --> 00:00:06,720
- 在这段视频中，我们来了解一下
- In this video, we take a look

5
00:00:06,720 --> 00:00:09,483
在称为困惑度的评估指标上。
at the mysterious sounding metric called perplexity.

6
00:00:11,070 --> 00:00:12,630
你在研究生成模型时
You might have encountered perplexity

7
00:00:12,630 --> 00:00:14,970
可能遇到过困惑度。
when reading about generative models.

8
00:00:14,970 --> 00:00:16,680
你可以在这里看到两个例子，
You can see two examples here,

9
00:00:16,680 --> 00:00:18,577
一个来自最初的 transformer 论文，
one from the original transformer paper,

10
00:00:18,577 --> 00:00:19,950
“你需要的只是注意力，”
"Attention is all you need,"

11
00:00:19,950 --> 00:00:23,340
另一个来自最近的 GPT-2 论文。
and the other one from the more recent GPT-2 paper.

12
00:00:23,340 --> 00:00:25,740
困惑度是衡量语言模型的性能
Perplexity is a common metric to measure the performance

13
00:00:25,740 --> 00:00:27,150
的常用指标。
of language models.

14
00:00:27,150 --> 00:00:30,000
它的值越小，性能越好。
The smaller its value, the better the performance.

15
00:00:30,000 --> 00:00:32,950
但它究竟意味着什么，我们又该如何计算得到它呢？
But what does it actually mean and how can we calculate it?

16
00:00:34,440 --> 00:00:36,180
机器学习中非常常见的量
A very common quantity in machine learning

17
00:00:36,180 --> 00:00:37,650
是相似度。
is the likelihood.

18
00:00:37,650 --> 00:00:39,240
我们可以计算似然性
We can calculate the likelihood

19
00:00:39,240 --> 00:00:42,390
作为每个词元的概率的乘积。
as the product of each token's probability.

20
00:00:42,390 --> 00:00:44,730
这意味着对于每个词元，
What this means is that for each token,

21
00:00:44,730 --> 00:00:47,340
我们使用语言模型基于之前的词元
we use the language model to predict its probability

22
00:00:47,340 --> 00:00:49,560
来预测它的概率。
based on the previous tokens.

23
00:00:49,560 --> 00:00:52,050
最后，我们将所有概率相乘
In the end, we multiply all probabilities

24
00:00:52,050 --> 00:00:53,253
从而得到似然性。
to get the likelihood.

25
00:00:55,892 --> 00:00:57,000
通过似然性，
With the likelihood,

26
00:00:57,000 --> 00:00:59,340
我们可以计算另一个重要的量，
we can calculate another important quantity,

27
00:00:59,340 --> 00:01:01,200
交叉熵。
the cross-entropy.

28
00:01:01,200 --> 00:01:03,450
当你接触损失函数时
You might have already heard about cross-entropy

29
00:01:03,450 --> 00:01:05,670
可能已经听说过交叉熵。
when looking at loss function.

30
00:01:05,670 --> 00:01:09,210
它通常在分类中作为损失函数使用。
It is often used as a loss function in classification.

31
00:01:09,210 --> 00:01:11,610
在语言建模中，我们基于之前的词元
In language modeling, we predict the next token

32
00:01:11,610 --> 00:01:12,930
预测下一个词元，
based on the previous token,

33
00:01:12,930 --> 00:01:15,810
这也是一个分类任务。
which is also a classification task.

34
00:01:15,810 --> 00:01:17,340
因此，如果我们想计算
Therefore, if we want to calculate

35
00:01:17,340 --> 00:01:19,290
一个例子的交叉熵，
the cross-entropy of an example,

36
00:01:19,290 --> 00:01:21,090
我们可以简单地将它传递给模型
we can simply pass it to the model

37
00:01:21,090 --> 00:01:23,580
以其输入作为标签。
with its inputs as labels.

38
00:01:23,580 --> 00:01:26,433
其损失与交叉熵相关。
The loss then corresponds to the cross-entropy.

39
00:01:29,130 --> 00:01:31,110
现在对于计算困惑度
We are now only a single operation away

40
00:01:31,110 --> 00:01:33,510
我们现在只差一个操作了。
from calculating the perplexity.

41
00:01:33,510 --> 00:01:37,710
通过对交叉熵取幂，我们得到了困惑度。
By exponentiating the cross-entropy, we get the perplexity.

42
00:01:37,710 --> 00:01:40,260
所以你可以发现困惑度和损失
So you see that the perplexity is closely related

43
00:01:40,260 --> 00:01:41,163
是密切相关的。
to the loss.

44
00:01:42,060 --> 00:01:43,380
插入以前的结果
Plugging in previous results

45
00:01:43,380 --> 00:01:47,010
表明这相当于对每个词元
shows that this is equivalent to exponentiating

46
00:01:47,010 --> 00:01:51,033
的负平均锁定概率取幂值。
the negative average lock probability of each token.

47
00:01:52,050 --> 00:01:54,630
请记住，损失只是针对模型
Keep in mind that the loss is only a weak proxy

48
00:01:54,630 --> 00:01:57,360
生成高质量文本的能力的一个弱代理
for a model's ability to generate quality text

49
00:01:57,360 --> 00:02:00,510
困惑度也是如此。
and the same is true for perplexity.

50
00:02:00,510 --> 00:02:02,550
出于这个原因，人们通常也计算
For this reason, one usually also calculates

51
00:02:02,550 --> 00:02:03,840
更复杂的指标
more sophisticated metrics

52
00:02:03,840 --> 00:02:07,413
例如生成式任务上的 BLEU 或 ROUGE。
such as BLEU or ROUGE on generative tasks.

53
00:02:08,551 --> 00:02:11,468
（屏幕呼啸）
(screen whooshing)

