1
00:00:00,000 --> 00:00:02,333
（嘶嘶声）
(whooshing)

2
00:00:05,250 --> 00:00:07,230
- 让我们看一下如何针对掩码语言建模
- Let's see how we can preprocess our data

3
00:00:07,230 --> 00:00:08,703
预处理我们的数据。
for masked language modeling.

4
00:00:10,230 --> 00:00:12,570
提醒一下，屏蔽语言建模
As a reminder, masked language modeling

5
00:00:12,570 --> 00:00:15,333
主要在模型需要填补句子中的空白时使用。
is when a model needs to fill the blanks in a sentence.

6
00:00:16,530 --> 00:00:19,650
为此，你只需要文本，不需要标签，
To do this, you just need texts, no labels,

7
00:00:19,650 --> 00:00:22,200
因为这是一个自监督的问题。
as this is a self-supervised problem.

8
00:00:22,200 --> 00:00:23,670
要将其应用于您自己的数据，
To apply this on your own data,

9
00:00:23,670 --> 00:00:25,740
只要确保您在数据集的一列中
just make sure you have all your texts gathered

10
00:00:25,740 --> 00:00:27,603
收集了所有的文本。
in one column of your dataset.

11
00:00:28,440 --> 00:00:30,480
在开始随机掩码处理之前，
Before we start randomly masking things,

12
00:00:30,480 --> 00:00:33,090
我们需要以某种方式使所有这些文本的长度相同
we will need to somehow make all those texts the same length

13
00:00:33,090 --> 00:00:34,263
从而将它们一起批处理。
to batch them together.

14
00:00:35,640 --> 00:00:38,490
使所有文本长度相同的第一种方法
The first way to make all the texts the same length

15
00:00:38,490 --> 00:00:40,590
和我们在文本分类中所使用的相同。
is the one we used in text classification.

16
00:00:41,430 --> 00:00:44,163
让我们填充短文本并截断长文本。
Let's pad the short texts and truncate the long ones.

17
00:00:45,030 --> 00:00:45,900
正如我们所看到的
As we have seen

18
00:00:45,900 --> 00:00:48,690
当我们处理文本分类数据时，
when we processed data for text classification,

19
00:00:48,690 --> 00:00:49,923
这都是由我们的分词器完成的
this is all done by our tokenizer

20
00:00:49,923 --> 00:00:53,130
并且配置相应的填充和截断选项。
with the right options for padding and truncation.

21
00:00:53,130 --> 00:00:56,100
如果与我们选择的上下文长度相比，
This will however make us lose a lot of texts

22
00:00:56,100 --> 00:00:58,620
我们数据集的示例很长，
if the examples in our dataset are very long,

23
00:00:58,620 --> 00:01:00,960
就会使我们丢失很多文本。
compared to the context length we picked.

24
00:01:00,960 --> 00:01:03,393
在这里，所有标记灰色部分都丢失了。
Here, all the portion in gray is lost.

25
00:01:04,410 --> 00:01:06,660
这就是为什么第二种生成文本样本的方法
This is why a second way to generate samples of text

26
00:01:06,660 --> 00:01:08,820
具有相同的长度是为了在上下文长度中
with the same length is to chunk our text

27
00:01:08,820 --> 00:01:10,560
为我们的文本分块
in pieces of context lengths,

28
00:01:10,560 --> 00:01:14,010
而不是在第一个数据块之后丢弃所有内容。
instead of discarding everything after the first chunk.

29
00:01:14,010 --> 00:01:15,420
大概会有余数
There will probably be a remainder

30
00:01:15,420 --> 00:01:17,700
长度小于上下文大小，
of length smaller than the context size,

31
00:01:17,700 --> 00:01:20,493
我们可以选择保留并填充或者忽略。
which we can choose to keep and pad or ignore.

32
00:01:21,570 --> 00:01:23,790
这是我们如何在实践中应用它，
Here is how we can apply this in practice,

33
00:01:23,790 --> 00:01:26,460
只需在我们调用分词器时添加 return overflowing tokens
by just adding the return overflowing tokens option

34
00:01:26,460 --> 00:01:28,200
选项
in our tokenizer call.

35
00:01:28,200 --> 00:01:30,243
请注意这样会为我们提供更大的数据集！
Note how this gives us a bigger dataset!

36
00:01:31,560 --> 00:01:34,260
如果你所有的文本很长，
This second way of chunking is ideal if all your texts

37
00:01:34,260 --> 00:01:36,270
这里第二种分块方式是理想的，
are very long, but it won't work

38
00:01:36,270 --> 00:01:39,900
但如果你的课文有不同的长度，那么效果就不尽人意。
as nicely if you have a variety of lengths in the texts.

39
00:01:39,900 --> 00:01:41,040
在这种情况下，
In this case,

40
00:01:41,040 --> 00:01:44,280
最好的选择是将所有标记的文本组合成为一个大的数据流
the best option is to concatenate all your tokenized texts

41
00:01:44,280 --> 00:01:46,560
附加一个特殊的标记
in one big stream, with a special tokens

42
00:01:46,560 --> 00:01:49,800
表明你何时从一份文件转到另一份文件，
to indicate when you pass from one document to the other,

43
00:01:49,800 --> 00:01:52,503
然后才将该数据流分成数据块。
and only then split the big stream into chunks.

44
00:01:53,760 --> 00:01:55,620
这是用代码完成的方法，
Here is how it can be done with code,

45
00:01:55,620 --> 00:01:58,230
用一个循环连接所有文本
with one loop to concatenate all the texts

46
00:01:58,230 --> 00:01:59,673
另一个将它分块。
and another one to chunk it.

47
00:02:00,780 --> 00:02:02,850
注意在我们这里的数据集中，
Notice how it reduces the number of samples

48
00:02:02,850 --> 00:02:04,230
它是如何减少样本数量的
in our dataset here,

49
00:02:04,230 --> 00:02:06,580
一定有大量短条目！
there must have been quite a few short entries!

50
00:02:07,710 --> 00:02:11,130
完成此操作后，掩码处理就很容易了。
Once this is done, the masking is the easy part.

51
00:02:11,130 --> 00:02:13,400
在 Transformers 库中有专门为此设计的
There is a data collator designed specifically for this

52
00:02:13,400 --> 00:02:15,540
数据整理器。
in the Transformers library.

53
00:02:15,540 --> 00:02:17,700
你可以直接在 Trainer 中使用它，
You can use it directly in the Trainer,

54
00:02:17,700 --> 00:02:20,400
或者将你的数据集转换为 tensorflow 数据集时
or when converting your datasets to tensorflow datasets

55
00:02:20,400 --> 00:02:23,703
在执行 Keras.fit 之前，使用 to_tf_dataset 方法。
before doing Keras.fit, with the to_tf_dataset method.

56
00:02:24,992 --> 00:02:27,325
（嘶嘶声）
(whooshing)

