1
00:00:00,449 --> 00:00:01,559
（空气呼啸）
(air whooshing)

2
00:00:01,559 --> 00:00:02,767
（徽标弹出）
(logo popping)

3
00:00:02,767 --> 00:00:05,670
（金属滑动）
(metal sliding)

4
00:00:05,670 --> 00:00:08,470
- 让我们看看如何预处理翻译数据集。
- Let's see how to preprocess a dataset for translation.

5
00:00:09,540 --> 00:00:12,420
这个任务需要将句子用另一种语言
This is a task of well translating a sentence

6
00:00:12,420 --> 00:00:14,310
进行恰当的翻译。
in another language.

7
00:00:14,310 --> 00:00:17,100
本视频将重点介绍当你把句子按照下面的格式组织时
This video will focus on how to preprocess your dataset

8
00:00:17,100 --> 00:00:19,950
如何预处理你的数据集。
once you've managed to put it in the following format.

9
00:00:19,950 --> 00:00:23,730
一列用于输入文本，一列用于目标文本。
One column for input texts and one for the target texts.

10
00:00:23,730 --> 00:00:25,980
以下是我们如何使用 Datasets 库实现此目的
Here is how we can achieve this with the Datasets library

11
00:00:25,980 --> 00:00:29,643
以及用于英语到法语翻译的 KDE4 数据集。
and the KDE4 dataset for English to French translation.

12
00:00:30,870 --> 00:00:33,240
只要你能够让数据按照这样的形式组织，
As long as you manage to have your data look like this,

13
00:00:33,240 --> 00:00:35,440
就应该可以按照相同的步骤操作。
you should be able to follow the same steps.

14
00:00:36,630 --> 00:00:39,210
这一次，我们对于某些类的标签不是整数
For once, our labels are not integers

15
00:00:39,210 --> 00:00:42,210
而是纯文本。
corresponding to some classes, but plain texts.

16
00:00:42,210 --> 00:00:45,810
因此，我们需要将它们词元化，就像我们的输入一样。
We will thus need to tokenize them, like our inputs.

17
00:00:45,810 --> 00:00:47,370
虽然那里有一个陷阱，
There is a trap there though,

18
00:00:47,370 --> 00:00:49,890
好像你按照输入那样词元化你的目标，
as if you tokenize your targets like your inputs,

19
00:00:49,890 --> 00:00:51,690
就会遇到一个问题。
you will hit a problem.

20
00:00:51,690 --> 00:00:54,090
即使你不会说法语，你也可能会注意到
Even if you don't speak French, you might notice

21
00:00:54,090 --> 00:00:57,270
目标词元化后出现了一些奇怪的东西。
some weird things in the tokenization of the targets.

22
00:00:57,270 --> 00:01:00,510
大多数单词都被处理为几个子词元，
Most of the words are tokenized in several subtokens,

23
00:01:00,510 --> 00:01:03,180
而 fish，文本中唯一的英语单词，
while fish, one of the only English word,

24
00:01:03,180 --> 00:01:05,670
被词元化为一个单独的词。
is tokenized as a single word.

25
00:01:05,670 --> 00:01:08,703
那是因为我们的输入已经按照英语进行词元化。
That's because our inputs have been tokenized as English.

26
00:01:09,660 --> 00:01:11,430
由于我们的模型涉及两种语言，
Since our model knows two languages,

27
00:01:11,430 --> 00:01:13,800
你必须在词元化目标时提示它
you have to warn it when tokenizing the targets

28
00:01:13,800 --> 00:01:16,230
从而切换到法语模式。
so it switches in French mode.

29
00:01:16,230 --> 00:01:20,010
这是通过 as_target_tokenizer 上下文管理器完成的。
This is done with the as_target_tokenizer context manager.

30
00:01:20,010 --> 00:01:23,343
你可以看到它如何导致更紧凑的词元化。
You can see how it results in a more compact tokenization.

31
00:01:24,810 --> 00:01:25,890
处理整个数据集
Processing the whole dataset

32
00:01:25,890 --> 00:01:28,440
然后配合 map 函数使用非常简单。
is then super easy with the map function.

33
00:01:28,440 --> 00:01:30,207
你可以为输入和目标输出
You can pick different maximum lengths

34
00:01:30,207 --> 00:01:32,100
选择不同的最大长度，
for the inputs and targets,

35
00:01:32,100 --> 00:01:34,530
并通过设置 padding=max_length
and choose to pad at this stage to that maximum length

36
00:01:34,530 --> 00:01:36,273
在此阶段填充到最大长度。
by setting padding=max_length.

37
00:01:37,230 --> 00:01:39,300
由于还需要再多一步
Here we'll show you to pad dynamically

38
00:01:39,300 --> 00:01:41,013
所以这里我们将向你展示动态填充。
as it requires one more step.

39
00:01:42,450 --> 00:01:43,470
你的输入和目标结果
Your inputs and targets

40
00:01:43,470 --> 00:01:46,080
都是各种长度的句子。
are all sentences of various lengths.

41
00:01:46,080 --> 00:01:48,510
由于输入和目标结果的最大长度可能会不同，
We will pad the inputs and targets separately,

42
00:01:48,510 --> 00:01:50,460
我们将分别填充输入
as the maximum lengths of the inputs and targets

43
00:01:50,460 --> 00:01:51,483
和目标结果。
might be different.

44
00:01:52,620 --> 00:01:54,540
然后我们用 pad token 填充输入
Then we pad the inputs with the pad token

45
00:01:54,540 --> 00:01:57,060
以及索引为 -100 的目标输出
and the targets with the -100 index

46
00:01:57,060 --> 00:01:58,890
确保它们不被包含在
to make sure they're not taken into account

47
00:01:58,890 --> 00:02:00,123
损失计算中。
in the loss computation.

48
00:02:01,320 --> 00:02:02,153
一旦完成上述操作，
Once this is done,

49
00:02:02,153 --> 00:02:04,340
批处理输入和目标输出变得超级容易。
batching inputs and targets become super easy.

50
00:02:05,670 --> 00:02:08,220
Transformers 库为我们提供了数据整理器
The Transformers library provides us with data collator

51
00:02:08,220 --> 00:02:10,500
自动完成这一切。
to do this all automatically.

52
00:02:10,500 --> 00:02:13,800
之后你可以在你的 Keras 模型上使用 model.fit () 之前
You can then pass it to the Trainer with your datasets

53
00:02:13,800 --> 00:02:15,960
将它与你的数据集一起传递给 Trainer
or use it in the to_tf_dataset method

54
00:02:15,960 --> 00:02:18,560
或者在 to_tf_dataset 方法中使用它。
before using model.fit () on your Keras model.

55
00:02:21,057 --> 00:02:23,724
（空气呼啸）
(air whooshing)

