1
00:00:05,730 --> 00:00:07,590
- 让我们研究如何为词元分类
- Let's study how to preprocess a dataset

2
00:00:07,590 --> 00:00:09,063
预处理数据集！
for token classification!

3
00:00:10,560 --> 00:00:12,660
任何对于单词进行标记
Token classification regroups any task

4
00:00:12,660 --> 00:00:14,940
或者对词元进行标记的任务
that can be framed as labeling each word

5
00:00:14,940 --> 00:00:17,190
均可以由词元分类进行重新组合，
or token in a sentence,

6
00:00:17,190 --> 00:00:19,530
比如识别人物、组织
like identifying the persons, organizations

7
00:00:19,530 --> 00:00:21,093
和位置之类。
and locations for instance.

8
00:00:22,170 --> 00:00:25,470
对于本例来说，我们将使用 Conll 数据集，
For our example, we will use the Conll dataset,

9
00:00:25,470 --> 00:00:27,900
在其中我们删除了我们不会使用的列
in which we remove columns we won't use

10
00:00:27,900 --> 00:00:29,940
并重命名其他的以获取数据集
and rename the other ones to get to a dataset

11
00:00:29,940 --> 00:00:32,943
其中只有两列，即 words 和 labels。
with just two columns, words and labels.

12
00:00:34,200 --> 00:00:36,750
如果你有自己的词元分类数据集，
If you have your own dataset for token classification,

13
00:00:36,750 --> 00:00:39,870
请确保你清理数据以达到相同的效果，
just make sure you clean your data to get to the same point,

14
00:00:39,870 --> 00:00:43,290
包括 words 列内含字符列表
with one column containing words as list of strings

15
00:00:43,290 --> 00:00:45,540
和另一个 labels 列内含整数
and another containing labels as integers

16
00:00:45,540 --> 00:00:48,513
从零到你的标签数量减一。
spanning from zero to your number of labels minus one.

17
00:00:49,740 --> 00:00:52,290
确保将标签名称存储在某处。
Make sure you have your label names stored somewhere.

18
00:00:52,290 --> 00:00:54,810
在这里，我们从数据集特征中获取它们。
Here we get them from the dataset features.

19
00:00:54,810 --> 00:00:57,660
所以你检查数据时
So you are able to map the integers to some real labels

20
00:00:57,660 --> 00:00:58,960
可以将整数映射到一些真实的标签。
when inspecting your data.

21
00:01:00,690 --> 00:01:03,510
这里我们正在做命名实体识别，
Here we are doing named entity recognitions,

22
00:01:03,510 --> 00:01:05,430
所以对于不属于任何实体的词
so ours labels are either O

23
00:01:05,430 --> 00:01:08,310
我们的标签可能是 O 。
for words that do not belong to any entity.

24
00:01:08,310 --> 00:01:13,310
LOC 代表位置，PER 代表个人，ORG 代表组织
LOC for location, PER for person, ORG for organization

25
00:01:13,860 --> 00:01:15,603
和 MISC 杂项。
and MISC for miscellaneous.

26
00:01:16,650 --> 00:01:18,540
每个标签有两个版本。
Each label has two versions.

27
00:01:18,540 --> 00:01:21,960
B 标签表示以实体开始的单词
The B labels indicate a word that begins an entity

28
00:01:21,960 --> 00:01:25,503
而 I 标签表示实体内部的单词。
while the I labels indicate a word that is inside an entity.

29
00:01:27,180 --> 00:01:28,830
预处理我们数据的第一步
The first step to preprocess our data

30
00:01:28,830 --> 00:01:30,660
是将单词词元化。
is to tokenize the words.

31
00:01:30,660 --> 00:01:33,120
使用分词器很容易做到这一点。
This is very easily done with the tokenizer.

32
00:01:33,120 --> 00:01:35,370
我们只需要告诉它我们已经预先词元化了数据
We just have to tell it we have pre-tokenized the data

33
00:01:35,370 --> 00:01:37,503
带有标志 is_split_into_words=True。
with the flag is_split_into_words=True.

34
00:01:38,520 --> 00:01:40,380
然后是困难的部分。
Then comes the hard part.

35
00:01:40,380 --> 00:01:42,360
由于我们添加了特殊词元
Since we have added special tokens

36
00:01:42,360 --> 00:01:45,270
每个单词可能被拆分成几个词元，
and each word may have been split into several tokens,

37
00:01:45,270 --> 00:01:48,090
我们的标签将不再与词元匹配。
our labels won't match the tokens anymore.

38
00:01:48,090 --> 00:01:50,670
这是我们的快速分词器提供的单词 ID
This is where the word IDs our fast tokenizer provides

39
00:01:50,670 --> 00:01:51,723
前来救援。
come to the rescue.

40
00:01:53,040 --> 00:01:55,500
他们将每个词元与其所属的单词匹配
They match each token to the word it belongs to

41
00:01:55,500 --> 00:01:58,470
这允许我们将每个词元映射到它的标签。
which allows us to map each token to its label.

42
00:01:58,470 --> 00:01:59,303
我们只需要确保
We just have to make sure

43
00:01:59,303 --> 00:02:01,710
我们针对里面的词元将 B 标签
we change the B labels to their I counterparts

44
00:02:01,710 --> 00:02:03,450
更改为对应的 I
for tokens that are inside

45
00:02:03,450 --> 00:02:05,433
但不是在单词的开头。
but not at the beginning of a word.

46
00:02:06,330 --> 00:02:09,120
特殊词元的标签为 -100，
The special tokens get a label of -100,

47
00:02:09,120 --> 00:02:11,070
这就是当计算损失时我们告诉 Transformer 损失函数
which is how we tell the Transformer loss functions

48
00:02:11,070 --> 00:02:14,607
要忽略它们。
to ignore them when computing the loss.

49
00:02:14,607 --> 00:02:16,890
然后代码非常简单明了。
The code is then pretty straightforward.

50
00:02:16,890 --> 00:02:18,660
我们针对可以自定义的单词内的词元
We write a function that shifts the labels

51
00:02:18,660 --> 00:02:21,840
写了一个函数来移动标签
for tokens that are inside a word that you can customize

52
00:02:21,840 --> 00:02:24,490
并在为每个词元生成标签时调用它。
and use it when generating the labels for each token.

53
00:02:25,830 --> 00:02:28,260
一旦编写了创建标签的函数，
Once that function to create our labels is written,

54
00:02:28,260 --> 00:02:31,920
我们可以使用 map 函数预处理整个数据集。
we can preprocess the whole dataset using the map function.

55
00:02:31,920 --> 00:02:33,360
使用选项 batched=True，
With the option batched=True,

56
00:02:33,360 --> 00:02:35,793
我们释放了快速分词器的速度。
we unleash the speed of out fast tokenizers.

57
00:02:37,110 --> 00:02:40,350
当我们需要创建一个批处理任务时，最后一个问题就来了。
The last problem comes when we need to create a batch.

58
00:02:40,350 --> 00:02:42,150
除非你改变了预处理函数
Unless you changed the preprocessing function

59
00:02:42,150 --> 00:02:43,890
增加一些固定的填充，
to apply some fixed padding,

60
00:02:43,890 --> 00:02:45,900
否则我们会得到各种长度的句子，
we will get sentences of various lengths,

61
00:02:45,900 --> 00:02:47,900
我们需要将其填充到相同的长度。
which we need to pad to the same length.

62
00:02:48,930 --> 00:02:50,730
填充处理需要在输入
The padding needs to be applied to the inputs

63
00:02:50,730 --> 00:02:51,900
以及标签上进行，
as well as the labels,

64
00:02:51,900 --> 00:02:53,950
因为每个词元应该有一个标签。
since we should have one label per token.

65
00:02:54,870 --> 00:02:58,260
同样，-100 表示应忽略的标签
Again, -100 indicates the labels that should be ignored

66
00:02:58,260 --> 00:02:59,510
用于损失计算。
for the loss computation.

67
00:03:00,420 --> 00:03:01,560
所有这一切
This is all done for us

68
00:03:01,560 --> 00:03:04,050
都通过 DataCollatorForTokenClassification 为我们完成了，
by the DataCollatorForTokenClassification,

69
00:03:04,050 --> 00:03:06,740
你可以在 PyTorch 或 TensorFlow 中使用它。
which you can use in PyTorch or TensorFlow.

70
00:03:06,740 --> 00:03:08,880
完成上述内容后，你就可以准备发送数据了
With all of this, you are either ready to send your data

71
00:03:08,880 --> 00:03:11,190
并将此数据整理器传递给 Trainer，
and this data collator to the Trainer,

72
00:03:11,190 --> 00:03:13,320
或者使用 to_tf_dataset 方法
or use the to_tf_dataset method

73
00:03:13,320 --> 00:03:15,333
以及模型的拟合方法。
and the fit method of your model.

