1
00:00:00,227 --> 00:00:01,432
（嘶嘶声）
(whooshing sound)

2
00:00:01,432 --> 00:00:02,420
（贴纸弹出）
(sticker popping)

3
00:00:02,420 --> 00:00:05,340
（嘶嘶声）
(whooshing sound)

4
00:00:05,340 --> 00:00:07,833
- 数据集加上数据帧等于真爱。
- Datasets and DataFrames equals love.

5
00:00:08,790 --> 00:00:11,010
虽然 Datasets 库的处理函数
Although the processing functions of the Datasets library

6
00:00:11,010 --> 00:00:14,040
将涵盖训练模型所需的大部分情况，
will cover most of the cases needed to train a model,

7
00:00:14,040 --> 00:00:15,660
有时你需要切换到其它库
there are times when you'll need to switch to a library

8
00:00:15,660 --> 00:00:18,240
比如 Pandas 来获得更强大的功能
like Pandas to access more powerful features

9
00:00:18,240 --> 00:00:20,970
或针对计算机视觉使用更高级的 API。
or high level APIs for visualization.

10
00:00:20,970 --> 00:00:23,220
幸运的是，Datasets 库是专门为了
Fortunately, the Datasets library is designed

11
00:00:23,220 --> 00:00:25,710
和外部库进行互操作所设计的，比如 Pandas
to be interoperable with libraries like Pandas,

12
00:00:25,710 --> 00:00:29,790
此外还有 NumPy、PyTorch、TensorFlow 和 JAX。
as well as NumPy, PyTorch, TensorFlow and JAX.

13
00:00:29,790 --> 00:00:30,930
在本视频中，我们将了解
In this video, we'll take a look

14
00:00:30,930 --> 00:00:32,550
我们如何将数据格式快速切换到 Pandas DataFrames 
at how we can quickly switch our data

15
00:00:32,550 --> 00:00:34,263
并且再切换回来。
to Pandas DataFrames and back.

16
00:00:36,120 --> 00:00:38,310
例如，假设我们正在分析
As an example, let's suppose we're analyzing

17
00:00:38,310 --> 00:00:40,830
来自瑞士的最高法院案件。
Supreme Court cases from Switzerland.

18
00:00:40,830 --> 00:00:43,020
像往常一样，我们使用 load_dataset 函数
As usual, we download our dataset from the hub

19
00:00:43,020 --> 00:00:44,940
从 hub 下载我们的数据集。
using the load_dataset function.

20
00:00:44,940 --> 00:00:46,980
你可以看到训练集的第一个元素
And you can see that the first element of the training set

21
00:00:46,980 --> 00:00:48,510
是一个普通的 Python 字典类型
is an ordinary Python dictionary

22
00:00:48,510 --> 00:00:50,110
里面包含各种我们需要的字段。
with various fields of interest.

23
00:00:51,690 --> 00:00:53,670
现在，假设在我们训练任何模型之前，
Now, suppose that before we train any models,

24
00:00:53,670 --> 00:00:55,590
我们想稍微浏览一下数据。
we'd like to explore the data a bit.

25
00:00:55,590 --> 00:00:57,390
例如，我们可能希望知道
For example, we might be interested in knowing

26
00:00:57,390 --> 00:00:59,820
哪些法律领域最常见
which legal areas are the most common

27
00:00:59,820 --> 00:01:01,380
或者我们可能想知道语言
or we might wanna know how the languages

28
00:01:01,380 --> 00:01:02,930
在各个地区是如何分布的。
are distributed across regions.

29
00:01:04,500 --> 00:01:05,333
使用原生 Arrow 类型
Answering these questions

30
00:01:05,333 --> 00:01:07,530
很难回答这些问题，
with the native Arrow format isn't easy,

31
00:01:07,530 --> 00:01:10,500
但我们可以快速切换到 Pandas 来获得答案。
but we can quickly switch to Pandas to get our answers.

32
00:01:10,500 --> 00:01:13,500
通过使用 set_format 方法即可实现，
The way this works is that by using the set_format method,

33
00:01:13,500 --> 00:01:15,480
我们将更改数据集的输出格式
we will change the output format of the dataset

34
00:01:15,480 --> 00:01:18,930
从 Python 字典类型到 Pandas DataFrame 类型。
from Python dictionaries to Pandas DataFrames.

35
00:01:18,930 --> 00:01:20,130
正如你在此示例中所见，
As you can see in this example,

36
00:01:20,130 --> 00:01:22,890
数据集中的每一行都表示为一个 DataFrame，
each row in the dataset is represented as a DataFrame,

37
00:01:22,890 --> 00:01:24,540
所以我们可以对整个数据集进行切片
so we can slice the whole dataset

38
00:01:24,540 --> 00:01:26,583
来获取语料库的单个 DataFrame。
to get a single DataFrame of the corpus.

39
00:01:28,080 --> 00:01:29,520
其底层工作原理
The way this works under the hood,

40
00:01:29,520 --> 00:01:31,080
是 datasets 库改变了
is that the datasets library changes

41
00:01:31,080 --> 00:01:33,900
数据集中神奇的 __getitem__ 方法。
the magic __getitem__ method of the dataset.

42
00:01:33,900 --> 00:01:35,640
对于 Python 容器来说 __getitem__ 方法
The __getitem__ method is a special method

43
00:01:35,640 --> 00:01:37,320
是一种特殊的方法，它允许你指定
for Python containers that allows you

44
00:01:37,320 --> 00:01:39,870
以何种方式使用索引。
to specify how indexing works.

45
00:01:39,870 --> 00:01:42,540
在这种情况下，原始数据集的 __getitem__ 方法
In this case, the __getitem__ method of the raw dataset

46
00:01:42,540 --> 00:01:45,150
首先返回一个 Python 字典
starts off by returning a Python dictionary

47
00:01:45,150 --> 00:01:47,520
然后在调用 set_format 之后，
and then after applying set_format,

48
00:01:47,520 --> 00:01:50,283
我们更改 __getitem__ 以返回 DataFrames。
we change __getitem__ to return DataFrames instead.

49
00:01:52,080 --> 00:01:54,690
如果你想做格式转换  
The Datasets library also provides a to_pandas method

50
00:01:54,690 --> 00:01:56,250
并一次性对数据集进行切片
if you wanna do the format conversion

51
00:01:56,250 --> 00:01:58,113
Datasets 库还提供了一个 to_pandas 方法。
and slicing of the dataset in one go.

52
00:02:00,090 --> 00:02:01,590
一旦你有了一个 DataFrame，
And once you have a DataFrame,

53
00:02:01,590 --> 00:02:03,990
你可以找到各种复杂问题的答案
you can find the answers to all sorts of complex questions

54
00:02:03,990 --> 00:02:06,740
或使用你最喜欢的可视化库绘制图表。
or make plots with your favorite visualization library.

55
00:02:07,890 --> 00:02:08,850
唯一要记住的
The only thing to remember

56
00:02:08,850 --> 00:02:10,830
是一旦你完成了 Pandas 分析，
is that once you're done with your Pandas analysis,

57
00:02:10,830 --> 00:02:14,460
你应该将输出格式重置回 Arrow 表。
you should reset the output format back to Arrow tables.

58
00:02:14,460 --> 00:02:16,350
如果不这样做，当你尝试词元化你的文本
If you don't, you can run into problems

59
00:02:16,350 --> 00:02:17,910
可能会遇到问题
if you try to tokenize your text

60
00:02:17,910 --> 00:02:19,260
因为它不再是字典中的
because it is no longer represented

61
00:02:19,260 --> 00:02:20,610
字符串。
as strings in a dictionary.

62
00:02:21,750 --> 00:02:24,780
通过重置输出格式，我们得到 Arrow 表
By resetting the output format we get back Arrow tables

63
00:02:24,780 --> 00:02:26,580
我们可以毫无顾虑地进行词元化。
and we can tokenize without problem.

64
00:02:27,513 --> 00:02:30,346
（嘶嘶声）
(whooshing sound)

