﻿1
00:00:00,227 --> 00:00:01,432
(whooshing sound)

2
00:00:01,432 --> 00:00:02,420
(sticker popping)

3
00:00:02,420 --> 00:00:05,340
(whooshing sound)

4
00:00:05,340 --> 00:00:07,833
- Datasets and DataFrames equals love.

5
00:00:08,790 --> 00:00:11,010
Although the processing
functions of the Datasets library

6
00:00:11,010 --> 00:00:14,040
will cover most of the cases
needed to train a model,

7
00:00:14,040 --> 00:00:15,660
there are times when you'll
need to switch to a library

8
00:00:15,660 --> 00:00:18,240
like Pandas to access
more powerful features

9
00:00:18,240 --> 00:00:20,970
or high level APIs for visualization.

10
00:00:20,970 --> 00:00:23,220
Fortunately, the Datasets
library is designed

11
00:00:23,220 --> 00:00:25,710
to be interoperable with
libraries like Pandas,

12
00:00:25,710 --> 00:00:29,790
as well as NumPy, PyTorch,
TensorFlow and JAX.

13
00:00:29,790 --> 00:00:30,930
In this video, we'll take a look

14
00:00:30,930 --> 00:00:32,550
at how we can quickly switch our data

15
00:00:32,550 --> 00:00:34,263
to Pandas DataFrames and back.

16
00:00:36,120 --> 00:00:38,310
As an example, let's
suppose we're analyzing

17
00:00:38,310 --> 00:00:40,830
Supreme Court cases from Switzerland.

18
00:00:40,830 --> 00:00:43,020
As usual, we download
our dataset from the hub

19
00:00:43,020 --> 00:00:44,940
using the load_dataset function.

20
00:00:44,940 --> 00:00:46,980
And you can see that the first
element of the training set

21
00:00:46,980 --> 00:00:48,510
is an ordinary Python dictionary

22
00:00:48,510 --> 00:00:50,110
with various fields of interest.

23
00:00:51,690 --> 00:00:53,670
Now, suppose that before
we train any models,

24
00:00:53,670 --> 00:00:55,590
we'd like to explore the data a bit.

25
00:00:55,590 --> 00:00:57,390
For example, we might
be interested in knowing

26
00:00:57,390 --> 00:00:59,820
which legal areas are the most common

27
00:00:59,820 --> 00:01:01,380
or we might wanna know how the languages

28
00:01:01,380 --> 00:01:02,930
are distributed across regions.

29
00:01:04,500 --> 00:01:05,333
Answering these questions

30
00:01:05,333 --> 00:01:07,530
with the native Arrow format isn't easy,

31
00:01:07,530 --> 00:01:10,500
but we can quickly switch to
Pandas to get our answers.

32
00:01:10,500 --> 00:01:13,500
The way this works is that by
using the set_format method,

33
00:01:13,500 --> 00:01:15,480
we will change the output
format of the dataset

34
00:01:15,480 --> 00:01:18,930
from Python dictionaries
to Pandas DataFrames.

35
00:01:18,930 --> 00:01:20,130
As you can see in this example,

36
00:01:20,130 --> 00:01:22,890
each row in the dataset is
represented as a DataFrame,

37
00:01:22,890 --> 00:01:24,540
so we can slice the whole dataset

38
00:01:24,540 --> 00:01:26,583
to get a single DataFrame of the corpus.

39
00:01:28,080 --> 00:01:29,520
The way this works under the hood,

40
00:01:29,520 --> 00:01:31,080
is that the datasets library changes

41
00:01:31,080 --> 00:01:33,900
the magic __getitem__
method of the dataset.

42
00:01:33,900 --> 00:01:35,640
The __getitem__ method is a special method

43
00:01:35,640 --> 00:01:37,320
for Python containers that allows you

44
00:01:37,320 --> 00:01:39,870
to specify how indexing works.

45
00:01:39,870 --> 00:01:42,540
In this case, the __getitem__
method of the raw dataset

46
00:01:42,540 --> 00:01:45,150
starts off by returning
a Python dictionary

47
00:01:45,150 --> 00:01:47,520
and then after applying set_format,

48
00:01:47,520 --> 00:01:50,283
we change __getitem__ to
return DataFrames instead.

49
00:01:52,080 --> 00:01:54,690
The Datasets library also
provides a to_pandas method

50
00:01:54,690 --> 00:01:56,250
if you wanna do the format conversion

51
00:01:56,250 --> 00:01:58,113
and slicing of the dataset in one go.

52
00:02:00,090 --> 00:02:01,590
And once you have a DataFrame,

53
00:02:01,590 --> 00:02:03,990
you can find the answers to
all sorts of complex questions

54
00:02:03,990 --> 00:02:06,740
or make plots with your
favorite visualization library.

55
00:02:07,890 --> 00:02:08,850
The only thing to remember

56
00:02:08,850 --> 00:02:10,830
is that once you're done
with your Pandas analysis,

57
00:02:10,830 --> 00:02:14,460
you should reset the output
format back to Arrow tables.

58
00:02:14,460 --> 00:02:16,350
If you don't, you can run into problems

59
00:02:16,350 --> 00:02:17,910
if you try to tokenize your text

60
00:02:17,910 --> 00:02:19,260
because it is no longer represented

61
00:02:19,260 --> 00:02:20,610
as strings in a dictionary.

62
00:02:21,750 --> 00:02:24,780
By resetting the output format
we get back Arrow tables

63
00:02:24,780 --> 00:02:26,580
and we can tokenize without problem.

64
00:02:27,513 --> 00:02:30,346
(whooshing sound)

