i’m working on distilbert for text classification, and want to save the tensors into a data frame so i can use it again as feature generator
to illustrate my question more:
below are token IDs for one sentence in a row in csv file i’m working on :
Token IDs: tensor([  101, 11113,  2080, 25876,  2542, 15009, 11290, 22291,  3370,  2478,
7037, 22160,  3215,  1998,  5760,  5001, 10464, 24895, 15009,  2157,
2002, 17585,  6593, 16940,  1037,  2553,  3189,  1026,  1013,  1055,
1028,  1026,  1013,  1055,  1028, 11581,  2063,  2138,  1997,  1996,
15843,  1997, 10181, 17843,  2542, 15009, 11290, 22291,  3370, 25510,
7096,  2038,  2468,  1996,  2364,  7709,  2000,  7438,  5022,  2007,
2203,  1011,  2754,   102])
1- i want to replace the text in the csv file into their tokens by converting these tensors into np.array and save this CSV file
2- then i need to know how to convert this column back from np.array to tensors (input ids and attention masks) so i can use them for fine tuning tasks