Hello Peter,
(I hope that’s your intended name)
Am doing something similar but, my target isn’t just a class it’s rather a tensor of classes.
data1 = ([‘EU’, ‘rejects’, ‘German’, ‘call’, ‘to’, ‘boycott’, ‘British’, ‘lamb’, ‘.’],[‘B-ORG\n’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘O\n’])
The first list is the sentence while the second is the list of classes for each word.
This is further passed into a windower to give an output like so: (with relevant tags)
[’<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘EU’, ‘rejects’, ‘German’, ‘call’, ‘to’]
[’<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘EU’, ‘rejects’, ‘German’, ‘call’, ‘to’, ‘boycott’]
[’<PAD>’, ‘<PAD>’, ‘EU’, ‘rejects’, ‘German’, ‘call’, ‘to’, ‘boycott’, ‘British’]
[’<PAD>’, ‘EU’, ‘rejects’, ‘German’, ‘call’, ‘to’, ‘boycott’, ‘British’, ‘lamb’]
[‘EU’, ‘rejects’, ‘German’, ‘call’, ‘to’, ‘boycott’, ‘British’, ‘lamb’, ‘.’]
Now sampling is messing up with my sentence structure in a very wrong way and hence the word contexts for an NER task are lost.
Check an example below:
[(’<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘German’, ‘EU’, ‘<PAD>’, ‘German’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘rejects’, ‘<PAD>’, ‘<PAD>’, ‘to’, ‘German’, ‘rejects’, ‘<PAD>’, ‘<PAD>’), (’<PAD>’, ‘EU’, ‘EU’, ‘call’, ‘rejects’, ‘<PAD>’, ‘call’, ‘<PAD>’, ‘<PAD>’, ‘EU’, ‘<PAD>’, ‘EU’, ‘German’, ‘<PAD>’, ‘<PAD>’, ‘boycott’, ‘call’, ‘German’, ‘<PAD>’, ‘<PAD>’), (‘EU’, ‘rejects’, ‘rejects’, ‘to’, ‘German’, ‘<PAD>’, ‘to’, ‘<PAD>’, ‘<PAD>’, ‘rejects’, ‘EU’, ‘rejects’, ‘call’, ‘EU’, ‘EU’, ‘British’, ‘to’, ‘call’, ‘<PAD>’, ‘<PAD>’), (‘rejects’, ‘German’, ‘German’, ‘boycott’, ‘call’, ‘<PAD>’, ‘boycott’, ‘<PAD>’, ‘<PAD>’, ‘German’, ‘rejects’, ‘German’, ‘to’, ‘rejects’, ‘rejects’, ‘lamb’, ‘boycott’, ‘to’, ‘<PAD>’, ‘<PAD>’), (‘German’, ‘call’, ‘call’, ‘British’, ‘to’, ‘Peter’, ‘British’, ‘Peter’, ‘Peter’, ‘call’, ‘German’, ‘call’, ‘boycott’, ‘German’, ‘German’, ‘.’, ‘British’, ‘boycott’, ‘Peter’, ‘Peter’), (‘call’, ‘to’, ‘to’, ‘lamb’, ‘boycott’, ‘Blackburn’, ‘lamb’, ‘Blackburn’, ‘Blackburn’, ‘to’, ‘call’, ‘to’, ‘British’, ‘call’, ‘call’, ‘<PAD>’, ‘lamb’, ‘British’, ‘Blackburn’, ‘Blackburn’), (‘to’, ‘boycott’, ‘boycott’, ‘.’, ‘British’, ‘<PAD>’, ‘.’, ‘<PAD>’, ‘<PAD>’, ‘boycott’, ‘to’, ‘boycott’, ‘lamb’, ‘to’, ‘to’, ‘<PAD>’, ‘.’, ‘lamb’, ‘<PAD>’, ‘<PAD>’), (‘boycott’, ‘British’, ‘British’, ‘<PAD>’, ‘lamb’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘British’, ‘boycott’, ‘British’, ‘.’, ‘boycott’, ‘boycott’, ‘<PAD>’, ‘<PAD>’, ‘.’, ‘<PAD>’, ‘<PAD>’), (‘British’, ‘lamb’, ‘lamb’, ‘<PAD>’, ‘.’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘lamb’, ‘British’, ‘lamb’, ‘<PAD>’, ‘British’, ‘British’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’, ‘<PAD>’)] [(‘PAD’, ‘PAD’, ‘PAD’, ‘B-MISC\n’, ‘B-ORG\n’, ‘PAD’, ‘B-MISC\n’, ‘PAD’, ‘PAD’, ‘PAD’, ‘PAD’, ‘PAD’, ‘O\n’, ‘PAD’, ‘PAD’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘PAD’, ‘PAD’), (‘PAD’, ‘B-ORG\n’, ‘B-ORG\n’, ‘O\n’, ‘O\n’, ‘PAD’, ‘O\n’, ‘PAD’, ‘PAD’, ‘B-ORG\n’, ‘PAD’, ‘B-ORG\n’, ‘B-MISC\n’, ‘PAD’, ‘PAD’, ‘O\n’, ‘O\n’, ‘B-MISC\n’, ‘PAD’, ‘PAD’), (‘B-ORG\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘B-MISC\n’, ‘PAD’, ‘O\n’, ‘PAD’, ‘PAD’, ‘O\n’, ‘B-ORG\n’, ‘O\n’, ‘O\n’, ‘B-ORG\n’, ‘B-ORG\n’, ‘B-MISC\n’, ‘O\n’, ‘O\n’, ‘PAD’, ‘PAD’), (‘O\n’, ‘B-MISC\n’, ‘B-MISC\n’, ‘O\n’, ‘O\n’, ‘PAD’, ‘O\n’, ‘PAD’, ‘PAD’, ‘B-MISC\n’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘PAD’, ‘PAD’), (‘B-MISC\n’, ‘O\n’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘B-PER\n’, ‘B-MISC\n’, ‘B-PER\n’, ‘B-PER\n’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘O\n’, ‘B-MISC\n’, ‘B-MISC\n’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘B-PER\n’, ‘B-PER\n’), (‘O\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘I-PER\n’, ‘O\n’, ‘I-PER\n’, ‘I-PER\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘O\n’, ‘PAD’, ‘O\n’, ‘B-MISC\n’, ‘I-PER\n’, ‘I-PER\n’), (‘O\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘B-MISC\n’, ‘PAD’, ‘O\n’, ‘PAD’, ‘PAD’, ‘O\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘PAD’, ‘O\n’, ‘O\n’, ‘PAD’, ‘PAD’), (‘O\n’, ‘B-MISC\n’, ‘B-MISC\n’, ‘PAD’, ‘O\n’, ‘PAD’, ‘PAD’, ‘PAD’, ‘PAD’, ‘B-MISC\n’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘O\n’, ‘O\n’, ‘PAD’, ‘PAD’, ‘O\n’, ‘PAD’, ‘PAD’), (‘B-MISC\n’, ‘O\n’, ‘O\n’, ‘PAD’, ‘O\n’, ‘PAD’, ‘PAD’, ‘PAD’, ‘PAD’, ‘O\n’, ‘B-MISC\n’, ‘O\n’, ‘PAD’, ‘B-MISC\n’, ‘B-MISC\n’, ‘PAD’, ‘PAD’, ‘PAD’, ‘PAD’, ‘PAD’)
Regards