Hi
I asked this issue on mapping one entry to multiple entries with huggingface dataset library:

  

      github.com/huggingface/datasets
  

  
    
  
	  
  

  
    
      equal operation to perform unbatch for huggingface datasets  2
    

    
      
        opened 
        
          
        
        Aug 7, 2021
      
      


      
        
          
          dorooddorood606
        
      
    

    
        
          bug
        
    
  


  
    Hi
I need to use "unbatch" operation in tensorflow on a huggingface dataset, I …could not find this operation, could you kindly direct me how I can do it, here is the problem I am trying to solve:

I am considering "record" dataset in SuperGlue and I need to replicate each entery of the dataset for each answer, to make it similar to what T5 originally did:

https://github.com/google-research/text-to-text-transfer-transformer/blob/3c58859b8fe72c2dbca6a43bc775aa510ba7e706/t5/data/preprocessors.py#L925

Here please find an example:

  For example, a typical example from ReCoRD might look like
  {
      'passsage': 'This is the passage.',
      'query': 'A @placeholder is a bird.',
      'entities': ['penguin', 'potato', 'pigeon'],
      'answers': ['penguin', 'pigeon'],
  }
  and I need a prosessor which would turn this example into the following two examples:
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'penguin',
  }
  and
  {
      'inputs': 'record query: A @placeholder is a bird. entities: penguin, '
                'potato, pigeon passage: This is the passage.',
      'targets': 'pigeon',
  }


For doing this, one need unbatch, as each entry can map to multiple samples depending on the number of answers, I am not sure how to perform this operation with  huggingface datasets library and greatly appreciate your help

@lhoestq 

Thank you very much.
  

  

  
    
    
  

  


Not sure if this is better to be asked here, please let me know to close the issue in the repo, or here.
thank you for your kind response and assistance on how one can model this with huggingface datasets.
really appreciate your response @lhoestq