Thanks @nielsr
I am using ViTFeatureextractor and TorchVision to resize the image, made sure the images are a batch of 3D arrays like you pointed out.
I have added all snippets, here
  

      github.com/huggingface/transformers
  

  
    
  
	  
  

  
    
      Issues with Training VisionEncoderDecoder with Seq2SeqTrainer 3
    

    
      
        opened 
        
          
        
        Nov 18, 2021
      
      


      
        
          
          PrithivirajDamodaran
        
      
    

    
    
  


  
    Hello Team, 

I am trying to code up a VisionEncoderDecoder with VIT + BERT an…d finetune it with `Seq2SeqTrainer.` (Presuming the support exists).

```python
from transformers import VisionEncoderDecoderModel
from transformers import ViTFeatureExtractor, AutoTokenizer
import torch 

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
feature_extractor = ViTFeatureExtractor.from_pretrained('google/vit-base-patch16-224-in21k')
vitbert = VisionEncoderDecoderModel.from_encoder_decoder_pretrained('google/vit-base-patch16-224-in21k', 'bert-base-uncased')
vitbert.to(device)
print("Model loaded")
```
VITModel accepts `pixel_values`  but the VisionEncoderDecoder model apparently fails to recognize the preprocessed inputs with image encodings as `pixel_values` and `labels`. So I changed the `pixel_values` into `input_ids` for the model accept? (not sure if I am in the right direction) .

**The Pre-Process snippet** 
```python

input_encodings = feature_extractor(images=inputs, return_tensors="pt")
input_encodings["input_ids"] = input_encodings["pixel_values"]
```

**Feature casting snippet**
```python
from datasets import Features, Array3D, Sequence, Value
features = Features({
  "input_ids": Array3D(dtype="float32", shape=(3, 224, 224)),
  "labels": Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)
})
preprocessed_train_ds = tokenized_train_datasets.map(ds_preprocess_function, batched=True, features=features)
preprocessed_val_ds = tokenized_eval_datasets.map(ds_preprocess_function, batched=True, features=features)
```

But `Trainer.train` throws the below error.

```python
ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['labels']
```

Not sure what I am missing here. 

Please advice.
  

  

  
    
    
  

  


Along with the issue