fill-mask pipeline with tables (TapasForMaskedLM) fails DataFrame type assertion
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0
- Platform: macOS
- Python version: 3.9.2
- PyTorch version (GPU?): 1.8.9 (N/A)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@LysandreJik 
@NielsRogge

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using: **TapasForMaskedLM**

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ X ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Slight modification of [example](https://huggingface.co/transformers/v4.5.1/model_doc/tapas.html#tapasformaskedlm) to include `fill-mask` pipeline
2. Script to run:
```
if __name__ == '__main__':

    from transformers import TapasConfig,TapasTokenizer,TapasForMaskedLM
    from transformers import pipeline
    import pandas as pd
    import numpy as np
    import torch
    import sys

    config = TapasConfig.from_pretrained(
        'google/tapas-base-finetuned-wtq',from_pt=True)
    model = TapasForMaskedLM.from_pretrained(
        'google/tapas-base-finetuned-wtq', config=config)
    tokenizer=TapasTokenizer.from_pretrained(
        "google/tapas-base-finetuned-wtq", from_pt=True)

    data= {
        "actors": ["brad pitt", "leonardo di caprio", "george clooney"],
        "age": ["56", "45", "59"],
        "number of movies": ["87", "53", "69"],
        "date of birth": ["7 february 1967", "10 june 1996", "28 november 1967"]
    }

    table = pd.DataFrame.from_dict(data)

    queries=[
        f"The number of movies Brad Pitt acted in is {tokenizer.mask_token}",
        f"Leonardo di caprio's age is {tokenizer.mask_token}"]

    nlp = pipeline(task="fill-mask",framework="pt",model=model, tokenizer=tokenizer)
    test = nlp(queries, table=table)
```

3. From a short debugging it seems that `pandas/core/frame.py` is called and the following code overwrites `table` to a list:

```
        if isinstance(data, DataFrame):
            data = data._mgr

        if isinstance(data, BlockManager):
            if index is None and columns is None and dtype is None and copy is False:
                # GH#33357 fastpath
                NDFrame.__init__(self, data)
                return
```

## Expected behavior

Input table should not be overwritten to a list. Is this call to `frame.py` expected? If not what is the required steps to overcome this?
<!-- A clear and concise description of what you would expect to happen. -->

Hi,

TAPAS is not supported by the `FillMaskPipeline`, only by the `TableQuestionAnsweringPipeline`. 

`TapasForMaskedLM` was defined, but I did not include the weights of language modeling head when converting the checkpoints (I only loaded the weights of `TapasModel`, `TapasForQuestionAnswering` and `TapasForSequenceClassification`). However, one could also load the weights of a `TapasForMaskedLM` by updating [this function](https://github.com/huggingface/transformers/blob/d3c3e722d69627d6334d7ef8faaced7df3103174/src/transformers/models/tapas/modeling_tapas.py#L127).<|||||>Thank you Niels.

Not familiar how this should work. If you have any example scripts that can do this updating I'd appreciate the help. 

Anyhow thanks for answering.<|||||>So, to convert a TAPAS Tensorflow checkpoint to PyTorch, you can use [this script](https://github.com/huggingface/transformers/blob/master/src/transformers/models/tapas/convert_tapas_original_tf_checkpoint_to_pytorch.py). You can run it in a command line, like so (running it from the src/transformers/models/tapas directory of this repo):

```
python convert_tapas_original_tf_checkpoint_to_pytorch.py --task="MLM" --tf_checkpoint_path="path_to_the_tf_checkpoint" --tapas_config_file="path_to_the_json_file" --pytorch_dump_path="path_to_where_you_want_to_dump_the_pytorch_model"
```

However, it might be that you encounter an error as not all weights are correctly converted. In that case, you need to update the `load_tf_weights_in_tapas` function which the script uses (and which is defined in `modeling_tapas.py`).<|||||>Thanks Niels,

Actually I am encountering an import error for `load_tf_weights_in_tapas`. I played around a bit with `__init__.py` to adjust the `_import_structure` to include `modeling_tapas.py` + the function but it still wont import. Are you aware of this issue? 

```
Traceback (most recent call last):
  File "/Users/pafitis/miniforge3/envs/comp0087/lib/python3.9/site-packages/transformers/models/tapas/convert_tapas_original_tf_checkpoint_to_pytorch.py", line 20, in <module>
    from transformers import (
ImportError: cannot import name 'load_tf_weights_in_tapas' from 'transformers' (/Users/pafitis/miniforge3/envs/comp0087/lib/python3.9/site-packages/transformers/__init__.py)
```
I can bypass it if I manually change the import call to `from transformers.models.tapas.modeling_tapas import load_tf_weights_in_tapas`

The issue is within `convert_tapas_original_tf_checkpoint_to_pytorch.py` lines 20-28<|||||>There's also some remnants of your own path structure left over. Just FYI

Lines 95-96
```
# dir_name = r"C:\Users\niels.rogge\Documents\Python projecten\tensorflow\Tensorflow models\SQA\Base\tapas_sqa_inter_masklm_base_reset"
    # tokenizer = TapasTokenizer(vocab_file=dir_name + r"\vocab.txt", model_max_length=512)
```

<|||||>Hi,

Yeah I know how to solve the import issue. Let me create a branch that you can use<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Are there any updates on this?<|||||>Hi, 

I'll first fix the issue that you can't import `load_tf_weights_in_tapas`. You can then use it.<|||||>Good news: I've successfully converted the `TapasForMaskedLM` checkpoints. 

I've already uploaded some on the hub: 

* google-tapas-base-masklm: https://huggingface.co/google/tapas-base-masklm
* google-tapas-large-masklm: https://huggingface.co/google/tapas-large-masklm

Note: it will not work with the current version of Transformers, you'll need to install from the PR I will open soon.<|||||>Thank you Niels!<|||||>Hi @NielsRogge, I wanted to see if this performance is expected. Using this branch (same as PR linked): https://github.com/NielsRogge/transformers/tree/fix_tapas_conversion_script


The example:

```
tokenizer = TapasTokenizer.from_pretrained("google/tapas-large-masklm")
model = TapasForMaskedLM.from_pretrained("google/tapas-large-masklm")

data = {'Actors': ["Brad Pitt", "Leonardo Di Caprio", "George Clooney"],
        'Age': ["56", "45", "59"],
        'Number of movies': ["87", "53", "69"]
}

queries = ['Brad [MASK] played in 87 movies.',
           'George Clooney is [MASK] years old.']
table = pd.DataFrame.from_dict(data)


# prepare inputs
inputs = tokenizer(table=table, queries=query, padding="max_length", return_tensors="pt")

# forward pass
outputs = model(**inputs)

# return top 5 values and predictions
masked_index = torch.nonzero(inputs.input_ids.squeeze() == tokenizer.mask_token_id, as_tuple=False)
logits = outputs.logits[0, masked_index.item(), :]
probs = logits.softmax(dim=0)
values, predictions = probs.topk(5)

for value, pred in zip(values, predictions):
  print(f"{tokenizer.decode([pred])} with confidence {value}")
```

The results I get:

**FOR google/tapas-large-masklm:**
```
##gned with confidence 0.0003957822045776993
brodie with confidence 0.00031843443866819143
scanned with confidence 0.0002803522511385381
##kshi with confidence 0.0002378804492764175
scanning with confidence 0.0002144851314369589
```

**FOR google/tapas-base-masklm**
```
[CLS] with confidence 0.7544503808021545
[SEP] with confidence 0.000950647983700037
[MASK] with confidence 0.00019540438370313495
, with confidence 6.406998727470636e-05
the with confidence 5.370331200538203e-05
```



**IS THIS BEHAVIOUR EXPECTED? SEEMS VERY POOR!**<|||||>It runs fine for me. I get the following answers respectively (using google/tapas-large-masklm):

* first query: 'Brad [MASK] played in 87 movies.'
```
pitt with confidence 0.9996523857116699
has with confidence 0.00017903841217048466
have with confidence 1.926756158354692e-05
had with confidence 8.52907123771729e-06
lee with confidence 7.179685326264007e-06
```
* second query: 'George Clooney is [MASK] years old.'
```
59 with confidence 0.9172192215919495
58 with confidence 0.02275438793003559
69 with confidence 0.005611400585621595
60 with confidence 0.005492867436259985
57 with confidence 0.004567734897136688
```
There's probably a bug in your installation of Transformers.<|||||>Thank you @NielsRogge. Indeed, issue on my side.