Add PromptTemplate and allow for default PromptTemplate in model configuration
### Feature request

As a user, I want to be able to load a model and feed it my input in such a way that it matches the prompt template that it saw during training. I want to be able to load the default prompt with few lines of code and without having to look up how the model was trained. Additionally, I want to be able to modify the prompt to be different from the default prompt.

The specific implementation is up for discussion. I imagine something like this:
```
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoPromptTemplate

model_id = "meta-llama/Llama-2-xb-chat-hf"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
prompt_template = AutoPromptTemplate.from_pretrained(model_id)

inputs = {
   "system_prompt":"You are a helpful assistant",
   "interactions":[
      {"user":"What is the fastest sea mammal?"},
      {"assistant":"The fastest sea mammal is the peregrine falcon"},
      {"user":"the peregrine falcon is not a mammal"}
   ]
}

output = model(**tokenizer(prompt_template(inputs)))
```

### Motivation

The huggingface hub is accumulating many finetuned models, which have been trained with a specific prompt template in mind. However, this prompt template is often difficult to find, and even more often the prompt template is missing entirely from the model card. If the model is invoked with a different template, the model performance can be severely affected. The community would benefit from a PromptTemplate class that can be loaded from the model configuration that handles the prompt templating for the end user.

At this very moment, there are likely many users that are using the `meta-llama/Llama-2-xb-chat-hf` models with a prompting style that differs from how the model is intended to be used.

### Your contribution

I am happy to be a part of the discussion for implementation and testing.
cc @ArthurZucker <|||||>This is 100% needed!<|||||>Hey! Thanks for opening this. Not sure if you have seen this but we have the [`ConversationalPipeline` ](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.Conversation) along with the `Conversation` object, which can pretty easily handle conversations. You just need to override the `_build_conversation_input_ids` of the `tokenizer` that you are using. This allows for anyone to properly build their inputs and share the modeling code on the hub. 
Having an entirely new `Auto` module just for that is an overkill, and not really the intent of `transformers`.  

However adding support for `system_prompts` in the `Conversation` object or the `ConversationalPipeline` can be done. We where not entirely sure of whether it would be highly requested or not. <|||||>Hi @ArthurZucker , thanks for your reply. I was unaware of the ConversationalPipeline, so thanks for putting it on my radar. However, neither the ConversationalPipeline nor the Conversation class handle the templating that is really the core of this feature request. Perhaps illustration with some examples will be helpful:

The `Llama-2-xb-chat` models use a very specific format [of the following type](https://huggingface.co/spaces/ysharma/Explore_llamav2_with_TGI/blob/main/app.py):
```
input_prompt = f"[INST] <<SYS>>\n{system_message}\n<</SYS>>\n\n "
for interaction in chatbot:
    input_prompt = input_prompt + str(interaction[0]) + " [/INST] " + str(interaction[1]) + " </s><s> [INST] "
```

Instead, `oasst1` models often use a format of [the following type](https://huggingface.co/OpenAssistant/llama2-13b-orca-8k-3319):
```
input_prompt = f"""<|system|>{system_message}</s><|prompter|>{user_prompt}</s><|assistant|>"""
```

Even models that are not chat models can have very specific prompt templates, such as [this sql model](https://huggingface.co/juierror/text-to-sql-with-table-schema):
```
table_prefix = "table:"
question_prefix = "question:"
join_table = ",".join(table)
input_prompt = f"{question_prefix} {question} {table_prefix} {join_table}"
```

I hope this illustrates that many models (not just chat models) on the Hugging Face hub come with an implicit specific prompt template. However, there is currently no way (that I know off) to instruct users to follow that specific prompt template, other than to describe the template on the model card. With this feature request, I am suggesting to create a more standardised way for model creators to add a prompt template to their model page.

Note that [llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) has no mention of the expected prompt template. I think it is therefore likely that a significant portion of users are currently using the model with a different prompt template and are observing reduced model performance as a consequence.

If `transformers` would provide a standardised way to add prompt templates, I believe this would create an incentive for model creators to add their prompt template. This, combined with an easy way to use said template, would make it easier for users to get the best out of models on Hugging Face Hub.

For the implementation it is probably not necessary to have an entirely new `Auto` module. I'll let the developers be the judge of how to best implement this.<|||||>Hi @vincentmin! We did some internal discussion and we decided this was a great idea. We're still discussing the specifics, but our current plan is to add a `prompt` field to `tokenizer_config.json`. The method that formats conversational prompts is `Tokenizer._build_conversation_input_ids()`, which is called by `ConversationPipeline`. Therefore, we think the `tokenizer_config.json` is the right place to add fields that override the behaviour of the underlying `Tokenizer`.

The specific fields in `prompt` would be class-specific, but for conversational models they would be e.g. `system_message_start`, `system_message_end`, etc. We think breaking up the prompt into string fields will work, and avoids the need to store full templates in the config files. These fields will be read by the tokenizer and used in `_build_conversation_input_ids()` to customize input prompts correctly.

Since `_build_conversation_input_ids()` is currently a private method that we mostly use internally in the `Pipeline` code, we may also look at ways to expose the prompt information through other properties or methods.

WDYT? The details are still flexibile, but we're planning to finalize a concrete plan soon!<|||||>> Hi @vincentmin! We did some internal discussion and we decided this was a great idea. We're still discussing the specifics, but our current plan is to add a `prompt` field to `tokenizer_config.json`. The method that formats conversational prompts is `Tokenizer._build_conversation_input_ids()`, which is called by `ConversationPipeline`. Therefore, we think the `tokenizer_config.json` is the right place to add fields that override the behaviour of the underlying `Tokenizer`.
> 
> The specific fields in `prompt` would be class-specific, but for conversational models they would be e.g. `system_message_start`, `system_message_end`, etc. We think breaking up the prompt into string fields will work, and avoids the need to store full templates in the config files. These fields will be read by the tokenizer and used in `_build_conversation_input_ids()` to customize input prompts correctly.
> 
> Since `_build_conversation_input_ids()` is currently a private method that we mostly use internally in the `Pipeline` code, we may also look at ways to expose the prompt information through other properties or methods.
> 
> WDYT? The details are still flexibile, but we're planning to finalize a concrete plan soon!

@Rocketknight1 How to use `ConversationPipeline`  for llama2 chat？I want to do multi-turn chat. Could you show an example？ My code example :
```
from transformers import AutoTokenizer, LlamaTokenizerFast
from transformers import pipeline, Conversation
import torch

model = "/home/model_zoo/LLM/llama2/Llama-2-7b-chat-hf"

tokenizer = LlamaTokenizerFast.from_pretrained(model)
pipeline = pipeline(
    "conversational",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.float16,
    device_map="auto",
)

conversation_1 = Conversation("Going to the movies tonight - any suggestions?")
conversation_2 = Conversation("What's the last book you have read?")

print(pipeline([conversation_1, conversation_2]))
```
However it can not return normal response. 