|  | --- | 
					
						
						|  | title: Template-free prompt construction | 
					
						
						|  | description: "Template-free prompt construction with the `input_output` format" | 
					
						
						|  | --- | 
					
						
						|  |  | 
					
						
						|  | <!-- TOC --> | 
					
						
						|  |  | 
					
						
						|  | - [Background]( | 
					
						
						|  | - [Masking Inputs]( | 
					
						
						|  | - [You may not want prompt templates]( | 
					
						
						|  | - [The `input_output` format]( | 
					
						
						|  | - [Usage]( | 
					
						
						|  | - [1. Prepare Data]( | 
					
						
						|  | - [2. Use `type: input_output`]( | 
					
						
						|  | - [3. Check the prompts]( | 
					
						
						|  |  | 
					
						
						|  | <!-- /TOC --> | 
					
						
						|  |  | 
					
						
						|  | <a id="markdown-background" name="background"></a> | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | <a id="markdown-masking-inputs" name="masking-inputs"></a> | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | One of the most popular features of | 
					
						
						|  | [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) is | 
					
						
						|  | setting the following configuration value: | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | ```yaml | 
					
						
						|  | train_on_inputs: false | 
					
						
						|  | ``` | 
					
						
						|  |  | 
					
						
						|  | If you declare a [dataset formats](https://github.com/OpenAccess-AI-Collective/axolotl?tab=readme-ov-file | 
					
						
						|  | such as `alpaca` or `chatml`, axolotl knows what is an input | 
					
						
						|  | (i.e. human) vs. an output (i.e. the assistant) and masks the input | 
					
						
						|  | labels so that your model can focus on predicting the outputs only. | 
					
						
						|  |  | 
					
						
						|  | <a id="markdown-you-may-not-want-prompt-templates" name="you-may-not-want-prompt-templates"></a> | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | However, there are many situations where you don't want to use one of | 
					
						
						|  | these formats or templates. This is because they can: | 
					
						
						|  |  | 
					
						
						|  | -   Add unnecessary boilerplate to your prompts. | 
					
						
						|  | -   Create artifacts like special delimiters `<|im_start|>` that can | 
					
						
						|  | quickly become footguns if you don't include them correctly at | 
					
						
						|  | inference time. | 
					
						
						|  | -   Enforce a *chat* interface when you do not want one. Sometimes you | 
					
						
						|  | just want to fine-tune a model to a very specific task and do NOT | 
					
						
						|  | want multi-turn conversations, roles, etc. | 
					
						
						|  | -   Limit you to only certain roles that the template allows. | 
					
						
						|  |  | 
					
						
						|  | <a id="markdown-the-inputoutput-format" name="the-inputoutput-format"></a> | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | You can construct your prompts without a template by using the | 
					
						
						|  | `input_output` format, by setting `type: input_output` in your | 
					
						
						|  | configuration file like this: | 
					
						
						|  |  | 
					
						
						|  | **config.yml** | 
					
						
						|  |  | 
					
						
						|  | ```yaml | 
					
						
						|  | train_on_inputs: false | 
					
						
						|  | datasets: | 
					
						
						|  | - path: output.jsonl | 
					
						
						|  | type: input_output | 
					
						
						|  | ``` | 
					
						
						|  |  | 
					
						
						|  | Unlike `type: completion`, which is also template-free, | 
					
						
						|  | `type: input_output` allows you to mask segments of your text. More | 
					
						
						|  | details on how this works are described below. | 
					
						
						|  |  | 
					
						
						|  | <a id="markdown-usage" name="usage"></a> | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | This is how you can use the `input_output` format: | 
					
						
						|  |  | 
					
						
						|  | <a id="markdown-1-prepare-data" name="1-prepare-data"></a> | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | To use the `input_output` format, collect your data in the following | 
					
						
						|  | format into a jsonl file (below is the first row from the file | 
					
						
						|  | `output`.jsonl` pretty printed): | 
					
						
						|  |  | 
					
						
						|  | ```bash | 
					
						
						|  | $ head -n1 output.jsonl | python -m json.tool | 
					
						
						|  | ``` | 
					
						
						|  |  | 
					
						
						|  | :::{.cell-output .cell-output-stdout} | 
					
						
						|  | { | 
					
						
						|  | "segments": [ | 
					
						
						|  | { | 
					
						
						|  | "label": true, | 
					
						
						|  | "text": "<s>Hello\n" | 
					
						
						|  | }, | 
					
						
						|  | { | 
					
						
						|  | "label": true, | 
					
						
						|  | "text": "hi there!. " | 
					
						
						|  | }, | 
					
						
						|  | { | 
					
						
						|  | "label": false, | 
					
						
						|  | "text": "goodbye " | 
					
						
						|  | }, | 
					
						
						|  | { | 
					
						
						|  | "label": true, | 
					
						
						|  | "text": "farewell</s>" | 
					
						
						|  | } | 
					
						
						|  | ] | 
					
						
						|  | } | 
					
						
						|  | ::: | 
					
						
						|  |  | 
					
						
						|  | Set `label:false` when you want to mask a segment of text so that the | 
					
						
						|  | model isn't trained on it. Some things to keep in mind: | 
					
						
						|  |  | 
					
						
						|  | > [!IMPORTANT] | 
					
						
						|  | > 1.  **EOS, BOS, spaces, newlines etc. are entirely up to you. Axolotl | 
					
						
						|  | concatenates all the segments as-is.** The tokenizer doesn't add | 
					
						
						|  | anything additional. Notice how I added spaces, newlines, `<s>` | 
					
						
						|  | (BOS), and `</s>` (EOS) myself. | 
					
						
						|  | > 2.  Make sure you check the materialized output to validate that the | 
					
						
						|  | prompt is getting assembled how you like. | 
					
						
						|  |  | 
					
						
						|  | <a id="markdown-2-use-type-inputoutput" name="2-use-type-inputoutput"></a> | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | Let's materialize data with our `output.jsonl` file by setting | 
					
						
						|  | `type: input_output` in our axolotl config: | 
					
						
						|  |  | 
					
						
						|  | ```yaml | 
					
						
						|  | # training_config.yaml | 
					
						
						|  | base_model: mistralai/Mistral-7B-v0.1 | 
					
						
						|  | data_seed: 49 | 
					
						
						|  | seed: 49 | 
					
						
						|  |  | 
					
						
						|  | datasets: | 
					
						
						|  | - path: output.jsonl | 
					
						
						|  | type: input_output | 
					
						
						|  | val_set_size: 0.1 | 
					
						
						|  |  | 
					
						
						|  | sequence_len: 896 | 
					
						
						|  | sample_packing: false | 
					
						
						|  |  | 
					
						
						|  | micro_batch_size: 2 | 
					
						
						|  | gradient_accumulation_steps: 3 | 
					
						
						|  | eval_batch_size: 2 | 
					
						
						|  | num_epochs: 1 | 
					
						
						|  | learning_rate: 0.0002 | 
					
						
						|  |  | 
					
						
						|  | train_on_inputs: false | 
					
						
						|  | special_tokens: | 
					
						
						|  | bos_token: "<s>" | 
					
						
						|  | eos_token: "</s>" | 
					
						
						|  | unk_token: "<unk>" | 
					
						
						|  | ``` | 
					
						
						|  |  | 
					
						
						|  | You can use the following command to materialize your data. The | 
					
						
						|  | `--debug` flag will print the tokens, along with the labels so you can | 
					
						
						|  | verify that the correct items are being ignored: | 
					
						
						|  |  | 
					
						
						|  | ```bash | 
					
						
						|  | $ python -m axolotl.cli.preprocess training_config.yaml --debug | 
					
						
						|  |  | 
					
						
						|  | ... | 
					
						
						|  | [2024-03-05 23:36:46,969] [INFO] [axolotl.check_example_labels:35] [PID:607731] [RANK:0] <s>(1, 1) Hello(22557, 22557) | 
					
						
						|  | (13, 13) hi(12014, 12014) there(736, 736) !(28808, 28808) .(28723, 28723) (28705, 28705) good(-100, 1179) bye(-100, 17664) (-100, 28705) fare(19111, 19111) well(5458, 5458) </s>(2, 2) | 
					
						
						|  |  | 
					
						
						|  | ``` | 
					
						
						|  |  | 
					
						
						|  | The format is `decoded_token`(`label`, `token_id`), for example, | 
					
						
						|  | `<s>(1, 1)` means that the token is `<s>`, the label is `1` and the | 
					
						
						|  | token_id is `1`. When the label is `-100` then that token is ignored for | 
					
						
						|  | training. | 
					
						
						|  |  | 
					
						
						|  | <a id="markdown-3-check-the-prompts" name="3-check-the-prompts"></a> | 
					
						
						|  |  | 
					
						
						|  | ### 3. Check the prompts | 
					
						
						|  |  | 
					
						
						|  | Here is another way to check the materialized output: | 
					
						
						|  |  | 
					
						
						|  | ```python | 
					
						
						|  | from transformers import AutoTokenizer | 
					
						
						|  | from datasets import load_from_disk | 
					
						
						|  | import yaml | 
					
						
						|  |  | 
					
						
						|  | directory = !ls last_run_prepared/ | 
					
						
						|  | with open('training_config.yaml', 'r') as f: | 
					
						
						|  | cfg = yaml.safe_load(f) | 
					
						
						|  | model_id = cfg['base_model'] | 
					
						
						|  | tok = AutoTokenizer.from_pretrained(model_id) | 
					
						
						|  | ds = load_from_disk(f'last_run_prepared/{directory[0]}/') | 
					
						
						|  | ``` | 
					
						
						|  |  | 
					
						
						|  | ```python | 
					
						
						|  | >>> row = ds[0] | 
					
						
						|  | >>> print(tok.decode(row['input_ids'])) | 
					
						
						|  | <s> Hello | 
					
						
						|  | hi there!.  goodbye  farewell</s> | 
					
						
						|  | ``` | 
					
						
						|  |  | 
					
						
						|  | We can check that the right tokens are ingored by comparing the labels | 
					
						
						|  | to each token: | 
					
						
						|  |  | 
					
						
						|  | ```python | 
					
						
						|  | import pandas as pd | 
					
						
						|  | pd.DataFrame([{'token': tok.decode(i), 'label': l, 'id':i} for i,l in | 
					
						
						|  | zip(row['input_ids'], row['labels'])]) | 
					
						
						|  | ``` | 
					
						
						|  |  | 
					
						
						|  | | token | label | id    | | 
					
						
						|  | |-------|-------|-------| | 
					
						
						|  | | 0     | \<s\> | 1     | | 
					
						
						|  | | 1     | Hello | 22557 | | 
					
						
						|  | | 2     | \\n   | 13    | | 
					
						
						|  | | 3     | hi    | 12014 | | 
					
						
						|  | | 4     | there | 736   | | 
					
						
						|  | | 5     | !     | 28808 | | 
					
						
						|  | | 6     | .     | 28723 | | 
					
						
						|  | | 7     |       | 28705 | | 
					
						
						|  | | 8     | good  | -100  | | 
					
						
						|  | | 9     | bye   | -100  | | 
					
						
						|  | | 10    |       | -100  | | 
					
						
						|  | | 11    | fare  | 19111 | | 
					
						
						|  | | 12    | well  | 5458  | | 
					
						
						|  | | 13    | \</s\>| 2     | | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | If we look at the input data, the above table seems correct! (The jsonl | 
					
						
						|  | version is repeated below for reference): | 
					
						
						|  |  | 
					
						
						|  |  | 
					
						
						|  | ```bash | 
					
						
						|  | $ head -n1 output.jsonl | python -m json.tool | 
					
						
						|  | ``` | 
					
						
						|  |  | 
					
						
						|  | :::{.cell-output .cell-output-stdout} | 
					
						
						|  | { | 
					
						
						|  | "segments": [ | 
					
						
						|  | { | 
					
						
						|  | "label": true, | 
					
						
						|  | "text": "<s>Hello\n" | 
					
						
						|  | }, | 
					
						
						|  | { | 
					
						
						|  | "label": true, | 
					
						
						|  | "text": "hi there!. " | 
					
						
						|  | }, | 
					
						
						|  | { | 
					
						
						|  | "label": false, | 
					
						
						|  | "text": "goodbye " | 
					
						
						|  | }, | 
					
						
						|  | { | 
					
						
						|  | "label": true, | 
					
						
						|  | "text": "farewell</s>" | 
					
						
						|  | } | 
					
						
						|  | ] | 
					
						
						|  | } | 
					
						
						|  | ::: | 
					
						
						|  |  |