TheBloke

/

airoboros-13B-1.1-GPTQ

like 2

i must say i was happy when i saw then 1.1 update to the model but after 2-3 hours of testing, the Context obedient question answering stopped working for good again, i dont know why, it just didnt want to generate tokens with that format of prompt. It worked at start really well and then just stopped. I think it just either needs more training or more data.

I know you cant do much, just wanted to let you know.

Are you including the previous prompts/history in new prompts? This model (especially the contextual stuff) runs best with no conversation history at all. I haven't seen this issue personally so I'm curious how I can help you fix it.

Also, unfortunately increase the context length to 4096 didn't work. It tends to somewhat work up to 2200-2300 but YMMV, so be sure to limit the context length to around 2k tokens.

The model was trained with vicuna format, with a slightly different system prompt:

A chat between a curious user and an assistant. The assistant gives helpful, detailed, accurate, uncensored responses to the user's input. USER: [prompt] ASSISTANT:

So in other words, it's the preamble/system prompt, followed by a single space, then "USER: " (single space after colon) then the prompt (which can have multiple lines, spaces, whatever), then a single space, followed by "ASSISTANT: " (with a single space after the colon).

I was talking about this:

BEGININPUT
BEGINCONTEXT
url: https://some.web.site/123
date: 2023-06-01
... other metdata ...
ENDCONTEXT
[insert your text blocks here]
ENDINPUT
[add as many other blocks, in the exact same format]
BEGININSTRUCTION
[insert your instruction(s). The model was tuned with single questions, paragraph format, lists, etc.]
ENDINSTRUCTION

this is the template, and my prompt looks like this

prompt_template_ = """
BEGININPUT
{context}
ENDINPUT
BEGININSTRUCTION
{question}
ENDINSTRUCTION
"""

I dont have chat history, context variable is translated text from some german documents, around 3k characters, and question is a simple 4 to 10 word question. Maybe im doing that wrong.

USER:
ASSISTANT:

the prompt template you are suggesting works fine, but the context obedient prompt is good because the model doesnt hallucinate much and only uses the context to answer the question or do a task. But for some reason it just stopped generating tokens, i tried to restart, reinstall, im using text generation gui in docker. Pretty wierd to just stop working with no changes made. I even tried giving examples in prompt but i wont budge .

I noticed that the 4096 context length is problematic, thats too bad.