Actually works.
Don't know what kind of black magic you did, but it worked! You managed to make NeMo 12b less repetitive.
Not like this fixed the problem, as you still need to reroll replies. But at least now, NeMo can generate different structures!
Still, something is wrong with the way Mistral trains their models. If to compare it to Llama 3, NeMo doesn't want to execute instructions at all.
Like for example, Llama 3 8b can automatically track, update, and generate new quests, events, character cards, and current statuses with custom formatting if explained properly.
NeMo can't do it. Seems like it just sticks to the last context messages rather than the beginning and the end as usual.
The worst part is that it actually can(!) execute them too, but only if the USER orders it to do so directly.
...I feel like Llama 3 12-14b would be more ideal for RP and pseudo-scripting, tbh.
Maybe I'm doing something wrong here?
Found out that you can insert an extra hint of what AI must check and do every reply.
As close to the end of the context as possible... works so far... and it means that NeMo truly likes sticking to the end of context so much.
Like I need to kick it in the ass every time to make it work (β―β΅β‘β²)β―οΈ΅β»ββ»
Thanks for the feedback! I am happy to hear you think it actually does become less repetitive.
Interesting observation about Nemo really liking to just stick to the end of the context instructions, I have several ideas that I want to try that should improve this for the next iteration.