PicoStories-853K

PicoStories-853K is an ultra-compact, "pico-sized" language model based on the Mistral architecture, trained from scratch by Pico-Kittens. It is designed for minimal-resource environments where standard LLMs cannot operate.

Model Details

• Developed by: Pico-Kittens
• Model type: Mistral-based Language Model
• Training Status: Trained from scratch (Not a fine-tune)
• Parameters: 853,120
• Language(s): English
• License: apache-2.0

Capabilities & Limitations

PicoStories-853K is a specialized model for short-form narrative generation. At the 853,120 parameter scale, the model exhibits the following characteristics:

• Syntax: Capable of maintaining basic English sentence structures and punctuation.
• Narrative Flow: Follows a story-like progression, though logic can become surreal or abstract as the generation length increases.
• Vocabulary: Demonstrates "creative" word usage (e.g., "spenty fedful") when navigating low-probability token sequences.
• Coherence: Best suited for short, single-paragraph generations.

Generation Sample

Input: Once upon a time

PicoStories-853K:

Once upon a time there was an old man who lived in a big house with his family. His parents had a very special toy and he loved it so much that everyone loved him.

One day, the old man's family decided to go to the park. As they were walking she saw a new lady, his dad said "All on the ladder for us!" Everyone asked the old lady why she was so excited she didn't like her dad's car.

The old man told her about being so grown-ups at all! He was so impressed by how happy the old kind of lady was. She gave him a spenty fedful way to give him some milk instead!

How to Get Started

import torch
from transformers import MistralForCausalLM, AutoTokenizer

model_id = "PicoKittens/PicoStories-853K"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = MistralForCausalLM.from_pretrained(model_id, torch_dtype=torch.float32, device_map="auto")

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")

output = model.generate(
    **inputs, 
    max_new_tokens=150, 
    do_sample=True, 
    temperature=0.7, 
    top_p=0.9, 
    repetition_penalty=1.2
)

print(tokenizer.decode(output[0], skip_special_tokens=True))