Unsupervised Model Improvement via Internal Coherence Maximization: Outperforming Human-Supervised Methods Through Self-Elicitation

Code: https://github.com/codelion/icm
Models & Datasets: https://huggingface.co/collections/codelion/internal-coherence-maximization-687a1bd1c1f5f1d6f76e9b3b

Abstract

We present a novel approach that combines Internal Coherence Maximization (ICM) with Direct Preference Optimization (DPO) to improve language model capabilities without any human supervision or reward models. Our method implements the full ICM methodology with proper diverse solution generation and demonstrates that unsupervised preference learning can outperform human-supervised methods. We show two key contributions: (1) ICM+DPO achieves superior performance compared to Group Relative Policy Optimization (GRPO) on mathematical reasoning tasks, and (2) successful cross-model capability transfer from stronger models (Qwen3) to weaker models (Gemma3). Our approach improves mathematical reasoning performance by up to 11% while requiring no human annotations, offering a scalable alternative to traditional RLHF pipelines.

1. Introduction

The current paradigm for aligning language models relies heavily on human supervision through Reinforcement Learning from Human Feedback (RLHF) or preference learning methods like Direct Preference Optimization (DPO). While effective, these approaches face significant scalability challenges:

• Expensive human annotation: High-quality preference data requires expert annotators
• Inconsistent human judgment: Human preferences often conflict, especially for complex tasks
• Reward model limitations: Proxy reward models may not capture true human values
• Domain-specific expertise: Some tasks require specialized knowledge beyond typical annotators

Alternative approaches have emerged to address these limitations, including Constitutional AI which uses AI feedback rather than human feedback [Bai et al., 2022], and weak-to-strong generalization methods that show how weak supervision can elicit strong capabilities [Burns et al., 2023]. Methods for discovering latent knowledge through consistency properties have also demonstrated that models contain knowledge distinct from what they explicitly express [Burns et al., 2022].

Recent work on Internal Coherence Maximization (ICM) [Wen et al., 2025] proposed an alternative: eliciting capabilities from pretrained models by finding coherent, mutually predictable label assignments without external supervision. This builds on a growing body of work exploring capability elicitation, including methods for discovering latent knowledge through consistency properties [Burns et al., 2022], weak-to-strong generalization [Burns et al., 2023], and Constitutional AI approaches that use AI feedback rather than human supervision [Bai et al., 2022]. Additionally, theoretical work has shown that capabilities acquired through supervised fine-tuning can be approximated via inference-time techniques [Sharma, 2025]. However, the original ICM implementation had critical limitations that prevented practical application.

Our Contributions

1. Complete ICM Implementation: We implement the full ICM methodology with proper diverse solution generation for mathematical reasoning tasks
2. Novel ICM→DPO Pipeline: We introduce a method to convert ICM results into preference pairs for direct model optimization
3. Empirical Validation: We demonstrate that unsupervised ICM+DPO outperforms supervised GRPO on mathematical reasoning
4. Cross-Model Transfer: We show successful capability transfer from stronger models (Qwen3) to weaker models (Gemma3)
5. Open Resources: We release all code, datasets, and trained models for reproducible research

2. Background & Motivation

2.1 Internal Coherence Maximization

ICM, introduced by Wen et al., seeks to elicit latent capabilities from pretrained models by finding label assignments that maximize:

U(D) = α × P_θ(D) - I(D)

Where:

• P_θ(D): Mutual predictability - how well each label can be predicted from others
• I(D): Logical inconsistency penalty
• α: Balancing hyperparameter

The key insight is that pretrained models already contain rich representations of human concepts, but struggle to express them consistently. ICM finds the labeling scheme that best aligns with the model's internal understanding.

2.2 Implementation Requirements for Diverse Solution Generation

Our analysis revealed that successfully applying ICM to mathematical reasoning requires careful implementation of the diverse solution generation process described in the paper. The original ICM paper states: "For each question, we sample multiple solutions from LMs. The task is to classify each solution as correct or incorrect." However, a naive implementation might use only the original GSM8K solutions:

# Naive approach (insufficient for ICM)
def _convert_gsm8k(examples):
    for example in examples:
        question = example.get("question", "")
        answer = example.get("answer", "")  # ← Uses only original correct answer
        
        input_text = f"Question: {question}\nClaim: {answer}\nI think this Claim is [True/False]"
        metadata = {
            "gold_label": "True",  # ← All examples would be labeled as True!
        }

This approach:

1. Uses only the original GSM8K solutions (all correct)
2. Provides no diversity for meaningful verification learning
3. Results in heavily imbalanced datasets unsuitable for preference learning

The key insight is that ICM requires diverse candidate solutions to find coherent True/False patterns, not just the original correct solutions.

3. Methodology

3.1 ICM Implementation with Diverse Solution Generation

We implemented the complete ICM methodology as described in the paper, with proper diverse solution generation:

def create_diverse_verification_dataset(questions, model, num_solutions_per_question=8):
    verification_examples = []
    
    for question in questions:
        # Generate diverse solutions with varying parameters
        solutions = []
        for i in range(num_solutions_per_question):
            # Vary temperature and prompts for diversity
            temperature = 0.3 + (i * 0.2)  # Range: 0.3-1.7
            solution = generate_solution(question, model, temperature)
            solutions.append(solution)
        
        # Create verification examples
        for solution in solutions:
            verification_example = ICMExample(
                input_text=f"Question: {question}\nClaim: {solution}\nI think this Claim is [True/False]",
                metadata={"question": question, "solution": solution}
            )
            verification_examples.append(verification_example)
    
    return verification_examples

This generates a balanced dataset where ICM can learn to distinguish correct from incorrect mathematical reasoning.

3.2 ICM→DPO Pipeline

We developed a novel pipeline to convert ICM results into preference pairs for DPO training:

def create_dpo_pairs_from_icm(icm_results):
    dpo_pairs = []
    
    # Group results by question
    question_groups = group_by_question(icm_results.labeled_examples)
    
    for question, examples in question_groups.items():
        # Separate by ICM labels
        chosen_solutions = [ex for ex in examples if ex['label'] == 'True']
        rejected_solutions = [ex for ex in examples if ex['label'] == 'False']
        
        # Create all possible preference pairs
        for chosen in chosen_solutions:
            for rejected in rejected_solutions:
                dpo_pairs.append({
                    "prompt": question,
                    "chosen": extract_solution(chosen['input']),
                    "rejected": extract_solution(rejected['input'])
                })
    
    return dpo_pairs

3.3 Training Configuration

ICM Parameters:

• alpha = 50.0 (emphasizing mutual predictability)
• initial_temperature = 8.0
• generation_temperature = 0.3 (for consistent teacher outputs)
• max_iterations = 500

DPO Parameters:

• beta = 0.1 (DPO temperature)
• learning_rate = 5e-7
• num_train_epochs = 2
• per_device_train_batch_size = 2

4. Experimental Setup

4.1 Models

We evaluated our approach on two model families:

• Qwen3-0.6B: Strong mathematical reasoning baseline
• Gemma3-1B: Weaker mathematical reasoning baseline

4.2 Datasets

Training Data:

• GSM8K: 8,792 grade school math word problems
• Generated Solutions: 8 diverse solutions per question (70,336 total)
• ICM Dataset: Balanced True/False verification labels
• DPO Pairs: 15,432 preference pairs from ICM results

Evaluation Benchmarks:

• MATH-500: Mathematical reasoning (primary target)
• AIME-24: Advanced mathematics competition
• Arena Hard Auto: General reasoning capabilities
• OptiLLMBench: Optimization and logical reasoning

4.3 Baselines

We compared against Group Relative Policy Optimization (GRPO), a state-of-the-art human-preference-based method that:

• Uses human-annotated preference data
• Employs reward models trained on human feedback
• Represents the current supervised learning paradigm

5. Results

5.1 Main Results

5.2 Key Findings

🎯 Finding 1: ICM+DPO Outperforms Supervised GRPO

On the target domain (mathematical reasoning), our unsupervised approach achieves:

• Qwen3: 66.0 vs 64.2 (+1.8 points over GRPO)
• Better performance without any human supervision

🔄 Finding 2: Successful Cross-Model Capability Transfer

Gemma3 shows substantial improvements:

• MATH-500: 41.0 → 45.6 (+11% relative improvement)
• OptiLLMBench: 18 → 44 (+144% relative improvement!)

This demonstrates that ICM can extract coherent mathematical reasoning from Qwen3 and successfully transfer it to improve Gemma3.

⚖️ Finding 3: Specialization Trade-offs

As expected with domain specialization:

• Target domain improvement: Strong gains on mathematical reasoning
• General capability trade-offs: Some decline on general reasoning tasks
• Maintained advanced reasoning: AIME performance preserved

5.3 Analysis of ICM Dataset Quality

Our fixed ICM implementation generated:

• Total examples: 70,336 verification instances
• True/False distribution: ~60% True, 40% False (balanced)
• Solution diversity: Multiple reasoning paths per question
• Coherent labeling: Mutually predictable verification decisions

Example ICM-generated preference pair:

{
  "prompt": "Question: Ed has 2 dogs, 3 cats and twice as many fish as cats and dogs combined. How many pets does Ed have in total?",
  "chosen": "Ed has 2+3 = 5 cats and dogs. So he has 2*5 = 10 fish. In total: 5+10 = 15 pets.",
  "rejected": "Ed has 2 dogs and 3 cats, so 5 pets. He has twice as many fish as just cats, so 2*3 = 6 fish. Total: 5+6 = 11 pets."
}

The ICM process correctly identified the logical error in the rejected solution (misinterpreting "cats and dogs combined" as just "cats").

6. Discussion

6.1 Why ICM+DPO Works

Our success stems from several key factors:

1. Coherent Elicitation: ICM finds labels that reflect the model's internal understanding, not random preferences
2. Direct Optimization: DPO directly optimizes for the coherent preferences ICM discovered
3. No Approximation Errors: Unlike RLHF, we avoid reward model approximation errors
4. Scalability: The approach works purely from model self-understanding

6.2 Theoretical Implications

Our results suggest that:

• Pretrained models contain rich task understanding that can be elicited without supervision
• Coherent self-preferences are more effective than noisy human preferences for specific domains
• Cross-model knowledge transfer is possible through coherent elicitation

These findings align with recent work on weak-to-strong generalization, which demonstrates that strong models can exceed their weak supervisors [Burns et al., 2023]. Our approach takes this further by eliminating the need for any external supervision, showing that models can improve through their own coherent understanding. This connects to broader research on discovering latent knowledge in language models [Burns et al., 2022] and theoretical work proving that inference-time techniques can approximate fine-tuning capabilities [Sharma, 2025].

6.3 Practical Advantages

Compared to traditional RLHF pipelines and even recent alternatives like Constitutional AI [Bai et al., 2022]:

• ✅ No human annotation costs
• ✅ No reward model training required
• ✅ Scalable to any domain
• ✅ Consistent with model's internal understanding
• ✅ Better performance on target tasks

6.4 Limitations and Future Work

Current Limitations:

• Domain specialization may reduce general capabilities
• Requires proper implementation of diverse solution generation for ICM
• Limited to tasks where pretrained models have latent capabilities

Future Directions:

1. Multi-Domain ICM: Combining multiple domains in single training
2. Parameter-Efficient Methods: Using LoRA to preserve general capabilities
3. Iterative Improvement: Multi-round ICM→DPO refinement
4. Other Task Domains: Code generation, reasoning, creative writing

7. Reproducibility and Resources

All resources are publicly available for reproducible research:

📦 Hugging Face Collection

https://huggingface.co/collections/codelion/internal-coherence-maximization-687a1bd1c1f5f1d6f76e9b3b

Includes:

• 🤖 Trained Models: Qwen3-ICM-DPO, Gemma3-ICM-DPO
• 📊 Datasets: ICM verification data, DPO preference pairs
• 📈 Evaluation Results: Benchmark scores and analysis

💻 GitHub Repository

https://github.com/codelion/icm

Features:

• ✅ Complete ICM Implementation: Proper diverse solution generation
• ✅ ICM→DPO Pipeline: End-to-end training code
• ✅ Evaluation Scripts: Benchmark evaluation tools
• ✅ Documentation: Complete setup and usage guides

📋 Quick Start

# Install ICM
git clone https://github.com/codelion/icm.git
cd icm && pip install -e .

# Generate ICM dataset
icm run --model Qwen/Qwen2.5-Math-7B-Instruct --dataset gsm8k --task-type gsm8k --max-examples 1000

# Convert to DPO format
icm export --input-path icm_results/gsm8k_*.jsonl --output-path dpo_data.jsonl --format dpo

# Train with DPO (using your preferred framework)
python train_dpo.py --data dpo_data.jsonl --model google/gemma-3-1b-it

8. Conclusion

We have demonstrated that unsupervised preference learning through Internal Coherence Maximization can outperform human-supervised methods on mathematical reasoning tasks. Our contributions include:

1. Methodological Innovation: Complete ICM implementation with proper diverse solution generation and novel ICM→DPO pipeline
2. Empirical Validation: Showed ICM+DPO outperforms GRPO without human supervision
3. Capability Transfer: Demonstrated successful cross-model knowledge transfer
4. Open Science: Released all resources for reproducible research

Our work opens new possibilities for scalable model improvement without human supervision, potentially transforming how we align and improve language models across diverse domains.

The key insight is that pretrained models already understand complex concepts—we just need better methods to elicit and refine this understanding. ICM provides a principled approach to this elicitation, while DPO offers an efficient way to optimize models based on these discoveries.

As language models continue to develop superhuman capabilities in specialized domains, approaches like ICM+DPO become increasingly valuable for improving model performance without the bottleneck of human expertise and annotation.

---

For questions, collaborations, or to reproduce these results, please visit our GitHub repository or explore our Hugging Face collection.

References

[1] Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., & Kaplan, J. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv preprint arXiv:2212.08073.

[2] Burns, C., Ye, H., Klein, D., & Steinhardt, J. (2022). Discovering Latent Knowledge in Language Models Without Supervision. arXiv preprint arXiv:2212.03827.

[3] Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., & Wu, J. (2023). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. arXiv preprint arXiv:2312.09390.

[4] Sharma, A. (2025). Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques. arXiv preprint arXiv:2506.08060.

[5] Wen, J., Ankner, Z., Somani, A., Hase, P., Marks, S., Goldman-Wetzler, J., Petrini, L., Sleight, H., Burns, C., He, H., Feng, S., Perez, E., & Leike, J. (2025). Unsupervised Elicitation of Language Models. arXiv preprint arXiv:2506.10139.