Eliciting Fine-Tuned Transformer Capabilities via Inference-Time
  Techniques
Large language models have transformed natural language processing, yet
supervised fine-tuning (SFT) remains computationally intensive. This paper
formally proves that capabilities acquired through SFT can be approximated by a
base transformer model using inference-time techniques, specifically in-context
learning (ICL), without altering model parameters, under idealized assumptions
including unbounded computational resources and access to the fine-tuning
dataset. We extend these results to practical scenarios with finite context
lengths and partial dataset access. For text generation tasks with fixed output
length l, datasets of size Oleft( m V{varepsilon^2} log
m{delta} right) or, with bounded context, Oleft( l
log V{varepsilon^2} log 1{delta} right) suffice to approximate
fine-tuned behavior across m contexts within error varepsilon, where V
is the vocabulary size and delta is the failure probability. For linear
classification, datasets of size Oleft( d{varepsilon}
right) or, with fixed context, Oleft( 1{varepsilon^2} log
1{delta} right) are sufficient, where d is the input dimension.
Grounded in the Turing completeness of transformers, these results provide a
theoretical foundation for resource-efficient deployment of large language
models, with practical techniques like retrieval-augmented generation bridging
theory to real-world applications.
6848e0b042e4f9106973f280
Asankhaya Sharma
claimed_verified
2025-06-11T08:35:08.045Z
62f32eab52ad88c930bb3f3b
https://cdn-avatars.huggingface.co/v1/production/uploads/1677134945205-62f32eab52ad88c930bb3f3b.png
Asankhaya Sharma
user
codelion
Eliciting Fine-Tuned Transformer Capabilities via Inference-Time
  Techniques
Large language models have transformed natural language processing, yet
supervised fine-tuning (SFT) remains computationally intensive. This paper
formally proves that capabilities acquired through SFT can be approximated by a
base transformer model using inference-time techniques, specifically in-context
learning (ICL), without altering model parameters, under idealized assumptions
including unbounded computational resources and access to the fine-tuning
dataset. We extend these results to practical scenarios with finite context
lengths and partial dataset access. For text generation tasks with fixed output
length l, datasets of size Oleft( m V{varepsilon^2} log
m{delta} right) or, with bounded context, Oleft( l
log V{varepsilon^2} log 1{delta} right) suffice to approximate
fine-tuned behavior across m contexts within error varepsilon, where V
is the vocabulary size and delta is the failure probability. For linear
classification, datasets of size Oleft( d{varepsilon}
right) or, with fixed context, Oleft( 1{varepsilon^2} log
1{delta} right) are sufficient, where d is the input dimension.
Grounded in the Turing completeness of transformers, these results provide a
theoretical foundation for resource-efficient deployment of large language
models, with practical techniques like retrieval-augmented generation bridging
theory to real-world applications.
Transformers can approximate supervised fine-tuning capabilities through in-context learning without altering model parameters, supported by theoretical bounds and practical techniques.
supervised fine-tuning in-context learning base transformer model Turing completeness retrieval-augmented generation text generation linear classification