Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema
  Adherence
In this paper, we address the challenge of enforcing strict schema adherence
in large language model (LLM) generation by leveraging LLM reasoning
capabilities. Building on the DeepSeek R1 reinforcement learning framework, our
approach trains structured reasoning skills of a 1.5B parameter model through a
novel pipeline that combines synthetic reasoning dataset construction with
custom reward functions under Group Relative Policy Optimization (GRPO).
Specifically, we first perform R1 reinforcement learning on a 20K sample
unstructured-to-structured dataset, mirroring the original DeepSeek R1 methods,
to establish core reasoning abilities. Subsequently, we performed supervised
fine-tuning on a separate 10K reasoning sample dataset, focusing on refining
schema adherence for downstream tasks. Despite the relatively modest training
scope, requiring approximately 20 hours on an 8xH100 GPU cluster for GRPO
training and 3 hours on 1xA100 for SFT, our model demonstrates robust
performance in enforcing schema consistency. We compare our ThinkJSON approach
against the original DeepSeek R1 (671B), distilled versions of DeepSeek R1
(Qwen-1.5B and Qwen-7B), and Gemini 2.0 Flash (70B), showcasing its
effectiveness in real-world applications. Our results underscore the practical
utility of a resource-efficient framework for schema-constrained text
generation.
67bbe0520aabd5d571a723e7
Bhavik Agarwal
claimed_verified
2025-02-24T16:38:48.442Z
6739c3675d150a0c7e3c0014
/avatars/87a0d9d39b0c854c467a3cdd46fa0ce1.svg
Bhavik Agarwal
user
bhaviktheslider
67bbe0520aabd5d571a723e8
Ishan Joshi
admin_assigned
2025-02-24T15:55:31.357Z
647e85b6e4d52fe0e0210718
/avatars/ee9ed51b64486b898ce0b58b22db32d5.svg
Ishan Joshi
user
IshanJoshi
67bbe0520aabd5d571a723e9
Viktoria Rojkova
admin_assigned
2025-02-24T15:55:40.802Z
62c1b81e8b647bdc24f78027
/avatars/b994d31a0c4161f715bb2153c0f0a83f.svg
Viktoria Rojkova
user
vrojkova
Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema
  Adherence
In this paper, we address the challenge of enforcing strict schema adherence
in large language model (LLM) generation by leveraging LLM reasoning
capabilities. Building on the DeepSeek R1 reinforcement learning framework, our
approach trains structured reasoning skills of a 1.5B parameter model through a
novel pipeline that combines synthetic reasoning dataset construction with
custom reward functions under Group Relative Policy Optimization (GRPO).
Specifically, we first perform R1 reinforcement learning on a 20K sample
unstructured-to-structured dataset, mirroring the original DeepSeek R1 methods,
to establish core reasoning abilities. Subsequently, we performed supervised
fine-tuning on a separate 10K reasoning sample dataset, focusing on refining
schema adherence for downstream tasks. Despite the relatively modest training
scope, requiring approximately 20 hours on an 8xH100 GPU cluster for GRPO
training and 3 hours on 1xA100 for SFT, our model demonstrates robust
performance in enforcing schema consistency. We compare our ThinkJSON approach
against the original DeepSeek R1 (671B), distilled versions of DeepSeek R1
(Qwen-1.5B and Qwen-7B), and Gemini 2.0 Flash (70B), showcasing its
effectiveness in real-world applications. Our results underscore the practical
utility of a resource-efficient framework for schema-constrained text
generation.
ThinkJSON reinforces schema adherence in large language models using a novel reinforcement learning pipeline and supervised fine-tuning, demonstrating robust performance compared to larger models.
DeepSeek R1 reinforcement learning structured reasoning synthetic reasoning dataset custom reward functions Group Relative Policy Optimization (GRPO) supervised fine-tuning schema adherence ThinkJSON DeepSeek R1 Qwen-1.5B Qwen-7B Gemini 2.0 Flash