ernie-research/HH-RLHF-Gemma-2B-MA-PPO-Fixed5

Model Card for MA-RLHF

This repository contains the official checkpoint for Reinforcement Learning From Human Feedback with Macro Actions (MA-RLHF).

Model Description

MA-RLHF is a novel framework that integrates macro actions into conventional RLHF. The macro actions are sequences of tokens or higher-level language constructs, with can be computed through different defined termination conditions, like n-gram based, perplexity-based, or parsing-based termination conditions. By introducing macro actions into RLHF, we reduce the number of decision points and shorten decision trajectories, alleviating the credit assignment problem caused by long temporal distances.

Model	Checkpoint	Base Model	Dataset
TLDR-Gemma-2B-MA-PPO-Fixed5	🤗 HF Link	google/gemma-2b	openai/summarize_from_feedback
TLDR-Gemma-7B-MA-PPO-Fixed5	🤗 HF Link	google/gemma-7b	openai/summarize_from_feedback
TLDR-Gemma-2-27B-MA-PPO-Fixed5	🤗 HF Link	google/gemma-2-27b	openai/summarize_from_feedback
HH-RLHF-Gemma-2B-MA-PPO-Fixed5	🤗 HF Link	google/gemma-2b	Dahoas/full-hh-rlhf
HH-RLHF-Gemma-7B-MA-PPO-Fixed5	🤗 HF Link	google/gemma-7b	Dahoas/full-hh-rlhf
APPS-Gemma-2B-MA-PPO-Fixed10	🤗 HF Link	google/codegemma-2b	codeparrot/apps
APPS-Gemma-7B-MA-PPO-Fixed10	🤗 HF Link	google/codegemma-7b-it	codeparrot/apps

Model Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "baidu/HH-RLHF-Gemma-2B-MA-PPO-Fixed5"

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto", torch_dtype='auto', trust_remote_code=True)

input_text = """
Human: Would you be able to explain the differences between the Spanish
and Italian language? Assistant: Of course. Can you tell me more about
the specific areas where you’re interested in knowing more? Human: I’m
thinking between the Spanish spoken in Mexico and Italian spoken in Italy.
Assistant: 
"""

input_ids = tokenizer(input_text, return_tensors='pt').to(model.device)
output_ids = model.generate(**input_ids, max_new_tokens=20)
response = tokenizer.decode(output_ids[0], skip_special_tokens=True)

print(response)

Citation

@inproceedings{
  chai2025marlhf,
  title={{MA}-{RLHF}: Reinforcement Learning from Human Feedback with Macro Actions},
  author={Yekun Chai and Haoran Sun and Huang Fang and Shuohuan Wang and Yu Sun and Hua Wu},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=WWXjMYZxfH}
}