class trl.WinRateCallbacktrl.WinRateCallbackhttps://github.com/huggingface/trl/blob/v0.23.1/trl/trainer/callbacks.py#L229[{"name": "judge", "val": ": BasePairwiseJudge"}, {"name": "trainer", "val": ": Trainer"}, {"name": "generation_config", "val": ": typing.Optional[transformers.generation.configuration_utils.GenerationConfig] = None"}, {"name": "num_prompts", "val": ": typing.Optional[int] = None"}, {"name": "shuffle_order", "val": ": bool = True"}, {"name": "use_soft_judge", "val": ": bool = False"}]- **judge** (`BasePairwiseJudge`) -- The judge to use for comparing completions. - **trainer** (`Trainer`) -- Trainer to which the callback will be attached. The trainer's evaluation dataset must include a `"prompt"` column containing the prompts for generating completions. If the `Trainer` has a reference model (via the `ref_model` attribute), it will use this reference model for generating the reference completions; otherwise, it defaults to using the initial model. - **generation_config** (`GenerationConfig`, *optional*) -- The generation config to use for generating completions. - **num_prompts** (`int` or `None`, *optional*, defaults to `None`) -- The number of prompts to generate completions for. If not provided, defaults to the number of examples in the evaluation dataset. - **shuffle_order** (`bool`, *optional*, defaults to `True`) -- Whether to shuffle the order of the completions before judging. - **use_soft_judge** (`bool`, *optional*, defaults to `False`) -- Whether to use a soft judge that returns a win probability between 0 and 1 for the first completion vs the second.0 A [TrainerCallback](https://huggingface.co/docs/transformers/v4.57.0/en/main_classes/callback#transformers.TrainerCallback) that computes the win rate of a model based on a reference. It generates completions using prompts from the evaluation dataset and compares the trained model's outputs against a reference. The reference is either the initial version of the model (before training) or the reference model, if available in the trainer. During each evaluation step, a judge determines how often the trained model's completions win against the reference using a judge. The win rate is then logged in the trainer's logs under the key `"eval_win_rate"`. Usage: ```python trainer = DPOTrainer(...) judge = PairRMJudge() win_rate_callback = WinRateCallback(judge=judge, trainer=trainer) trainer.add_callback(win_rate_callback) ```

class trl.LogCompletionsCallbacktrl.LogCompletionsCallbackhttps://github.com/huggingface/trl/blob/v0.23.1/trl/trainer/callbacks.py#L425[{"name": "trainer", "val": ": Trainer"}, {"name": "generation_config", "val": ": typing.Optional[transformers.generation.configuration_utils.GenerationConfig] = None"}, {"name": "num_prompts", "val": ": typing.Optional[int] = None"}, {"name": "freq", "val": ": typing.Optional[int] = None"}]- **trainer** (`Trainer`) -- Trainer to which the callback will be attached. The trainer's evaluation dataset must include a `"prompt"` column containing the prompts for generating completions. - **generation_config** (`GenerationConfig`, *optional*) -- The generation config to use for generating completions. - **num_prompts** (`int` or `None`, *optional*) -- The number of prompts to generate completions for. If not provided, defaults to the number of examples in the evaluation dataset. - **freq** (`int` or `None`, *optional*) -- The frequency at which to log completions. If not provided, defaults to the trainer's `eval_steps`.0 A [TrainerCallback](https://huggingface.co/docs/transformers/v4.57.0/en/main_classes/callback#transformers.TrainerCallback) that logs completions to Weights & Biases and/or Comet. Usage: ```python trainer = DPOTrainer(...) completions_callback = LogCompletionsCallback(trainer=trainer) trainer.add_callback(completions_callback) ```

class trl.MergeModelCallbacktrl.MergeModelCallbackhttps://github.com/huggingface/trl/blob/v0.23.1/trl/trainer/callbacks.py#L517[{"name": "merge_config", "val": ": typing.Optional[ForwardRef('MergeConfig')] = None"}, {"name": "merge_at_every_checkpoint", "val": ": bool = False"}, {"name": "push_to_hub", "val": ": bool = False"}]- **merge_config** (`MergeConfig`, *optional*, defaults to `None`) -- Configuration used for the merging process. If not provided, the default `MergeConfig` is used. - **merge_at_every_checkpoint** (`bool`, *optional*, defaults to `False`) -- Whether to merge the model at every checkpoint. - **push_to_hub** (`bool`, *optional*, defaults to `False`) -- Whether to push the merged model to the Hub after merging.0 A [TrainerCallback](https://huggingface.co/docs/transformers/v4.57.0/en/main_classes/callback#transformers.TrainerCallback) that merges the policy model (the model being trained) with another model based on a merge configuration. Example: ```python from trl.mergekit_utils import MergeConfig from trl import MergeModelCallback config = MergeConfig() merge_callback = MergeModelCallback(config) trainer = DPOTrainer(..., callbacks=[merge_callback]) ```

class trl.BEMACallbacktrl.BEMACallbackhttps://github.com/huggingface/trl/blob/v0.23.1/trl/trainer/callbacks.py#L578[{"name": "update_freq", "val": ": int = 400"}, {"name": "ema_power", "val": ": float = 0.5"}, {"name": "bias_power", "val": ": float = 0.2"}, {"name": "lag", "val": ": int = 10"}, {"name": "update_after", "val": ": int = 0"}, {"name": "multiplier", "val": ": float = 1.0"}, {"name": "min_ema_multiplier", "val": ": float = 0.0"}, {"name": "device", "val": ": str = 'cpu'"}]- **update_freq** (`int`, *optional*, defaults to `400`) -- Update the BEMA weights every X steps. Denoted this as \$ \phi \$ in the paper. - **ema_power** (`float`, *optional*, defaults to `0.5`) -- Power for the EMA decay factor. Denoted \$ \kappa \$ in the paper. To disable EMA, set this to `0.0`. - **bias_power** (`float`, *optional*, defaults to `0.2`) -- Power for the BEMA scaling factor. Denoted \$ \eta \$ in the paper. To disable BEMA, set this to `0.0`. - **lag** (`int`, *optional*, defaults to `10`) -- Initial offset in the weight decay schedule that controls early-stage smoothness by acting as a virtual starting age for the updates. Denoted as \$ \rho \$ in the paper. - **update_after** (`int`, *optional*, defaults to `0`) -- Burn-in time before starting to update the BEMA weights. Denoted \$ \tau \$ in the paper. - **multiplier** (`float`, *optional*, defaults to `1.0`) -- Initial value for the EMA decay factor. Denoted as \$ \gamma \$ in the paper. - **min_ema_multiplier** (`float`, *optional*, defaults to `0.0`) -- Minimum value for the EMA decay factor. - **device** (`str`, *optional*, defaults to `"cpu"`) -- Device to use for the BEMA buffers, e.g. `"cpu"` or `"cuda"`. Note that in most cases, this device SHOULD BE DIFFERENT from the device used for training in order to avoid OOM.0 A [TrainerCallback](https://huggingface.co/docs/transformers/v4.57.0/en/main_classes/callback#transformers.TrainerCallback) that implements [BEMA](https://huggingface.co/papers/2508.00180) (Bias-Corrected Exponential Moving Average) by [Adam Block](https://huggingface.co/abblock) and [Cyril Zhang](https://huggingface.co/cyrilzhang). Code from https://github.com/abblock/bema under MIT license. BEMA computes model weights that scale like: $$ \theta_t' = \alpha_t \cdot (\theta_t - \theta_0) + \text{EMA}_t $$ where \$ \theta_t \$ is the current model weights, \$ \theta_0 \$ is a snapshot of the model weights at the first `update_after` step, \$ \text{EMA}_t \$ is the exponential moving average of the model weights, and \$ \alpha_t \$ is a scaling factor that decays with the number of steps \$ t \$ as $$ \alpha_t = (\rho + \gamma \cdot t)^{-\eta}. $$ The EMA is computed as: $$ \text{EMA}_t = (1 - \beta_t) \cdot \text{EMA}_{t-1} + \beta_t \cdot \theta_t $$ where \$ \beta_t \$ is a decay factor that decays with the number of steps \$ t \$ as $$ \beta_t = (\rho + \gamma \cdot t)^{-\kappa}. $$ Example: ```python from trl import BEMACallback trainer = Trainer(..., callbacks=[BEMACallback()]) ```