GFlowNet Training by Policy Gradients
<|reference_start|>GFlowNet Training by Policy Gradients: Generative Flow Networks (GFlowNets) have been shown effective to generate combinatorial objects with desired properties. We here propose a new GFlowNet training framework, with policy-dependent rewards, that bridges keeping flow balance of GFlowNets to optimizing the expected accumulated reward in traditional Reinforcement-Learning (RL). This enables the derivation of new policy-based GFlowNet training methods, in contrast to existing ones resembling value-based RL. It is known that the design of backward policies in GFlowNet training affects efficiency. We further develop a coupled training strategy that jointly solves GFlowNet forward policy training and backward policy design. Performance analysis is provided with a theoretical guarantee of our policy-based GFlowNet training. Experiments on both simulated and real-world datasets verify that our policy-based strategies provide advanced RL perspectives for robust gradient estimation to improve GFlowNet performance.<|reference_end|>
@article{niu2024gflownet,
    title={GFlowNet Training by Policy Gradients},
    author={Puhua Niu, Shili Wu, Mingzhou Fan, Xiaoning Qian},
    journal={arXiv preprint arXiv:2408.05885},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.05885},
    primaryClass={cs.LG stat.ML}
}