ProximalPolicyOptimizationAlgorithmsJohnSchulman,FilipWolski,PrafullaDhariwal,AlecRadford,OlegKlimovOpenAI{joschu,filip,prafulla,alec,oleg}@openai.comAbstractWeproposeanewfamilyofpolicygradientmethodsforreinforcementlearning,whichal-ternatebetweensamplingdatathroughinteractionwiththeenvironment,andoptimizinga“surrogate”objectivefunctionusingstochasticgradientascent.Whereasstandardpolicygra-dientmethodsperformonegradientupdateperdatasample,weproposeanovelobjectivefunctionthatenablesmultipleepochsofminibatchupdates.Thenewmethods,whichwecallproximalpolicyoptimization(PPO),havesomeofthebeneﬁtsoftrustregionpolicyoptimiza-tion(TRPO),buttheyaremuchsimplertoimplement,moregeneral,andhavebettersamplecomplexity(empirically).OurexperimentstestPPOonacollectionofbenchmarktasks,includ-ingsimulatedroboticlocomotionandAtarigameplaying,andweshowthatPPOoutperformsotheronlinepolicygradientmethods,andoverallstrikesafavorablebalancebetweensamplecomplexity,simplicity,andwall-time.1IntroductionInrecentyears,s
PPO