eperformthefollowingstepsineachpolicyupdate:•UsingseveralepochsofminibatchSGD,optimizetheKL-penalizedobjectiveLKLPEN(θ)=ˆEt(cid:20)πθ(at|st)πθold(at|st)ˆAt−βKL[πθold(·|st),πθ(·|st)](cid:21)(8)•Computed=ˆEt[KL[πθold(·|st),πθ(·|st)]]–Ifd<dtarg/1.5,β←β/2–Ifd>dtarg×1.5,β←β×2Theupdatedβisusedforthenextpolicyupdate.Withthisscheme,weoccasionallyseepolicyupdateswheretheKLdivergenceissigniﬁcantlydiﬀerentfromdtarg,however,thesearerare,andβquicklyadjusts.Theparameters1.5and2abovearechosenheuristically,butthealgorithmisnotverysensitivetothem.Theinitialvalueofβisaanotherhyperparameterbutisnotimportantinpracticebecausethealgorithmquicklyadjustsit.5AlgorithmThesurrogatelossesfromtheprevioussectionscanbecomputedanddiﬀerentiatedwithaminorchangetoatypicalpolicygradientimplementation.Forimplementationsthatuseautomaticdif-ferentation,onesimplyconstructsthelossLCLIPorLKLPENinsteadofLPG,andoneperformsmultiplestepsofstochasticgradientascentonthisobjective.Mosttechniquesforcomputingvariance-reducedadvantage-f
PPO