|
|
|
|
|
\documentclass{article} |
|
|
\usepackage{iclr2023_conference,times} |
|
|
|
|
|
|
|
|
\input{math_commands.tex} |
|
|
|
|
|
\usepackage{url} |
|
|
|
|
|
\usepackage[utf8]{inputenc} |
|
|
\usepackage[T1]{fontenc} |
|
|
\usepackage{hyperref} |
|
|
\usepackage{url} |
|
|
\usepackage{booktabs} |
|
|
\usepackage{amsfonts} |
|
|
\usepackage{nicefrac} |
|
|
\usepackage{microtype} |
|
|
\usepackage{xcolor} |
|
|
\usepackage{mathtools} |
|
|
|
|
|
\usepackage{amssymb} |
|
|
\usepackage{amsthm} |
|
|
\usepackage{amsmath} |
|
|
\newtheorem{theorem}{Theorem}[section] |
|
|
\newtheorem{proposition}{Proposition} |
|
|
\newtheorem{lemma}{Lemma} |
|
|
\newtheorem{corollary}{Corollary} |
|
|
|
|
|
\newtheorem{definition}{Definition} |
|
|
\newtheorem{assumption}{Assumption} |
|
|
|
|
|
\newtheorem{remark}{Remark} |
|
|
\usepackage{bbm} |
|
|
\usepackage[linesnumbered,ruled,vlined]{algorithm2e} |
|
|
\usepackage{algorithmic} |
|
|
|
|
|
\SetKwInput{KwInput}{Input} |
|
|
\SetKwInput{KwOutput}{Output} |
|
|
|
|
|
|
|
|
|
|
|
\usepackage{mathtools} |
|
|
\usepackage{listings} |
|
|
\usepackage{caption} |
|
|
\usepackage{subcaption} |
|
|
\usepackage{float} |
|
|
\usepackage{enumitem} |
|
|
\newcommand{\jin}[1]{\textbf{\textcolor{black}{[MJ: #1]}}} |
|
|
\newcommand{\VK}[1]{\textcolor{black}{#1}} |
|
|
\newcommand{\hl}[1]{\textcolor{black}{#1}} |
|
|
|
|
|
\newcommand{\yuhao}[1]{\textbf{\textcolor{red}{[Yuhao: #1]}}} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\newcommand{\edit}[1]{{\color{blue}{#1}}} |
|
|
|
|
|
\title{A CMDP-within-online framework for Meta-Safe Reinforcement Learning} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\author{Vanshaj Khattar \\ |
|
|
Virginia Tech\\ |
|
|
Blacksburg, VA 24061\\ |
|
|
\texttt{[email protected]} \\ |
|
|
\And |
|
|
Yuhao Ding \\ |
|
|
UC Berkeley \\ |
|
|
Berkeley, CA 94709 \\ |
|
|
\texttt{yuhao\textunderscore [email protected]} \\ |
|
|
\And |
|
|
Bilgehan Sel \\ |
|
|
Virginia Tech\\ |
|
|
Blacksburg, VA 24061\\ |
|
|
\texttt{[email protected]}\\ |
|
|
\AND |
|
|
Javad Lavaei\\ |
|
|
UC Berkeley\\ |
|
|
Berkeley, CA 94709\\ |
|
|
\texttt{[email protected]}\\ |
|
|
\And |
|
|
Ming Jin \thanks{Corresponding author}\\ |
|
|
Virginia Tech\\ |
|
|
Blacksburg, VA 24061\\ |
|
|
\texttt{[email protected]} |
|
|
} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\newcommand{\fix}{\marginpar{FIX}} |
|
|
\newcommand{\new}{\marginpar{NEW}} |
|
|
|
|
|
\iclrfinalcopy |
|
|
\begin{document} |
|
|
|
|
|
|
|
|
\maketitle |
|
|
|
|
|
\begin{abstract} |
|
|
Meta-reinforcement learning has widely been used as a learning-to-learn framework to solve unseen tasks with limited experience. However, the aspect of constraint violations has not been adequately addressed in the existing works, making their application restricted in real-world settings. In this paper, we study the problem of meta-safe reinforcement learning (Meta-SRL) through the CMDP-within-online framework to establish the \emph{first provable guarantees} in this important setting. We obtain task-averaged regret bounds for the reward maximization (optimality gap) and constraint violations using gradient-based meta-learning and show that the task-averaged optimality gap and constraint satisfaction improve with task-similarity in a static environment or task-relatedness in a dynamic environment. Several technical challenges arise when making this framework practical. To this end, we propose a meta-algorithm that performs inexact online learning on the upper bounds of within-task optimality gap and constraint violations estimated by off-policy stationary distribution corrections. Furthermore, we enable the learning rates to be adapted for every task and extend our approach to settings with a competing dynamically changing oracle. Finally, experiments are conducted to demonstrate the effectiveness of our approach. |
|
|
\end{abstract} |
|
|
|
|
|
\section{Introduction} |
|
|
\label{sec:Introduction} |
|
|
|
|
|
The field of meta-reinforcement learning (meta-RL) has recently evolved as one of the promising directions that enables reinforcement learning (RL) agents to learn quickly in dynamically changing environments \citep{finn2017model,mitchell2021offline,zintgraf2021exploration}. |
|
|
|
|
|
Many real-world applications, nevertheless, have safety constraints that should rarely be violated, which existing works do not fully address. |
|
|
|
|
|
|
|
|
Safe RL problems are often modeled as constrained Markov decision processes (CMDPs), where the agent aims to maximize the value function while satisfying given constraints on the trajectory \citep{altman1999constrained}. However, unlike meta-learning, CMDP algorithms are not designed to generalize efficiently over unseen tasks \citep{paternain2022safe,ding2021provably,ding2022provably} |
|
|
|
|
|
In this paper, we study how meta-learning can be principally designed to help safe RL algorithms adapt quickly while satisfying safety constraints. |
|
|
|
|
|
|
|
|
There are several unique challenges involved in meta-learning for the CMDP settings. First, multiple losses are incurred at each time step, i.e., reward and constraints, which are typically nonconvex and coupled through dynamics. Hence, adapting existing theories developed for stylized settings such as online convex optimization \citep{hazan2016introduction} |
|
|
|
|
|
is not straightforward. Second, it is unrealistic to assume the computation of a globally optimal policy for CMDPs (unlike online learning \citep{hazan2016introduction}). |
|
|
|
|
|
Thus, classical online learning algorithms that assume exact or unbiased estimator of the loss function do not apply \citep{khodak2019adaptive}. |
|
|
|
|
|
Overall, there is an interplay among nonconvexity, the stochastic nature of the optimization problem, as well as algorithm and generalization considerations, posing significant complexity to leverage inter-task dependency \citep{denevi2019learning}. |
|
|
|
|
|
|
|
|
To this end, we propose a provably low-regret online learning framework that extends the current meta-learning algorithms to safe RL settings. |
|
|
|
|
|
|
|
|
Our main contributions are as follows: |
|
|
\begin{enumerate}[wide, labelindent=0pt] |
|
|
\item \textbf{Inexact CMDP-within-online framework:} We propose a novel CMDP-within-online framework where the within-task is CMDP, and the meta-learner aims to learn the meta-initialization and learning rate. In our framework, the meta-learner only requires the inexact optimal policies for each within-task CMDP and the approximate state visitation distributions estimated using collected offline trajectories to construct the upper bounds on the suboptimality gap and constraint violations. An upper bound on these estimation errors is established in Theorem \ref{thm:dualDICE}. |
|
|
|
|
|
\item \textbf{Task-averaged regret in terms of empirical task-similarity:} We show that the task-averaged regrets for optimality gap (TAOG) and constraint violations (TACV) (Def. \ref{def:taog}) diminish with respect to both the number of steps in the within-task algorithm $M$ and the number of tasks $T$. Specifically, task-averaged regret of $\mathcal{O} \left(\frac{1}{\sqrt{M}}\sqrt{\frac{\mathcal{E}_T}{\sqrt{T}}+ \hat{D}^{*2} } \right)$ holds, where $\mathcal{E}_T$ is the total inexactness in online learning and $\hat{D}^*$ is the empirical task-similarity (Theorem \ref{thm:InexactTAOG}). |
|
|
|
|
|
\item \textbf{Adapting to a dynamic environment:} We adapt the learning rates for each task to \hl{environments that entail dynamically changing meta-initialization policies.} An improved rate of $\mathcal{O}\left(\frac{1}{M^{3/4}\sqrt{T}}\left(\mathcal{E}_T+ \sqrt{\frac{\mathcal{E}_T}{T} +\hat{V}_\psi^2 } \right) \right)$ for TAOG and TACV are shown, where $\hat{V}_\psi$ is the empirical task-relatedness with respect to a sequence of changing comparator policies $\{\psi_t^*\}_{t=1}^T$ (Corollary \ref{cor:CorollaryAdpativeRate}). |
|
|
\end{enumerate} |
|
|
|
|
|
Incorporating all these components makes our Meta-safe RL (Meta-SRL) approach highly practical and theoretically appealing for potential adaption to different RL settings. \VK{Furthermore, we remark on some {key technical contributions} that support the above developments, which may be of independent interest:} \emph{1)} \VK{We study the \emph{optimization landscape} of CMDP (Theorem |
|
|
\ref{thm:dualDICE}) that is algorithmic-agnostic, which differs from the existing work of \citep{mei2020global}[Lemmas 3 and 15] that is restricted to the setting of policy gradient. This is achieved by developing {new techniques} based on tame geometry and subgradient flow systems}; \emph{2)} \VK{we provide static and dynamic regret bounds for \emph{inexact online gradient descent} (see Appendix \ref{sec:inexact-app}), which we leverage to obtain our final theoretical results in Theorems \ref{thm:InexactTAOG}, \ref{thm:UtSim1}, and Corollary \ref{cor:CorollaryAdpativeRate}.} Due to the space restrictions, the related work can be found in Appendix \ref{sec:RelatedWork}. |
|
|
|
|
|
|
|
|
|
|
|
\section{CMDP-within-online framework} |
|
|
\label{sec:Preliminaries} |
|
|
|
|
|
In this section, we introduce the CMDP-within-online framework for the Meta-SRL problems. In this framework, a within-task algorithm (such as CRPO \citep{xu2021crpo}) for some CMDP task $t\in[T]$ is encapsulated in an online learning algorithm (meta-learning algorithm), which decides upon a sequence of initialization policy $\phi_t$ and learning rate $\alpha_t >0$ |
|
|
|
|
|
for each within-task algorithm. The goal of the meta-learning algorithm is to minimize some notion of task-averaged performance regret to facilitate provably efficient adaptation to a new task. |
|
|
\subsection{CMDP and the primal approach} |
|
|
\textbf{Model.} For each task $t\in[T]$, a CMDP $\mathcal{M}_t$ is defined by the state space $\mathcal{S}$, the action space $\mathcal{A}$, discount factor $\gamma$, initial state distribution over the state-space $\rho_t$, the transition kernel $P_t(s'|s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{S}$, reward functions $c_{t,0}: \mathcal{S} \times \mathcal{A} \rightarrow [0,1]$ and cost functions $c_{t,i}:\mathcal{S} \times \mathcal{A} \rightarrow [0,1]$ for $i = 1, ..., p$. The actions are chosen according to a stochastic policy $\pi_{t}:\mathcal{S} \rightarrow \Delta(\mathcal{A})$ where $\Delta(\mathcal{A})$ is the simplex over the action space. We use $\Delta(\mathcal{A})^{|\mathcal{S}|}$ to denote the simplex over all states $\mathcal{S}$. The initial policy for task $t$ is denoted as $\pi_{t,0}$. The discounted state visitation distribution of a policy $\pi$ is defined as |
|
|
$\nu_{t,s_{0}}^{\pi}(s):=(1-\gamma) \sum_{m=0}^{\infty} \gamma^{m} {P_t}\left(s_{m}=s \mid \pi, s_{0}\right)$ and we write $\nu_t^*(s) := \mathbb{E}_{s_0\sim \rho_t} \left[\nu^{\pi^*}_{t,s_0}(s)\right]$ as the visitation distribution when the initial state follows $\rho_t$ at task $t$. \VK{We denote $\pi_t^*$ as an optimal policy for task $t$ and $\nu_t^*(s) := \mathbb{E}_{s_0\sim \rho_t} \left[\nu^{\pi_t^*}_{t,s_0}(s)\right]$ is the corresponding state visitation distribution induced by policy $\pi_t^*$ when the initial state $s_0$ is sampled from initial state distribution $\rho_t$ at task $t$.} |
|
|
|
|
|
\textbf{Policy parametrization.} |
|
|
We consider the softmax parametrization. For any $\theta \in \mathbb{R}^{|\mathcal{S}| \times|\mathcal{A}|}$, the corresponding softmax policy $\pi_{\theta}$ is defined as |
|
|
$\pi_{\theta}(a \mid s):=\frac{\exp (\theta(s, a))}{\sum_{a^{\prime} \in \mathcal{A}} \exp \left(\theta\left(s, a^{\prime}\right)\right)}, \forall(s, a) \in \mathcal{S} \times \mathcal{A}.$ We neglect the dependence on $\theta$ to alleviate the notational burden. |
|
|
|
|
|
|
|
|
|
|
|
\textbf{Value function.} For task $t$ and a policy $\pi$, we define the state-value function as $V_{t,\pi}^{i}(s)=\mathbb{E}_t\left[\sum_{m=0}^{\infty} \gamma^{m} c_{t,i}\left(s_{m}, a_{m}, s_{m+1}\right) \mid s_{0}=s, \pi\right]$ and the action-value function as $Q_{t,\pi}^{i}(s, a)=$ $\mathbb{E}_t\left[\sum_{m=0}^{\infty} \gamma^{m} c_{t,i}\left(s_{m}, a_{m}, s_{m+1}\right) \mid s_{0}=s, a_{0}=a, \pi \right]$, where $m$ denotes the time steps. |
|
|
Furthermore, the expected total reward/cost functions are $J_{t,i}(\pi)=\mathbb{E}_{\rho_t}\left[V_{t,\pi}^{i}(s)\right]=\mathbb{E}_{\rho_t \cdot \pi}\left[Q_{t,\pi}^{i}(s, a)\right] .$ |
|
|
|
|
|
\textbf{CMDP.} In each task $t$, the goal of the agent is to solve the following CMDP problem |
|
|
\begin{equation}\label{eq:CRPOsafeRL} |
|
|
\underset{\pi}{\max} \hspace{0.1cm} J_{t,0}(\pi) \hspace{0.3cm} \text{s.t.} \hspace{0.2cm} J_{t,i}(\pi) \leq d_{t,i}, \hspace{0.3cm} \forall i = 1,...,p, |
|
|
\end{equation} |
|
|
where $d_{t,i}$ is a fixed limit on the expected total cost $J_{t,i}$ for task $t$ and constraint $i$ (among a total of $p$ constraints). We denote the optimal solution of \eqref{eq:CRPOsafeRL} for the task $t$ as $\pi_t^*$ which can be non-unique. |
|
|
|
|
|
|
|
|
\textbf{Primal approach.} In this work, we focus on the primal approach, CRPO \citep{xu2021crpo}, as an exemplary algorithm with guarantees for a single-task CMDP. \VK{CRPO is a primal-based online CMDP algorithm, which performs policy optimization (natural gradient ascent on the reward) when constraints are not violated, or constraint minimization (natural gradient descent on the constraint function) for one of the violated constraints. } Specifically, with softmax parametrization and carefully chosen parameters, the {suboptimality gap} and {constraint violation} for task $t$ are bounded as follows (if the exact action-value function $\{Q_{t,\pi}^i\}_{i=0}^p$ are available for all $\pi$)\footnote{This regret is slightly different from \citep{xu2021crpo} as we assume an exact critic estimation for simplicity. |
|
|
|
|
|
For more details about the CRPO algorithm, choice of parameters, and the convergence analysis for a single task, we refer the reader to Appendices \ref{sec:crpo} and \ref{sec:prelim-app}.}: |
|
|
\begin{equation}\label{eq:RegretRandC} |
|
|
\begin{aligned} |
|
|
& R_0 = J_{t,0}(\pi_t^*) - \mathbb{E}[J_{t,0}(\hat{\pi}_t)]\leq \frac{2}{\alpha_t M}\mathbb{E}_{s \sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})]+\frac{4 \alpha_t c_{max}^2|\mathcal{S}| |\mathcal{A}|}{(1-\gamma)^3},\\ |
|
|
& R_{i} = \mathbb{E}[J_{t,i}(\hat{\pi}_t)]- d_{t,i} \leq \frac{2}{\alpha_t M}\mathbb{E}_{s \sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})]+\frac{4 \alpha_t c_{max}^2 |\mathcal{S}| |\mathcal{A}|}{(1-\gamma)^3}, \forall i = 1,...,p. |
|
|
\end{aligned} |
|
|
\end{equation} |
|
|
where $\hat{\pi}_t$ is the policy returned by running CRPO for $M$ steps with learning rate $\alpha_t$ in task $t$, $\pi^*_t$ is the optimal policy, $c_{max}$ is the upper bound on reward/cost function, and |
|
|
$D_{KL}(\cdot|\cdot)$ is the KL divergence. |
|
|
|
|
|
|
|
|
\subsection{Meta-SRL problem setup}\label{subsec:Adaptation} |
|
|
We now consider the lifelong extension of CMDPs in which safe RL tasks arrive one at a time, and $t=1,2,\ldots,T$ denotes the index for a sequence of online learning problems. In each single task $t$, the agent must sequentially optimize the policy $\{\pi_{t,i}\}_{i=0}^{M}$ so that the corresponding sub-optimality and the constraint violation, |
|
|
given in \eqref{eq:RegretRandC}, decays sub-linearly in $M$. Beyond the single task, the meta-learner should aim to optimize the upper bounds in \eqref{eq:RegretRandC} over the initial policy $\pi_{t,0}$ and the learning rate $\alpha_t$ so that the task-averaged sub-optimality and constraint violation are expected to improve as the meta-learner solves more tasks. Therefore, we will aim to minimize the task-averaged sub-optimality gap and the task-averaged constraint violation defined as follows: |
|
|
\begin{definition}\label{def:taog} |
|
|
The \textbf{task-averaged optimality gap (TAOG)} $\bar{R}_0$ and the \textbf{task-averaged constraint-violation (TACV)} $\bar{R}_i$ of a safe RL algorithm after $T$ tasks are |
|
|
\begin{equation}\label{eq:TARandTACV} |
|
|
\begin{aligned} |
|
|
\bar{R}_0 = \frac{1}{T}\sum_{t=1}^T\bigg[ J_{t,0}(\pi_t^*) - \mathbb{E}[J_{t,0}(\hat{\pi}_t)] \bigg], |
|
|
\ \bar{R}_{i} = \frac{1}{T} \sum_{t=1}^T\bigg[ \mathbb{E}[J_{t,i}(\hat{\pi}_t)] - d_{t,i} \bigg], \ \forall i = 1,...,p, |
|
|
\end{aligned} |
|
|
\end{equation} |
|
|
where $\hat{\pi}_t$ is the policy returned by running some safe RL algorithm for $M$ time-steps at task $t$ where the expectation is taken with respect to the randomness of the algorithm and environment. |
|
|
\end{definition} |
|
|
\hl{We can observe from (\eqref{eq:RegretRandC}) that the task-averaged regrets can be upper bounded by terms based on the policy initializations $\{\pi_{t,0}\}_{t=1}^T$ and the learning rates $\{\alpha_t\}_{t=1}^T$. The crux of our idea is to design a meta-algorithm that can sequentially update the initial policy $\pi_{t,0}$ and the learning rate of the CRPO algorithm $\alpha_t$ by performing online learning on the upper bounds, i.e., we consider the right-hand sides of (\eqref{eq:RegretRandC}) as the individual loss function. This enables us to bound the dynamic regrets (TAOG and TACV), which are measured against a dynamic sequence of optimal policies $\{\pi_t^*\}_{t=1}^T$, \emph{via the static regret, which is measured against a fixed initial policy} $\phi$.} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Task-similarity}\label{subsec:taskSimilarity} |
|
|
In Meta-SRL, we expect TAOG and TACV to improve with the similarity among the online CMDP tasks. We now discuss the notions of similarity in a static environment; an extension to a dynamic environment is introduced in Sec. \ref{sec:dynamic-regret}. |
|
|
|
|
|
Given optimal polices $\{\pi_t^\ast\}_{t=1}^T$, where $\pi_t^\ast \in \Pi_t^\ast$ for every $t$, the \textbf{task-similarity} can be measured as $D^{*2}=\underset{\phi \in \Delta(\mathcal{A})^{|\mathcal{S}|} } {\min} \frac{1}{T} \sum_{t=1}^T \mathbb{E}_{s \sim \nu_t^*}[D_{KL}(\pi_t^*|\phi)]$. If the optimal policy is not unique, we take the worst case for $D^*$, i.e., a set of policies for which $D^{*2}$ is maximum. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This notion of task-similarity in the static environment is natural for studying gradient-based meta-learning, as it implies that there exists a meta initialization $\phi$ with respect to which optimal policies for individual tasks are all close together. In particular, when the tasks are all identical, i.e., $\{\pi_t^\ast\}_{t=1}^T$ are all equal, we have $D^{*2}=0$. In the practical scenario where only suboptimal policies are accessible, we denote the \textbf{empirical task-similarity} as $\hat{D}^{*2} \coloneqq \underset{\phi \in \Delta(\mathcal{A})^{|\mathcal{S}|} } {\min} \frac{1}{T} \sum_{t=1}^T \mathbb{E}_{s \sim \hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\phi)]$, which depends on the suboptimal policies $\{\hat{\pi}_t\}_{t=1}^T$ returned by a within-task algorithm. |
|
|
|
|
|
As it is desirable that the meta-initialization policy $\pi_{t,0}$ has good exploration properties, we need the initial policy to have full support over $\mathcal{S} \times \mathcal{A}$. We introduce the following assumption: |
|
|
\begin{assumption} \label{asmptn:newAsmptn1} The meta-initialization policy $\pi_{t,0}$ for any task $t$ lies inside a \textbf{shrinkage simplex set}, i.e., $\pi_{t,0}(\cdot | s) \in \Delta \mathcal{A}_{\varrho} \coloneqq \left\{a_1e_1 + ... + a_{n_a}e_{n_a}| \sum_{i=0}^{n_a}a_i = 1, a_i \geq \varrho \hspace{0.3cm} \forall i = 1, ..., n_a \right\}$ for all $s \in \mathcal{S}$, where $e_1,...,e_{n_a} \in \mathbb{R}^{n_a}$ are one-hot vectors for each action (e.g., $e_1$ is the vector of all $0s$ except $1$ at the first location) and $\varrho > 0$. In particular, $\Delta \mathcal{A}_\varrho$ lies inside the regular simplex set $\Delta \mathcal{A}$ ($\varrho=0$). |
|
|
\end{assumption} |
|
|
|
|
|
|
|
|
|
|
|
\hl{Technically, Assumption \ref{asmptn:newAsmptn1} is a minimal requirement for the CRPO to provide any guarantees in a single task. This can be seen in (\eqref{eq:RegretRandC}): if $\pi_{t,0}$ does not have full support over the state/action space, then there may be a state $s$ and an action $a$ where $\pi_t^*(a|s) > 0$ but $\pi_{t,0}(a|s) = 0$, which would make the KL divergence term in (\eqref{eq:RegretRandC}) infinite.} Furthermore, Assumption \ref{asmptn:newAsmptn1} ensures the following holds for the meta-initialization policy $\pi_{t,0}$ for any state $s \in \mathcal{S}$ with positive constants $C_\pi$, $L_g$, $L_\pi$ and $\mu_\pi$: {(1)} $|D_{KL}(\pi_t^*(\cdot|s)|\pi_{t,0}(\cdot|s))|$, $|D_{KL}(\hat{\pi}_t(\cdot|s)|\pi_{t,0}(\cdot|s))|\leq C_{\pi}$; {(2)} $D_{KL}(\pi_t^*(\cdot|s)|\pi_{t,0}(\cdot|s))$ is $L_g$-Lipschitz and $L_\pi$-smooth in $\pi_{t,0}(\cdot|s)$; {(3)} $D_{KL}(\pi_t^*(\cdot|s)|\pi_{t,0}(\cdot|s))$ is $\mu_\pi$-strongly convex in $\pi_{t,0}(\cdot|s)$. We use these conditions in the proof of Lemma \ref{lemma: ideal setting}, Lemma \ref{cor:dynamicRegretOGD}, and Theorem \ref{thm:InexactTAOG}. |
|
|
|
|
|
In this work, we develop algorithms whose TAOG and TACV scale with the task-similarity, which implies that the method will do well if tasks are similar. To understand the CMDP-within-online framework and the impact of task-similarity on the upper bounds of TAOG and TACV for Meta-SRL, we first present a simplified result under the ideal setting where $\{\nu_t^*\}_{t=1}^T$ and $\{\pi_t^*\}_{t=1}^T$ are available for each task $t$ and the task-similarity $D^{*2}$ is known. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\begin{lemma}\label{lemma: ideal setting} |
|
|
Assume $\{\nu_t^\ast\}_{t=1}^T$ and $\{\pi_t^\ast\}_{t=1}^T$ are given after each task and the task-similarity $D^{*2}$ is known. |
|
|
For each task $t$, we run CRPO for $M$ iterations with $\alpha = \frac{(1-\gamma)^{\frac{3}{2}}}{\sqrt{2M |\mathcal{S}||\mathcal{A}| }} \sqrt{\frac{L_g^2(\log T + 1)}{\mu_\pi T}+ D^{*2} }$. In addition, the initialization $\{\pi_{t,0}\}_{t=1}^T$ are determined by playing \textit{Follow-the-Regularized-Leader} (FTRL) or \textit{online gradient descent} (OGD) on the functions $\mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \cdot \right)\right], \text{ for } t=1,\ldots, T$.\footnote{When online learning is played on $\mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \pi \right)\right]$ to determine $\pi_{t+1,0}$, we treat $\theta$ in $\pi_{\theta}$ for all $s\in\mathcal{S}$ and $a\in\mathcal{A}$ as the decision variable. For simplicity, we will refer to $\pi$ as the decision variable.} |
|
|
Then, it holds that |
|
|
\begin{align*} |
|
|
& \Bar{R}_0 \leq \mathcal{O}\left(\frac{1}{\sqrt{M}} \sqrt{\frac{\log T}{T} + D^{*2} } \right), \Bar{R}_i \leq \mathcal{O}\left(\frac{1}{\sqrt{M}} \sqrt{\frac{\log T}{T} + D^{*2} } \right) \ \forall i=1,\ldots,p. |
|
|
\end{align*} |
|
|
\end{lemma} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The above result reveals an interesting benefit brought by including more tasks (the regret decays at a rate of $\log(T)/T$) with more similarity (i.e., lower $D^*$), which improves upon single-task guarantee and serves as the initial point of our study. |
|
|
However, there are several limitations. First, if the optimal policies $\pi_t^*$ and the induced state distributions $\nu_t^*$ are not revealed after each task, it is not likely that the plug-in estimator $\{\mathbb{E}_{s \sim \hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\cdot)]\}_{t=1}^T$ with the learned policy $\hat{\pi}_t$ and estimated visitation distribution $\hat{\nu}_t$ is an unbiased estimator, ruling out existing analysis for FTRL or OGD in the bandit setting. Besides, |
|
|
while the knowledge of $D^*$ used to determine the learning rate can be relaxed \citep{khodak2019adaptive,balcan2019provable}, the resulting scheme is complex to implement---we expect that the learning rates can be chosen \emph{adaptively} for different tasks based on losses observed in the past. We aim to address these challenges with a series of developments in the next section. |
|
|
|
|
|
|
|
|
\begin{algorithm}[t] |
|
|
|
|
|
|
|
|
\caption{Inexact CMDP-within-online framework (exemplified with CRPO \citep{xu2021crpo} as the within-task safe RL algorithm)} |
|
|
\begin{algorithmic}[1] |
|
|
\STATE Initialize actor policy $\pi_{1,0}$ and learning rate $\alpha_1$ |
|
|
\FOR{task $t \in [T]$} |
|
|
\STATE Run CRPO with initializations for actor policy $\pi_{t,0}$ and learning rates $\alpha_{t}$ to obtain a policy $\hat{\pi}_t$ |
|
|
\STATE Estimate the discounted state visitation distribution $\hat{\nu}_t$ of $\hat{\pi}_t$ based on trajectory data collected within-task $t$ with DualDICE \citep{nachum2019dualdice} |
|
|
\STATE Run one or multiple steps of OGD on |
|
|
\begin{enumerate} |
|
|
\item[(a)] $INIT$: $\hat{f}_{t}^{init}(\phi) = \mathbb{E}_{\hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\phi)]$. |
|
|
\item[(b)] SIM: $\hat{f}_t^{sim}(\kappa) = \frac{c_1^t \mathbb{E}_{\hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\pi_{t,0})]}{\kappa} + \kappa (c_2^tM + c_4^t \sqrt{M})+c_3^t\sqrt{M}$ |
|
|
\end{enumerate} |
|
|
to obtain $\pi_{t+1,0}$ and $\alpha_{t+1}$. Here $c_1^t=2$, $c_2^t=\frac{4c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}$, $c_3^t=\frac{3+(1-\gamma)^2}{(1-\gamma)^2}$, and $c_4^t=\frac{3 c_{max}}{(1-\gamma)^2}$. |
|
|
\ENDFOR |
|
|
\end{algorithmic} |
|
|
\label{alg:MetaSRL} |
|
|
\end{algorithm} |
|
|
|
|
|
\section{Provable guarantees for practical CMDP-within-online framework} |
|
|
\label{sec:Methodology} |
|
|
|
|
|
|
|
|
|
|
|
\subsection{Inexact CMDP-within-online framework} |
|
|
\label{subsec:Inexact} |
|
|
|
|
|
One of the key steps to generalize the online-within-online methodology \citep{balcan2019provable} |
|
|
|
|
|
to Meta-SRL is to relax the assumption of accessing the exact upper bounds of within-task performance by designing algorithms to estimate and update their inexact versions. |
|
|
|
|
|
\textbf{Estimation of upper bounds.} Once a CMDP task $t$ is complete, the meta-learner only has access to a suboptimal policy $\hat{\pi}_t$ and the trajectory dataset $\mathcal{D}_t$ produced by some safe RL algorithm. Let $\tilde{\nu}_t$ denote the discounted state visitation distribution induced by policy $\hat{\pi}_t$. To obtain an estimate $\hat{\nu}_t$ from $\mathcal{D}_t$, recent methods often rely on estimating discounted state visitation distribution corrections \citep{liu2018breaking,gelada2019off}. |
|
|
|
|
|
However, the main issues are that $\mathcal{D}_t$ is collected by multiple behavior policies during the learning period, and depending on how far these behavior policies are from the target policy, the per-step importance ratios involved in these methods may have large variance, which may result in a detrimental effect on stochastic algorithms. In this work, we use a methods from the distribution correction estimation (DICE) family, namely DualDICE \citep{nachum2019dualdice}, which is agnostic to the number of behavior policies used and does not involve any per-step importance ratios, thus is less likely to be affected by their high variance. In particular, for each state-action pair $(s,a)$, the method aims to estimate the quantity $\omega_{\pi/\mathcal{D}_t}(s,a) = \frac{d^\pi(s,a)}{d^{\mathcal{D}_t}(s,a)}$, i.e., the likelihood that the target policy $\pi$ will experience the state-action pair normalized by the probability with which the state-action pair appears in the off-policy data $\mathcal{D}_t$. Thereby, we estimate $\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi)]$ with $\mathbb{E}_{\hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\pi)]$ by plugging in $\hat{\pi}_t$ from the within-task CMDP and $\hat{\nu}_t$ from DualDICE in lieu of the optimal policy $\pi_t^*$ and the corresponding discounted state visitation distribution $\nu_t^*$. |
|
|
|
|
|
\textbf{Bounding the estimation error.} We breakdown the error by sources of origin: |
|
|
\begin{align} |
|
|
\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi)] &- \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\hat{\pi}_{t}|\pi)]=\underbrace{\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)]}_{(A)}\label{eq:error_decompose}\\ |
|
|
&+\underbrace{\mathbb{E}_{ \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)]}_{(B)}+\underbrace{\mathbb{E}_{\hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\hat{\pi}_{t}|\pi)]}_{(C)},\nonumber |
|
|
\end{align} |
|
|
where $(A)$ accounts for the mismatch between the discounted state visitation distributions of an optimal policy $\pi_t^*$ and a suboptimal one $\hat{\pi}_t$, $(B)$ originates from the estimation error of DualDICE, and $(C)$ is due to the difference between $\pi_t^*$ and $\hat{\pi}_t$ measured according to $\hat{\pi}_t$. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
To bound $(A)$, we need to control the distance between $\nu_t^*$ and $\Tilde{\nu}_t$, which can be bounded by the distance between the inducing policy parameters as long as they are Lipschitz continuous \cite[Lemma 3]{xu2020improving}. In addition, the bound on $(C)$ also depends on the distance between policies. Controlling the distance between a policy to an optimal policy based on the suboptimality gap requires the optimization to have some curvatures around the optima (e.g., H{\"o}lderian growth \citep{johnstone2020faster}). However, to the best of our knowledge, available results are algorithm-dependent PL inequalities for policy gradient \citep{mei2020global} or quadratic growth with entropy regularization \citep{ding2021beyond}. Given some mild assumptions on the objective/constraint functions and policy parametrization, we can show that a growth condition holds broadly for any CMDP problem. |
|
|
\begin{assumption}\label{asmptn:Definable} |
|
|
The functions $J_{t,i}(\cdot)$ for $i =0,1,...,p$ and $t \in [T]$ and parametric policy $\pi_{\theta}$ are definable in some o-minimal structure \citep{van1996geometric}. |
|
|
\end{assumption} |
|
|
\hl{The definition of ``o-minimal structure'' is given in Appendix \ref{subsec:app_F1}. Assumption \ref{asmptn:Definable} is mild as practically all functions from real-world applications, including deep neural networks, are definable in some o-minimal structures. |
|
|
|
|
|
For Assumption \ref{asmptn:Definable} to hold, a sufficient condition requires that the reward and utility functions belong to the same o-minimal structure.} \hl{ We use Assumption \ref{asmptn:Definable} to bound the terms $(A)$ and $(C)$ as definable sets admit the property of Whitney stratification, and any stratifiable function enjoys a nonsmooth Kurdyka-Lojasiewicz inequality, which implies some curvature around the local/global minima. Details on tame geometry and proof of Theorem \ref{thm:dualDICE} is given in Appendix \ref{sec:kl-bound}.} |
|
|
\begin{theorem}[KL divergence estimation error bound] \label{thm:dualDICE} The following bound holds: |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\quad & |\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\pi)]| \\ \quad & \leq \mathcal{O}\bigg(h\left(\frac{1}{\sqrt{M}}\right)+\frac{1}{\sqrt{M}}+ \sqrt{\epsilon_{opt}}+\sqrt{\epsilon_{approx}(\mathcal{F},\mathcal{H})}\bigg) = \epsilon_t, |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where $h$ is a strictly increasing continuous function with the property that $h(0)=0$ as specified in Proposition \ref{prop:Bolte}, $\epsilon_{approx}(\mathcal{F},\mathcal{H})$ and $\epsilon_{opt}$ are the approximation error and optimization error of DualDICE, defined in \Eqref{eq:eps_FH} and \Eqref{eq:opt}, respectively. |
|
|
\end{theorem} |
|
|
\begin{remark} |
|
|
\label{rem:cum_inexactness} |
|
|
We define cumulative inexactness $\mathcal{E}_T \coloneqq \sum_{t=1}^T \epsilon_t$. This quantity decays with $M$ at a rate of $\mathcal{O} \left(h\left(\frac{1}{\sqrt{M}}\right)+ \frac{1}{\sqrt{M}} \right)$ up to some approximation and optimization errors $\epsilon_{approx}$ and $\epsilon_{opt}$. Moreover, there is a trade-off between $\epsilon_{approx}$ and $\epsilon_{opt}$: if the parametrization functions $\mathcal{F}$ and $\mathcal{H}$ used to solve DualDICE optimization are chosen as neural networks, then $\epsilon_{approx}$ can be reduced at the cost of increasing $\epsilon_{opt}$. If we use stochastic gradient descent as an optimization algorithm in DualDICE with $K$ steps, then $\epsilon_{opt}$ decays at a rate of $\mathcal{O}(1/K)$. Note that $h$ is a definable function used in the Kurdyka–\L{}ojasiewicz (KL) inequality (see, e.g., \cite[Thm. 14]{bolte2007clarke}). |
|
|
\end{remark} |
|
|
With the above uniform bound on estimation error, our next step is to develop static regret bounds for the inexact online gradient descent, which are used to furnish the upper bounds on TAOG and TACV of the proposed inexact CMDP-within-online algorithm. |
|
|
|
|
|
|
|
|
|
|
|
\begin{lemma}[Static regret bound for inexact OGD] |
|
|
\label{cor:staticRegretOGD} |
|
|
Denote $f_t(\pi_{t,0}) \coloneqq \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})]$ for all $t \in [T]$. For any fixed comparator $\pi^*_{0} = \underset{\pi_{0} \in \Delta \mathcal{A}_{\varrho}^{|\mathcal{S}|}}{\argmin} \sum_{t=1}^T f_t(\pi_{0})$, if OGD is run on a sequence of loss functions $\{\hat{f}_t\}_{t \in [T]}$, where $\hat{f}_t(\pi_{t,0}) \coloneqq \mathbb{E}_{\hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\pi_{t,0})]$ for all $t \in [T]$ with the step-size of $\mathcal{O}(1/\sqrt{T})$, then the following bound holds for static regret: |
|
|
\begin{equation*} |
|
|
\sum_{t=1}^T f_t(\pi_{t,0}) - \sum_{t=1}^Tf_t(\pi_{0}^*) \leq \mathcal{O} \left(\sqrt{T}+ \mathcal{E}_T \right), |
|
|
\end{equation*} |
|
|
where $\mathcal{E}_T \coloneqq \sum_{t=1}^T \epsilon_t$ is the cumulative inexactness, and $\epsilon_t$ is the upper bound from Theorem \ref{thm:dualDICE}. |
|
|
\end{lemma} |
|
|
\VK{The static regret analyzed above is defined with respect to the optimal \emph{initial policy} $\pi_{0}^*$ in hindsight, not the \emph{final learned policy}. |
|
|
|
|
|
A static regret with respect to an initial policy provides freedom for the safe RL algorithm to adapt the initial policy based on observations within the task.} Once the static regret for the inexact OGD is established, we can obtain the upper bounds on TAOG and TACV for the proposed inexact CMDP-within-online algorithm in terms of the empirical task-similarity $\hat{D}^*$. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\begin{theorem} |
|
|
\label{thm:InexactTAOG} |
|
|
For each task $t$, we run CRPO for $M$ iterations with $\alpha = \mathcal{O} \left(\frac{1}{\sqrt{M}} \sqrt{\frac{1}{\sqrt{T}} + \frac{\mathcal{E}_T}{T} + \hat{D}^{*2}} \right) $ and we obtain $\{\hat{\nu}_t\}_{t=1}^T$ and $\{\hat{\pi}_t\}_{t=1}^T$. |
|
|
In addition, the initialization $\{\pi_{t,0}\}_{t=1}^T$ for each task $t$ are determined by playing \textit{inexact OGD} (Algorithm \ref{alg:InexactOGD}) on $\mathbb{E}_{ \hat{\nu}_t}\left[D_{KL}\left(\hat{\pi}_t | \cdot \right)\right], \text{ for } t=1,\ldots, T$. Then, the following holds for TAOG ($i=0$) and TACV ($i=1,...,p$): |
|
|
\begin{align*} |
|
|
& \Bar{R}_i \leq \mathcal{O}\left(\frac{1}{\sqrt{M}} \left(\sqrt{\frac{1}{\sqrt{T}}+ \frac{\mathcal{E}_T}{T} + \hat{D}^{*2}} \right) \right) \ \forall i=0,1,\ldots,p. |
|
|
\end{align*} |
|
|
\end{theorem} |
|
|
The benefit of task-similarity is preserved when we perform directly on the plug-in estimator $\mathbb{E}_{ \hat{\nu}_t}\left[D_{KL}\left(\hat{\pi}_t | \cdot \right)\right]$, though we incur an additional cost on the inexactness $\mathcal{E}_T$ and the dependence on $T$ is worse compared to Lemma \ref{lemma: ideal setting}. As $\mathcal{E}_T/T$ diminishes when the learned policy becomes optimal across tasks, e.g., by increasing within-task steps $M$ or if meta-initialization is chosen such that a few steps suffice to reach optimal, we expect the inexactness to have a limited effect on the performance. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Dynamic regret and task-relatedness} |
|
|
\label{sec:dynamic-regret} |
|
|
\hl{In many settings, we have a changing environment, so it is natural to study dynamic regret and compare with a sequence of potentially time-varying \emph{initial policies} $\{\psi_t^*\}_{t=1}^T$.} To measure task-similarity in this case, we define \textbf{task-relatedness} which can be measured by $V^2_{\psi}=\frac{1}{T}\sum_{t=1}^T \mathbb{E}_{s \sim \nu_t^*}[D_{KL}(\pi_t^*|\psi_t^*)]$. This notion of task-relatedness gives the measure of how far optimal policies are in each task from some time-varying comparator. We denote \textbf{empirical task-relatedness} as $\hat{V}_\psi^2 \coloneqq \frac{1}{T}\sum_{t=1}^T \mathbb{E}_{s \sim \hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\psi_t^*)]$, which depends on the suboptimal policy returned by the within-task algorithm. |
|
|
|
|
|
To measure the performance of Meta-SRL in dynamic settings, we analyze the dynamic regret bound, i.e., $U_T\coloneqq\sum_{t=1}^Tf_t(\phi_t)-\sum_{t=1}^T f_t(\psi^*_t)$, where $\psi^*_t\in\arg\min_{x\in\mathcal{X}}f_t(x)$ is a sequence of minimizers for each loss, and $f_t(\cdot) = \mathbb{E}_{s \sim \nu_t^*}[D_{KL}(\pi_t^*|\cdot)]$. By exploiting the strong convexity of the loss function (KL divergence in our case), previous studies have shown that the dynamic regret can be upper bounded by the path-length of the comparator sequence, defined as $\mathcal{P}_T\coloneqq\sum_{t=2}^T\|\psi_t^*-\psi_{t-1}^*\|$, which captures the cumulative difference between successive comparators \citep{zhao2020dynamic}. The bound can be further improved for strongly convex functions as the minimum of the path-length and the squared path-length, $\mathcal{S}_T\coloneqq\sum_{t=2}^T\|\psi_t^*-\psi_{t-1}^*\|^2$, which can be much smaller than the path-length \citep{zhang2017improved}. We extend these results to the settings of inexact online gradient descent by allowing the learner to query the inexact gradient of the function. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\begin{lemma}[Dynamic regret bound for inexact OGD]\label{cor:dynamicRegretOGD} |
|
|
Denote $f_t(\phi_{t}) \coloneqq \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\phi_{t})]$ for all $t \in [T]$. For any dynamically varying comparator $\psi^*_{t}$ , if single-step inexact OGD is run with the step-size $\beta \leq \frac{1}{2\mu_\pi}$ on a sequence of loss functions $\{\hat{f}_t\}_{t \in [T]}$, where $\hat{f}_t(\phi_t) = \mathbb{E}_{\hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\phi_{t})]$ , then the following bound holds for dynamic regret: |
|
|
\begin{equation*} |
|
|
\sum_{t=1}^T f_t(\phi_{t}) - \sum_{t=1}^Tf_t(\psi_{t}^*) \leq \mathcal{O}\left(\min(\mathcal{S}_T+\mathcal{E}_T,\mathcal{P}_T+\tilde{\mathcal{E}}_T) \right), |
|
|
\end{equation*} |
|
|
where $\mathcal{P}_T \coloneqq \sum_{t=2}^T\|\psi_{t}^* - \psi_{t-1}^*\|$ is the path-length of the comparator sequence, $\mathcal{S}_T \coloneqq \sum_{t=2}^T\|\psi_{t}^* - \psi^*_{t-1}\|^2$ is the squared path-length, $\mathcal{E}_T \coloneqq \sum_{t=1}^T \epsilon_t$ is the cumulative inexactness, $\tilde{\mathcal{E}}_T \coloneqq \sum_{t=1}^T \sqrt{\epsilon_t}$ is the cumulative square root of inexactness, and $\epsilon_t$ is the upper bound from Theorem \ref{thm:dualDICE}. |
|
|
\end{lemma} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Dynamic regret with adaptive learning rates} \label{subsec:adapt_learning_rates} |
|
|
|
|
|
\label{subsec:DifferentLearningRates} |
|
|
It can be observed from the last section that to set the learning rate $\alpha_t$ for the within-task algorithm CRPO, knowledge of optimal/suboptimal policies from all $T$ tasks is used. This makes the algorithm less applicable in online settings where tasks are encountered sequentially. Moreover, when the task-environment changes dynamically, a fixed policy initialization $\psi$ may not be the best candidate comparator, where it is natural to study dynamic regret by competing with a potentially time-varying sequence $\{\psi_t^*\}_{t=1}^T$. Also, the tasks may share some common aspects of the optimization landscape, |
|
|
so adapting learning rates based on prior experience may further improve performance. This is the direction we pursue. Recall the regret for suboptimality and constraint violation of the CRPO: |
|
|
\begin{equation}\label{eq:fullRegretCRPO} |
|
|
\begin{aligned} |
|
|
U_{t}(\pi_{t,0},\alpha_t)&\coloneqq \frac{c_1^t}{\alpha_{t}}\mathbb{E}_{s\sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + \alpha_t(c_2^tM + c_4^t \sqrt{M}) + c_3^t\sqrt{M} , |
|
|
\end{aligned} |
|
|
\end{equation} |
|
|
where the constants $\{c_i^t\}_{i=1,\ldots,4}$ are given in Algorithm \ref{alg:MetaSRL}. We assume that $\alpha_t \in \Lambda\coloneqq\{\alpha_t\mid\alpha_t\geq\zeta\}$ for some $\zeta>0$, where $\Lambda$ is a convex set. |
|
|
Overall, the goal of the meta-learner is to make a sequence of decisions, collected by $x_t = \{\pi_{t,0}\in \Delta \mathcal{A}_{\varrho}^{|\mathcal{S}|},\alpha_{t}\in\Lambda\}$, such that TAOG and TACV are minimized. |
|
|
|
|
|
To design the adaptive algorithm, we consider the following two parallel sequences of loss functions over initial policy $\phi$, $f_{t}^{init}(\phi) = \mathbb{E}_{{\nu_t^*}}[D_{KL}({\pi}_t^*|\phi)]$, and learning rate $\kappa$, |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
f_t^{sim}(\kappa) &= \frac{c_1^t\mathbb{E}_{{\nu_t^*}}[D_{KL}({\pi}^*_t|\pi_{t,0})]}{\kappa}+\underbrace{\kappa(c_2^tM + c_4^t \sqrt{M}) + c_3^t\sqrt{M}}_{ f_{t}^{rate}(\kappa)}. |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
Note that $f_t^{sim}(\alpha_{t})=U_t(\pi_{t,0}, \alpha_{t})$ matches the upper bound in \eqref{eq:fullRegretCRPO}. |
|
|
We also denote the inexact versions $\hat{f}_{t}^{init}(\phi)$ and $\hat{f}_t^{sim}(\kappa)$ by replacing $\mathbb{E}_{{\nu_t^*}}[D_{KL}({\pi}_t^*|\phi)]$ with $\mathbb{E}_{{\hat{\nu}_t}}[D_{KL}({\hat{\pi}}_t|\phi)]$ in the above. Inspired by \citep{khodak2019adaptive}, |
|
|
|
|
|
instead of running one online algorithm on $U_{t}(\pi_{t},\alpha_t)$, we will run two online algorithms separately for the function sequences $\hat{f}_t^{init}$ and $\hat{f}_t^{sim}$ by taking actions on the initial policy and learning rates, respectively, such that the overall regret can be bounded by an expression that depends on the regrets for each sequence. |
|
|
Let INIT and SIM be two algorithms, such that the actions $\pi_{t+1,0} \coloneqq \mathrm{INIT}(t)$ are taken over $\hat{f}_{t}^{init}$ and the actions $\alpha_{t+1} \coloneqq \mathrm{SIM}(t)$ are taken over $\hat{f}_t^{sim}$; these actions will then be used as policy initialization and learning rates for the next CMDP. We assume the following regret upper bounds for each algorithm:\footnote{While we run the online learning algorithm on the inexact versions of the loss $\{\hat{f}_t\}_{t=1}^T$, the dynamic/static regret is the standard one measured using the exact losses: $U_T=\sum_{t=1}^Tf_t(\phi_t)-\sum_{t=1}^T f_t(\psi^*_t)$.} |
|
|
\begin{enumerate} |
|
|
\item $U_{T}^{init}(\{\psi_t^*\}_{t=1}^T)$: upper bound on the dynamic regret for $\mathrm{INIT}$ over functions $\{\hat{f}_t^{init}\}_{t=1}^T$ with respect to a time-varying sequence $\{\psi_t^*\}_{t=1}^T$; |
|
|
|
|
|
\item $U_{T}^{sim}(\kappa)$: upper bound on the static regret for $\mathrm{SIM}$ over functions $\{\hat{f}_t^{sim}\}_{t=1}^T$ with respect to a comparator $\kappa > 0$. |
|
|
|
|
|
\end{enumerate} |
|
|
|
|
|
\begin{theorem}\label{thm:UtSim1} |
|
|
Let each within-task CMDP $t$ run $M$ steps of CRPO, initialized by policy $\pi_{t,0}\coloneqq \mathrm{INIT}(t)$ and learning rates $\alpha_{t} \coloneqq \mathrm{SIM}(t)$. Let $\kappa^* \coloneqq \argmin L(\kappa)$, where |
|
|
\begin{equation} \label{eq:appLkappa} |
|
|
L(\kappa) =U_T^{sim}(\kappa)+ \frac{U_T^{init}(\{\psi_t^*\}_{t=1}^T)}{\kappa} + \frac{\mathcal{E}_T}{\kappa} + \sum_{t=1}^T\bigg[ \frac{\hat{f}_t^{init}(\psi_t^*)}{\kappa} + f_t^{rate}(\kappa)\bigg], |
|
|
\end{equation} |
|
|
and $\{\psi_t^*\}_{t=1}^T$ is any comparator sequence. Then, |
|
|
the following bounds on TAOG and TACV hold: |
|
|
\begin{equation} |
|
|
\bar{R}_{i} \leq \frac{L(\kappa^*)}{T},\qquad\qquad \forall\; i=0,...,p. |
|
|
\end{equation} |
|
|
\end{theorem} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\VK{Note that the terms $U_T^{init}$ and $U_T^{sim}$ are simply placeholders for upper bounds on the respective regrets for some inexact online algorithms. In particular, INIT and SIM can be any inexact online algorithms in Algorithm \ref{alg:MetaSRL}, and the results of Theorem \ref{thm:UtSim1} can be instantiated by plugging in the respective $U_T^{init}$ and $U_T^{sim}$. |
|
|
} |
|
|
The following corollary presents the TAOG, and TACV regret bounds when $\mathrm{INIT}$ and $\mathrm{SIM}$ are inexact OGD over the loss functions $\{\hat{f}_t^{init}\}_{t=1}^T$ and $\{\hat{f}_t^{sim}\}_{t=1}^T$ respectively. |
|
|
|
|
|
\begin{corollary} \label{cor:CorollaryAdpativeRate} |
|
|
If $\mathrm{INIT}(t)$ and $\mathrm{SIM}(t)$ are inexact OGD, and are run over the sequences $\{\hat{f}_t^{init}\}_{t=1}^T$ and $\{\hat{f}_t^{sim}\}_{t=1}^T$, then, the following bounds on TAOG and TACV hold for all $i = 0,\ldots,p$: |
|
|
\begin{equation}\label{eq:upper-bound-adapt} |
|
|
\begin{aligned} |
|
|
\bar{R}_{i} \leq & \mathcal{O}\left(\frac{1}{\sqrt{M}} \left( \frac{1}{\sqrt{MT}} + \frac{\mathcal{E}_T}{T\sqrt{M}} + \frac{1}{M^{1/4}\sqrt{T}} \sqrt{\frac{\min(\mathcal{S}_T + \mathcal{E}_T, \mathcal{P}_T + \tilde{\mathcal{E}}_T)+\mathcal{E}_T}{T} + \hat{V}_{\psi}^2 } \right) \right). |
|
|
\end{aligned} |
|
|
\end{equation} |
|
|
\end{corollary} |
|
|
|
|
|
\begin{remark} |
|
|
The bounds are improved in terms of $M$ and $T$ due to the adaptive learning rate. Specifically, the bounds diminish at a rate $\mathcal{O}\left(\frac{1}{M^{3/4}\sqrt{T}}\left(\mathcal{E}_T+ \sqrt{\frac{\mathcal{E}_T}{T} +\hat{V}_\psi^2 } \right) \right)$ as compared to the previous rate $\mathcal{O}\left(\frac{1}{\sqrt{M}}\left(\sqrt{\frac{\mathcal{E}_T}{\sqrt{T}} + \hat{D}^{*2} } \right) \right)$. Note that $\hat{V}_\psi$ is same as $\hat{D}^*$ in the case of a fixed comparator $\psi^*$. Moreover, a practical aspect of our algorithm is that it does not require the knowledge of quantities like $\mathcal{S}_T, \mathcal{P}_T$ and $\mathcal{E}_T$ to decide the value of learning rate $\alpha_t$. |
|
|
\end{remark} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\begin{figure}[t] |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{FrozenLake/FrozenLakeLowSimilarity.pdf} |
|
|
\caption{Frozen lake results for reward maximization and constraint violations when the task-relatedness is low. The Blue dashed line represents the averaged thresholds for the constraint violations. We do $10$ runs on each baseline to get the performance plots with variance.} |
|
|
\label{fig:FrozenLake} |
|
|
\end{figure} |
|
|
|
|
|
\begin{figure}[t] |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{Acrobot/Acrobot_low_similarity2.pdf} |
|
|
\caption{Acrobot results for reward maximization and constraint violations when the task-relatedness is low. Blue dashed line represents the averaged thresholds for the constraint violations.} |
|
|
\label{fig:Acrobot} |
|
|
\end{figure} |
|
|
|
|
|
\vspace{-0.1cm} |
|
|
\section{Experiments}\label{sec:experiments} |
|
|
|
|
|
In this section, we show the effectiveness of the proposed Meta-SRL framework and compare it with the following baselines: simple averaging (i.e., initialize with the average of learned policies from past CMDPs), pre-trained (i.e., initialize test task with the suboptimal policy from another CMDP), Follow the Average Leader (FAL), and random initialization strategies. Note that simple averaging takes the average of previous suboptimal policies obtained from random initializations on all CMDP tasks, while FAL does this in an online manner while tasks arrive sequentially. Different CMDPs are generated using a probability distribution over the parameters of CMDPs (e.g., rewards, transition dynamics), similar to the latent CMDP model \citep{chen2021understanding}. \hl{We consider the Frozen lake, acrobot, half-Cheetah, and humanoid environments from the OpenAI gym \citep{brockman2016openai} and MuJoco \cite{todorov2012mujoco} under constrained settings. } For more details on experimental setups, distribution shift, and extra experiments on Mujoco, please refer to Appendix \ref{sec:ExpDetails}. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
We can observe from Figure \ref{fig:FrozenLake} that Meta-SRL achieves higher rewards and lower constraint violations more quickly than baseline initializations. The baseline FAL which simply takes the average of previous suboptimal policies, performs poorly. This illustrates the benefit of incorporating stationary distribution correction estimation and adaptive learning rates. Indeed, for Frozen lake, different locations of the hole can result in different stationary distributions---it is more sensible to put higher weights on policies that frequently visit a particular state since it implies that the corresponding strategies can have a substantial impact on the case of low task-similarity conditions. We also observe similar trends for the Acrobot in Figure \ref{fig:Acrobot}, where Meta-SRL achieves higher rewards quickly and zero constraint violations as compared to other baseline initializations under low task-relatedness settings. The pre-trained baseline was able to achieve higher rewards but did not achieve constraint satisfaction for both constraints. Under high task-similarity settings, we expected all the methods (except vanilla CRPO) to perform well; however, we noticed that simple averaging does poorly even in this setting, possibly due to adverse interference among different tasks. |
|
|
|
|
|
\section{Conclusion and future directions} |
|
|
\vspace{-0.2cm} |
|
|
We introduced a novel framework, Meta-SRL, for meta-learning over CMDPs. The proposed framework does not assume access to globally optimal policies from the training tasks, and instead performs online learning over inexact within-task bounds estimated by stationary distribution correction. Moreover, strategies for learning rate adaptation are designed to further exploit task-relatedness. One limitation of the proposed method is that it only considers CRPO as the within-task algorithm; nevertheless, our framework can be potentially adapted to more single-task algorithms by making the dependence of guarantees on initial policy/step sizes explicit. |
|
|
|
|
|
|
|
|
Some potential future directions could be to design Meta-SRL with zero constraint violation \citep{liu2021fast}, |
|
|
|
|
|
non-stationary environments \citep{ding2022provably}, and multi-agent settings \citep{de2021constrained}. |
|
|
|
|
|
|
|
|
\section{Broader Impact Statements}\label{sec:BroaderImpact} |
|
|
\VK{There is an increasing need to address fairness as a constraint in learning settings. Existing works that aim to achieve zero-shot generalization without any task-specific adaptation have limited capability to adapt to shifting environments. While online meta-learning is a principled technique to learn good priors over model parameters for fast adaptation in a sequential setting, existing methods often do not address constraints and thus have limited applications in fairness-aware learning.} |
|
|
|
|
|
\VK{ |
|
|
The proposed CMDP-within-online framework can potentially be adapted to reinforcement learning tasks with fairness constraints in a \textbf{non-stationary environment}. In practice, this can provide a strategy that learns priors over policy parameters not only to master the current fairness-aware task but also to become proficient with quick adaptation at learning newly arrived tasks. Our theoretical analysis can be leveraged to provide a sublinear bound on the “task-averaged fairness violation” regret. Similar ideas have been explored by \citep{zhao2021fairness} in the supervised learning setting, while we are not aware of any work on the reinforcement learning counterpart. Thus, it can be an extension for future work to explore the extent to which our method can address this important problem.}\VK{ Nevertheless, fairness constraints present a unique challenge for meta-safe RL settings, as fairness constraints should rarely be violated in a real-world setting due to the implicated discrimination or bias. Additional efforts, such as incorporating pessimism \cite{bai2021achieving} or developing offline methods, may be entailed to reduce fairness violations during initial deployment.} |
|
|
|
|
|
\section{Acknowledgments} |
|
|
|
|
|
The authors acknowledge the generous support by NSF, the Commonwealth Cyber Initiative (CCI), 4-VA collaborative research grant, C3.ai Digital Transformation Institute, and the U.S. Department of Energy. We would also like to thank the anonymous reviewers, which helped us to improve the quality of the manuscript and Tengyu Xu for providing the code for CRPO. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\bibliography{iclr2023_conference} |
|
|
\bibliographystyle{iclr2023_conference} |
|
|
|
|
|
\newpage |
|
|
\appendix |
|
|
\section*{Appendix} |
|
|
In this section, we start with a summary of related work in Sec. \ref{sec:RelatedWork} followed by a brief recapitulation of the CRPO algorithm in Sec. \ref{sec:crpo}. Note that CRPO will be our focus as the exemplary within-task safe RL algorithm. We also introduce notations therein that will be used in later analysis. In Sec. \ref{sec:inexact_cmdp-app}, we give the pseudo-code of our inexact CMDP-within-online algorithm, with further discussions on key aspects. Sec. \ref{sec:prelim-app} provides the proof for Sec. \ref{sec:Preliminaries} of the main paper, which focuses on an elementary yet illustrative example of the CMDP-within-online approach. We start by providing a simplified proof to help the reader understand the main approach of CRPO, and then demonstrate the potential improvement by exploiting inter-task-relatedness (Lemma~\ref{lemma: appIdeal setting}). Sec. \ref{sec:inexact-app} contains the key developments in extending online learning approaches, specifically online gradient descent, to the case of inexact loss functions. We start with some preliminaries on $\epsilon$-subgradient (Sec. \ref{sec:basic-subgradient}). Then, we conduct the analysis for static regret (Thm. \ref{prop:InexactUpperBound-app}) and dynamic regret (Thm. \ref{prop:DynamicRegret}). In Sec. \ref{sec:kl-bound}, we provide a detailed analysis of the KL divergence estimation error bound, which contributes to one of our main contributions in understanding the key aspects of the proposed inexact CMDP-within-online framework. Our development leverages the seminal results developed for tame geometry, which we briefly review in Sec. \ref{sec:prelim-tame}. We also briefly set up the notations and recall basic properties of subgradient flow systems \ref{sec:subflow}. Through a series of bounds, the final result is obtained in Thm. \ref{thm:dualDICEAppendix}. We then provide proofs for Sec. \ref{subsec:adapt_learning_rates}. In Sec. \ref{subsec:appAdaptiveRate}, we first extend the analysis of CRPO to the case of adaptive learning rates. Then, we provide the proof for Thm. \ref{thm:UtSim1} and Corollary \ref{cor:CorollaryAdpativeRate} in Sec. \ref{subsec:appProofsAdaptive}. Experimental details are provided in Sec. \ref{sec:ExpDetails}. Frequently used notations and constants are listed in Sec.\ref{sec:notations}. |
|
|
|
|
|
|
|
|
|
|
|
\section{Related work} |
|
|
\label{sec:RelatedWork} |
|
|
|
|
|
\textbf{Meta-reinforcement learning:} Current state-of-the-art meta-RL includes learning the initial conditions \citep{finn2017model}, hyperparameters \citep{jaderberg2019human}, step directions \citep{li2017meta} and stepsizes \citep{young2018metatrace}, and training recurrent neural networks to embed previous task experience \citep{duan2016rl} (see also \citep{chen2021understanding} for sim-to-real transfer, and \cite{suilen2022robust} for the extension to robust MDPs), with recent developments on improving meta-optimization \citep{rothfuss2018promp,liu2019taming,song2019maml} (see \citep{hospedales2020meta} for a review). Recently, \citep{fallah2021convergence, ji2022theoretical} provided theoretical studies on the convergence of model-agnostic meta-RL. However, these works all focus on the unconstrained meta-RL and their local optimality convergence, while our work is the first to obtain provable guarantees for optimality and constraint satisfaction for CMDPs. |
|
|
|
|
|
\textbf{Online meta-learning/learning-to-learn (LTL).} |
|
|
|
|
|
Most initialization-based meta-learning studies focus on the setting with decomposable within-task loss functions that are often convex \citep{finn2019online,denevi2019learning,balcan2019provable}; nonconvex within-task settings are studied usually for multi-task representation learning \citep{balcan2015efficient,maurer2016benefit,du2020few,tripuraneni2020theory}. Theoretically, our work is inspired by the Average Regret-Upper-Bound Analysis (ARUBA) strategy \citep{khodak2019adaptive} for obtaining a meta-procedure, which has been recently extended to learning nonconvex piecewise-Lipschitz functions \citep{balcan2021learning}; the main technical advance in our work is in providing the guarantees for CMDPs, which is challenging due to the interplay between the nonconvexity and stochasticity of the optimization and the complexity of the within-task safe RL algorithms that involve policy update, critic learning, and the proper choice of stepsizes for reward/constraints. |
|
|
|
|
|
\textbf{Inexact online learning.} Online learning with access to inexact loss/gradient information has been studied for stochastic zero-biased noise \citep{cesa2011online,yang2016tracking,bedi2018tracking,dixit2019online}, deterministic error/nonzero-biased stochastic noise \citep{bedi2018tracking,dixit2019online}, and adversarial perturbation \citep{resler2019adversarial}. Our analysis for static regret uses the formalism of $\epsilon-$subgradient \cite[Chap. XI]{jean2010convex}; for dynamic regret, we extend the work \citep{zhang2017improved} to |
|
|
the inexact setting allowing multiple updates per round and provide improved rates than prior results \citep{bedi2018tracking,dixit2019online}. |
|
|
|
|
|
\textbf{safe RL and CMDP.} Direct policy search methods have had substantial empirical successes in solving CMDPs \citep{borkar2005actor,uchibe2007constrained,bhatnagar2012online,achiam2017constrained,chow2017risk} (see, e.g., \citep{garcia2015comprehensive} for a survey of safe RL). Recently, major progress in understanding the theoretical nonasymptotic global convergence behavior of policy-based methods for CMDPs has also been achieved |
|
|
\citep{chow2018lyapunov,paternain2022safe,efroni2020exploration,ding2021provably,ding2022provably, ying2021dual, yu2019convergent,xu2021crpo,chen2021primal,liu2021fast, liu2021learning}. |
|
|
However, most of these works only study a single CMDP task and don't seek to make the algorithm perform well on new, potentially related CMDP tasks. In addition, while our work uses the constraint-rectified policy optimization (CRPO) algorithm proposed in \citep{xu2021crpo} as a building block, our framework can be potentially adapted to most of the existing RL literature by making the dependence of guarantees on initial policy/step sizes explicit, e.g., safe exploration \citep{efroni2020exploration}, regularization \citep{geist2019theory}, off-policy evaluation \citep{duan2020minimax,tennenholtz2020off}, and offline RL under constraints \citep{le2019batch,wu2021offline,lee2021optidice,thomas2021multi}. |
|
|
|
|
|
\section{CRPO Algorithm and notations} |
|
|
\label{sec:crpo} |
|
|
We provide some preliminaries and notations for the CRPO algorithm for the sake of completeness. CRPO \citep{xu2021crpo} is a primal-based CMDP algorithm, which performs policy optimization (natural gradient ascent on the reward) when constraints are not violated, or constraint minimization (natural gradient descent on the constraint function) for the corresponding violated constraint. There are three crucial components in the overall strategy to solve the CMDP problem \eqref{eq:CRPOsafeRL}: |
|
|
\begin{enumerate} |
|
|
\item \textbf{Policy evaluation:} In each step $m$ of task $t$, for a certain policy $\pi_{t,m}$, the action-value functions $Q_{t,\pi_{t,m}}^i$ are estimated for the reward ($i=0$) and constraints ($i = 1,...,p$). TD-learning is employed for critic evaluation \citep{bhandari2018finite}. |
|
|
\item \textbf{Estimation of constraint violation:} Once the Q-estimates $\Bar{Q}_{t,\pi_{t,m}}^i(s,a)$ are obtained, then a weighted average is taken to estimate expected constraint violation $\Bar{J}_{t,i}(\pi_{t,m})$ under a given policy $\pi_{t,m}$. |
|
|
|
|
|
\item \textbf{Policy optimization:} After the constraint estimation, it is checked if the expected constraint violation $\Bar{J}_{t,\pi_{t,m}}^i$ exceeds the given safety threshold, i.e., if $\Bar{J}_{t,i}(\pi_{t,m}) \leq d_{t,i} + \eta_t$ for all $i=1,\ldots,p$. If none of the constraints are violated, then one step of natural policy gradient ascent is performed to maximize the objective. If one or more constraints are violated, then one step of natural policy gradient descent is conducted to minimize one of the unsatisfied constraints. |
|
|
\end{enumerate} |
|
|
|
|
|
The set of time steps the policy optimization for reward maximization takes place is denoted by $\mathcal{N}_{t,0}$, and the set of time steps constraint minimization takes place is denoted by $\mathcal{N}_{t,i}$. Thus $|\mathcal{N}_{t,0}|+\sum_{i=1}^p|\mathcal{N}_{t,i}| = M$ for any task $t$. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{ Inexact CMDP-within-online Algorithm} |
|
|
\label{sec:inexact_cmdp-app} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Algorithm \ref{alg:MetaSRL} presents the inexact-CMDP-within-online algorithm for Meta-SRL. The first step in the algorithm is to initialize with some random actor policy $\phi_1$, and the learning rate $\alpha_1$ for the first task. Then, for each task $t$, a within-task algorithm (i.e., CRPO) is run for $M$ steps to obtain a policy $\hat{\pi}_t$. The discounted state visitation distribution $\hat{\nu}_t$ induced by $\hat{\pi}_t$ is then estimated using the trajectory data collected within task $t$. Afterward, an inexact OGD method is run on the new loss functions to update the meta-initialization policy $\phi_{t+1}$, and the learning rate $\alpha_{t+1}$. The online learning loop is iterated for all tasks $t\in[T]$. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{Proof in Section \ref{sec:Preliminaries}} |
|
|
\label{sec:prelim-app} |
|
|
We first present a simplified proof for the results in Equation \eqref{eq:RegretRandC}. \VK{This result also shows in (\eqref{eq: condition for two events hold}) how the safety threshold $\eta_t$ can be chosen to achieve sublinear convergence rate in $M$.} |
|
|
\begin{lemma} \label{lemma: three events hold} |
|
|
For CRPO \citep{xu2021crpo} with the softmax parametrization and the exact critic estimation (i.e., no critic evaluation error), if we have |
|
|
\begin{align} \label{eq: condition for two events hold} |
|
|
\eta_t \geq & \frac{2}{\alpha M} \left(\mathbb{E}_{s \sim \nu^{*}_t}\left[D\left(\pi^{*}_t | \pi_{{t,0}}\right)\right] +\frac{2 M \alpha^{2} c_{max}^2 |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}} \right), |
|
|
\end{align} |
|
|
then the following holds |
|
|
\begin{enumerate} |
|
|
\item $\mathcal{N}_{t,0} \neq \emptyset$, i.e., $\hat{\pi}_t$ is well-defined, |
|
|
|
|
|
\item $J_{t,0}\left(\pi_t^{*}\right)-J_{t,0}\left(\hat{\pi}_t\right) \leq \eta_t.$ |
|
|
\item $J_{t,i}\left(\pi_t^{*}\right)-J_{t,i}\left(\hat{\pi}_t\right) \leq \eta_t$, for $i=1,\ldots, p$. |
|
|
\end{enumerate} |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
The following inequality holds due to Lemma 7 in \citep{xu2021crpo}: |
|
|
\begin{align} \label{eq: summation bound of gap in Lemma two events hold} |
|
|
&\alpha \sum_{m \in \mathcal{N}_{t,0}} \left(J_{t,0}\left(\pi_t^{*}\right)-J_{t,0}\left(\pi_{{t,m}}\right)\right)+\alpha \eta_t \sum_{i=1}^p \left|\mathcal{N}_{t,i}\right| |
|
|
\leq \mathbb{E}_{s \sim \nu^{*}_t}\left[D\left(\pi^{*}_t | \pi_{{t,0}}\right)\right] +\frac{2 M \alpha^{2} |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}}. |
|
|
\end{align} |
|
|
We first verify item 1. If $\mathcal{N}_{t,0}=\emptyset$, then $\sum_{i=1}^p \left|\mathcal{N}_{t,i}\right|=M$, and \eqref{eq: summation bound of gap in Lemma two events hold} implies that |
|
|
\begin{align*} |
|
|
\alpha \eta_t M \leq & \mathbb{E}_{s \sim \nu^{*}_t}\left[D\left(\pi^{*}_t | \pi_{{t,0}}\right)\right] +\frac{2 M \alpha^{2} |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}} |
|
|
\end{align*} |
|
|
which contradicts \eqref{eq: condition for two events hold}. Thus, we must have $\mathcal{N}_{t,0} \neq \emptyset$. |
|
|
|
|
|
We then proceed to verify item 2. |
|
|
If $\sum_{m \in \mathcal{N}_{t,0}}\left(J_{t,0}\left(\pi_t^{*}\right)-J_{t,0}\left(\pi_{t,m}\right)\right) > \eta_t |\mathcal{N}_{t,0}|$ , then \eqref{eq: summation bound of gap in Lemma two events hold} implies that |
|
|
$$ |
|
|
\alpha \eta_t M \leq \mathbb{E}_{s \sim \nu^{*}_t}\left[D\left(\pi^{*}_t | \pi_{{t,0}}\right)\right] +\frac{2 M \alpha^{2} |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}}, |
|
|
$$ |
|
|
which contradicts \eqref{eq: condition for two events hold}. Hence, item 2 holds. |
|
|
|
|
|
Finally, the item 3 holds obviously since $\hat{\pi}_t$ is sampled from $\mathcal{N}_{t,0}$. This completes the proof. |
|
|
\end{proof} |
|
|
|
|
|
We now prove Lemma \ref{lemma: ideal setting} in Section \ref{sec:Preliminaries}. |
|
|
|
|
|
\begin{lemma}\label{lemma: appIdeal setting} |
|
|
Assume $\{\nu_t^\ast\}_{t=1}^T$ and $\{\pi_t^\ast\}_{t=1}^T$ are given. |
|
|
For each task $t$, we run CRPO for $M$ iterations with $\alpha = \frac{(1-\gamma)^{\frac{3}{2}}}{\sqrt{2M |\mathcal{S}||\mathcal{A}| }} \sqrt{\left(\frac{L_g^2(\log T + 1)}{\mu_\pi T}+ D^{*2} \right)}$. |
|
|
In addition, the initialization $\{\pi_{t,0}\}_{t=1}^T$ are determined by playing FTRL or OGD on the functions $\mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \cdot \right)\right], \text{ for } t=1,\ldots, T$. Then, it holds that |
|
|
\begin{align*} |
|
|
\Bar{R}_i \leq \frac{\sqrt{8|\mathcal{S}||\mathcal{A}|}}{\sqrt{M(1-\gamma)^{3}}}\sqrt{\left(\frac{L_g^2(\log T + 1)}{\mu_\pi T}+ D^{*2} \right)}, \ \forall i=1,\ldots,p. |
|
|
\end{align*} |
|
|
\end{lemma} |
|
|
|
|
|
\begin{proof} |
|
|
By the within-task guarantee for CMDP, we know that $\Bar{R}_0$ and $\{\Bar{R}_i\}_{i=1}^p$ are well-defined. In addition, it holds that |
|
|
\begin{align*} |
|
|
\Bar{R}_0 \leq &\frac{1}{T}\sum_{t=1}^T \left(\frac{2 \mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \pi_{{t,0}}\right)\right]}{\alpha M} +\frac{ 4\alpha |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}} \right)\\ |
|
|
=& \frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \nu^{*}_t}\left[D_{\mathrm{KL}}\left(\pi^{*}_t | \phi_t \right)\right]- \mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \phi^\ast \right)\right]}{\alpha M } \right)\\ |
|
|
+& \frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \phi^\ast \right)\right]}{\alpha M}+\frac{2 \alpha |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}} \right). |
|
|
\end{align*} |
|
|
where $\phi_t=\pi_{t,0}$. |
|
|
The first inequality follows from the choice of $\eta_t$ from Lemma \ref{lemma: three events hold}. The key step is the last step, which splits the total loss into the loss of the meta-update algorithm and the the loss if we had always initialized at $\phi^\ast$. |
|
|
|
|
|
Since each $\mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \cdot \right)\right]$ is $\mu_\pi$-strongly convex due to Assumption \ref{asmptn:newAsmptn1}, and each $\phi_{t}$ is determined by playing OGD, we have that: |
|
|
\begin{align*} |
|
|
& \frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \phi_t \right)\right]- \mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \phi^\ast \right)\right]}{\alpha M} \right)\leq \frac{2L_g^2 (\log T + 1)}{\mu_\pi \alpha MT}, |
|
|
\end{align*} |
|
|
where $L_g$ is the upper bound on $\nabla_\phi \mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \phi \right)\right]$, $\mu_\pi$ is the strong convexity parameter for the KL divergence of the softmax policy. |
|
|
|
|
|
Since $\phi^\ast=\argmin_{\phi} \sum_{t=1}^T \mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \phi \right)\right]$, by the definition of $D^\ast$, we have $\mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \phi^* \right)\right] \leq D^{\ast 2}$. Thus, by substituting the definition of $\phi^\ast$, it holds that |
|
|
\begin{align*} |
|
|
\frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \nu^{*}_t}\left[D_{KL}\left(\pi^{*}_t | \phi^\ast \right)\right]}{\alpha M}+\frac{2 \alpha |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}} \right)& = \frac{2D^{\ast 2}}{\alpha M} + \frac{4 \alpha |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}}. |
|
|
\end{align*} |
|
|
|
|
|
Setting the value of $\alpha = \frac{\sqrt{\left(\frac{L_g^2(\log T + 1)}{\mu_\pi T}+ D^{*2} \right)(1-\gamma)^{3}} }{\sqrt{2M|\mathcal{S}||\mathcal{A}|}}$, we can obtain the TAOG $\bar{R}_0$ as: |
|
|
\begin{align*} |
|
|
\bar{R}_0 \leq \frac{\sqrt{8\left(\frac{L_g^2(\log T + 1)}{\mu_\pi T}+ D^{*2} \right)|\mathcal{S}||\mathcal{A}|} }{\sqrt{M (1-\gamma)^{3}}} |
|
|
\end{align*} |
|
|
The bound for $\Bar{R}_i$ can be derived similarly. |
|
|
\end{proof} |
|
|
|
|
|
\VK{The lemma above shows how the parameters like learning rate $\alpha$ and safety threshold $\eta_t$ can be chosen to achieve decreasing TAOG and TACV in the number of updates per task $M$ and the number of tasks $T$.} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{Inexact online gradient descent} |
|
|
\label{sec:inexact-app} |
|
|
\subsection{Basics for \texorpdfstring{$\epsilon$}{[1]}-subgradient} |
|
|
\label{sec:basic-subgradient} |
|
|
We start with some basics for $\epsilon$-subdifferential used in the subsequent analysis. This material is based on \cite[Chap. XI]{jean2010convex}. Throughout this section, we consider a convex, closed, and proper function $f:\mathbb{R}^d \rightarrow \mathbb{R} \cup \{+\infty\}$ with domain $\mathrm{Dom}(f)$. We always consider a positive $\epsilon>0$. |
|
|
|
|
|
\begin{definition}[$\epsilon$-subgradient \citep{jean2010convex}] Given $\hat{x} \in \mathrm{Dom}(f)$, the vector $u \in \mathbb{R}^d$ is called $\epsilon$-subgradient of $f$ at $\hat{x}$ when the following property holds for any $x \in \mathbb{R}^d$: |
|
|
\begin{align*} |
|
|
f(x) \geq f(\hat{x}) + \langle u,x - \hat{x} \rangle - \epsilon. |
|
|
\end{align*} |
|
|
The set of all $\epsilon$-subgradients of $f$ at $\hat{x}$ is the $\epsilon$-subdifferential of f at $\hat{x}$, denoted by $\partial_{\epsilon}f(\hat{x})$. |
|
|
\end{definition} |
|
|
|
|
|
In view of the exact subdifferential $\partial f(x)$, $\partial_{\epsilon}f(\hat{x})$ can be called an approximate subdifferential, which is a set-valued function with a convex graph. For practical use, $\partial_{\epsilon}f(\hat{x})$ can be used to characterize the $\epsilon$-solution to a convex minimization problem. |
|
|
|
|
|
\begin{lemma}(\cite[Thm. 1.1.5]{jean2010convex}) The following two properties are equivalent. |
|
|
\begin{align*} |
|
|
0 \in \partial_{\epsilon}f(\hat{x}) \iff f(\hat{x}) \leq f(x) + \epsilon,\qquad \text{for all }x \in \mathbb{R}^d. |
|
|
\end{align*} |
|
|
\end{lemma} |
|
|
|
|
|
One useful result that stems directly from the definition is to link the $\epsilon$-subdifferential of two uniformly close functions (e.g., an expectation of a function and its empirical version). |
|
|
|
|
|
\begin{lemma}\label{prop:Appsubdifferential}\label{lem:app2epsilon} |
|
|
Consider two convex functions $f$ and $g$, with the property that $\|f-g\|_{\infty} \leq \epsilon$, where $\|f-g\|_{\infty} = \max_x|f(x) - g(x)|$. Then, for any $x\in\mathbb{R}^d$ and $u \in \partial f(x)$ in the subdifferential of $f$ at $x$, it is also in the $2\epsilon$-subdifferential of $g$ at $x$, i.e., $u \in \partial_{2\epsilon} g(x)$. |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
The proof follows directly by convexity and the uniform condition: |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
g(y) & \geq f(y) - \epsilon \\ & \geq f(x) + \langle s,y-x \rangle - \epsilon \\ & \geq g(x) + \langle s, y-x\rangle - 2\epsilon, |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where the second inequality is by the convexity of $f$, and the first and last inequalities are due to the supremum norm condition. |
|
|
\end{proof} |
|
|
|
|
|
Our next result is concerned with bounding the distance (measured in $\ell_2$ norm) between the true gradient and the $\epsilon$-subgradient of the function, assuming the function is differentiable and smooth. |
|
|
\begin{lemma}\label{prop:Appsubdifferential2} |
|
|
Suppose a function $f$ is convex, differentiable, and $L$-smooth over $\mathrm{Dom}(f)$, and $u \in \partial_{\epsilon}f(x)$ is an $\epsilon$-subgradient of $f$ at $x\in \mathrm{Dom}(f)$. Then, |
|
|
\begin{equation*} |
|
|
\|u - \nabla f(x)\|_2^2 \leq \frac{2 \epsilon}{2 C_1 - C_1^2 L}, |
|
|
\end{equation*} |
|
|
for any $C_1 \in \{ c \in(0, \frac{2}{L}): x + c(u - \nabla f(x)) \in \mathrm{Dom}(f)\}$. In particular, if $x + \frac{1}{L}(u - \nabla f(x)) \in \mathrm{Dom}(f)$, then $\|u - \nabla f(x)\|_2^2 \leq 2 \epsilon L$. |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
Since $u$ is an $\epsilon$-gradient, $f(y) \geq f(x) + \langle u, y-x \rangle - \epsilon$ for all $ y \in \mathrm{Dom}(f)$. Thus, |
|
|
\begin{align*} |
|
|
0 &\leq f(x) - f(y)+ \langle \nabla f(x), y-x \rangle + \frac{L}{2} \|y - x\|^2\\ |
|
|
&\leq \langle \nabla f(x)-u, y-x \rangle + \frac{1}{2} \|y - x\|^2+\epsilon |
|
|
\end{align*} |
|
|
Choose $y = x+ c(u - \nabla f(x))$ for $c\in(0,\frac{2}{L})$ such that $x + c(u - \nabla f(x)) \in \mathrm{Dom}(f)$, we have that |
|
|
\begin{equation*} |
|
|
\|u - \nabla f(x)\|_2^2 \leq \frac{2 \epsilon}{2 c - c^2 L}. |
|
|
\end{equation*} |
|
|
\end{proof} |
|
|
The smoothness condition in the above seems necessary, as we can construct counterexamples that drive the distance of an $\epsilon$-subgradient and its exact counterpart arbitrarily large without the smoothness condition. In fact, it is known that the set-valued mapping $(x,\epsilon)\rightarrow\partial_\epsilon f(x)$ is inner semi-continuous for a Lipschitz-continuous $f$, which is implied by the fact that the distance (using the Hausdorff distance for sets) between any two subdifferential $\partial_\epsilon f(x)$ and $\partial_{\epsilon'}f(x')$ for all $x,x'\in\mathbb{R}^d$ and $\epsilon,\epsilon'$ is positive, and shown to be bounded by $\mathcal{O}\left(\frac{1}{\min\{\epsilon,\epsilon'\}}(\|x-x'\|+|\epsilon-\epsilon'|\right)$ \cite[Thm. 4.1.3]{jean2010convex}. While the exact gradient can be interpreted as $\epsilon$-subgradient driving $\epsilon\rightarrow 0^+$, the existing bound provided by \cite[Thm. 4.1.3]{jean2010convex} is vacuous in this case; on the other hand, the bound provided in Lemma \ref{prop:Appsubdifferential2} remains meaningful. |
|
|
|
|
|
\subsection{Static regret for the inexact OGD algorithm} |
|
|
\label{sec:static-regret-app} |
|
|
|
|
|
|
|
|
|
|
|
\begin{algorithm}[t] |
|
|
\label{alg} |
|
|
|
|
|
|
|
|
\caption{Inexact OGD Algorithm} |
|
|
\KwInput{Learning rate $\alpha$, $x_1=0$} |
|
|
\begin{algorithmic}[1] |
|
|
|
|
|
\FOR{$t = 1,..,T$} |
|
|
\STATE Incur loss $\ell_t(x_t)$ and compute $\epsilon$-gradient $\hat{\nabla}_t\ell_t(x_t)$ |
|
|
\STATE $x_{t+1} = P_X(x_t - \alpha \hat{\nabla}_t \ell_t(x_t))$ |
|
|
\ENDFOR |
|
|
\end{algorithmic} |
|
|
\label{alg:InexactOGD} |
|
|
\end{algorithm} |
|
|
|
|
|
In the following, we consider the online learning setup, where a sequence of loss functions $\{\ell_t\}_{t \in [T]}$ are revealed sequentially, and the performance of the OGD algorithm (see Algorithm \ref{alg:InexactOGD}) is measured against a static decision in hindsight: |
|
|
\begin{equation} |
|
|
\text{(static regret)}\quad \sum_{t=1}^T\ell_t(x_t)-\min_{x\in X}\sum_{t=1}^T\ell_t(x) \quad |
|
|
\end{equation} |
|
|
where $\{x_t\in X\}_{t\in[T]}$ is a sequence of actions played by the online algorithm. For simplicity, we assume that $X$ belongs to the domains of $\ell_t$ for all $t\in [T]$. Furthermore, we define the following cumulative inexact error bounds: |
|
|
\begin{equation} |
|
|
\mathcal{E}_T\coloneqq\sum_{t=1}^T {\epsilon}_t, |
|
|
\end{equation} |
|
|
where $\epsilon_t$ corresponds to the inexactness of the $\epsilon_t$-subgradient in each round of OGD. |
|
|
|
|
|
\begin{theorem}[Static regret bound for the inexact OGD]\label{prop:InexactUpperBound-app} |
|
|
Assume that $\{\ell_t\}_{t \in [T]}$ are convex and $L_2$-smooth, with bounded gradient, i.e., $\|\nabla \ell_t(x)\|_2\leq L_1$ for all $t\in[T]$ and all $x\in X$. Then, for any comparator $x\in X$, with the stepsize $\alpha\coloneqq \frac{\|x\|}{L_1\sqrt{2T}}$, we have that |
|
|
\begin{equation*} |
|
|
\sum_{t=1}^T\ell_t(x_t)-\sum_{t=1}^T\ell_t(x) \leq L_1\|x\|\sqrt{2T} + \left( 1+\frac{\sqrt{2}cL_1L_2\|x\|}{\sqrt{T}}\right)\sum_{t=1}^T \epsilon_t, |
|
|
\end{equation*} |
|
|
where $\epsilon_t$ is the amount of inexactness at each step $t$. |
|
|
\end{theorem} |
|
|
|
|
|
\begin{proof} |
|
|
|
|
|
By convexity and the fact that $\hat{\nabla}_t$ is an $\epsilon_t$-subgradient of $\ell_t$ at $x_t$, we have that |
|
|
\begin{equation*} |
|
|
\ell_t(x_t) - \ell_t(x) \leq \langle\hat{\nabla}_t, x_t-x \rangle + \epsilon_t, \quad\forall x\in X |
|
|
\end{equation*} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Hence, summing over $t = 1, ..., T$, we get |
|
|
\begin{equation*} |
|
|
\frac{1}{T}\sum_{t=1}^T \ell_t(x_t) - \ell_t(x) \leq \frac{1}{T} \sum_{t=1}^T \langle \hat{\nabla}_t, x_t-x\rangle + \epsilon_t. |
|
|
\end{equation*} |
|
|
To bound the RHS, observe that |
|
|
\begin{align*} |
|
|
\|x_{t+1} - x\|^2 & \leq \|x_t - \alpha \hat{\nabla}_t - x\|^2 \\ \quad & = \|x_t - x\|^2 - 2\alpha \langle x_t-x, \hat{\nabla}_t \rangle + \alpha^2 \|\hat{\nabla}_t\|^2, |
|
|
\end{align*} |
|
|
where the first inequality is due to the OGD update rule and the nonexpansiveness of the projection operator. Thus, rearranging the terms and exploiting the telescopic sum over $t \in [T]$, we have that |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\sum_{t=1}^T \langle x_t - x, \hat{\nabla}_t \rangle & \leq \frac{1}{2 \alpha}(\|x_1 - x\|^2 - \|x_{T+1} - x\|^2 ) + \frac{\alpha}{2}\sum_{t=1}^T \|\hat{\nabla}_t\|^2 \\ \quad & \leq \frac{1}{2\alpha} \|x_1 - x\|^2 + \frac{\alpha}{2} \sum_{t=1}^T \|\hat{\nabla}_t\|^2. |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
Furthermore, since $\ell_t$ is $L_2$-smooth with bounded gradient, and $\hat{\nabla}_t$ is an $\epsilon_t$-gradient for any $t \in [T]$, by Lemma \ref{prop:Appsubdifferential2}, the following holds: |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\|\hat{\nabla}_t\|^2 & \leq 2 \|\nabla_t\|^2 + 2 \|\nabla_t - \hat{\nabla}_t\|^2 \\ \quad & \leq 2 L_1^2 + 2cL_2\epsilon_t, |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where the constant $c$ is specified by Lemma \ref{prop:Appsubdifferential2}. |
|
|
Hence, by combining the above relations, we get |
|
|
|
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\frac{1}{T}\sum_{t=1}^T \ell_t(x_t) - \ell_t(x) & \leq \frac{1}{2 \alpha T}\|x_1 - x\|^2 + \frac{1}{T}\sum_{t=1}^T \Bigg(\frac{\alpha}{2} \|\hat{\nabla}_t\|^2 +\epsilon_t \Bigg) \\ |
|
|
& \leq \frac{1}{2 \alpha T}\|x_1 - x\|^2 + \alpha L_1^2+ \Bigg(\frac{\alpha c L_2+1}{T} \Bigg) \sum_{t=1}^T \epsilon_t. |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
|
|
|
Let $\alpha = \frac{\|x\|}{L_1\sqrt{2T}}$, then we get the RHS as |
|
|
\begin{equation*} |
|
|
L_1\|x\|\sqrt{\frac{2}{T}} + \frac{1+\frac{\sqrt{2}cL_1L_2\|x\|}{\sqrt{T}}}{T}\sum_{t=1}^T \epsilon_t. |
|
|
\end{equation*} |
|
|
\end{proof} |
|
|
|
|
|
\begin{remark} |
|
|
We can relax the dependence of setting the stepsize on $T$ by using a standard doubling trick (first proposed in \citep{auer2002finite}, see also, e.g., \citep{balcan2019provable,khodak2019adaptive}). |
|
|
\end{remark} |
|
|
|
|
|
After establishing the static regret for the inexact OGD, we can use this result to obtain the proof of Lemma \ref{cor:staticRegretOGD}, which gives the static regret if we run inexact OGD over the loss sequences $\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})]$ for all $t \in [T]$. Overall, the following regret bound will eventually help us establish the proof of Theorem \ref{thm:InexactTAOG}, where we will utilize the static regret upper bound for inexact OGD over the loss sequences $\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})]$ for all $t \in [T]$. Also, we denote the norm of the policy with respect to the state distribution $\nu$ as $\|\pi\|_\nu = \sum_{s \in \mathcal{S}} \nu(s) \pi(s)$. Now we proceed to present the proof for Lemma \ref{cor:staticRegretOGD}. |
|
|
\begin{lemma} |
|
|
\label{cor:appstaticRegretOGD} |
|
|
Denote $\ell_t(\pi_{t,0}) \coloneqq \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})]$ for all $t \in [T]$. For any fixed comparator $\pi^*_{0} = \underset{\pi_{0} \in \Delta \mathcal{A}_{\varrho}^{|\mathcal{S}|}}{\argmin} \sum_{t=1}^T \ell_t(\pi_{0})$, if OGD is run on a sequence of loss functions $\{\hat{\ell}_t\}_{t \in [T]}$, where $\hat{\ell}_t \coloneqq \mathbb{E}_{\hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\pi_{t,0})]$ with the step-size $\alpha \coloneqq \frac{\|\pi_{0}^*\|}{L_g \sqrt{2T}}$, then the following bound holds for static regret: |
|
|
\begin{equation*} |
|
|
\sum_{t=1}^T \ell_t(\pi_{t,0}) - \sum_{t=1}^T\ell_t(\pi_{0}^*) \leq \sqrt{2T}L_g\|\pi_{0}^* \| + \left(1 + \frac{4\sqrt{2} L_gL_\pi \|\pi_{0}^*\|}{(2C_1 - C_1^2 L_\pi) \sqrt{T}} \right) \mathcal{E}_T, |
|
|
\end{equation*} |
|
|
for any $C_1 \in \{c \in \left(0, \frac{2}{L_\pi}\right): \pi_{0}^* + c(\hat{\nabla}_t - \nabla_t ) \in \Delta \mathcal{A}_{\varrho}^{|\mathcal{S}|}\}$ where $\hat{\nabla}_t$ and $\nabla_t$ are an $\epsilon_t$-subgradient and exact subgradient of $\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})]$ at $\pi_{t,0}$, respectively, $\mathcal{E}_T \coloneqq \sum_{t=1}^T \epsilon_t$ is the cumulative inexactness. |
|
|
\end{lemma} |
|
|
|
|
|
\begin{proof} |
|
|
The proof follows directly after substituting $c = \frac{4}{2C_1 - C_1^2 L_\pi}$ and other appropriate constants in Theorem \ref{prop:InexactUpperBound-app}. |
|
|
\end{proof} |
|
|
|
|
|
Note that the inexactness bound $\epsilon_t$ can be obtained from Theorem \ref{thm:dualDICE}. Establishing the above Corollary, we can finally provide the proof for Theorem \ref{thm:InexactTAOG}. |
|
|
|
|
|
\textbf{Proof of Theorem \ref{thm:InexactTAOG}} |
|
|
|
|
|
\begin{theorem} |
|
|
\label{thm:appTAOGinexactStatic} |
|
|
Let $\hat{D}^{*2}=\underset{\phi \in \Delta \mathcal{A}_{\varrho}^{|\mathcal{S}|}} {\min} \frac{1}{T} \sum_{t=1}^T \mathbb{E}_{s \sim \hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\phi)]$ be the empirical task-similarity, and let $c_1 = \sqrt{2}L_g\|\phi^*\|$, and $c_2 = \left(2 + \frac{4\sqrt{2} L_gL_\pi \|\phi^*\|}{(2C_1 - C_1^2 L_\pi) \sqrt{T}} \right)$, where $\phi^*$ is the fixed optimal meta-initialization for all the tasks given by $\phi^* = \underset{\phi \in \Delta \mathcal{A}_{\varrho}^{|\mathcal{S}|}} {\argmin} \frac{1}{T} \sum_{t=1}^T \mathbb{E}_{s \sim \hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\phi)]$. For each task $t$, we run CRPO for $M$ iterations with $\alpha = \sqrt{\frac{|\mathcal{S}|\mathcal{A}|}{2M(1-\gamma)^3}} \sqrt{\left(\frac{c_1}{\sqrt{T}} + \frac{c_2 \mathcal{E}_T}{T}+ \hat{D}^{*2} \right)}$, and we obtain $\{\hat{\nu}_t\}_{t=1}^T$ and $\{\hat{\pi}_t\}_{t=1}^T$. |
|
|
In addition, the initialization $\{\pi_{t,0}\}_{t=1}^T$ are determined by playing OGD on the functions $\mathbb{E}_{s \sim \hat{\nu}_t}\left[D_{KL}\left(\hat{\pi}_t | \cdot \right)\right], \text{ for } t=1,\ldots, T$. Then, it holds that |
|
|
\begin{align*} |
|
|
& \Bar{R}_i \leq \frac{\sqrt{8 |\mathcal{S}||\mathcal{A}|} }{ \sqrt{M}(1-\gamma)^{3/2}} \left(\sqrt{ \frac{\sqrt{2}L_g\|\phi^* \|}{\sqrt{T}} + \left(2 + \frac{4\sqrt{2} L_gL_\pi \|\phi^*\|}{(2C_1 - C_1^2 L_\pi) \sqrt{T}} \right) \frac{\mathcal{E}_T}{T} + \hat{D}^{*2} }\right) \hspace{0.2cm} \forall i=0,\ldots,p. |
|
|
\end{align*} |
|
|
\end{theorem} |
|
|
|
|
|
\begin{proof} |
|
|
We know that $\Bar{R}_0$ and $\{\Bar{R}_i\}_{i=1}^p$ are well-defined. In addition, it holds that |
|
|
\begin{align*} |
|
|
\Bar{R}_0 \leq &\frac{2}{T}\sum_{t=1}^T \left(\frac{ \mathbb{E}_{s \sim \nu^*_t}\left[D_{KL}\left(\pi^*_t| \phi_t\right)\right]}{\alpha M} +\frac{ 2\alpha |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}} \right)\\ |
|
|
=& \frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \nu^*_t}\left[D_{\mathrm{KL}}\left(\pi^*_t | \phi_t \right)\right]- \mathbb{E}_{s \sim \nu^*_t}\left[D_{KL}\left(\pi^*_t | \phi^\ast \right)\right]}{\alpha M } \right)\\ |
|
|
&+ \frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \nu^*_t}\left[D_{KL}\left(\pi^*_t | \phi^\ast \right)\right]}{\alpha M}+\frac{ 2\alpha |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}} \right) \\ |
|
|
\leq & \frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \nu^*_t}\left[D_{\mathrm{KL}}\left(\pi^*_t | \phi_t \right)\right]- \mathbb{E}_{s \sim \nu^*_t}\left[D_{KL}\left(\pi^*_t | \phi^\ast \right)\right]}{\alpha M } \right)\\ |
|
|
&+ \frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \hat{\nu}_t}\left[D_{KL}\left(\hat{\pi}_t | \phi^\ast \right)\right]\pm \epsilon_t}{\alpha M}+\frac{ 2\alpha |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}} \right). |
|
|
\end{align*} |
|
|
where $\phi_t=\pi_{t,0}$. |
|
|
Second equality follows from the fact that the total loss can be split into the loss of the meta-update algorithm and the the loss if we had always initialized at $\phi^\ast$. Last inequality follows from the KL-divergence estimation error bound in Theorem \ref{thm:dualDICE}. |
|
|
|
|
|
Since each $\mathbb{E}_{s \sim \hat{\nu}_t}\left[D_{KL}\left(\hat{\pi}_t | \cdot \right)\right]$ is $\mu_\pi$-strongly convex due to Assumption 1, and since each $\phi_{t}$ is determined by playing FTL or inexact OGD, the following term can be upper bounded using Lemma \ref{cor:appstaticRegretOGD} as follows: |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \nu^*_t}\left[D_{KL}\left(\pi^*_t | \phi_t \right)\right]- \mathbb{E}_{s \sim \nu^*_t}\left[D_{KL}\left(\pi^*_t | \phi^\ast \right)\right]}{\alpha M} \right) \leq \\ \frac{2}{\alpha M} \left(\frac{\sqrt{2}L_g\|\phi^* \|}{\sqrt{T}} + \left(1 + \frac{4\sqrt{2} L_gL_\pi \|\phi^*\|}{(2C_1 - C_1^2 L_\pi) \sqrt{T}} \right) \frac{\mathcal{E}_T}{T} \right), |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where the constants are from the Corollary \ref{cor:appstaticRegretOGD}. Now, we will upper bound the second term. Since $\phi^\ast=\argmin_{\phi} \frac{1}{T}\sum_{t=1}^T \mathbb{E}_{s \sim \hat{\nu}_t}\left[D_{KL}\left(\hat{\pi}_t | \phi \right)\right]$, by the definition of $\hat{D}^*$, we have $\mathbb{E}_{s \sim \hat{\nu}_t}\left[D_{KL}\left(\hat{\pi}_t | \phi \right)\right] \leq \hat{D}^{\ast 2}$. Thus, by substituting the definition of $\phi^\ast$, it holds that |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\frac{2}{T}\sum_{t=1}^T \left( \frac{\mathbb{E}_{s \sim \hat{\nu}_t}\left[D_{KL}\left(\hat{\pi}_t | \phi^\ast \right)\right] \pm \epsilon_t}{\alpha M}+\frac{2\alpha |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}} \right)& \leq \frac{2\hat{D}^{\ast 2}}{\alpha M} + \frac{2\mathcal{E}_T}{T\alpha M}+ \frac{ 4\alpha |\mathcal{S}||\mathcal{A}|}{(1-\gamma)^{3}}. |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
Setting the value of $\alpha = \frac{(1-\gamma)^{3/2}\sqrt{\frac{c_1}{\sqrt{T}}+ \frac{c_2\mathcal{E}_T}{T} + \hat{D}^{*2} } }{\sqrt{2M|\mathcal{S}||\mathcal{A}|}}$, where $c_1 = \sqrt{2}L_g\|\phi^*\|$, $c_2 = \left(2 + \frac{4\sqrt{2} L_gL_\pi \|\phi^*\|}{(2C_1 - C_1^2 L_\pi) \sqrt{T}} \right)$, we can obtain the TAOG $\bar{R}_0$ as follows: |
|
|
\begin{align*} |
|
|
\bar{R}_0 \leq \frac{\sqrt{8\left(\frac{c_1}{\sqrt{T}}+ \frac{c_2\mathcal{E}_T}{T} + \hat{D}^{*2} \right)|\mathcal{S}||\mathcal{A}|} }{\sqrt{M(1-\gamma)^{3}}} |
|
|
\end{align*} |
|
|
The bound for $\Bar{R}_i$ can be derived similarly. |
|
|
|
|
|
\end{proof} |
|
|
|
|
|
\subsection{Dynamic regret for the inexact OGD algorithm} |
|
|
\label{sec:dynamic-regret-app} |
|
|
In the following, we consider a stronger notion of regret that measures the performance of the OGD algorithm (see Algorithm \ref{alg:InexactOGD}) against a dynamically changing sequence in hindsight (see, e.g.,~\citep{zinkevich2003online,jadbabaie2015online,zhang2017improved}): |
|
|
\begin{equation} |
|
|
\text{(dynamic regret)}\quad \sum_{t=1}^T\ell_t(x_t)-\sum_{t=1}^T\ell_t(x^*_t) \quad |
|
|
\end{equation} |
|
|
where $x^*_t\in\arg\min_{x\in X}\ell_t(x)$ is the optimal decision for the loss $\ell_t$. It is well known that in the worst case, it is impossible to achieve a sub-linear dynamic regret bound, due to the arbitrary fluctuation in the functions \citep{zinkevich2003online,besbes2015non,yang2016tracking}. Thus, it is common to upper bound the dynamic regret in terms of a certain regularity of the comparator sequence. One possible regularity condition is the path length of the comparator sequence \citep{zinkevich2003online,jadbabaie2015online}: |
|
|
\begin{equation} |
|
|
\mathcal{P}_T\coloneqq \sum_{t=2}^T\|x^*_t-x^*_{t-1}\|, |
|
|
\end{equation} |
|
|
which captures the cumulative Euclidean norm of the difference between successive comparators (note that we will use $\|\cdot\|$ for the Euclidean norm, unless otherwise specified). The path-length measure is also the regularity condition used in existing inexact OGD literature \citep{bedi2018tracking,dixit2019online}. However, as remarked in \citep{zhang2017improved}, a potentially tighter bound can be achieved by examining the squared path-length measure: |
|
|
\begin{equation} |
|
|
\mathcal{S}_T\coloneqq \sum_{t=2}^T\|x^*_t-x^*_{t-1}\|^2, |
|
|
\end{equation} |
|
|
which can be much smaller than $\mathcal{P}_T$ when the local variations are small. For example, when $\|x^*_t-x^*_{t-1}\|=\Theta(1/\sqrt{T})$ for all $t \in[T]$, we have $\mathcal{P}_T=\Theta(\sqrt{T})$ but $\mathcal{S}_T=\Theta(1)$. In this section, we provide analysis with respect to both measures for strongly convex and smooth functions. Furthermore, we propose to apply inexact OGD multiple times in each round, and demonstrate that the dynamic regret is reduced from $\mathcal{O}(\mathcal{P}_T+\mathcal{E}_T)$ to $\mathcal{O}(\min\{\mathcal{P}_T+\mathcal{E}_T,\mathcal{S}_T+\tilde{\mathcal{E}}_T\})$, where |
|
|
\begin{equation*} |
|
|
\tilde{\mathcal{E}}_T\coloneqq\sum_{t=1}^T\sqrt{\epsilon_t} |
|
|
\end{equation*} |
|
|
is the cumulative square root inexactness bounds. Note that our results improve over existing bounds for inexact online learning \citep{bedi2018tracking,dixit2019online} and can be regarded as a generalization of \citep{zhang2017improved} to the inexact settings. We start with a result that will be used in later analysis. |
|
|
|
|
|
\begin{lemma}\label{lem:DynamicLemma1} |
|
|
Assume that $f:X \rightarrow \mathbb{R}$ is $\lambda$-strongly convex and $L$-smooth, and let $x^* = \underset{x \in X}{\argmin} f(x)$ be the unique optimal solution. Let $v = P_X(x - \alpha \hat{\nabla} f(x))$, where $\hat{\nabla}f(u) \in \partial_{\epsilon}f(u)$ and $\alpha \leq \frac{1}{2L}$, we have that |
|
|
\begin{align*} |
|
|
\|v - x^*\|^2 \leq \frac{1}{\lambda \alpha +1}\|x^* - x\|^2+ \frac{c\alpha+2L\alpha}{\lambda L \alpha + L}\epsilon, |
|
|
\end{align*} |
|
|
where the constant $c$ is specified in Lemma \ref{prop:Appsubdifferential2}. |
|
|
\end{lemma} |
|
|
|
|
|
\begin{proof} |
|
|
By the update rule, we have that |
|
|
\begin{equation} |
|
|
\label{eq:updaterule} |
|
|
v = \underset{x' \in X}{\argmin} f(x) + \langle \hat{\nabla}f(x),x'-x \rangle + \frac{1}{2\alpha}\|x'-x\|^2. |
|
|
\end{equation} |
|
|
|
|
|
By strong convexity of the objective above, |
|
|
\begin{equation}\label{eq:strongConvexity} |
|
|
\langle \hat{\nabla}f(x), v-x \rangle + \frac{1}{2\alpha}\|v-x\|^2 \leq \langle \hat{\nabla} f(x),x^* - x \rangle + \frac{1}{2\alpha}\|x^* - x\|^2 - \frac{1}{2\alpha}\|v - x^*\|^2. |
|
|
\end{equation} |
|
|
|
|
|
Since, $f(x)$ is $\lambda$-strongly convex and $L$-smooth, we have that |
|
|
\begin{equation}\label{eq:3} |
|
|
\begin{aligned} |
|
|
f(x^*) - \frac{\lambda}{2}\|x^* - x\|^2 \geq f(x) + \langle \nabla f(x), x^* - x \rangle, |
|
|
\end{aligned} |
|
|
\end{equation} |
|
|
and |
|
|
\begin{equation}\label{eq:4} |
|
|
f(x^*) \leq f(x) + \langle \nabla f(x), x^*-x \rangle + \frac{L}{2}\|x^*-x\|^2. |
|
|
\end{equation} |
|
|
Also, since $\hat{\nabla}f(x)$ is an $\epsilon$-subgradient, we can write |
|
|
\begin{equation}\label{eq:SubgradientProperty} |
|
|
f(x^*) \geq f(x) + \langle \hat{\nabla}f(x), x^* - x \rangle - \epsilon. |
|
|
\end{equation} |
|
|
Combining \eqref{eq:3}, \eqref{eq:4} and \eqref{eq:SubgradientProperty}, we have that |
|
|
\begin{equation*} |
|
|
f(x^*) + \frac{L-\lambda}{2}\|x^* - x\|^2 \geq f(x) + \langle \hat{\nabla} f(x), x^* - x\rangle - \epsilon. |
|
|
\end{equation*} |
|
|
Combining the above relations, we have that |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
f(v) &\leq f(x) + \langle \nabla f(x), v-x \rangle + \frac{L}{2} \|v-x\|^2 \\ & = f(x) + \langle \hat{\nabla}f(x),v-x\rangle + \frac{L}{2} \|v-x\|^2 + \langle \nabla f(x) - \hat{\nabla}f(x),v-x \rangle \\ |
|
|
& \overset{(i)}{\leq} f(x) + \langle \hat{\nabla}f(x), x^* - x \rangle + \left(\frac{L}{2} - \frac{1}{2\alpha} \right)\|v-x\|^2 \\ |
|
|
&\qquad\qquad\qquad\qquad+ \frac{1}{2\alpha}\|x^* -x\|^2 - |
|
|
\frac{1}{2\alpha}\|v - x^*\|^2 + \langle \nabla f(x) - \hat{\nabla}f(x),v-x \rangle \\ |
|
|
& \overset{(ii)}{\leq} f(x^*) + \left(\frac{L}{2} - \frac{1}{2\alpha} \right)\|v-x\|^2 + \frac{1}{2\alpha}\|x^* - x\|^2 \\ |
|
|
&\qquad\qquad\qquad\qquad- |
|
|
\frac{1}{2\alpha}\|v - x^*\|^2 + \langle \nabla f(x) - \hat{\nabla}f(x),v-x \rangle + \epsilon\\ |
|
|
& \overset{(iii)}{\leq} f(v) - \left( \frac{\lambda}{2} + \frac{1}{2\alpha}\right)\|v - x^*\|^2 + \left(\frac{L}{2} - \frac{1}{2\alpha} \right)\|v-x\|^2 \\ |
|
|
&\qquad\qquad\qquad\qquad+ \frac{1}{2\alpha}\|x^* - x\|^2+\langle \nabla f(x) - \hat{\nabla}f(x),v-x \rangle + \epsilon\\ |
|
|
& \overset{(iv)}{\leq} f(v) - \left( \frac{\lambda}{2} + \frac{1}{2\alpha}\right)\|v - x^*\|^2 + \left(\frac{L}{2} - \frac{1}{2\alpha} \right)\|v-x\|^2\\ |
|
|
&\qquad\qquad\qquad\qquad + \frac{1}{2\alpha}\|x^* - x\|^2+ \|\nabla f(x) - \hat{\nabla}f(x)\|\|v-x\| + \epsilon \\ |
|
|
& \overset{(v)}{\leq} f(v) - \left( \frac{\lambda}{2} + \frac{1}{2\alpha}\right)\|v - x^*\|^2 \\ |
|
|
&\qquad\qquad\qquad\qquad + \left(\frac{L}{2} - \frac{1}{2\alpha} + \frac{\kappa}{2} \right)\|v-x\|^2 + \frac{1}{2\alpha}\|x^* -x\|^2 + \left( \frac{c}{2\kappa}+1\right)\epsilon, |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where the first inequality is due to $L$-smoothness, $(i)$ follows from \eqref{eq:strongConvexity}, $(ii)$ is due to convexity, $(iii)$ is due to strong convexity, $(iv)$ follows from Cauchy-Schwarz inequality, and $(v)$ is due to the inequality $ab\leq \frac{1}{2\kappa}a^2+\frac{\kappa}{2}b^2$ for $a,b\geq 0$ and $\kappa>0$ and the constant $c$ comes from Lemma \ref{prop:Appsubdifferential2}. |
|
|
Choosing $\kappa= L$, $\alpha \leq \frac{1}{2L}$, and rearranging the above, we have then proved the claim. |
|
|
\end{proof} |
|
|
|
|
|
|
|
|
|
|
|
\begin{algorithm}[t] |
|
|
|
|
|
|
|
|
\caption{Inexact Online Multiple Gradient Descent Algorithm} |
|
|
\KwInput{Learning rate $\alpha$, $x_1=0$} |
|
|
\begin{algorithmic}[1] |
|
|
|
|
|
\FOR{$t = 1,..,T$} |
|
|
\STATE Incur loss $\ell_t(x_t)$ |
|
|
\STATE $z_{t}^1 = x_t$ |
|
|
\FOR{$k = 1,...,K$} |
|
|
|
|
|
\STATE $z_{t}^{k+1} = P_X(x_{t} - \alpha \hat{\nabla}\ell_t(z_t^k))$ |
|
|
|
|
|
|
|
|
|
|
|
\ENDFOR |
|
|
\STATE $x_{t+1} = z_t^{K+1}$ |
|
|
\ENDFOR |
|
|
|
|
|
\end{algorithmic} |
|
|
\label{alg:InexactMOGD} |
|
|
\end{algorithm} |
|
|
|
|
|
\begin{theorem}[Dynamic regret for inexact OGD with multiple updates]\label{prop:DynamicRegret} |
|
|
Assume that $\ell_t: X \rightarrow \mathbb{R}$ is $\lambda$-strongly convex, $L_1$-Lipschitz, and $L_2$-smooth for all $t \in [T]$. By setting $\alpha \leq \frac{1}{2L_2}$, $K\coloneqq\ceil{\frac{\ln 2}{\ln (1+\lambda \alpha)}}$, then, for any $\beta>0$, we have that |
|
|
|
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\sum_{t=1}^T\ell_t(x_t) - \ell_t(x_t^*) \leq \min\bigg(C_1 \|x_1 - x_1^*\|^2 + C_2&\mathcal{E}_T + C_3 S_T + \frac{1}{2\beta}\sum_{t=1}^T\|\nabla \ell_t(x_t^*)\|^2, \\ \quad & C_4 \|x_1 - x_{1}^*\|+C_5\sum_{t=1}^T \sqrt{\epsilon_t} + C_4 P_T\bigg), |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where $C_1 = 2(L_2+\beta)$, $C_2 =(L_2+\beta)\frac{3c\alpha+6\alpha L_2}{2\lambda \alpha L_2}$, $C_3 = 3(L_2+\beta)$, $C_4 = \frac{2L_1}{2-\sqrt{2}}$ and $C_5 = \frac{2L_1}{2-\sqrt{2}}\sqrt{\frac{c\alpha+2L_2\alpha}{2\alpha \lambda L_2}}$. |
|
|
\end{theorem} |
|
|
|
|
|
\begin{proof} |
|
|
The proof has two parts, where we use different techniques to bound the dynamic regret by $\mathcal{S}_T$ and $\mathcal{E}_T$, as well as $\mathcal{P}_T$ and $\tilde{\mathcal{E}}_T$. Then the final bound is obtained by taking the minimum between the two bounds. |
|
|
|
|
|
\textbf{Bounding the dynamic regret by $\mathcal{S}_T$ and ${\mathcal{E}}_T$.} Since $\ell_t$ is $L_2$-smooth, we have that |
|
|
\begin{align} |
|
|
\ell_t(x_t) - \ell_t(x_t^*) & \leq \langle \nabla \ell_t(x_t^*), x_t - x_t^* \rangle + \frac{L_2}{2} \|x_t - x_t^*\|^2 \\ |
|
|
& \leq \|\nabla \ell_t(x_t^*)\|\|x_t - x_t^*\| + \frac{L_2}{2} \|x_t - x_t^*\|^2 \\ |
|
|
& \leq \frac{1}{2\beta} \|\nabla \ell_t(x_t^*)\|^2 + \frac{L_2+\beta}{2}\|x_t - x_t^*\|^2,\label{eq:dynamic_bd0} |
|
|
\end{align} |
|
|
where the second inequality is due to Cauchy–Schwartz and the third inequality is due to $ab\leq \frac{1}{2\beta}a^2+\frac{\beta}{2}b^2$ for $a,b\geq 0$ and $\beta>0$. |
|
|
|
|
|
Now, using $\|x-y\|^2\leq (1 + \iota)\|x-z\|^2+\left(1 + \frac{1}{\iota}\right)\|z-y\|^2$, we can bound |
|
|
\begin{equation}\label{eq:dynamic_bd1} |
|
|
\sum_{t=1}^T \|x_t - x_t^*\|^2 \leq \|x_1 - x_1^*\|^2 + \sum_{t=2}^T(1 + \iota)\|x_t - x_{t-1}^*\|^2 + \left(1 + \frac{1}{\iota}\right)\|x_t^* - x_{t-1}^*\|^2. |
|
|
\end{equation} |
|
|
Recall the updating rule $z_{t-1}^{j+1} = P_X(z_{t-1}^j - \alpha \hat{\nabla}f_{t-1}(z_{t-1}^j))$, $j = 1, ..., K$; then, we can write that |
|
|
\begin{align} |
|
|
\|x_t - x_{t-1}^*\|^2&=\|z_{t-1}^{K+1}- x_{t-1}^*\|^2\label{eq:dynamic_bd2}\\ |
|
|
&\leq \left(\frac{1}{\lambda \alpha +1}\right)^{K}\|x_{t-1}-x_{t-1}^*\|^2+ \frac{1-\left(\frac{1}{\lambda \alpha +1}\right)^{K}}{1-\frac{1}{\lambda \alpha +1}}\frac{c\alpha+2L_2\alpha}{\lambda L_2 \alpha + L_2}\epsilon_{t-1},\nonumber |
|
|
\end{align} |
|
|
where we recursively apply the result from Lemma \ref{lem:DynamicLemma1}. Thus, by plugging in \eqref{eq:dynamic_bd2} into \eqref{eq:dynamic_bd1}, and using the definitions of $\mathcal{S}_T$ and $\mathcal{S}_T$, we have that |
|
|
\begin{align} |
|
|
\sum_{t=1}^T\|x_t - x_t^*\|^2 &\leq \|x_1 - x_1^*\|^2 + (1+\iota) \left(\frac{1}{\lambda \alpha +1} \right)^K \sum_{t=1}^T\|x_t - x_t^*\|^2 \label{eq:dynamic_bd3}\\ |
|
|
&\qquad\qquad\qquad\qquad+ (1+\iota)\frac{1-\left(\frac{1}{\lambda \alpha +1}\right)^{K}}{1-\frac{1}{\lambda \alpha +1}}\frac{c\alpha+2L_2\alpha}{\lambda L_2 \alpha + L_2}\mathcal{E}_T + \left( 1 +\frac{1}{\iota}\right)\mathcal{S}_T.\nonumber |
|
|
\end{align} |
|
|
Rearranging the terms, the above relation implies that |
|
|
\begin{align*} |
|
|
\sum_{t=1}^T\|x_t - x_t^*\|^2 &\leq \frac{(1+\lambda \alpha)^K}{(1+\lambda \alpha)^K - (1+\iota)} \|x_1 -x_1^* \|^2+\left( 1 +\frac{1}{\iota}\right)\frac{(1+\lambda \alpha)^K}{(1+\lambda \alpha)^K - (1+\iota)} \mathcal{S}_T\\ |
|
|
&\qquad\qquad\qquad\qquad+(1+\iota)\frac{(1+\lambda \alpha)^K-1}{(1+\lambda \alpha)^K - (1+\iota)}\frac{c\alpha+2L_2\alpha}{\lambda L_2 \alpha }\mathcal{E}_T |
|
|
\end{align*} |
|
|
Let $\iota = \frac{1}{2}$ and choose $K = \ceil{\frac{\log 2}{\log (1+\lambda \alpha)}}$, we have |
|
|
\begin{equation*} |
|
|
\sum_{t=1}^T\|x_t - x_t^*\|^2 \leq 4 \|x_1 - x_1^*\|^2 + \frac{3c\alpha+ 6 L_2 \alpha}{\lambda \alpha L_2} \mathcal{E}_T + 6 \mathcal{S}_T. |
|
|
\end{equation*} |
|
|
Combine the above with \eqref{eq:dynamic_bd0}, and summing over $t\in[T]$, we have that |
|
|
\begin{align*} |
|
|
&\sum_{t=1}^T \ell_t(x_t) - \ell_t(x_t^*) \\ |
|
|
&\leq \frac{1}{2\beta}\sum_{t=1}^T\|\nabla \ell_t(x_t^*)\|^2 + 3(L_2+\beta)\mathcal{S}_T + (L_2+\beta)\frac{3c\alpha+6L\alpha}{2\lambda \alpha L}\mathcal{E}_T + 2 (L_2+\beta)\|x_1 - x_1^*\|^2, |
|
|
\end{align*} |
|
|
which holds true for any positive $\beta>0$. |
|
|
|
|
|
|
|
|
\textbf{Bounding the dynamic regret by $\mathcal{P}_T$ and $\tilde{\mathcal{E}}_T$.} By \eqref{eq:dynamic_bd3} and the choice of $K= \ceil{\frac{\log 2}{\log (1+\lambda \alpha)}}$, we have that: |
|
|
\begin{equation*} |
|
|
\|x_t - x_{t-1}^*\|^2 \leq \frac{1}{2}\|x_{t-1} - x_{t-1}^*\|^2 + \frac{c\alpha+2L_2\alpha}{2\alpha \lambda L_2}\epsilon_{t-1}. |
|
|
\end{equation*} |
|
|
Thus, |
|
|
\begin{align} |
|
|
\|x_t - x_{t-1}^*\| & \leq \sqrt{\frac{1}{2}\|x_{t-1} - x_{t-1}^*\|^2+ \frac{c\alpha+2L_2\alpha}{2\alpha \lambda L_2}\epsilon_{t-1}}\nonumber \\ |
|
|
& \leq \frac{1}{\sqrt{2}}\|x_{t-1} - x_{t-1}^*\| + \sqrt{\frac{c\alpha+2L_2\alpha}{2\alpha \lambda L_2}} \sqrt{\epsilon_{t-1}},\label{eq:dynamic_bd4} |
|
|
\end{align} |
|
|
where the last inequlity follows from $\sqrt{a+b}\leq\sqrt{a}+\sqrt{b}$. |
|
|
Due to the bounded gradient assumption, we have that |
|
|
\begin{equation} |
|
|
\label{eq:dynamic_bd5} |
|
|
\sum_{t=1}^T \ell_t(x_t) - \ell_t(x_t^*) \leq L_1 \sum_{t=1}^T\|x_t - x_t^*\| |
|
|
\end{equation} |
|
|
|
|
|
To bound $\sum_{t=1}^T\|x_t - x_t^*\|$, notice that |
|
|
\begin{align*} |
|
|
\sum_{t=1}^T\|x_t - x_t^*\| & \leq \|x_1 - x_1^*\| + \sum_{t=2}^T\|x_t - x_{t-1}^*\|+ \|x_{t-1}^* - x_t^*\|\\ & \leq \|x_1 - x_1^*\| + \frac{1}{\sqrt{2}}\sum_{t=1}^T\|x_t - x_t^*\| + \sqrt{\frac{c\alpha+2L_2\alpha}{2\alpha \lambda L_2}} \tilde{\mathcal{E}}_T+\mathcal{P}_T, |
|
|
\end{align*} |
|
|
which implies that |
|
|
\begin{equation*} |
|
|
\sum_{t=1}^T \|x_t - x_t^*\| \leq \frac{2}{2-\sqrt{2}}\|x_1 - x_1^*\| + \frac{2}{2-\sqrt{2}}\sqrt{\frac{c\alpha+2L_2\alpha}{2\alpha \lambda L_2}} \tilde{\mathcal{E}}_T + \frac{2}{2-\sqrt{2}} \mathcal{P}_T. |
|
|
\end{equation*} |
|
|
Plugging the above in \eqref{eq:dynamic_bd5} proves the claim. |
|
|
\end{proof} |
|
|
|
|
|
In the above result, the number of OGD updates per round is on the order of $\mathcal{O}(L_2/\alpha)$, where $L_2/\alpha$ is the condition number of each loss function. Below, we also provide a dynamic regret bound for standard OGD (single update per round); as a result, we only provide the bound in terms of $\mathcal{P}_T$ (similar to \citep{jadbabaie2015online,mokhtari2016online}. |
|
|
|
|
|
|
|
|
After establishing the dynamic regret for the inexact OGD, we can use this result to obtain the proof of Lemma \ref{cor:dynamicRegretOGD} in the main paper, which provides the dynamic regret of inexact OGD over the loss sequences $\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\phi_{t})]$ for all $t \in [T]$. |
|
|
Here, we present the full statement with constants for the Lemma \ref{cor:dynamicRegretOGD}. |
|
|
|
|
|
\begin{lemma}[Dynamic regret bound for inexact OGD] |
|
|
\label{cor:appdynamicRegretOGD} |
|
|
Denote $\ell_t(\phi_{t}) \coloneqq \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\phi_{t})]$ for all $t \in [T]$. For any dynamically varying comparator $\psi^*_{t} = \underset{\psi_{t} \in \Delta \mathcal{A}_{\varrho}^{|\mathcal{S}|}}{\argmin} \sum_{t=1}^T \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\phi_{t})]$ if OGD is run on a sequence of loss functions $\hat{\ell}_t(\phi_{t})$, where $\hat{\ell}_t(\phi_t) = \mathbb{E}_{\nu_t^*}[D_{KL}(\hat{\pi}_t|\phi_{t})]$ for all $t \in [T]$ with the step-size $\alpha \leq \frac{1}{2\mu_\pi}$, number of iterations $K \coloneqq \ceil{\frac{\ln 2}{\ln (1 + \mu_\pi \alpha)}} $ then the following bound holds for dynamic regret for any $\beta > 0$: |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\sum_{t=1}^T \ell_t(\phi_{t}) - \sum_{t=1}^T\ell_t(\psi_{t}^*) \leq \min (C_1\|\phi_1 - \psi_1^*\|^2 + C_2 & \mathcal{E}_T + C_3 \mathcal{S}_T + \frac{1}{2 \beta}\sum_{t=1}^T \|\nabla \ell_t(\psi_t^*)\|^2, \\ |
|
|
& C_4\|\phi_1 - \psi_1^*\| + C_5 \tilde{\mathcal{E}}_T + C_4 \mathcal{P}_T ), |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where $C_1 = 2(L_\pi + \beta)$, $C_2 = (L_\pi +\beta)\frac{3C_6\alpha+6\alpha L_\pi}{2 \mu_\pi \alpha L_\pi}$, $C_3 = 3(L_\pi + \beta)$, $C_4 = \frac{2 L_g}{2-\sqrt{2}}$, and $C_5 = \frac{2L_g}{2-\sqrt{2}}\sqrt{\frac{C_6\alpha + 2L_\pi \alpha}{2\alpha \mu_\pi L_\pi}}$, for any $C_6 \in \{c \in \left(0, \frac{2}{L_\pi}\right): \psi_{t}^* + c(\hat{\nabla}_t - \nabla_t ) \in \Delta \mathcal{A}_{\varrho}^{|\mathcal{S}|}\}$ where $\hat{\nabla}_t$ and $\nabla_t$ are an $\epsilon_t$-subgradient and exact subgradient of $\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\psi_{t})]$ at $\psi_{t}$, respectively, $\mathcal{E}_T \coloneqq \sum_{t=1}^T \epsilon_t$ is the cumulative inexactness. |
|
|
\end{lemma} |
|
|
|
|
|
\begin{proof} |
|
|
The proof directly follows after plugging in the constants from Theorem \ref{prop:DynamicRegret}. |
|
|
\end{proof} |
|
|
|
|
|
\section{KL divergence estimation error bound} |
|
|
\label{sec:kl-bound} |
|
|
We recall the following notations. For each task $t$, the initial state distribution is denoted by $\rho_t$, the state distribution for the optimal policy $\pi_{t}^*$ is given by $\nu_t^*$, the state distribution for the policy $\hat{\pi}_t$ is denoted by $\Tilde{\nu}_t$, and the state distribution estimated using the trajectory sample dataset $\mathcal{D}_t$ is denoted as $\hat{\nu}_t$. |
|
|
|
|
|
In the main paper, we breakdown the KL divergence estimation error by the sources of origin: |
|
|
\begin{align} |
|
|
\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi)] &- \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\hat{\pi}_{t}|\pi)]=\underbrace{\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)]}_{(A)}\\ |
|
|
&+\underbrace{\mathbb{E}_{ \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)]}_{(B)}+\underbrace{\mathbb{E}_{\hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\hat{\pi}_{t}|\pi)]}_{(C)},\nonumber |
|
|
\label{eq:KLerrorDecompose} |
|
|
\end{align} |
|
|
where $(A)$ accounts for the mismatch between the discounted state visitation distributions of an optimal policy $\pi_t^*$ and a suboptimal one $\hat{\pi}_t$, $(B)$ originates from the estimation error of DualDICE, and $(C)$ is due to the difference between $\pi_t^*$ and $\hat{\pi}_t$ measured according to $\hat{\pi}_t$. By the triangle inequality, we can bound the total error by controlling each term separately. This decomposition is general in the sense that it provides a guideline to bound each term with potentially different strategies. In particular, the term $(B)$ can be bounded differently if we replace DualDICE with another stationary distribution estimation algorithm. To bound the terms $(A)$ and $(C)$, we have developed new techniques based on tame geometry and subgradient flow systems. To streamline the presentation, we consider the tabular setting with softmax parametrization. |
|
|
|
|
|
|
|
|
|
|
|
To bound $(A)$, we need to control the distance between $\nu_t^*$ and $\Tilde{\nu}_t$, which can be bounded by the distance between the inducing policy parameters as long as they are Lipschitz continuous \cite[Lemma 3]{xu2020improving}. In addition, the bound on $(C)$ also depends on the distance between policies. In general, controlling the distance between a policy to an optimal policy based on the suboptimality gap requires the optimization to have some curvatures around the optima (e.g., quadratic growth \citep{drusvyatskiy2018error} or H{\"o}lderian growth \citep{johnstone2020faster}). However, to the best of the knowledge of the authors, the only available results are algorithm-dependent PL inequalities for policy gradient \citep{mei2020global} or quadratic growth with entropy regularization \citep{ding2021beyond}. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\VK{ |
|
|
\textbf{Discussion on Assumption \ref{asmptn:newAsmptn1}:} As discussed in the main text, Assumption \ref{asmptn:newAsmptn1} implies boundedness and Lipschitzness of the KL divergence. We make use of this in bounding the terms $(A)$ and $(C)$ in (\eqref{eq:error_decompose}) and eventually obtain Theorems \ref{thm:dualDICE} and \ref{thm:UtSim1}.} |
|
|
\VK{ |
|
|
We expect that Assumption \ref{asmptn:newAsmptn1} is also needed in unconstrained meta-learning by adapting our method, i.e., the MDP-within-online framework. Technically, Assumption \ref{asmptn:newAsmptn1} is a minimum requirement even for single-task CRPO to provide provable guarantees. This can be seen in the convergence guarantee of the original CRPO method (Lemma \ref{lemma: three events hold} and Lemma \ref{lem:2_threeEventsHold} in our paper, or Theorem 3 in \citep{xu2021crpo} last line of their proof before the term $D_{KL}(\pi_t^*|\pi_{t,0})$ is submerged in the big-O notation). For example, as shown in our (\eqref{eq:RegretRandC}),} |
|
|
|
|
|
$$R_0 = J_{t,0}(\pi_t^*) - \mathbb{E}[J_{t,0}(\hat{\pi}_t)]\leq \frac{2}{\alpha_t M}\mathbb{E}_{s \sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})]+\frac{4 \alpha_t c_{max}^2|\mathcal{S}| |\mathcal{A}|}{(1-\gamma)^3}.$$ |
|
|
\VK{ |
|
|
To ensure that the bound is nontrivial, we need to bound the term $D_{KL}(\pi_t^*|\pi_{t,0}).$ However, if $\pi_{t,0}$ does not have full support over the state/action space, then there may be a state $s$ where $\pi_t^*(s) > 0$ but $\pi_{t,0}(s) = 0$, which would make the KL divergence infinite.} |
|
|
|
|
|
|
|
|
|
|
|
\subsection{Preliminaries on tame geometry}\label{subsec:app_F1} |
|
|
|
|
|
\label{sec:prelim-tame} |
|
|
For the sake of completeness, let us recall some fundamental concepts/results in tame geometry, which allows us to study the global geometry of the solution maps of a wide range of optimization problems, which will be used in bounding the estimation error for the KL divergence. More information can be found in \citep{davis2020stochastic,van1996geometric}. Recall that a class of functions on a bounded set is called $C^p$ smooth when it possesses the uniformly bounded partial derivatives up to order $p$. |
|
|
|
|
|
|
|
|
\begin{definition}[Whitney Stratification] |
|
|
\label{def:WhitneyStratification} |
|
|
A Whitney $C^k$ stratification of a set $I$ is a partition of $I$ into finitely many nonempty $C^k$ manifolds, called strata, satisfying the following compatibility conditions: |
|
|
|
|
|
\begin{enumerate} |
|
|
\item For any two strata $I_a$ and $I_b$, the implication $I_a \cap I_b \neq \emptyset$ implies that $I_a \subset \mathrm{cl} I_b$ holds, where $\mathrm{cl} I_b$ denotes the closure of the set $I_b$. |
|
|
|
|
|
\item For any sequence of points $x_k$ in a stratum $I_a$, converging to a point $x^{\star}$ in a stratum $I_b$, if the corresponding normal vectors $v_k \in N_{I_a}(x_k)$ converge to a vector $v$, then the inclusion $v \in N_{I_b}(x^{\star})$ holds. Here $N_{I_a}(x_k)$ denotes the normal cone to $I_a$ at $x_k$. |
|
|
\end{enumerate} |
|
|
|
|
|
\end{definition} |
|
|
|
|
|
Roughly speaking, stratification is a locally finite partition of a given set into differentiable manifolds, which fit together in a regular manner (property $1$ in Def. \ref{def:WhitneyStratification}). Whitney stratification as defined above is a special type of stratification for which the strata are such that their tangent spaces (as viewed from normal cones) also fit regularly (property $2$). |
|
|
|
|
|
There are several paths to verifying Whitney stratifiability. For instance, one can show that the function under study belongs to one of the well-known function classes, such as semialgebraic functions \citep{davis2020stochastic}, whose members are known to be Whitney stratifiable. However, to study the solution function of a general convex optimization problem, we need a far-reaching axiomatic extension of semialgebraic sets to classes of functions definable on ``o-minimal structures,'' which are very general classes and share several attractive analytic features as semialgebraic sets, including Whitney stratifiability \citep{davis2020stochastic,van1996geometric}. |
|
|
|
|
|
\begin{definition}[o-minimal structure] |
|
|
\label{def:0-minimal structure} |
|
|
\citep{van1996geometric} An o-minimal structure is defined as a sequence of Boolean algebras $O_v$ of subsets of $\mathbb{R}^{v}$, such that for each $n_v \in \mathbb{N}$, the following properties hold: |
|
|
|
|
|
\begin{enumerate} |
|
|
\item If some set $X$ belongs to $O_v$, then $X \times \mathbb{R}$ belong to $O_{v+1}$. |
|
|
|
|
|
\item Let $P_{proj}: \mathbb{R}^{v} \times \mathbb{R} \rightarrow \mathbb{R}^{v}$ denote the coordinate projection operator onto $\mathbb{R}^{v}$, then for any $X$ in $O_{v+1}$, the set $P_{proj}(X)$ belongs to $O_v$. |
|
|
|
|
|
\item $O_v$ contains all sets of the form $\{x \in \mathbb{R}^{v}: \hspace{0.1cm} y(x) = 0 \}$, where $y(x)$ is a polynomial in $\mathbb{R}^{v}$. |
|
|
|
|
|
\item The elements of $O_1$ are exactly the finite unions of intervals (possibly infinite) and points. |
|
|
\end{enumerate} |
|
|
Then all the sets that belong to $O_v$ are called definable in the o-minimal structure. |
|
|
|
|
|
\end{definition} |
|
|
|
|
|
Definable sets have broader applicability than semialgebraic sets (in the sense that the latter is a special kind of definable sets) but enjoys the same, remarkable stability property: the composition of definable mappings (including sum, inf-convolution, and several other classical operations of analysis involving a finite number of definable objects) in some o-minimal structure remains in the same structure. We will crucially exploit these properties in the following sections. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Basic properties of subgradient flow systems} |
|
|
\label{sec:subflow} |
|
|
We also recall some basic definitions and properties of the subgradient flow system (see, e.g., \cite[Thm. 13]{bolte2010characterizations}). Let $f: \mathbb{R}^d \rightarrow \mathbb{R} \cup \{+\infty\}$ be a proper lower semicontinuous function. |
|
|
|
|
|
\begin{definition}[Subgradient flow system]\label{def:subgflow} |
|
|
For every $x \in \mathrm{dom}(f)$, there exists a unique absolutely continuous curve (called trajectory or subgradient curve) $\theta(\tau):[0,+\infty) \rightarrow \mathbb{R}^d$ that satisfies |
|
|
\begin{equation} |
|
|
\begin{cases} |
|
|
\Dot{\theta}(\tau) \in - \partial f(\theta(\tau)) & \text{a.e. on}\hspace{0.3cm} (0,+\infty)\\ |
|
|
\theta(0) = \theta_0 \in \mathrm{dom}(f). |
|
|
\end{cases} |
|
|
\end{equation} |
|
|
\end{definition} |
|
|
|
|
|
Moreover, the trajectory also satisfies the following properties \cite[Thm. 13]{bolte2010characterizations}: |
|
|
\begin{enumerate} |
|
|
\item $\theta(\tau) \in \mathrm{dom}(\partial f)$ for all $\tau \in (0,+\infty)$. |
|
|
\item For all $\tau>0$, the right derivative $\Dot{\theta}(\tau^+)$ is well defined and equal to |
|
|
\begin{align*} |
|
|
\Dot{\theta}(\tau^+) = - \partial^0 f(\theta(\tau)), |
|
|
\end{align*} |
|
|
where $\partial^0 f(\theta)$ is the minimum norm subgradient in $\partial f(\theta)$. In particular, we have that $\Dot{\theta}(\tau) = - \partial^0 f(\theta(\tau))$, for almost all $\tau$. |
|
|
\end{enumerate} |
|
|
|
|
|
|
|
|
\subsection{Bounding the distance \texorpdfstring{$\|\hat{\theta}_t-\theta^*_t\|$}{[1]}} \label{subsec:app_F3} |
|
|
|
|
|
Recall Assumption \ref{asmptn:Definable}, which requires that the objective/constraint functions and policy parametrization are definable in some o-minimal structure \citep{van1996geometric}. This is a mild assumption as practically all functions from real-world applications, including deep neural networks, are definable in some o-minimal structure \citep{davis2020stochastic}; also, the composition of mappings, along with the sum, inf-convolution, and several other classical operations of analysis involving a finite number of definable objects in some o-minimal structure remains in the same structure \citep{van1996geometric}. The far-reaching consequence of definability, exploited in this study, is that definable sets and functions admit, for each $k \geq 1$, a $C^k$–Whitney stratification with finitely many strata (see, for instance, \cite[Result 4.8]{van1996geometric}). This remarkable property, combined with the result that any stratifiable functions enjoys a nonsmooth Kurdyka–\L{}ojasiewicz inequality \citep{bolte2007clarke}, provides the foundation to bound the distance $\|\pi^*_t-\hat{\pi}_t\|$ by the suboptimality gap. Note that without further specifications, $\pi^*_t$ is understood as one of the optimal policies that are closest to the policy $\hat{\pi}_t$ (i.e., the projection of $\hat{\pi}_t$ onto the optimal policy set). |
|
|
|
|
|
We start with the following elementary result. Here and throughout the section, we use $\mathcal{F}_{t,\Tilde{d}} = \{\pi_{t,\theta}: J_{t,i}(\pi_{t,\theta}) \leq \Tilde{d}_{t,i}\}$ to denote the feasible set with upper bounds $\Tilde{d}$. Note that $\mathcal{F}_{t,{d}}$ is the original feasible set. We also let $\mathbb{I}_{\mathcal{F}_{t,\Tilde{d}}}(\cdot)$ be the indicator function for the set $\mathcal{F}_{t,\Tilde{d}}$. |
|
|
\begin{lemma}\label{prop:Definable} |
|
|
The function (with variable $\theta$) $J_{t,0}(\pi_{t,\theta})+ \mathbb{I}_{\mathcal{F}_{t,\Tilde{d}}}(\pi_{t,\theta})$, where $\Tilde{d}_{t}$ is any vector such that $\mathcal{F}_{t,\Tilde{d}}$ is non-empty, is definable. |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
Since $J_{t,i}(\cdot)$ is definable for $i = 1,...,p$, by the rule of composition, which is due to the definable counterpart of the Tarski-Seidenberg theorem, $J_{t,i}(\pi_{t,\theta}) - d_{t,i}$ is definable for $i = 1,...,p$. Thus, $\mathcal{F}_{t,\Tilde{d}}$ is definable on the same o-minimal structure by definition. Furthermore, $\mathbb{I}_{\mathcal{F}_{t,\Tilde{d}}}(\cdot)$ is definable as the indicator of $\mathcal{F}_{t,\Tilde{d}}$. The definability of $J_{t,0}(\pi_{t,\theta})$ follows similarly. Since definability is preserved under addition, the function $J_{t,0}(\pi_{t,\theta})+ \mathbb{I}_{\mathcal{F}_{t,\Tilde{d}}}(\pi_{t,\theta})$ is definable. |
|
|
\end{proof} |
|
|
|
|
|
For the convenience of the reader, we restate the result for non-smooth Kurdyka–\L{}ojasiewicz (KL) inequality from \cite[Thm. 14]{bolte2007clarke}. |
|
|
|
|
|
\begin{proposition}[Non-smooth Kurdyka–\L{}ojasiewicz inequality] \label{prop:Bolte} |
|
|
Let $f$ be a lower semicontinuous definable function. There exists $\rho>0$, a strictly increasing continuous definable function $h:[0,\rho]\rightarrow (0,\infty)$ which is $C^1$ smooth on $(0,\rho)$, with $h(0) = 0$, and a continuous definable function $\mathcal{X}:\mathbb{R}_+ \rightarrow (0,\rho)$ such that |
|
|
\begin{equation*} |
|
|
\|\partial^0 f(x)\| \geq \frac{1}{h'(|f(x)|)}, |
|
|
\end{equation*} |
|
|
whenever $0 < |f(x)| \leq \mathcal{X}(\|x\|)$. |
|
|
\end{proposition} |
|
|
|
|
|
|
|
|
Let ${\theta}$ and $\theta_t^*$ denote the parameters of a policy ${\pi_\theta}$ and $\pi_t^*$, respectively. Directly bounding the distance between ${\theta}$ and $\theta_t^*$ is difficult because $\pi$ may be infeasible (this is even true for $\hat\pi_t$, since it is only guaranteed to approximately satisfy the constraints), i.e., $\theta \notin \mathcal{F}_{t,d}$. Thus, the typical approach of following the subgradient flow of $J_{t,0}(\pi_\theta)+\mathbb{I}_{\mathcal{F}_{t,{d}}}(\pi_\theta)$ to reach $\theta_t^*$ is not applicable. The idea is to enlarge the feasible set $\mathcal{F}_{t,d}$ by increasing the violation threshold $\Tilde{d}_{t,i} \geq d_{t,i} + \delta$, for any $\delta >0$, such that with high probability, $\theta \in \mathcal{F}_{t,\Tilde{d}}$. Then by following the subgradient flow for $J_{t,0}(\pi_\theta)+ \mathbb{I}_{\Tilde{\mathcal{F}}_t}(\pi_\theta)$, we can arrive at a critical point $\Tilde{\theta}_t^*$ (corresponding to the policy $\Tilde{\pi}_t^*$), which is most likely different from $\theta_t^*$. It then remains to bound the distance between $\theta_t^*$ and $\Tilde{\theta}_t^*$, which is possible due to the preservation of definability through $\inf$ projection. This is the roadmap we will follow. A graphical illustration of the approach is shown in Fig. \ref{fig:EnlargedFeasibleSet}. |
|
|
|
|
|
|
|
|
\begin{figure}[t] |
|
|
\centering |
|
|
\includegraphics[width=.4\columnwidth]{illustrate.pdf} |
|
|
\caption{To bound the distance between $\pi_t^*$ and $\hat{\pi}_t$, we first bound the distance between $\Tilde{\pi}_t^*$ and the optimal policy with respect to a larger feasible set $\mathcal{F}_{t,\tilde{d}}$ by an argument based on subgradient flow curve. Note that $\hat{\pi}_t\in\mathcal{F}_{t,\tilde{d}}$ may be infeasible with respect to the original set of constraints but feasible with respect to the relaxed constraints. We then bound the distance between the optimal policies $\pi_t^*$ and $\Tilde{\pi}_t^*$, which correspond to the original feasible set $\mathcal{F}_{t,{d}}$ and the enlarged set $\mathcal{F}_{t,\tilde{d}}$. By the triangle inequality, we can then derive the desired bound on the distance between $\pi_t^*$ and $\hat{\pi}_t$. Note that for better visualization, we vertically separate the sets $\mathcal{F}_{t,{d}}$ and $\mathcal{F}_{t,\tilde{d}}$, which also aims to indicate that in general the optimal solution $\Tilde{\pi}_t^*$ has a higher objective than $\pi_t^*$ due to the relaxed constraints. } |
|
|
\label{fig:EnlargedFeasibleSet} |
|
|
\end{figure} |
|
|
|
|
|
|
|
|
\textbf{Bounding the term $\|\theta_t^* - \Tilde{\theta}_t^*\|$.} In this part, we will bound the term $\|\theta_t^* - \Tilde{\theta}_t^*\|$, which will be used to bound $\|\pi_t^* - \Tilde{\pi}_t^*\|$. Firstly, we will prove that the parameter $\theta_t$, which represents the solution map of an optimization with definable objective and constraints, is definable. |
|
|
|
|
|
\begin{proposition} \label{prop:thetaDdefinable} |
|
|
Let $\theta_t(d) \in {\arg\min} \left\{ J_{t,0}(\pi_{t,\theta}), \text{ s.t. } J_{t,i}(\pi_{t,\theta}) \leq d_{t,i}, \forall i = 1,...,p\right\}$ be the solution map of the constraint parameters $d$. Then, the function $\theta_t(d)$ is continuous and definable. Furthermore, there exists a finite partition of the space such that the restriction of $\theta_t(d)$ to each partition is $C^p$ smooth. |
|
|
\end{proposition} |
|
|
|
|
|
\begin{proof} |
|
|
First, it can be seen that the solution map $\theta_t(d) \in \argmin J_{t,0}(\pi_{t,{\theta}})+\mathbb{I}_{\mathcal{F}_{t,{d}}}(\pi_{t,\theta})$. Let $\phi_t(d) = \min J_{t,0}(\pi_{t,{\theta}}) + \mathbb{I}_{\mathcal{F}_{t,{d}}}(\pi_{t,\theta})$ be the optimal value function. Since $J_{t,0}(\pi_{t,\theta}) + \mathbb{I}_{\mathcal{F}_{t,{d}}}(\pi_{t,\theta})$ is definable by Proposition \ref{prop:Definable}, and definability is preserved under $\inf$ projection, $\phi_t(d)$ is definable. Since $\theta_t(d) = \{\theta:J_{t,0}(\pi_{t,\theta})+\mathbb{I}_{\mathcal{F}_{t,{d}}}(\pi_{t,\theta}) = \phi_t(d)\}$, by the Tarski-Seidenberg Theorem, $\theta_t(d)$ is definable. The continuity property follows directly from Berge's Maximum theorem. |
|
|
|
|
|
|
|
|
Following the discussion of Whitney stratifications in \citep{bolte2007clarke}, |
|
|
since the graph of a definable function is Whitney stratifiable, we can construct a partition by projecting the stratification into the function domain, which will be a Whitney $C^p$-stratification by the constant rank theorem. Furthermore, the restriction of the definable function to each stratum is $C^p$-smooth. Alternatively, we can directly use the fact that for any definable function, there exists a $C^p$-decomposition which has a finite number of cells, and the restriction to each cell is $C^p$-smooth \citep{van1996geometric}. This completes the proof. |
|
|
\end{proof} |
|
|
|
|
|
Now that we have proved that the function $\theta_t(d)$ is definable, we can obtain the bound $\|\theta_t^* - \Tilde{\theta}_t^*\|$. Intuitively, our proof exploits the fact that continuous and definable functions exhibit controlled behaviors along any path, even if it crosses over a finite number of Whitney strata. |
|
|
\begin{lemma}\label{lem:DefinableTheta} |
|
|
For any $\tilde{d}$ such that $\mathcal{F}_{t,\tilde d}$ is non-empty, the following holds: |
|
|
\begin{align*} |
|
|
\|\theta_t^* - \Tilde{\theta}_t^*\| = \|\theta(d) - \theta(\Tilde{d})\| = \mathcal{O}(\|d-\tilde{d}\|). |
|
|
\end{align*} |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
Since every smooth function over a bounded set is Lipschitz, let us denote $L_d$ as the maximum of the Lipschitz constants for all the cells of the Whitney stratification of $\theta_t(d)$. Let $d(\lambda) = \lambda d + (1-\lambda)\Tilde{d}$, where $0 \leq \lambda \leq 1$, be the curve that connects between $d$ and $\Tilde{d}$. Also, let $0 = \lambda_1 \leq ... \leq \lambda_n = 1$ be the partition such that $\theta_t(d(\lambda))$ belongs to one cell for all $\lambda_i < \lambda < \lambda_{i+1}$ for $i = 1, ..., n-1$. We know that $n < \infty$ since $d(\theta_t)$ is Whitney stratifiable. Thus, |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\|\theta_t(d) - \theta_t(\Tilde{d})\| & \leq \sum_{i=1}^{n-1}\|\theta_t(d(\lambda_i)) - \theta_t(d(\lambda_{i+1}))\| \\ |
|
|
& \leq L_d \sum_{i=1}^{n-1}\|d(\lambda_i) - d(\lambda_{i+1})\| \\ |
|
|
& \leq L_d \|d - \Tilde{d}\|\sum_{i=1}^{n-1}|\lambda_{i+1} - \lambda_i| \\ |
|
|
& = L_d\|d - \Tilde{d}\| |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where the first inequality is due to triangle inequality, the second inequality is due to Lipschitz continuity, the third inequality is due to the definition of $d(\lambda)$, the first equality is due to the non-decreasing sequence of $\lambda_i$. |
|
|
\end{proof} |
|
|
|
|
|
\textbf{Bounding the term $\|\Tilde{\theta}_t^* - \hat{\theta}_t \|$.} Recall that $\hat{\theta}_t$ is the parameter for $\hat{\pi}_t$ (output of within-task CRPO), and $\Tilde{\theta}_t^*$ is the parameter for an optimal solution with an enlarged feasible set $\mathcal{F}_{t,\tilde d}$. In this subsection, we will obtain the upper bound for the term $\|\Tilde{\theta}_t^* - \hat{\theta}_t \|$. Let $f(\theta,\tilde{d}) = J_{t,0}(\pi_{\theta}) + \mathbb{I}_{{\mathcal{F}}_{t,\tilde{d}}}(\pi_\theta)$, and choose $\tilde{d}_{i}=\mathcal{O}(1/\sqrt{M})$ for $i=1,...,p$, which coincides with the upper bound on constraint violation for within-task CRPO such that $\hat\theta_t\in {\mathcal{F}}_{t,\tilde{d}}$ with high probability. In the next result, we will condition on this high-probability event. |
|
|
|
|
|
\begin{lemma}\label{lem:theta-tilde-bd} |
|
|
With the choice of $\tilde{d}=d+\delta$, where $\delta=\mathcal{O}(1/\sqrt{M})$ coincides with the upper bound on constraint violation for within-task CRPO such that $\hat\theta_t\in {\mathcal{F}}_{t,\tilde{d}}$, the following holds: |
|
|
\begin{equation*} |
|
|
\|\Tilde{\theta}_t^* - \hat{\theta}_t \| \leq \mathcal{O}\left(h\left({1}/{\sqrt{M}}\right)\right), |
|
|
\end{equation*} |
|
|
where $h$ is a strictly increasing continuous function with the property that $h(0)=0$ as specified in Lemma \ref{prop:Bolte}. |
|
|
|
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
Without loss of generality, consider $f(\theta) = J_{t,0}(\pi_{\theta}) + \mathbb{I}_{{\mathcal{F}}_{t,\tilde{d}}}(\pi_\theta)+c$, where $c \coloneqq - \inf J_{t,0}(\pi_{\theta}) + \mathbb{I}_{\Tilde{\mathcal{F}}_t}(\pi_\theta)$, so that the minimal value of $f$ is translated to $0$. For simplicity, also assume that $f(\hat{\theta}_t) \leq \mathcal{X}(\|\hat{\theta}_t\|)$. Note that this assumption can be relaxed by using the concept of ``curves of maximal slope'' at the cost of slightly more complicated analysis and bounds \citep{ioffe2009invitation}. |
|
|
|
|
|
Now, consider a subgradient flow $\Dot{\theta}(\tau) \in -\partial f (\theta(\tau))$ (see Definition \ref{def:subgflow}), initialized at $\theta(0)=\hat{\theta}_t$ then, for any $0 \leq s'<s$, we have that |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
h \big(f(\theta(s'))\big) - h \big(f(\theta(s)) \big) & = \int_s^{s'} \frac{d}{d \tau}h\big(f(\theta(\tau)) \big)d\tau \\ |
|
|
& = \int_{s'}^s h'\big(f(\theta(\tau))\big)\|\Dot{\theta}(\tau)\|^2 d\tau \\ |
|
|
& \geq \int_{s'}^s \|\Dot{\theta}(\tau)\|d\tau \\ |
|
|
& \geq \Bigg\|\int_{s'}^s \Dot{\theta}(\tau)d\tau\Bigg\|\\ |
|
|
&=\|\theta(s)-\theta(s')\| |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where the second equality is due to the property of the subgradient flow (see Sec. \ref{sec:subflow}), the first inequality is due to $\|\partial^0(h f)\big(\theta(\tau)\big)\|\geq 1$ from Proposition \ref{prop:Bolte}, and the second inequality is due to the triangle inequality. Thus, by taking $s' = 0$ and $s \rightarrow \infty$, |
|
|
we have shown that |
|
|
\begin{align*} |
|
|
h(f(\hat{\theta}_t)) \geq \|\hat{\theta}_t - \Tilde\theta_t^*\|. |
|
|
\end{align*} |
|
|
Therefore, |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\|\hat{\theta}_t - \Tilde{\theta}_t^*\| & \leq h(f(\hat{\theta}_t) - f(\Tilde\theta_t^*)) \\ \quad & \overset{(i)}{\leq} h(J_{t,0}(\hat{\pi}_t) - J_{t,0}(\Tilde{\pi}_t^*)), |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where the first inequality is due to the optimality of $\Tilde{\pi}_t^*$, and $(i)$ follows since both $\hat{\pi}_t$ and $\Tilde{\pi}_t^*$ are feasible for $\mathcal{F}_{t, \Tilde{d}}$. Note that the suboptimality bound can be split as |
|
|
\begin{equation*} |
|
|
h(J_{t,0}(\hat{\pi}_t) - J_{t,0}(\Tilde{\pi}_t^*)) = h\bigg(J_{t,0}(\hat{\pi}_t) - J_{t,0}(\pi_t^*) + J_{t,0}(\pi_t^*) - J_{t,0}(\Tilde{\pi}_t^*) \bigg). |
|
|
\end{equation*} |
|
|
|
|
|
By CRPO, we can bound the value difference $J_{t,0}(\hat{\pi}_t) - J_{t,0}(\pi_t^*) \leq \mathcal{O}({1}/{\sqrt{M}} )$. Moreover, Since the value function is Lipschitz \cite[Lemma 4]{xu2020improving}, the value difference $J_{t,0}(\pi_t^*) - J_{t,0}(\Tilde{\pi}_t^*)$ can be bounded by the distance $\|\theta_t^* - \Tilde{\theta}_t^*\|$, which is bounded again by $ \mathcal{O}({1}/{\sqrt{M}} )$ according to Lemma \ref{lem:DefinableTheta}. Hence, recognizing that $h$ is strictly increasing, we have proved the claim. |
|
|
\end{proof} |
|
|
|
|
|
|
|
|
\textbf{Bounding the term $\|{\theta}_t^* - \hat{\theta}_t \|$.} Finally, we are able to bound the term of our original interests. |
|
|
\begin{lemma}\label{lem:bound-definable-final} |
|
|
Under assumption \ref{asmptn:Definable}, the following holds: |
|
|
\begin{equation*} |
|
|
\|{\theta}_t^* - \hat{\theta}_t \| \leq \mathcal{O}\left(h\left(\frac{1}{\sqrt{M}}\right)+\frac{1}{\sqrt{M}}\right), |
|
|
\end{equation*} |
|
|
where $h$ is a strictly increasing continuous function with the property that $h(0)=0$ as specified in Lemma \ref{prop:Bolte}. |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
The claim follows directly from Lemmas \ref{lem:DefinableTheta} and \ref{lem:theta-tilde-bd} and the triangle inequality. |
|
|
\end{proof} |
|
|
\begin{remark} |
|
|
Note that our strategy to bound $\|{\theta}_t^* - \hat{\theta}_t \|$ is algorithmic-agnostic as it only relies on the optimization landscape. The only place we rely on the algorithm is to bound the suboptimality gap, which is then converted to a bound on $\|\Tilde{\theta}_t^* - \hat{\theta}_t \|$ in Lemma \ref{lem:theta-tilde-bd}. Also, the enlargement of the feasible set should be viewed as a proof technique and has no implications for the algorithm design. Indeed, the motivation for the enlargement is to properly design a subgradient flow system. Thus, the result of Lemma \ref{lem:bound-definable-final} is not conditioned on how the enlargement is performed. Also, note that definability is used differently in Lemmas \ref{lem:DefinableTheta} and \ref{lem:theta-tilde-bd}. In the former case, we exploit the Whitney stratification property to provide an upper bound, while in the latter case, we exploit the KL property to obtain a lower bound, hence they serve different purposes. |
|
|
\end{remark} |
|
|
|
|
|
|
|
|
|
|
|
\subsection{Bounding term \textit{(A)}: \texorpdfstring{$|\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)]|$}{[1]}} |
|
|
|
|
|
The result from Lemma \ref{lem:bound-definable-final} can be used directly to provide bounds for \emph{(A)} and \emph{(C)}. We start with the term \emph{(A)}. |
|
|
\begin{lemma}\label{lem:BoundonA} |
|
|
The following bound holds: |
|
|
\begin{equation} |
|
|
{|\mathbb{E}_{s\sim \nu_t^*}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{s\sim \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)]|}=\mathcal{O}\left(h\left(\frac{1}{\sqrt{M}}\right)+\frac{1}{\sqrt{M}}\right) |
|
|
\end{equation} |
|
|
where $h$ is a strictly increasing continuous function with the property that $h(0)=0$ as specified in Lemma \ref{prop:Bolte}. |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
\begin{align*} |
|
|
& |\mathbb{E}_{s\sim \nu_t^*}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{s\sim \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)]| \\&= |
|
|
\bigg|\sum_{s \in \mathcal{S}_t}\big(\nu_t^*(s) - \Tilde{\nu}_t(s)\big)D_{KL}(\pi_t^*(s)|\pi(s)) \bigg| \\ &\leq C_{\pi}\|\nu_t^* - \Tilde{\nu}_t\|_1\\ |
|
|
&\leq 2C_{\pi}C_\nu\|\theta_t^*-\hat{\theta}_t\|_2 |
|
|
\end{align*} |
|
|
where the first equality is by definition, the first inequality is due to Assumption \ref{asmptn:newAsmptn1}, and the second inequality is due to \cite[Lem. 3]{xu2020improving}, which also specifies the constant $C_\nu$, and \cite[Prop. 4.2]{levin2017markov}. The result then follows by recalling the result from Lemma \ref{lem:bound-definable-final}. |
|
|
\end{proof} |
|
|
|
|
|
|
|
|
|
|
|
\subsection{Bounding term (C): \texorpdfstring{$|\mathbb{E}_{\hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\hat{\pi}_{t}|\pi)]|$}{[1]}} |
|
|
Similarly, we can prove the upper bound for the error term $(C)$. |
|
|
\begin{lemma}\label{lem:BoundonC} |
|
|
The following bound holds: |
|
|
\begin{equation} |
|
|
|\mathbb{E}_{s\sim \hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{s\sim \hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\pi)]|=\mathcal{O}\left(h\left(\frac{1}{\sqrt{M}}\right)+\frac{1}{\sqrt{M}}\right) |
|
|
\end{equation} |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
|
|
|
\begin{align*} |
|
|
& |\mathbb{E}_{s\sim \hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{s\sim \hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\pi)]| \\ |
|
|
&= \bigg| \sum_{s \in \mathcal{S}}\hat{\nu}_t(s) \bigg( D_{KL}(\pi_t^*|\pi) - D_{KL}(\hat{\pi}_t|{\pi}) \bigg)\bigg|\\ |
|
|
&\leq \sum_{s \in \mathcal{S}}\hat{\nu}_t(s) \bigg|D_{KL}(\pi_t^*|\pi) - D_{KL}(\hat{\pi}_t|{\pi}) \bigg| \\ |
|
|
&\leq L_g \sum_{s \in \mathcal{S}}\hat{\nu}_t(s) \|\pi_t^*(s) - \hat{\pi}_t(s)\|_2 \\ |
|
|
&\leq L_g\sum_{s \in \mathcal{S}}\hat{\nu}_t(s) \| \theta_t^* - \hat{\theta}_{t} - c'1\|_{\infty} \\\ |
|
|
&\leq L_g \| \theta_t^* - \hat{\theta}_{t}\|_2 |
|
|
\end{align*} |
|
|
where the first inequality is due to the non-negativity of $\hat{\nu}_t(s)$ and triangle inequality, the second inequality is due to Assumption \ref{asmptn:newAsmptn1}, the third inequality holds for any constant $c'$ and is due to \cite[Lem. 24]{mei2020global}, and the last inequality is due to $\sum_{s \in \mathcal{S}}\hat{\nu}_t(s)=1$ and by choosing $c'=0$. The result then follows by recalling the result from Lemma \ref{lem:bound-definable-final}. |
|
|
\end{proof} |
|
|
|
|
|
\subsection{Bounding term (B): \texorpdfstring{$\left|\mathbb{E}_{ \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)] \right|$}{[1]}} |
|
|
Now, we will upper bound the error term $(B)$. The proof follows DualDICE \citep{nachum2019dualdice}. We introduce the following notations. Let $\hat{\mathbb{E}}_{d^{\mathcal{D}_t}}$ denote an average of empirical samples where $\{s_i, a_i, r_i, s_i'\}_{i=1}^{N} \sim d^{\mathcal{D}_t}$, and $\rho_t$ be the initial state distribution for the CMDP task $t$. The number of data points $N=\mathcal{O}(M^{1+1/\sigma})$, where $\sigma$ is any positive number $\sigma\in(0,1)$. Note that the additional factor of $\mathcal{O}(M^{1/\sigma})$ results from the critic evaluation per policy update (see \cite[Thm. 1]{xu2021crpo}). We will roughly bound $N=\mathcal{O}(M^{2})$ in the following to simplify the presentation. The stationary distribution correction factor is denoted as $w_{\hat{\pi}_t/\mathcal{D}_t}(s,a) = \frac{\Tilde{\nu}_t(s,a)}{d^{\mathcal{D}_t}(s,a)}$. |
|
|
|
|
|
We make the following regularity assumption on the distribution $d^{\mathcal{D}_t}$ with respect to the target policy $\hat{\pi}_t$ \cite[Asm. 1]{nachum2019dualdice}. |
|
|
|
|
|
\begin{assumption}[Reference distribution property] |
|
|
For any $(s,a)$, $\tilde{\nu}_t(s,a)>0$ implies that $d^{\mathcal{D}_t}(s,a)>0$. Furthermore, the correction terms are bounded by some finite constant $C_{\omega}$: $\|\omega_{\Tilde{\nu}_t/\mathcal{D}_t}\|_\infty\leq C_{\omega}$. |
|
|
\end{assumption} |
|
|
|
|
|
|
|
|
For convenience, we recapitulate the key points from DualDICE, where we also omit the task dependence $t$ (i.e., we use $d^\mathcal{D}$, $\pi$, and $\rho$ in lieu of $d^{\mathcal{D}_t}$, $\hat{\pi}_t$, and $\rho_t$, respectively). The objective function is given by |
|
|
\begin{align} |
|
|
J(z, \zeta) = & \E_{(s,a,s'), a'\sim\pi(s')}\left[(z(s,a) - \gamma z(s',a'))\zeta(s,a) - \zeta(s,a)^2 / 2\right] \\ |
|
|
& - (1 - \gamma)~\E_{s_0\sim\beta,a_0\sim\pi(s_0)} \left[ z(s_0,a_0) \right]. |
|
|
\end{align} |
|
|
|
|
|
The objective in the form prior to introduction of $\zeta$ is denoted as $J(z)$: |
|
|
\begin{align} |
|
|
J(z) = \frac{1}{2}\E_{(s,a)}\left[(z - \mathcal{T}^{\pi}z)(s,a)^2\right] |
|
|
- (1 - \gamma)~\E_{s_0\sim\beta,a_0\sim\pi(s_0)} \left[ z(s_0,a_0) \right]. |
|
|
\label{eq:Jz-dualdice} |
|
|
\end{align} |
|
|
|
|
|
Let $\hat{J}(z, \zeta)$ denotes the empirical surrogate of $J(z, \zeta)$ with optimal solution as $(\hat z^*, \hat\zeta^*)$. We denote $z^*_\mathcal{F} = \argmin_{z\in\mathcal{F}} J(z)$ and $z^* = \argmin_{z: S\times A\rightarrow \mathbb{R}} J(z)$. We denote $L(z) = \max_{\zeta\in \mathcal{H}} J(z, \zeta)$ and $\hat L(z)=\max_{\zeta\in \mathcal{H}} \hat J(z, \zeta)$ as the primal objectives, and $\ell(\zeta) = \min_{z\in\mathcal{F}} J(z, \zeta)$, $\hat\ell(\zeta) = \min_{z\in\mathcal{F}} \hat J(z, \zeta)$ as the dual objectives. We apply some optimization algorithm $OPT$ for optimizing $\hat J(z, \zeta)$ with samples $\{s_i, a_i, r_i, s'_i\}_{i=1}^N$, $\{s^i_0\}_{i=1}^N\sim \beta$, and target actions $a'_i\sim\pi(s'_i),a^i_0\sim\pi(s^i_0)$ for $i=1,\dots,N$. |
|
|
The output of $OPT$ is denoted by $(\hat{z}, \hat{\zeta})$. We also make the following definitions to capture the error of approximation with $\mathcal{F}$ for $z$ and $\mathcal{H}$ for $\zeta$ in optimizing $\hat J(z, \zeta)$: |
|
|
\begin{align} |
|
|
\epsilon_{approx}(\mathcal{F})&\coloneqq\sup_{z\in S\times A\rightarrow\ \mathbb{R}}\inf_{z_{\mathcal{F}}\in\mathcal{F}}\left(\|z_{\mathcal{F}}-z\|_{\mathcal{D},1}+\| z_{\mathcal{F}}-z\|_{\rho\pi,1} \right)\\ |
|
|
\epsilon_{approx}(\mathcal{H})&\coloneqq\sup_{\zeta\in S\times A\rightarrow\ \mathbb{R}}\inf_{\zeta_{\mathcal{H}}\in\mathcal{H}}\left(\|\zeta_{\mathcal{H}}-\zeta\|_{\mathcal{D},1}+\| \zeta_{\mathcal{H}}-\zeta\|_{\rho\pi,1} \right)\\ |
|
|
\epsilon_{approx}(\mathcal{F},\mathcal{H})&\coloneqq\epsilon_{approx}(\mathcal{F})+\epsilon_{approx}(\mathcal{H})\label{eq:eps_FH} |
|
|
\end{align} |
|
|
We also define |
|
|
\begin{equation} |
|
|
\epsilon_{opt}\coloneqq{\|\hat\zeta - \hat{\zeta}^*\|^2_{\mathcal{D}_t} + \big \|\big( \hat{z}^* - \hat{\mathcal{B}}^{\pi}\hat{z}^*\ \big) - \big(\hat{z} - \hat{\mathcal{B}}^\pi \hat{z} \big)\big\|^2_{\mathcal{D}_t} }\label{eq:opt} |
|
|
\end{equation} |
|
|
as the optimization error of OPT from DualDICE. |
|
|
|
|
|
\begin{lemma}\label{lem:BoundonB} |
|
|
By estimating $\hat{\nu}_t$ with DualDICE, the following bound holds: |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\left|\mathbb{E}_{ \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)] \right|=\mathcal{O}\left(\sqrt{\frac{1}{M}+\epsilon_{opt}+\epsilon_{approx}(\mathcal{F},\mathcal{H})}\right), |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
\end{lemma} |
|
|
|
|
|
|
|
|
\begin{proof} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
We begin with the following decomposition: |
|
|
\begin{align*} |
|
|
&\left(\mathbb{E}_{ \Tilde{\nu}_t}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\pi_t^*|\pi)] \right)^2 \leq \\ |
|
|
& \qquad \underbrace{2\bigg(\hat{\mathbb{E}}_{d^{\mathcal{D}_t}} \sum_a\big((\hat{z} - \hat{\mathcal{B}}^{\hat{\pi}_t}\hat{z})(s,a) - (\hat{z}^* - \hat{\mathcal{B}}^{\hat{\pi}_t}\hat{z}^*)(s,a)\big)D_{KL}(\pi_{t}^*|{\pi}) \bigg)^2}_{\epsilon_1} \\ |
|
|
& \qquad+\underbrace{2\bigg(\hat{\mathbb{E}}_{d^{\mathcal{D}_t}} \sum_a (\hat{z}^* - \hat{\mathcal{B}}^{\hat{\pi}_t}\hat{z}^*)(s,a)D_{KL}(\pi_{t}^*|{\pi}) - \mathbb{E}_{d^{\mathcal{D}_t}}\sum_a\omega_{\hat{\pi}_t/\mathcal{D}_t}(s,a)D_{KL}(\pi_{t}^*|{\pi})) \bigg)^2}_{\epsilon_2} . |
|
|
\end{align*} |
|
|
We will bound each term above separately. |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\epsilon_1 & \leq 2C_{\pi}^2 \bigg(\hat{\mathbb{E}}_{d^{\mathcal{D}_t}}\sum_a\big(\hat{z} - \hat{\mathcal{B}}^{\hat{\pi}_t}\hat{z} \big)(s,a) - \big(\hat{z}^* - \hat{\mathcal{B}}^{\hat{\pi}_t}{\hat{z}^*} \big)(s,a) \bigg)^2 \\ |
|
|
& \leq 2C_\pi^2 \Bigg(\underbrace{\|\hat\zeta - \hat{\zeta}^*\|^2_{\mathcal{D}_t} + \big \|\big( \hat{z}^* - \hat{\mathcal{B}}^{\pi}\hat{z}^*\ \big) - \big(\hat{z} - \hat{\mathcal{B}}^\pi \hat{z} \big)\big\|^2_{\mathcal{D}_t} }_{\epsilon_{opt}}\Bigg), |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where the error $\epsilon_{opt}$ is induced by the optimization OPT. The error term $\epsilon_2$ can be decomposed as |
|
|
\begin{equation} |
|
|
\begin{aligned} |
|
|
\epsilon_2 \leq & 2\underbrace{C_\pi^2 \bigg(\hat{\mathbb{E}}_{d^{\mathcal{D}_t}}\sum_a\big(\hat{z}^* - \hat{\mathcal{B}}^{\hat{\pi}_t}{\hat{z}^*} \big)(s,a) - \mathbb{E}_{d^{\mathcal{D}_t}}\sum_a \big(\hat{z}^* - {\mathcal{B}}^{\hat{\pi}_t}\hat{z}^* \big)(s,a) \bigg)^2}_{\epsilon_{stat}} \\ |
|
|
& \qquad+ 2C_\pi^2\bigg({\mathbb{E}}_{d^{\mathcal{D}_t}}\sum_a\big(\big(\hat{z}^* - {\mathcal{B}}^{\hat{\pi}_t}{\hat{z}^*} \big)(s,a) - \omega_{\hat{\pi}_t/\mathcal{D}_t}(s,a)\big) \bigg)^2\\ |
|
|
= & 2{\epsilon_{stat}} + 2C_\pi^2\bigg({\mathbb{E}}_{d^{\mathcal{D}_t}}\sum_a\big(\big(\hat{z}^* - {\mathcal{B}}^{\hat{\pi}_t}{\hat{z}^*} \big)(s,a) - \big({z}^* - {\mathcal{B}}^{\hat{\pi}_t}{{z}^*} \big)(s,a)\big) \bigg)^2, |
|
|
\end{aligned} |
|
|
\end{equation} |
|
|
where the equality is due to the result that ${z}^* - {\mathcal{B}}^{\hat{\pi}_t}{{z}^*}(s,a)=\omega_{\hat{\pi}_t/\mathcal{D}_t}(s,a)$ (see \cite[Eq. 17]{nachum2019dualdice}) and $\epsilon_{stat}$ is the error due to the finite number error. By \cite[Lem. 7]{nachum2019dualdice}, $\epsilon_{stat}=\mathcal{O}\left(\frac{\log M+\log \frac{1}{\delta}}{M^2}\right)$ with probability at least $1-\delta$, where we use the bound on the number of data as $\mathcal{O}(M^2)$. To bound the second term, use the fact that $J(z)$ as defined in \eqref{eq:Jz-dualdice} is $1$-strongly convex. Hence, |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
&\bigg({\mathbb{E}}_{d^{\mathcal{D}_t}}\sum_a\big(\big(\hat{z}^* - {\mathcal{B}}^{\hat{\pi}_t}{\hat{z}^*} \big)(s,a) - \big(z^* - {\mathcal{B}}^{\hat{\pi}_t}z^* \big)(s,a)\big) \bigg)^2 \\ |
|
|
&\leq \|\big(\hat{z}^* - {\mathcal{B}}^{\hat{\pi}_t}{\hat{z}^*} \big) - \big(z^* - {\mathcal{B}}^{\hat{\pi}_t}z^* \big)\|_{\mathcal{D}_t}^2\\ |
|
|
& \leq 2\big(J(\hat{z}^*) - J(z^*)\big) |
|
|
|
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
|
|
|
where $(i)$ follows from \citep[Section D.1]{nachum2019dualdice} $\epsilon_{approx}(\mathcal{F})$ is the error due to the approximation with $\mathcal{F}$ for $z$, $\epsilon_{approx}(\mathcal{H})$ is the error due to the approximation with $\mathcal{H}$ for $\zeta$, and $\epsilon_{est}$ is the estimation error, and $L$ is the Lipschitz constant for $f$. |
|
|
|
|
|
|
|
|
To bound the error between $J(\hat{z}^*)$ and $J(z^*)$, we use the decomposition suggested in \citep{nachum2019dualdice}: |
|
|
\begin{equation} |
|
|
J(\hat{z}^*)-J(z^*)=\underbrace{J(\hat{z}^*)-L(\hat{z}^*)}_{(i)}+\underbrace{L(\hat{z}^*)-L({z}^*_{\mathcal{F}})}_{(ii)}+\underbrace{L({z}^*_{\mathcal{F}})-J({z}^*_{\mathcal{F}})}_{(iii)}+\underbrace{J({z}^*_{\mathcal{F}})-J(z^*)}_{(iv)}, |
|
|
\end{equation} |
|
|
where $(i)\leq\frac{2C_\omega}{1-\gamma}\|\zeta^*_\mathcal{H}-\zeta^*\|_{\mathcal{D}_t,1}\leq\frac{2C_\omega}{1-\gamma}\epsilon_{approx}(\mathcal{H})$, $(ii)=\mathcal{O}\left(\frac{\sqrt{\log M+\log\frac{1}{\delta}}}{M}\right)$ by \cite[Lem. 6]{nachum2019dualdice} (by also plugging in $N=\mathcal{O}(M^2)$), $(iii)\leq 0$ by definition, and $(iv)=\mathcal{O}(\epsilon_{approx}(\mathcal{F}))$. Note that we refer the reader to \cite[Sec. D.1]{nachum2019dualdice} for the above bounds. Therefore, we can bound $J(\hat{z}^*)-J(z^*)$ on the order of $\mathcal{O}\left(\epsilon_{approx}(\mathcal{H})+\epsilon_{approx}(\mathcal{F})+\frac{\sqrt{\log M+\log\frac{1}{\delta}}}{M}\right)$. |
|
|
|
|
|
Combining the above relations, while noting that $\frac{1}{\sqrt{M}}$ decreases slower than $\frac{1}{{M}}$ in terms of $M$ and is thus kept as the upper bound, we have shown the result. |
|
|
\end{proof} |
|
|
|
|
|
|
|
|
\subsection{Putting it together: bounding the KL divergence estimation error} |
|
|
\begin{theorem}[KL divergence estimation error bound] \label{thm:dualDICEAppendix} The following bound holds: |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\quad & |\mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi)] - \mathbb{E}_{ \hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\pi)]| \\ \quad & = \mathcal{O}\bigg(h\left(\frac{1}{\sqrt{M}}\right)+\frac{1}{\sqrt{M}}+ \sqrt{\epsilon_{opt}}+\sqrt{\epsilon_{approx}(\mathcal{F},\mathcal{H})}\bigg), |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where $h$ is a strictly increasing continuous function with the property that $h(0)=0$ as specified in Lemma \ref{prop:Bolte}, $\epsilon_{approx}(\mathcal{F},\mathcal{H})$ is defined in \eqref{eq:eps_FH}, and $\epsilon_{opt}$ is defined in \eqref{eq:opt}. |
|
|
\end{theorem} |
|
|
\begin{proof} |
|
|
The result follows by combining the upper bounds for the error terms $(A)$, $(B)$ and $(C)$, as specified by Lemmas \ref{lem:BoundonA}, \ref{lem:BoundonB}, and \ref{lem:BoundonC}. We also apply the elementary inequality $\sqrt{a+b+c}\leq\sqrt{a}+\sqrt{b}+\sqrt{c}$ to further simplify the bound. |
|
|
\end{proof} |
|
|
|
|
|
\begin{remark} |
|
|
The bound above depends on the number of iterations $M$ per task in different ways. By increasing $M$, we can expect to reduce the suboptimality gap, which can help reduce the distance between $\hat{\pi}_t$ to the optimal set of policies. Also, increasing $M$ results in a larger dataset used to estimate its stationary distribution offline by DualDICE, which reduces the estimation error. The bound indicates that the only terms that do not vanish as we increase the number of iterations per task are those due to the inherent optimization error $\epsilon_{opt}$ and function approximation error $\epsilon_{approx}(\mathcal{F},\mathcal{H})$. In the case those terms are negligible (which are possible in view of the recent breakthrough in overparametrized deep learning \citep{zhang2021understanding,neyshabur2019towards,li2018learning,zou2018stochastic,arora2019fine}, see also \citep{fan2021selective} for a survey), then the KL divergence estimation can be driven to arbitrary accuracy. |
|
|
\end{remark} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{Proofs for the section \ref{subsec:adapt_learning_rates}} |
|
|
|
|
|
\subsection{TAOG and TACV bounds for CRPO with adaptive learning rates} \label{subsec:appAdaptiveRate} |
|
|
This section presents the task-averaged regret upper bounds for the CRPO when the adaptive learning rates $\alpha_t$ are used for each task, and the Q-estimation error is accounted for. We also recall that $d_{t,i}$ is the constraint upper bound for $i=1,...,p$ and $\eta_t$ is the tolerance for constraint violation (i.e., increasing the upper bound to $d_{t,i}+\eta_t$). For a single run of CRPO in task $t$, we denote $\mathcal{N}_{t,0}$ as the set of time steps the reward is maximized and $\mathcal{N}_{t,i}$ as the set of time step constraint $i$ is minimized. The Q-function in the CRPO algorithm is learned through TD learning with the total number of iterations denoted by $K_{in}$. The Q-function of objective $i$ for policy $\pi_{t,m}$ at time step $m$ is denoted by $Q_{t,m}^i$, and the estimated Q-function is denoted by $\Bar{Q}_{t,m}^i$. The maximum value for both rewards and constraints is assumed to be $c_{max}$. |
|
|
|
|
|
With all notations for CRPO in place, |
|
|
we present the following result \citep{xu2021crpo} |
|
|
|
|
|
\begin{lemma}\label{lem:appCRPOLem8} For the CRPO algorithm in the tabular settings with learning rates $\alpha_{t}$, the following bound holds: |
|
|
\label{lem:Lem8_CRPO} |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
& \alpha_{t}\sum_{m \in \mathcal{N}_{t,0}}\Big(J_{t,0}(\pi_t^*) - J_{t,0}(\pi_{t,m}) \Big) + \eta_t \alpha_{t}\sum_{i=1}^p |\mathcal{N}_{t,i}| \\ & \leq \mathbb{E}_{s\sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + \frac{2c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}\alpha_{t}^2\sum_{i=0}^p|\mathcal{N}_{t,i}| \\ & + \sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\frac{\alpha_{t}(3+(1-\gamma)^2+3\alpha_{t}c_{max})}{(1-\gamma)^2}\|Q^i_{t,m} - \bar{Q}^{i}_{t,m}\|_2 |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
\end{lemma} |
|
|
|
|
|
\begin{proof} |
|
|
|
|
|
|
|
|
If $m \in \mathcal{N}_{t,0}$, by \citep[Lemma 7]{xu2021crpo}, we have that: |
|
|
\begin{align} |
|
|
\alpha_{t}\big(J_{t,0}(\pi_t^*) - J_{t,0}(\pi_{t,m}) \big) & \leq \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,m}) - D_{KL}(\pi_t^*|\pi_{t,m+1})]+ \frac{2c^2_{max}|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3} \alpha_{t}^2 \nonumber\\ &\qquad + \frac{3\alpha_{t}(1+\alpha_{t}c_{max})}{(1-\gamma)^2}\|Q_{t,m}^0 - \bar{Q}_{t,m}^0\|_2.\label{eq:appEq12CRPO} |
|
|
\end{align} |
|
|
Similarly, if $m \in \mathcal{N}_{t,i}$, we can write |
|
|
\begin{align} |
|
|
\alpha_{t}\big(J_{t,i}(\pi_{t,m}) - J_{t,i}(\pi_t^*) \big) & \leq \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,m}) - D_{KL}(\pi_t^*|\pi_{t,m+1})]+ \frac{2c^2_{max}|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3} \alpha_{t}^2 \nonumber \\ & \qquad+ \frac{3\alpha_{t}(1+\alpha_{t}c_{max})}{(1-\gamma)^2}\|Q_{t,m}^i - \bar{Q}_{t,m}^i\|_2.\label{eq:appEq13CRPO} |
|
|
\end{align} |
|
|
Taking the summation of \eqref{eq:appEq12CRPO} and \eqref{eq:appEq13CRPO} from $m = 0$ to $M-1$, we get |
|
|
\begin{align*} |
|
|
&\alpha_{t} \sum_{m \in \mathcal{N}_{t,0}}\big(J_{t,0}(\pi_t^*) - J_{t,0}(\pi_{t,m}) \big) + \sum_{i=1}^p\sum_{m \in \mathcal{N}_{t,i}}\alpha_{t}\big(J_{t,i}(\pi_{t,m}) - J_{t,i}(\pi_t^*) \big) \\ |
|
|
&\leq \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + \frac{2c^2_{max}|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3} \sum_{i=0}^p\alpha_{t}^2|\mathcal{N}_{t,i}| \\ |
|
|
&\qquad\qquad\qquad+ \sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\frac{3\alpha_{t}(1+\alpha_{t}c_{max})}{(1-\gamma)^2}\|Q_{t,m}^i - \bar{Q}_{t,m}^i\|_2 |
|
|
\end{align*} |
|
|
Since $J_{t,i}(\pi_{t,m}) - J_{t,i}(\pi_t^*) \geq \eta_t - \|Q_{t,m}^i - \bar{Q}_{t,m}^i\|$ \cite[Eq. 15]{xu2021crpo}, by rearranging the terms above, we obtain the result. |
|
|
\end{proof} |
|
|
|
|
|
Next, we study the condition on the maximum constraint violation threshold $\eta_t$ and how it affects $\mathcal{N}_{t,0}$ and the upper bounds for TAOG and TACV. We make the following assumption to proceed. |
|
|
\begin{assumption}\label{asm:crpo_cj} |
|
|
Assume that $\sum_{m\in\mathcal{N}_{t,0}}J_{t,0}(\pi_t^*)-J_{t,0}(\pi_{t,m})\geq c_J$ for some $c_J\in(-\frac{1}{2}\alpha_{t}\eta_tM,0]$. |
|
|
\end{assumption} |
|
|
The assumption above indicates that the policies in $\mathcal{N}_{t,0}$ do not have rewards higher than the optimal policy by more than $\frac{1}{2}\alpha_{t}\eta_t$ on average. Note that it is indeed possible to have rewards higher than the optimal policy if the corresponding policy does not satisfy some safety constraints (i.e., infeasible policy). However, it is not a strong assumption since we are comparing it with the optimal policy. The above assumption is not present in \citep{xu2021crpo}, which invalidates one of its derivation steps (in particular, \cite[Thm. 3]{xu2021crpo}), and is thus introduced to rectify the proof. |
|
|
|
|
|
\begin{lemma}\label{lem:appCRPOlem9} |
|
|
Suppose that the following condition holds: |
|
|
\begin{align} |
|
|
\frac{1}{2}\eta_t M\alpha_{t} & \geq \mathbb{E}_{ \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + \frac{2c^2_{max}|\mathcal{S}||\mathcal{A}|M}{(1-\gamma)^3} \sum_{i=0}^p \alpha_{t}^2 \nonumber\\ |
|
|
& + \sum_{i=0}^p \sum_{m \in \mathcal{N}_{t,i}} \frac{\alpha_{t}(3+(1-\gamma)^2+3\alpha_{t}c_{max})}{(1-\gamma)^{2}}\|Q_{t,m}^i - \bar{Q}_{t,m}^i\|_2.\label{eq:crpo_eta} |
|
|
\end{align} |
|
|
Then, we have that $\mathcal{N}_{t,0}\neq \emptyset$, i.e., $\hat{\pi}_t$ is well-defined; also, one the following two statements must hold, |
|
|
\begin{enumerate} |
|
|
\item $|\mathcal{N}_{t,0}| \geq M/2$, |
|
|
\item $\sum_{m \in \mathcal{N}_{t,0}}\Big(J_{t,0}(\pi_t^*) - J_{t,0}(\pi_{t,m}) \Big) \leq 0$. |
|
|
\end{enumerate} |
|
|
Under assumption \ref{asm:crpo_cj}, we also have the following holds: |
|
|
$$|\mathcal{N}_{t,0}| \geq \bigg(\frac{1}{2} - \kappa \bigg)M$$ |
|
|
for some $\kappa\in(0,\frac{1}{2})$. |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
The proof for $\mathcal{N}_{t,0}\neq \emptyset$ follows directly from \cite[Lem. 9]{xu2021crpo}. For the second statement, we consider the case that $\sum_{m \in \mathcal{N}_{t,0}}\Big(J_{t,0}(\pi_t^*) - J_{t,0}(\pi_{t,m}) \Big) > 0$. From Lemma \ref{lem:appCRPOLem8}, it implies that |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\eta_t\sum_{i=1}^p \alpha_{t}|\mathcal{N}_{t,i}| & \leq \mathbb{E}_{s\sim \nu^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + \frac{2c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}\sum_{i=0}^p\alpha_{t}^2|\mathcal{N}_{t,i}| \\ & + \sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\frac{\alpha_{t}(3+(1-\gamma)^2+3\alpha_{t}c_{max})}{(1-\gamma)^2}\|Q^i_{t,m} - \bar{Q}^{i}_{t,m}\|_2 . |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
Suppose that $|\mathcal{N}_{t,0}| < M/2$, then $\sum_{i=1}^p|\mathcal{N}_{t,i}| >M/2$, we have that |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\frac{1}{2} \alpha_{t}\eta_tM & < \mathbb{E}_{s\sim \nu^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + \frac{2c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}\sum_{i=0}^p\alpha_{t}^2|\mathcal{N}_{t,i}| \\ |
|
|
& \qquad\qquad + \sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\frac{\alpha_{t}(3+(1-\gamma)^2+3\alpha_{t}c_{max})}{(1-\gamma)^2}\|Q^i_{t,m} - \bar{Q}^{i}_{t,m}\|_2, |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
which contradicts \eqref{eq:crpo_eta}. Hence, we must have $|\mathcal{N}_{t,0}| \geq M/2$. |
|
|
|
|
|
|
|
|
Next, we show that $|\mathcal{N}_{t,0}| \geq \big(\frac{1}{2} - \kappa \big)M$ for some $\kappa\in(0,\frac{1}{2})$. Under assumption \ref{asm:crpo_cj} and by Lemma~\ref{lem:appCRPOLem8}, we have that |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\eta_t\sum_{i=1}^p \alpha_{t}|\mathcal{N}_{t,i}| & \leq \mathbb{E}_{s\sim \nu^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + \frac{2c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}\sum_{i=0}^p\alpha_{t}^2|\mathcal{N}_{t,i}| \\ & + \sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\frac{\alpha_{t}(3+(1-\gamma)^2+3\alpha_{t}c_{max})}{(1-\gamma)^2}\|Q^i_{t,m} - \bar{Q}^{i}_{t,m}\|_2-\alpha_{t}c_J . |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
|
|
|
Choose $\kappa\coloneqq \frac{-c_J}{\alpha_{t}\eta_tM}$. Since $-c_J<\frac{1}{2}\alpha_{t}\eta_tM$ by assumption, we have that $\kappa\in(0,\frac{1}{2})$. Consider the case that $|\mathcal{N}_{t,0}| < \big(\frac{1}{2} - \kappa \big)M$, which implies that $\sum_{i=1}^p|\mathcal{N}_{t,i}|>\big(\frac{1}{2} + \kappa \big)M$. This implies that |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
\big(\frac{1}{2} + \kappa \big)\alpha_{t}\eta_tM & \leq \mathbb{E}_{s\sim \nu^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + \frac{2c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}\sum_{i=0}^p\alpha_{t}^2|\mathcal{N}_{t,i}| \\ |
|
|
& \qquad\qquad + \sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\frac{\alpha_{t}(3+(1-\gamma)^2+3\alpha_{t}c_{max})}{(1-\gamma)^2}\|Q^i_{t,m} - \bar{Q}^{i}_{t,m}\|_2, |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
which again contradicts \eqref{eq:crpo_eta}. Hence, we must have $|\mathcal{N}_{t,0}| \geq\big(\frac{1}{2} - \kappa \big)M$. |
|
|
\end{proof} |
|
|
|
|
|
Now, we prove the upper bound of suboptimality and constraint violation per task. |
|
|
|
|
|
\begin{lemma}\label{lem:2_threeEventsHold} Let the violation tolerance be chosen as: |
|
|
\begin{equation*} |
|
|
\eta_t = \frac{2\mathbb{E}_{s\sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})]}{M\alpha_{t}} + \frac{\alpha_{t}4c^2_{max}|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3} + \sum_{i=0}^p \frac{2\alpha_{t}(3+(1-\gamma)^2+3\alpha_{t}c_{max})}{\sqrt{M}\alpha_{t}(1-\gamma)^{2}}, |
|
|
\end{equation*} |
|
|
Then, the following holds |
|
|
\begin{align} |
|
|
U_{t,0}(\pi_{t,0}, \alpha_{t})& = \frac{c_1^t}{\alpha_{t}M}\mathbb{E}_{s\sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + c_2^t \alpha_t + \sum_{i=0}^p \frac{c_3^t\alpha_{t}+c_4^t\alpha_{t}^2}{\alpha_{t}\sqrt{M} }\label{eq:U0-adapt}\\ |
|
|
U_{t,i}(\pi_{t,0}, \alpha_{t})& = |
|
|
U_{t,0}(\pi_{t,0}, \alpha_{t})+\frac{c_5^t}{\sqrt{M}},\label{eq:Ui-adapt} |
|
|
\end{align} |
|
|
where $c_1^t=2$, $c_2^t=\frac{4c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}$, $c_3^t=\frac{3+(1-\gamma)^2}{(1-\gamma)^2}$, $c_4^t=\frac{3 c_{max}}{(1-\gamma)^2}$, and $c_5^t=\frac{2\sqrt{(1-\gamma)|\mathcal{S}||\mathcal{A}|}}{1-2\kappa}$. |
|
|
\end{lemma} |
|
|
\begin{proof} |
|
|
|
|
|
From Lemma \ref{lem:appCRPOLem8}, we have that |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
& \alpha_{t}\sum_{m \in \mathcal{N}_{t,0}}\Big(J_{t,0}(\pi_t^*) - J_{t,0}(\pi_{t,m}) \Big) + \eta_t\sum_{i=1}^p \alpha_{t}|\mathcal{N}_{t,i}| \\ |
|
|
& \leq \mathbb{E}_{s\sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + \frac{2c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}\sum_{i=0}^p\alpha_{t}^2|\mathcal{N}_{t,i}| \\ & + \sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\frac{\alpha_{t}(3+(1-\gamma)^2+3\alpha_{t}c_{max})}{(1-\gamma)^2}\|Q^i_{t,\pi_m} - \bar{Q}^{i}_{t,\omega_m}\|_2 |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
|
|
|
If $\sum_{m \in \mathcal{N}_{t,0}}\Big(J_{t,0}(\pi_t^*) - J_{t,0}(\pi_{t,m}) \Big) \leq 0$, then $J_{t,0}(\pi_t^*) - \mathbb{E}[J_{t,0}(\hat{\pi}_t)]\leq 0$. If $\sum_{m \in \mathcal{N}_{t,0}}\Big(J_{t,0}(\pi_t^*) - J_{t,0}(\pi_{t,m}) \Big) > 0$, then, by Lemma \ref{lem:appCRPOlem9}, we have $|\mathcal{N}_{t,0}| \geq M/2$. Hence, |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
&J_{t,0}(\pi_t^*) - \mathbb{E}[J_{t,0}(\hat{\pi}_t)] = \frac{1}{|\mathcal{N}_{t,0}|} \sum_{m \in \mathcal{N}_{t,0}}[J_{t,0}(\pi_t^*) - J_{t,0}({\pi}_{t,m})] \\ |
|
|
& \leq \frac{2}{ \alpha_{t}M} \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + c_2^t \alpha_t + \sum_{i=0}^p \sum_{m \in \mathcal{N}_{t,i}} \frac{(c_3^t\alpha_{t}+c_4^t\alpha_{t}^2)}{M\alpha_{t}} \|Q^i_{t,\pi_m} - \bar{Q}^{i}_{t,\omega_m}\|_2, \\ |
|
|
&\leq\frac{2}{ \alpha_{t}M} \mathbb{E}_{\nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + c_2^t \alpha_t + \sum_{i=0}^p \sum_{m \in \mathcal{N}_{t,i}} \frac{(c_3^t\alpha_{t}+c_4^t\alpha_{t}^2)}{\sqrt{M}\alpha_{t}} |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
where the last inequality is due to the choice of $K_{in}=\Theta(M^{1/\sigma}\log^{2/\sigma}(|\mathcal{S}|^2|\mathcal{A}|^2M^{1+2/\sigma}/\delta))$ as specified by \cite[Lem. 8]{xu2021crpo} for critic evaluations. Here, the constants are chosen as $c_1^t=2$, $c_2^t=\frac{4c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}$, $c_3^t=\frac{3+(1-\gamma)^2}{(1-\gamma)^2}$, $c_4^t=\frac{3 c_{max}}{(1-\gamma)^2}$. |
|
|
For constraint violation, consider any $i=1,...,p$, we have |
|
|
\begin{align*} |
|
|
\mathbb{E}[J_{t,i}(\hat{\pi}_{t})] - d_{t,i} & = \frac{1}{|\mathcal{N}_{t,0}|}\sum_{m \in \mathcal{N}_{t,0}}J_{t,i}(\pi_{t,m}) -d_{t,i} \\ |
|
|
&\leq \frac{1}{|\mathcal{N}_{t,0}|}\sum_{m \in \mathcal{N}_{t,0}}\big(\bar{J}_{t,i}(\pi_{t,m}) -d_{t,i}\big)+\frac{1}{|\mathcal{N}_{t,0}|}\sum_{m \in \mathcal{N}_{t,0}}|\bar{J}_{t,i}(\pi_{t,m}) -{J}_{t,i}(\pi_{t,m})|\\ |
|
|
& \leq \eta_t + \frac{1}{|\mathcal{N}_{t,0}|}\sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\|Q_{t,\pi_m}^i - \Bar{Q}_{t,\pi_m}^i\|_2\\ |
|
|
&\leq \eta_t + \frac{2}{(1-2\kappa)M}\sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\|Q_{t,\pi_m}^i - \Bar{Q}_{t,\pi_m}^i\|_2 |
|
|
\end{align*} |
|
|
where the first inequality is due to triangle inequality, the second inequality is by the design of the CRPO algorithm, and the third inequality is due to Lemma \ref{lem:appCRPOlem9}, where $\kappa\in(0,\frac{1}{2})$. By the choice of $K_{in}$, we have that $\sum_{i=0}^p\sum_{m \in \mathcal{N}_{t,i}}\|Q_{t,\pi_m}^i - \Bar{Q}_{t,\pi_m}^i\|_2\leq\sqrt{(1-\gamma)|\mathcal{S}||\mathcal{A}|M}$. Plugging the value of $\eta_t$ yields the desired result. |
|
|
\end{proof} |
|
|
|
|
|
Finally, we are able to provide the following bounds on TAOG and TACV in the case of adaptive learning rates. |
|
|
\begin{theorem}[Bounds on TAOG and TACV] |
|
|
\label{thm:taog-tacv-adapt} |
|
|
Suppose we run the CRPO algorithm for $M$ steps per task $t$ with learning rates $\alpha_{t}$. Then, after $T$ tasks, the TAOG $\bar{R}_0$ is given by |
|
|
\begin{align} |
|
|
\bar{R}_0 = \frac{1}{T}\sum_{t=1}^T\bigg[\frac{c_1^t}{\alpha_{t}M}\mathbb{E}_{s\sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + c_2^t\alpha_t + \sum_{i=0}^p \frac{c_3^t\alpha_{t}+c_4^t\alpha_{t}^2}{\alpha_{t}\sqrt{M} } \bigg], |
|
|
\end{align} |
|
|
and the TACV $\bar{R}_i$ is given by |
|
|
\begin{align} |
|
|
\bar{R}_{i} = \frac{1}{T} \sum_{t=1}^T\bigg[ \frac{c_1^t}{\alpha_{t}M}\mathbb{E}_{s\sim \nu_t^*}[D_{KL}(\pi_t^*|\pi_{t,0})] + c_2^t\alpha_t + \sum_{i=0}^p \frac{c_3^t\alpha_{t}+c_4^t\alpha_{t}^2}{\alpha_{t}\sqrt{M} }+\frac{c_5^t}{\sqrt{M}} \bigg], |
|
|
\end{align} |
|
|
for $i=1,...,p$, where $c_1^t=2$, $c_2^t=\frac{4c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}$, $c_3^t=\frac{3+(1-\gamma)^2}{(1-\gamma)^2}$, $c_4^t=\frac{3 c_{max}}{(1-\gamma)^2}$, and $c_5^t=\frac{2\sqrt{(1-\gamma)|\mathcal{S}||\mathcal{A}|}}{1-2\kappa}$. |
|
|
\end{theorem} |
|
|
\begin{proof} |
|
|
The proof follows directly by summing the results \eqref{eq:U0-adapt} and \eqref{eq:Ui-adapt} over $t=1,...,T$. |
|
|
\end{proof} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Adapting to the dynamic regret using adaptive learning rates} \label{subsec:appProofsAdaptive} |
|
|
We restate the theorem below for convenience. |
|
|
|
|
|
\begin{theorem}\label{thm:appUtSim1} |
|
|
Let each within-task CMDP $t$ run $M$ steps of CRPO, initialized by policy $\pi_{t,0}\coloneqq \mathrm{INIT}(t)$ and learning rates $\{\alpha_{t}\}_{i=0}^p \coloneqq \mathrm{SIM}(t)$. Let $\kappa^* \coloneqq \argmin L(\kappa)$, where |
|
|
\begin{equation} |
|
|
\label{eq:appLkappa1} |
|
|
L(\kappa) =U_T^{sim}(\kappa)+ \frac{U_T^{init}(\{\psi_t\}_{t=1}^T)}{\kappa} + \frac{\mathcal{E}_T}{\kappa} + \sum_{t=1}^T\bigg[ \frac{\hat{f}_t^{init}(\phi_t)}{\kappa} + f_t^{rate}(\kappa)\bigg], |
|
|
\end{equation} |
|
|
and $\{\psi_t^*\}_{t=1}^T$ is any comparator sequence. Then, |
|
|
the following bounds on TAOG and TACV hold: |
|
|
\begin{equation}\label{eq:upper-bound-adapt1} |
|
|
\bar{R}_{i} \leq \frac{L(\kappa^*)}{T},\qquad\qquad \forall\; i=0,...,p. |
|
|
\end{equation} |
|
|
\end{theorem} |
|
|
\begin{proof} |
|
|
The idea of the proof is to freeze the learning rates first to obtain a dynamic regret bound based on policy initialization and then optimize over the learning rates to obtain a tighter characterization. Also, since TAOG and TACV only differ by a bias term that does not depend on either the learning rates or the initial policy, we can treat them indistinguishably. In particular, |
|
|
\begin{align} |
|
|
&\sum_{t=1}^T U_t(\pi_{t,0}, \alpha_{t}) = \sum_{t=1}^T f_t^{sim}(\alpha_{t})\\ & \leq \underset{\kappa}{\min}\hspace{0.2cm} U_T^{sim}(\kappa) + \sum_{t=1}^Tf_t^{sim}(\kappa)\\ |
|
|
& \leq \underset{\kappa}{\min} \hspace{0.2cm} U_T^{sim}(\kappa) + \frac{U_T^{init}(\Psi)}{\kappa} + \sum_{t=1}^T\bigg[ \frac{f_t^{init}(\psi_t)}{\kappa} + f_t^{rate}(\kappa)\bigg] \\ |
|
|
& \leq \underset{\kappa}{\min} \hspace{0.2cm} U_T^{sim}(\kappa) + \frac{U_T^{init}(\Psi)}{\kappa} + \frac{\mathcal{E}_T}{\kappa} +\sum_{t=1}^T\bigg[ \frac{\hat{f}_t^{init}(\phi_t)}{\kappa} + f_t^{rate}(\kappa)\bigg] .\label{eq:up-plug} |
|
|
\end{align} |
|
|
where $\Psi\coloneqq\{\phi_t\}_{t=1}^T$. Let |
|
|
\begin{equation} |
|
|
L(\kappa) = U_T^{sim}(\kappa) +\frac{U_T^{init}(\Psi)}{\kappa} + \frac{\mathcal{E}_T}{\kappa} + \sum_{t=1}^T\bigg[ \frac{\hat{f}_t^{init}(\phi_t)}{\kappa} + f_t^{rate}(\kappa)\bigg] |
|
|
\end{equation} |
|
|
and define |
|
|
\begin{equation} |
|
|
\label{eq:opt-rate-learning} |
|
|
\kappa^* = \argmin L(\kappa). |
|
|
\end{equation} |
|
|
Thus, plugging $\kappa = \kappa^*$ in \eqref{eq:up-plug} results in \eqref{eq:upper-bound-adapt1}. |
|
|
\end{proof} |
|
|
|
|
|
Now, we provide the regret upper bound for $U_T^{sim}(\kappa)$, when $\mathrm{SIM}(t)$ is OGD over the sequence $\hat{f}_t^{sim}(\kappa)$. |
|
|
|
|
|
\begin{corollary} |
|
|
\label{cor:app_fsimstaticRegretOGD} |
|
|
For any fixed comparator $\alpha^* = \underset{\kappa}{\argmin} \sum_{t=1}^T \hat{f}_t^{sim}$, if $\mathrm{SIM}(t)$ is OGD which is run on a sequence of loss functions $\{\hat{f}_t^{sim}\}_{t \in [T]}$ with the step-size $ \frac{\alpha^*}{K_\alpha \sqrt{2T}}$, then the following bound holds for static regret: |
|
|
\begin{equation*} |
|
|
\sum_{t=1}^T \hat{f}_t^{sim}(\kappa) - \sum_{t=1}^T\hat{f}_t^{sim}(\alpha^*) \leq \sqrt{2T}K_\alpha|\alpha^* | + \left(1 + \frac{4\sqrt{2} K_\alpha L_\alpha |\alpha^*|}{(2C_1 - C_1^2 L_\alpha) \sqrt{T}} \right) \mathcal{E}_T, |
|
|
\end{equation*} |
|
|
for any $C_1 \in \{c \in \left(0, \frac{2}{L_\alpha}\right): \alpha^* + c(u - \hat{\nabla}_t ) \in \Lambda\}$ where $u$ is an $\epsilon$-subgradient of $f_t^{sim}(\kappa)$ with respect to $\kappa$, $\mathcal{E}_T \coloneqq \sum_{t=1}^T \epsilon_t$ is the cumulative inexactness, $\epsilon_t$ is the upper bound from Theorem \ref{thm:dualDICE}. |
|
|
\end{corollary} |
|
|
|
|
|
\begin{proof} |
|
|
The proof follows directly after substituting $c = \frac{4}{2C_1 - C_1^2 L_\alpha}$ and other appropriate constants in Theorem \ref{prop:InexactUpperBound-app}. |
|
|
\end{proof} |
|
|
|
|
|
|
|
|
|
|
|
Next, we provide the proof of Corollary \ref{cor:CorollaryAdpativeRate} TAOG and TACV regret bounds when $\mathrm{INIT}$ and $\mathrm{SIM}$ are either FTL or inexact OGD. |
|
|
|
|
|
|
|
|
\begin{corollary} \label{cor:appCorollaryAdpativeRate} |
|
|
For any fixed comparator $\alpha^* = \underset{\kappa}{\argmin} \sum_{t=1}^T \hat{f}_t^{sim}$ and $c_3^t = \frac{3+(1-\gamma)^2}{(1-\gamma)^2}$. If $\mathrm{INIT}(t)$ and $\mathrm{SIM}(t)$ are FTL or inexact OGD over the sequences $\hat{f}_t^{init}$ and $\hat{f}_t^{sim}$, then, the following bounds on TAOG and TACV hold: |
|
|
\begin{equation} |
|
|
\begin{aligned} |
|
|
\bar{R}_{i} \leq & \frac{1}{\sqrt{M}} \left( \frac{\sqrt{2}K_\alpha|\alpha^* |}{\sqrt{MT}} + \left(c_3^t + \frac{4\sqrt{2} K_\alpha L_\alpha |\alpha^*|}{(2C_1 - C_1^2 L_\alpha) \sqrt{T}} \right) \frac{\mathcal{E}_T}{\sqrt{M}T} + \frac{1}{\sqrt{c_2^t}} \frac{\sqrt{U_T^{init}(\{\psi_t^*\}_{t \in [T]}) + \mathcal{E}_T + T\hat{V}_\psi^2} }{M^{1/4}T} \right), |
|
|
\end{aligned} |
|
|
\end{equation} |
|
|
for all $i = 0,\ldots,p$, where $\{\psi_t^*\}_{t \in [T]}$ is any comparator sequence, $c_3^t = \frac{3+(1-\gamma)^2}{(1-\gamma)^2}$, and $c_2^t=\frac{4c_{max}^2|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^3}$. |
|
|
\end{corollary} |
|
|
|
|
|
\begin{proof} |
|
|
Firstly, the following inexact upper bounds hold for $U_T^{init}$ and $U_T^{sim}$: |
|
|
\begin{equation} |
|
|
\begin{aligned} |
|
|
\label{eq:UTinit} |
|
|
U_T^{init}(\{\psi_t^*\}_{t \in T}) = \min \bigg(C_1\|\phi_1 - \psi_1^*\|^2 + C_2 & \mathcal{E}_T + C_3 \mathcal{S}_T + \frac{1}{2 \beta}\sum_{t=1}^T \|\nabla \ell_t(\psi_t^*)\|^2, \\ & C_4\|\phi_1 - \psi_1^*\| + C_5 \tilde{\mathcal{E}}_T + C_4 \mathcal{P}_T \bigg) , |
|
|
\end{aligned} |
|
|
\end{equation} |
|
|
\begin{equation} |
|
|
\label{eq:UTsim} |
|
|
U_T^{sim}(\kappa) = \sqrt{2T}K_\alpha|\alpha^* | + \left(1 + \frac{4\sqrt{2} K_\alpha L_\alpha |\alpha^*|}{(2C_1 - C_1^2 L_\alpha) \sqrt{T}} \right) \mathcal{E}_T , |
|
|
\end{equation} |
|
|
|
|
|
where the constants in the \ref{eq:UTinit} and \eqref{eq:UTsim} are from the Lemma \ref{cor:appdynamicRegretOGD} and Corollary \ref{cor:app_fsimstaticRegretOGD} respectively. |
|
|
|
|
|
Plugging these in the equation \eqref{eq:appLkappa1}, we can obtain the following: |
|
|
\begin{equation*} |
|
|
\begin{aligned} |
|
|
L(\kappa) \leq & \sqrt{2T}K_\alpha|\alpha^* | + \left(1 + \frac{4\sqrt{2} K_\alpha L_\alpha |\alpha^*|}{(2C_1 - C_1^2 L_\alpha) \sqrt{T}} \right) \mathcal{E}_T + \frac{1}{\kappa} \min \bigg(C_1\|\phi_1 - \psi_1^*\|^2 \\& + C_2 \mathcal{E}_T + C_3 \mathcal{S}_T + \frac{1}{2 \beta}\sum_{t=1}^T \|\nabla \ell_t(\psi_t^*)\|^2, C_4\|\phi_1 - \psi_1^*\| + C_5 \tilde{\mathcal{E}}_T + C_4 \mathcal{P}_T \bigg) \\ & + \frac{\mathcal{E}_T}{\kappa} + \frac{T\hat{V}_\psi^2}{\kappa} + f_t^{rate}(\kappa), |
|
|
\end{aligned} |
|
|
\end{equation*} |
|
|
|
|
|
where the relation $\hat{V}_\psi^2 = \frac{1}{T}\sum_{t=1}^T \mathbb{E}_{\hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\psi_t^*)]$ is used to get the second last term $\frac{T \hat{V}_\psi^2}{\kappa}$. To get $\kappa^*$ such that $L(\kappa)$ is minimized, we proceed using the KKT optimality conditions to get $\frac{dL}{d\kappa}$ as |
|
|
\begin{equation*} |
|
|
\frac{dL}{d\kappa} = \frac{-U_T^{init}(\{\psi\}_{t \in [T]})}{\kappa^2} - \frac{\mathcal{E}_T}{\kappa^2} - \frac{T \hat{V}_\psi}{\kappa^2} + c_2^tM+ c_4^t\sqrt{M}. |
|
|
\end{equation*} |
|
|
|
|
|
Therefore, we obtain $\kappa^* = \sqrt{\frac{U_T^{init}(\psi) + \mathcal{E}_T + T\hat{V}_\psi^2}{c_2^tM+c_4^t\sqrt{M}}}$. |
|
|
|
|
|
Substituting this value of $\kappa^*$ in \eqref{eq:appLkappa}, and using the approximation $\sqrt{\frac{1}{M+\sqrt{M}}} \approxeq \frac{1}{M^{1/4}}$, we can obtain the final result. |
|
|
\end{proof} |
|
|
|
|
|
\section{Additional experiments and details} \label{sec:ExpDetails} |
|
|
|
|
|
In this section, we describe our experimental setup and other additional details for the OpenAI gym experiments under constrained settings. We present the performance of Meta-SRL on the Acrobot and the Frozen lake under high task-similarity conditions. We first present some details on the baselines used to avoid any confusion. |
|
|
|
|
|
\textbf{Details of baselines used:} Denote the policy initialization for task $t$ as $\phi_t$. First baseline is the FAL, which initializes the policy $ \phi_{t+1}$ for the test task $t+1$ as $\phi_{t+1} = \frac{1}{t} \sum_{i=1}^t \phi_i$. This is done online as the tasks are encountered sequentially. |
|
|
|
|
|
For the simple averaging baseline, we run CRPO on $10$ tasks with the random initialization and evaluate the performance on a test task by initializing with the average of all the suboptimal policies from the batch of suboptimal policies, i.e., $\phi_T = \frac{1}{T-1} \sum_{i=1}^{T-1} \phi_i$, where $T$ is the test task. This can be seen as an offline method, where suboptimal policies from $T-1$ tasks are stored and averaged to get the initialization for the test task. |
|
|
|
|
|
The pre-trained baseline uses suboptimal policy from an already trained task to initialize the policy on the test task, i.e., $\phi_t = \hat{\pi}_{t'}$, where $\phi_t$ is the policy initialization for the task $t$, and $\hat{\pi}_{t'}$ is the suboptimal policy returned some pre-trained task $t'$. This baseline could be thought of as the Strawman initialization strategy, where the policy for the next task is initialized using the suboptimal policy from the previous task, i.e., $\phi_{t+1} = \hat{\pi}_t$. |
|
|
|
|
|
For the Meta-SRL in the discrete state action space (i.e., Frozen lake), we take the weighted average of the previous suboptimal policies weighted by the stationary distributions induced by each suboptimal policy over all the states. We also adapt the learning rates $\alpha_t$ for the CRPO algorithm for each task $t$. The difference between FAL and Meta-SRL is that Meta-SRL weights the suboptimal policies higher in the states which were encountered more frequently. |
|
|
|
|
|
Random initialization baseline is using random iniitalization for the within-task algorithm CRPO. |
|
|
|
|
|
|
|
|
|
|
|
\textbf{Experimental setup:} We run all the algorithms online where tasks are encountered sequentially and present the results for the test task after the policy initialization suggested by respective baselines. We do $10$ runs for each baseline on the test task and present the variance plots. On the test task, we train for $8$ steps on the Frozen lake and for $5$ steps on the Acrobot. In Frozen lake, each step corresponds to $5$ episodes, and the average rewards/costs are reported for each step in the performance and constraint violation plots. |
|
|
|
|
|
\subsection{Frozen lake} |
|
|
|
|
|
\textbf{Frozen lake:} For the Frozen lake, we randomly generate $T=10$ different orientations as tasks over the probability of a state being frozen or a hole and evaluate the performance for the scenarios with high task-similarity (low variance for the latent CMDP distribution) or low task-similarity (high variance for the latent CMDP distribution). The agent gets rewarded $+2$ when it reaches the goal state and incurs a cost $-1$ when it falls into a hole. We choose the constraint threshold $d_{t,i} = 0.3$. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\textbf{High task-similarity:} To generate tasks with high task-similarity, we start with a random frozen lake grid of $4 \times 4$, where the probability of each grid being frozen is $0.7$. Then, we generate $9$ different grids which differ from the first one by only one of the grids. This means the agent always encounters the new grid, which is very similar to the previous task. From Figure \ref{fig:FrozenLakeHigh}, we can observe that baselines pre-trained and FAL are competitive with the Meta-SRL in terms of reward maximization and almost zero constraint violations. The good performance of the pre-trained baseline can be explained using the fact that the the test task and the training task are very similar; the policy initialization will be close to the optimal policy of the test task. |
|
|
|
|
|
\begin{figure}[htbp] |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{FrozenLake/FrozenLakeHighSimilarity.pdf} |
|
|
\caption{Frozen lake results for reward maximization and constraint violations when the task-relatedness is high. The blue dashed line represents the averaged thresholds for the constraint violations.} |
|
|
\label{fig:FrozenLakeHigh} |
|
|
\end{figure} |
|
|
|
|
|
|
|
|
\textbf{Low task-similarity:} In this case, random tasks are generated where the probability of a tile being frozen is kept between $0.3$ and $0.7$. The tasks are less similar due to the high uncertainty associated with the changing orientations. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\subsection{Acrobot} |
|
|
|
|
|
\textbf{Acrobot:} Acrobot is a $2$ link robot OpenAI gym environment with continuous state space. The agent is rewarded when it achieves a certain height of the end link. Two constraints are introduced for two links, where a $-1$ cost is incurred if any link swings in the prohibited direction. We randomly generate $T=50$ different tasks with different mass links and centers of gravity. |
|
|
|
|
|
\textbf{High task-similarity:} To generate tasks with high similarity for the acrobot, we considered changing the mass of the links, the center of gravity (COG), and constraint threshold for each link. The changes in these quantities were done by adding noise to the default quantities. We considered a Gaussian noise with a low variance of $0.1$ to change the tasks only slightly. From Figure \ref{fig:AcrobotHigh}, we can observe that only pre-trained and FAL baselines are competitive with the Meta-SRL in terms of reward maximization and almost zero constraint violations. |
|
|
|
|
|
\begin{figure}[htbp] |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{Acrobot/Acrobot_high_similarity.pdf} |
|
|
\caption{Acrobot results for reward maximization and constraint violations when the task-relatedness is high. The blue dashed line represents the averaged thresholds for the constraint violations.} |
|
|
\label{fig:AcrobotHigh} |
|
|
\end{figure} |
|
|
|
|
|
\textbf{Low task-similarity:} To generate low similar tasks, we increased the variance of the Gaussian noise to $0.3$. We can observe from Figure \ref{fig:Acrobot} that the performance of the baselines was poor for constraint satisfaction, while Meta-SRL could converge quickly for reward maximization and constraint satisfaction. |
|
|
|
|
|
Note that, in real-world settings, tasks are likely to have low similarity in terms of how close their optimal policies are. The good performance of Meta-SRL under these settings highlights its potential to be extended to real-world settings where safety constraints are present. |
|
|
|
|
|
|
|
|
\subsection{Half cheetah} |
|
|
|
|
|
\hl{Half-cheetah is a simulation environment for a 2-dimensional robot. It consists of 9 links and eight joints, where the goal for the cheetah is to run at a certain velocity. It has a 17-dimensional state space and a 6-dimensional action space. The reward is calculated as the negative of the absolute difference between the current cheetah velocity and the desired velocity. The original HalfCheetah environment does not have any constraints. We introduce a constraint that penalizes the deviation of the cheetah’s head from some desired height:} |
|
|
|
|
|
$$h_{cheetah} - h_{target} \leq \epsilon,$$ |
|
|
|
|
|
\VK{where we specify the cumulative absolute difference between the cheetah head height and the desired height to be less than a tolerance $\epsilon$. The cheetah is trained on $T = 100$ tasks for both high and low task-similarity settings.} |
|
|
|
|
|
\VK{\textbf{High task-similarity:} To generate tasks with high similarity, the goal velocity for each training task is uniformly sampled from a range of $[0.35,0.65]m/s$. From Figure \ref{fig:CheetahHigh}, we can observe that under high task similarity settings, Meta-SRL is able to achieve high rewards and also able to keep the constraint violation of the cheetah's head height below the threshold. Under high task-similarity settings, pre-trained and Meta-SRL perform well, as expected. However, it can be observed that both simple averaging and random initialization perform poorly in this setting.} |
|
|
|
|
|
\begin{figure}[htbp] |
|
|
\centering |
|
|
\begin{subfigure}{.47\textwidth} |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{HalfCheetah/HalfCheetahReward_high_task_similarity_broken_axis.pdf} |
|
|
\caption{} |
|
|
\end{subfigure} |
|
|
\begin{subfigure}{.47\textwidth} |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{HalfCheetah/HalfCheetahCost_high_task_similarity.pdf} |
|
|
\caption{} |
|
|
\end{subfigure} |
|
|
\caption{Half-cheetah results for reward maximization and constraint violations when the task-relatedness is high. The blue dashed line represents the averaged thresholds for the constraint violations. Here, the simple averaging and random initialization perform the worst on the test task.} |
|
|
\label{fig:CheetahHigh} |
|
|
\end{figure} |
|
|
|
|
|
\VK{\textbf{Low task-similarity:} To generate tasks with low similarity for the half-cheetah, the goal velocity for each training task is uniformly sampled from a range of $[0.0,1.0]m/s$. Tasks are less similar due to the high variance of goal velocities that the cheetah is trained on. It can be observed from Figure \ref{fig:Halfcheetah} that Meta-SRL is able to achieve higher rewards and zero constraint violations quickly compared to other baseline initializations under low task-relatedness settings. The pre-trained baseline can achieve higher rewards, but similar to other baselines, it cannot achieve constraint satisfaction within 10 steps. It can also be observed that both simple averaging and random initialization perform poorly in reward maximization in this setting. Close inspection indicates that there is a high variance among the policy parameters learned from each task, which may result in interference among different tasks in the relatively high dimensional state space.} |
|
|
|
|
|
\begin{figure}[htbp] |
|
|
\centering |
|
|
\begin{subfigure}{.47\textwidth}\label{fig:FrozenLakeReward1} |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{HalfCheetah/HalfCheetahReward_low_task_similarity_broken_axis.pdf} |
|
|
\caption{} |
|
|
\end{subfigure} |
|
|
\begin{subfigure}{.47\textwidth}\label{fig:FrozenLakeReward2} |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{HalfCheetah/HalfCheetahCost_low_task_similarity.pdf} |
|
|
\caption{} |
|
|
\end{subfigure} |
|
|
\caption{Half-cheetah results for reward maximization and constraint violations when the task-relatedness is low. The blue dashed line represents the averaged thresholds for the constraint violations.} |
|
|
\label{fig:Halfcheetah} |
|
|
\end{figure} |
|
|
|
|
|
|
|
|
\subsection{Humanoid} |
|
|
|
|
|
\hl{Humanoid is a simulation environment of a 3D bipedal robot, which consists of a torso (abdomen) with two arms and legs. Each leg and arm has two links (representing the knees and elbows, respectively). It has a 376-dimensional observation space and a 17-dimensional action space. The goal of the humanoid is to walk forward as fast as possible without falling over.} |
|
|
|
|
|
\VK{The original humanoid environment does not have any constraints. We introduce a constraint that penalizes the deviation of the angles between the torso and the upper arm and the angle between the upper arm and the lower arm, such that humanoid motions are smooth and graceful. The cumulative constraint is given as:} |
|
|
$$\left|\theta_{tr} - \frac{\pi}{4}\right|+\left|\theta_{tl} - \frac{\pi}{4}\right|+\left|\theta_{r} - \frac{\pi}{4}\right|+\left|\theta_{l} - \frac{\pi}{4}\right| \leq \epsilon,$$ |
|
|
\VK{where $\theta_{tr}$, $\theta_{tl}$, $\theta_r$, and $\theta_l$ are the angles between the torso and right arm, the angle between the torso and the left arm, the angle between both the upper and lower right arms, and the angle between both the upper and lower left arms, respectively. We specify the cumulative absolute difference between all these angles and the desired angle to be less than a tolerance $\epsilon$. The reward is calculated on the basis of the dot product between the direction and the velocity vector of the Center of Gravity of the humanoid, multiplied by the default scaling value in the humanoid environment $W_f$. the humanoid as follows:} |
|
|
|
|
|
$$r = W_f(v_y \sin \theta + v_x \cos \theta),$$ |
|
|
|
|
|
\VK{where $r$ is the instantaneous reward that accounts for the amount of forward movement by the humanoid, $v_x$, and $v_y$ are the horizontal and lateral components of the velocity, and $\theta$ is the walking direction of the humanoid. The default value of $W_f$ is 1.25. The humanoid is trained on $T = 250$ tasks for both high and low task-similarity settings.} |
|
|
|
|
|
\VK{\textbf{High task-similarity:} We generate different tasks by changing the direction of motion of the humanoid. Possible direction angles in the humanoid environment range from $-\pi/2$ to $\pi/2$ (which varies from left to right). To generate tasks with high similarity, the goal direction of the humanoid for each training task is uniformly sampled from a range of $[-\pi/4,\pi/4]$. From Figure \ref{fig:HumanoidHigh}, we can observe that under high task similarity settings, Meta-SRL is able to achieve high rewards and maintain the constraint violation of the humanoid's hand and torso angles below the threshold of $\epsilon=4$. Under high task-similarity settings, both pre-trained and Meta-SRL perform well, as expected. However, it can be observed that both simple averaging and random initialization perform poorly in this setting. Moreover, the pre-trained baseline also fails to learn reward-maximizing behaviors.} |
|
|
|
|
|
|
|
|
\begin{figure}[htbp] |
|
|
\centering |
|
|
\begin{subfigure}{.47\textwidth} |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{Humanoid/HumanoidReward_high_task_similarity.pdf} |
|
|
\caption{} |
|
|
\end{subfigure} |
|
|
\begin{subfigure}{.47\textwidth} |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{Humanoid/HumanoidCost_high_task_similarity.pdf} |
|
|
\caption{} |
|
|
\end{subfigure} |
|
|
\caption{Humanoid results for reward maximization and constraint violations when the task-relatedness is high. The blue dashed line represents the averaged thresholds for the constraint violations.} |
|
|
\label{fig:HumanoidHigh} |
|
|
\end{figure} |
|
|
|
|
|
\VK{\textbf{Low task-similarity:} To generate tasks with low similarity for the humanoid, the goal direction for each training task is uniformly sampled from a range of $[-\pi/4,\pi/4]$. Tasks are less similar due to the high variance of goal direction that the humanoid is trained on. Figure \ref{fig:HumanoidLow} shows that Meta-SRL is able to quickly achieve higher rewards and zero constraint violations compared to other baseline initializations under low task-relatedness settings. The pre-trained baseline also achieves constraint satisfaction in this case but fails to learn behaviors to maximize the rewards within 10 steps. It can also be observed that both simple averaging and random initialization perform poorly in reward maximization in this setting. This can be attributed to a high variance among the policy parameters learned from each task, which may result in interference among different tasks in the relatively high-dimensional state space.} |
|
|
|
|
|
\begin{figure}[htbp] |
|
|
\centering |
|
|
\begin{subfigure}{.47\textwidth} |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{Humanoid/HumanoidReward_low_task_similarity.pdf} |
|
|
\caption{} |
|
|
\end{subfigure} |
|
|
\begin{subfigure}{.47\textwidth} |
|
|
\centering |
|
|
\includegraphics[width=\columnwidth]{Humanoid/HumanoidCost_low_task_similarity.pdf} |
|
|
\caption{} |
|
|
\end{subfigure} |
|
|
\caption{Humanoid results for reward maximization and constraint violations when the task-relatedness is low. The blue dashed line represents the averaged thresholds for the constraint violations.} |
|
|
\label{fig:HumanoidLow} |
|
|
\end{figure} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\section{Relation of Meta-SRL to hardness results in \texorpdfstring{\citep{kwon2021rl}}{[1]}}\label{sec:KwonRelation} |
|
|
\VK{ |
|
|
There is a key difference in the problem setting of meta-learning in our study and the latent MDP setting in \citep{kwon2021rl}. The latent MDP setting is more challenging in the sense that there is no clear boundary between tasks. In the latent MDP setting, each episode may come from an unknown MDP drawn from a distribution (as a special case of POMDP); in the meta-learning setting, the agent knows when a new task has arrived and is allowed to interact with the MDP over a set of episodes (the number is linear with respect to $M$ in our paper). Due to the above difference, the worst-case lower bound of requiring an exponential number of episodes to learn an $\epsilon$-optimal policy in \citep{kwon2021rl} does not hold in our case. |
|
|
Indeed, if the identity (referred to as ``context” in latent MDP) is revealed or can be inferred, \citep{kwon2021rl} is able to achieve a regret that is polynomial in the number of episodes (Thms. 3.3 and 3.4 from \citep{kwon2021rl}).} |
|
|
|
|
|
|
|
|
\VK{ |
|
|
Furthermore, a close examination of the bounds provided by \citep{kwon2021rl} also reveals some differences from our result. In particular, let $K$ be the number of contexts in a latent MDP and $N$ be the total number of episodes ($N$ is on the order of $TM$ in our case as we encounter $T$ tasks, each with $M$ episodes). Then \citep{kwon2021rl} is able to bound the regret (without dividing by the number of episodes $N$) as $\mathcal{O}(\sqrt{KN})$. To compare their bound with ours, we consider each task in the meta-learning setting as a context, so $T = \mathcal{O}(K)$. Therefore, their upper bound (after dividing by the number of episodes $N=TM$) becomes $\mathcal{O}(1/\sqrt{M})$, which does not diminish with the number of tasks $T$. Note that our bound (see the comment after Corollary \ref{cor:CorollaryAdpativeRate}) is $\mathcal{O}\left( \frac{\hat{V}_\psi}{M^{3/4}\sqrt{T}} \right)$ (after dividing by the number of episodes $N = TM$), where for simplicity we have assumed $\mathcal{E}_T = 0$, i.e., exact access to the loss function. Note that $\hat{V}_\psi$ is a measure of task-relatedness (a smaller value indicates more relatedness among tasks). It can be seen that while we have a worse order dependence on $M$, our bound scales with task relatedness $\hat{V}_\psi$ and diminishes with respect to the increasing number of tasks $T$. This is expected as we leverage the relatedness among contexts (in fact, the result of \citep{kwon2021rl} would hold when tasks are sufficiently different from each other to infer the contexts with spectral methods).} |
|
|
\VK{ |
|
|
In summary, we refer to \citep{kwon2021rl} as an example that achieving regret diminishing in the number of tasks $T$ is hard, even with the assumption of observing the task identities (contexts).} |
|
|
|
|
|
|
|
|
|
|
|
\section{Notations and constants} |
|
|
\label{sec:notations} |
|
|
|
|
|
|
|
|
\begin{table}[h] |
|
|
\centering |
|
|
\begin{tabular}{ll} |
|
|
\hline \textbf{Notation} & \textbf{Definition} \\ \hline |
|
|
$t$ & index of task\\ |
|
|
$k$ & index of OGD steps\\ |
|
|
$\phi_t$,$\pi_{t,0}$ & policy initialization for task $t$\\ |
|
|
$\alpha_t$ & learning rate of within-task algorithm\\ $t \in [T]$ & set of all tasks, where $[T] = \{1,\ldots,T\}$ |
|
|
\\ |
|
|
$\mathcal{M}_t$ &CMDP for task $t$\\ |
|
|
$\mathcal{S}$ & state space of CMDP $\mathcal{M}_t$ |
|
|
\\ |
|
|
$\mathcal{A} \in \mathbb{R}^{n_a}$ & action space of CMDP $\mathcal{M}_t$\\ $\rho_t$ & initial state distribution of task $t$ |
|
|
\\ |
|
|
$P_t(\cdot|s,a)$ & transition kernel for task $t$ \\ |
|
|
$c_{t,0}: \mathcal{S} \times \mathcal{A} \rightarrow [0,1]$ & reward function |
|
|
\\ |
|
|
$c_{t,i}: \mathcal{S} \times \mathcal{A} \rightarrow [0,1]$ & cost function $i$ for task $t$ \\ |
|
|
$p$ & total number of constraints |
|
|
\\ |
|
|
$m \in [M]$ & set of all timesteps for within-task algorithm \\ |
|
|
$\Delta (\mathcal{A})^{|\mathcal{S}|}$ & simplex over all state-action pairs |
|
|
\\ |
|
|
$\pi_t: \mathcal{S}\rightarrow \Delta(\mathcal{A})$ & stochastic policy for task $t$ \\ |
|
|
$\nu_t^\pi$ &state visitation distribution of policy $\pi$ at task $t$\\ |
|
|
$\theta$ & softmax policy parameters |
|
|
\\ |
|
|
$V_{t,\pi}^i(s)$ & state-value function for reward ($i=0$) or cost $i$ in task $t$ with policy $\pi$\\ $Q_{t,\pi}^i(s,a)$ & action-value function for reward ($i=0$) or cost $i$ in task $t$ with policy $\pi$ |
|
|
\\ |
|
|
$J_{t,i}(\pi)$ & expected total reward ($i=0$) or cost $i$ for task $t$ and policy $\pi$\\ $d_{t,i}$ & bound on the expected total cost $i$ for task $t$ |
|
|
\\ |
|
|
$\Pi_t^*$ & set of optimal solutions for task $t$\\ |
|
|
$\pi_t^*$ & \VK{optimal policy for task $t$}\\ |
|
|
$c_{max}$ & maximum value of reward/cost functions |
|
|
\\ |
|
|
$D_{KL}(\cdot|\cdot)$ & KL divergence\\ |
|
|
$\bar{R}_0, \bar{R}_i$ & TAOG and TACV\\ |
|
|
$\hat{\pi}_t$ & suboptimal policy returned by within-task algorithm in task $t$ |
|
|
\\ |
|
|
$D^*$, $\hat{D}^*$ & true and empirical task-similarity\\ |
|
|
$V_\psi$ ,$\hat{V}_\psi$ & true and empirical task-relatedness |
|
|
\\ |
|
|
$\Delta \mathcal{A}_\varrho$ & shrinkage simplex set inside $\Delta \mathcal{A}$\\ $L_g$, $L_\pi$ &Lipschitzness and smoothness parameter for KL divergence of policy $\pi$ w.r.t. initial policy |
|
|
\\ |
|
|
$C_\pi$ & maximum value of KL divergence of policy $\pi$ w.r.t. initial policy \\ |
|
|
$\omega_{\pi/ \mathcal{D}_t}(s,a)$ &stationary distribution correction for task $t$ at state $s$ action $a$ |
|
|
\\ |
|
|
$\mathcal{D}_t$ & off policy dataset for task $t$\\ |
|
|
$ \mu_\pi$ & strong convexity parameter for KL divergence of policy $\pi$ w.r.t. initial policy |
|
|
\\ |
|
|
$\{\psi_t^*\}_{t=1}^T$ & dynamically varying comparator sequence\\ |
|
|
$\epsilon_t$ &inexactness in the KL divergence estimation using DualDICE |
|
|
\\ |
|
|
$\hat{\nu}_t$ & state distribution induced by $\hat{\pi}_t$\\ $\epsilon_{opt}$ &optimization error in DualDICE |
|
|
\\ |
|
|
$\epsilon_{approx}$ & approximation error in DualDICE\\ |
|
|
$\mathcal{F}, \mathcal{H}$ &hypothesis class used in DualDICE |
|
|
\\ |
|
|
$\mathcal{E}_T$ & cumulative inexactness for KL divergence estimation given by $\sum_{t=1}^T \epsilon_t$ \\ $\tilde{\mathcal{E}}_T$ &cumulative square root of inexactness for KL divergence estimation given by $\sum_{t=1}^T \sqrt{\epsilon_t}$ |
|
|
\\ |
|
|
$h:[0,\rho] \rightarrow (0,\infty)$& strictly increasing continuous definable function used in Theorem \ref{thm:dualDICE} |
|
|
\\ |
|
|
$\nabla_t$, $\hat{\nabla}_t$& Exact and inexact gradient |
|
|
\\ |
|
|
$\beta$ &learning rate for inexact OGD\\ |
|
|
$P_X(\cdot)$ & Projection operator |
|
|
\\ |
|
|
$\mathcal{P}_T$ &path-length of the comparator $\psi_t^*$ |
|
|
\\ |
|
|
$K_{in}$ & number of iterations in the critic estimation of CRPO |
|
|
\\ |
|
|
$\eta_t$ & tolerance for the constraint violation $d_{t,i}$ for task $t$\\ |
|
|
$\mathcal{P}_T$, $\mathcal{S}_T$ &path-length and squared path-length of the comparator $\psi_t^*$ |
|
|
\\ |
|
|
$\partial_\epsilon f(\cdot)$ & $\epsilon$-subgradient of function $f$ |
|
|
\\ |
|
|
$Dom(f)$ & Domain of the function $f$ |
|
|
\\ |
|
|
$\hat{f}_t(\cdot)$ & loss functions for suboptimal policy $\mathbb{E}_{\hat{\nu}_t}[D_{KL}(\hat{\pi}_t|\pi_{t,0})]$ \\ |
|
|
\hline |
|
|
\end{tabular} \caption{Table of notations} |
|
|
\end{table} |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
\begin{table}[h] |
|
|
|
|
|
\centering |
|
|
|
|
|
\caption{Table of constants.} |
|
|
\begin{tabular}{llll} |
|
|
\hline \textbf{Notation} & \multicolumn{1}{l||}{\textbf{Definition}} & \textbf{Notation} & \textbf{Definition} \\ |
|
|
|
|
|
|
|
|
\hline |
|
|
$c_1^t$ & \multicolumn{1}{l||}{$2$} & $c_2^t$ & $\frac{4 c_{max}^2|\mathcal{S}| |\mathcal{A}|}{(1-\gamma)^3}$ |
|
|
\\ |
|
|
$c_3^t$ & \multicolumn{1}{l||}{$\frac{3+(1-\gamma)^2}{(1-\gamma)^2}$} & $c_4^t$ & $\frac{3c_{max}}{(1-\gamma)^2}$ |
|
|
\\ |
|
|
$c_5^t$ & \multicolumn{1}{l||}{$\frac{2\sqrt{(1-\gamma)}|\mathcal{S}||\mathcal{A}|}{1-2\kappa}$} & $c$ & $c \in \left(0, \frac{2}{L_\pi}\right)$ |
|
|
\\ |
|
|
$C_1$ & \multicolumn{1}{l||}{$2(L_\pi + \beta)$} & $C_2$ & $(L_\pi +\beta)\frac{3c\alpha+6\alpha L_\pi}{2 \mu_\pi \alpha L_\pi}$\\ |
|
|
$C_3$ & \multicolumn{1}{l||}{$3(L_\pi + \beta)$} & $C_4$ & $\frac{2 L_g}{2-\sqrt{2}}$\\ |
|
|
$C_5$ & \multicolumn{1}{l||}{$\frac{2L_g}{2-\sqrt{2}}\sqrt{\frac{c\alpha + 2L_\pi \alpha}{2\alpha \mu_\pi L_\pi}}$} & \\ |
|
|
\hline |
|
|
\end{tabular} |
|
|
\vspace{-0.5em} |
|
|
\label{tab:constant} |
|
|
\end{table} |
|
|
|
|
|
|
|
|
|
|
|
\end{document} |
|
|
|
|
|
|
|
|
|
