\section{Introduction} \label{SECTION: INTRODUCTION}
	A trading system is said to be \textit{robust} if it generates a robust return regardless of market direction. In particular, a consistently positive expected  trading gain-loss is often used as a robustness metric.
	It is known that the  so-called  \textit{Simultaneous Long-Short (SLS)} control scheme, see~\cite{barmish2008trading,
		barmish2011performance, 	 barmish2011arbitrage,
		barmish2015new,
		baumann2016stock, baumann2017simultaneously},
	guarantees the satisfaction of the so-called \textit{Robust Positive Expectation} (RPE); i.e., the cumulative trading gain-loss function is guaranteed to have positive expected value for a broad class of stock price processes.
	This fact attracts many new extensions and modifications to the SLS theory.

	For example, the SLS theory involves RPE results with respect to stock prices having time-varying drift and volatility; see~\cite{primbs2017robustness}, prices generated from Merton’s diffusion model~\cite{baumann2017simultaneously}, generalization from a static linear policy to the case of Proportional–Integral (PI) controllers~\cite{malekpour2018generalization}, and discrete-time systems with delays~\cite{malekpour2016stock}.
	More recently, in~\cite{deshpande2020simultaneous}, an SLS control with cross-coupling is proposed to trade two stocks.
	In~\cite{o2020generalized}, a generalized RPE Theorem to the case of an SLS controller which can have different parameters for the long and short sides of the trade is studied.

	In~\cite{maroni2019robust},  an~$\mathcal{H}_\infty$  approach for selecting the SLS controller parameters is proposed.

	To close this brief introduction, we mention some related studies using the control-theoretic approach in stock trading scenario; e.g., see \cite{zhang2001stock} for studying optimal selling rule, \cite{tie2018optimal, deshpande2018generalization} for studying optimal pair trading, and~\cite{barmish2021feedforward} for studying a new feedforward control in transaction level price model.

	While various extensions and ramifications are proposed to improve the SLS theory, the effect of transaction costs, which is believed to erode the desired positivity, is arguably lacking in the  existing study.
	To this end, we propose a new SLS-based trading structure, which we call the \textit{double linear policy}, in a discrete-time setting to study  positive expectation property closely when the transaction costs are involved.

	The contributions of this paper are threefold: First,   we propose a new double linear policy scheme  for a class of asset price processes involving independent returns; see Section~\ref{SECTION: Problem Formulation}.
	In the financial market with nonzero transaction costs, we derive explicit expressions for the expected value and variance of the cumulative gain-loss function using the proposed policy scheme; see Section~\ref{SECTION: Analysis of Trading Performance}.
	Second, we go beyond the existing literature by quantifying under what condition  the desired expected positivity can still be preserved when transaction costs are involved.
	A novel asymptote approach is used to obtain a relatively conservative positivity; see Theorem~\ref{theorem: Robust Positive Expectation} in Section \ref{SECTION: Analysis of Trading Performance}.
	Third, we   validate the theory via heavy Monte-Carlo simulations.  An empirical backtesting example using the historical data for cryptocurrency Bitcoin USD is also provided; see Section~\ref{SECTION: Illustrative Examples}.

	\section{Preliminaries}	\label{SECTION: Problem Formulation}
	In this section, we provide some necessary preliminaries that are useful for the analysis to follow.

	\subsection{Asset Prices and Returns}
	For $k=0, 1, 2, \dots$, let~$S(k) > 0$ be the \textit{price} of an underlying risky asset, often to be stock, at stage~$k$. Then the associated \textit{per-period return}, call it $X(k)$, is given by
	$
		X(k) := \frac{S(k+1) - S(k)}{S(k)}.
$
	Assuming that~$X_{\min} \leq X(k)\leq X_{\max}$ for all $k$ almost surely with known bounds
	$
	-1 < X_{\min} < 0 < X_{\max} < \infty
	$
	and~$X_{\min}$ and~$X_{\max}$ are in the support of $X(k)$.
	Unlike many existing papers in finance that models the asset price dynamics, we assume that~$X(k)$ are independent in $k$ and have \textit{arbitrary} distribution with common but unknown mean $\mathbb{E}[X(k)] := \mu  $ and \textit{unknown} variance
	$
	{\rm var}(X(k)) = \mathbb{E}[(X(k) - \mu)^2]:= \sigma^2 \geq 0
	$
	for all $k$.

	\begin{remark} \rm
		In practice, since the true mean $\mu$ and variance~$\sigma$ are unknown, one often works with the estimated surrogates such as sample mean and sample standard deviation; see \cite[Chapter 9]{luenberger2013investment} and  Section~\ref{subsection: trading with historical data} for an illustration.
		While this approach may incur some estimation errors, as seen in the sections to follow, we shall quantify a range for $\mu$ such that the desired positivity of expected return can be preserved.
	\end{remark}

	\subsection{Double Linear Policy and Transaction Costs}
	With initial account value $V(0):=V_0>0$, we split it into two parts as follows:
	Taking an \textit{allocation constant}~$\alpha \in [0,1]$, we define~$V_L(0) := \alpha V_0$ as the initial account value  for \textit{long} position   and~$V_S(0) := (1-\alpha)V_0$ for \textit{short} position.\footnote{In stock trading, a long position means that the trader purchases shares of stock in the hope of making a profit from a subsequent rise in the price of the underlying stock. On the other hand, a short position means that the trader borrows shares from a broker in the hope of making a profit from a subsequent fall in the price of the stock; see~\cite{luenberger2013investment, bodie2009investments}.}
	Then~$V(0) = V_L(0) +V_S(0) .$
	The~\emph{trading policy} or  \textit{controller}~$\pi(\cdot)$ is given~by
	$
	\pi(k):=\pi_L(k) +\pi_S(k)
	$
	where~$\pi_L$ and~$\pi_S$ are of \textit{double linear} forms:
	\begin{align} \label{eq: double linear feedback}
		\begin{cases}
			\pi_L( k ) = w V_L\left( k \right);\\
			\pi_S( k ) =  - w V_S \left( k \right)
		\end{cases}
	\end{align}
	where the weight $w$ satisfies

	$	w \in \mathcal{W}$ for some admissible set $\cal W$.
	Specifically,
	when there are transaction costs per trade with percentage rate $\varepsilon \in [0,1]$, we take $\varepsilon \pi_L(k) := \varepsilon w V_L(k)$ for long trade or~$\varepsilon |\pi_S(k)| := \varepsilon w V_S(k)$ for short trade. In the sequel, we take $\mathcal{W}:=[0, w_{\max}]$ with
	$$
	w_{\max}:=\min \left\{ \frac{1}{1+\varepsilon},\, \frac{1}{X_{\max}+\varepsilon} \right\} .
	$$
	 In the sequel, we refer to the control scheme above as the \textit{double linear policy} with pair~$(\alpha, w)$.

	\begin{remark} \rm \label{remark: admissible set}
		$(i)$ The double linear form can be viewed as a natural extension of the  pure long linear strategy by taking~$\alpha =1$; e.g., see~\cite{hsieh2019positive, hsieh2020necessary, hsieh2022generalization}.
		$(ii)$ The choice of the upper bound for~$\cal W$ above corresponds to assuring \textit{survival} trades, see Lemma~\ref{lemma: survivability} in the next Section~\ref{subsection: Survivability} to follow, and the trades are \textit{cash-financed}; i.e.,~$|\pi(k)| \leq V(k)$ for all $k$ with probability one; see Lemma~\ref{lemma: Cash-Financing}.
		As seen later in Section~\ref{SECTION: Analysis of Trading Performance} to follow, the double linear policy also enjoys some additional properties such as convexity and positive expectation.
	\end{remark}

	\subsection{Account Value  Dynamics}
	The associated account values under~$\pi_L$ and~$\pi_S$, denoted by~$V_L(k)$ and $V_S(k)$ respectively, are described by the following stochastic recursive equations:
	\begin{align} \label{eq: V_L and V_S}
	\begin{cases}
		{V_L}\left( {k + 1} \right) = V_L( k ) + X\left( k \right){\pi_L}\left( k \right) - \varepsilon \pi_L(k);\\
		{V_S}\left( {k + 1} \right) = V_S( k ) + X\left( k \right){\pi_S}\left( k \right) - \varepsilon |\pi_S(k)| .
	\end{cases}
	\end{align}
	This implies that
	$
	{V_L}( k ) = \prod_{j = 0}^{k - 1} {\left( {1 + w \widetilde{X}_L ( j )} \right)} {V_L}\left( 0 \right)$
	and
	$
	{V_S}( k ) = \prod_{j = 0}^{k - 1} {\left( {1 - w \widetilde{X}_S ( j )} \right)} {V_S}\left( 0 \right)
	$
	where $\widetilde{X}_L(k):=X(k) -\varepsilon$ and $\widetilde{X}_S(k) := X(k) + \varepsilon.$
	Therefore, the \textit{overall account value} at stage $k$ is given~by
	{\small \begin{align*}
			V(k)
			&= V_L(k) + V_S(k)  \\

			&= V_0\left(  \alpha {\prod_{j = 0}^{k - 1} \left( 1 + w \widetilde{X}_L( j ) \right)  + (1-\alpha)\prod_{j = 0}^{k - 1} \left( 1 - w \widetilde{X}_S( j ) \right) } \right).
		\end{align*}
	}Note that the account value $V(k)$ depends on the allocation constant~$\alpha$, the decision weight $w$, transaction costs rate $\varepsilon $ and return sequence $X=\{X(i)\}_{i=0}^{k-1}$.

	\begin{remark} \rm
		 $(i)$ According to classical SLS theory; e.g., see~\cite{barmish2015new, barmish2011arbitrage}, the associated control law can be written as
		\begin{align*}
			\begin{cases}
				\pi_L(k) = \pi_0 +w(V_L(k)-V_L(0)); \\
				\pi_S(k) = -\pi_0 - w(V_S(k) - V_S(0))
			\end{cases}
		\end{align*}
		for some initial investment $\pi_0>0$. It is readily verified that the classical~SLS controller is equivalent to the double linear policy with $\alpha = 1/2$ if one takes $\pi_0 := wV_0/2$ and $V_L(0) = V_S(0) := V_0/2$.
		$(ii)$ Our framework can easily adapt to involve the risk-free asset, such as a \textit{bond} or \textit{bank account}, with per-period interest rate $r(k) \geq 0$ for all $k$ with probability one; see \cite{cvitanic2004introduction}. Then the account value dynamics for the long trade can be modeled as
		\begin{align*}
			V_L( {k + 1} )
			&= (1 + r(k) )V_L( k ) + (X( k ) - \varepsilon - r(k) )\pi_L(k)
		\end{align*}
		and the account value dynamics $V_S(k)$  for short trade stays the same.  By taking the risk-free asset as a \textit{cash} which yields~$r(k) :=0$ for all $k$, we replicate Equation~(\ref{eq: V_L and V_S}).
	\end{remark}

	\section{Analysis of Trading Performance} \label{SECTION: Analysis of Trading Performance}
	In this section, we study the trading performance of the proposed double linear policy. Specifically, we provide several results such as survivability, convexity, positive expectation, and cash-financing lemma.

	\subsection{Survivability Considerations} \label{subsection: Survivability}
	As mentioned in the previous subsection, every $w \in \mathcal{W}$  assures \textit{survivability}, i.e., bankruptcy avoidance. That is, for all $k$, the~$w$-value that can potentially lead to $V(k)<0$ is disallowed.

	\begin{lemmarep}[Survivability] \label{lemma: survivability}
		Let $\varepsilon \in [0, 1]$.
		The double linear policy with a pair $(\alpha, w) \in (0,1] \times \mathcal{W}$ assures
		$
		V(k) > 0
		$  for~$k =0,1,\dots,$ with probability one.
	\end{lemmarep}

	\textit{Proof.} See Appendix.

	\begin{proof}
		Note that for $k=0$, $V(0) = V_0>0$.
		Fix an integer~$k > 0$. Observe that
		\begin{equation}
			\begin{aligned}
				&V(k) = V_L(k) + V_S(k) \\
				&= \left( \alpha \prod_{j = 0}^{k - 1} \left( {1 + w \widetilde{X}_L\left( j \right)} \right)  + (1-\alpha)\prod_{j = 0}^{k - 1} {\left( 1 - w \widetilde{X}_S( j ) \right)}  \right)V_0 \\
				&\geq \left( \alpha ( 1 + w (X_{\min}-\varepsilon)  )^k  + (1-\alpha)  ( 1 - w (X_{\max}+\varepsilon)  )^k  \right)V_0\\
				&:=V_{\min}^*(k).
			\end{aligned}
		\end{equation}
		To complete the proof, it suffices to show that $V_{\min}^*(k) > 0.$
		Since  $w \in \mathcal{W}$
		and~$X_{\min}>-1$, it implies that
$$
			1 + w (X_{\min} -\varepsilon)
			> 1 - w_{\max} (1 + \varepsilon) \geq 0
$$
		where the last inequality holds since~$w_{\max} \leq 1/(1+\varepsilon)$.
		Hence $(1+ w (X_{\min} - \varepsilon) )^k > 0$ for all $k$.
		On the other hand, since $0\leq w \leq 1/(X_{\max} + \varepsilon)$, it follows that
$
			1- w (X_{\max} + \varepsilon)\geq 0.
$
Hence, $(1- w (X_{\max} + \varepsilon))^k \geq 0$.
In combination with the results above and the fact that $\alpha \in (0,1]$, we conclude that $V_{\min}^*(k) > 0$ for all~$k$ with probability~one.

	\end{proof}

\subsection{Expected Trading Gain-Loss and Variance}

	We take
$
\mathcal{G}_k(\alpha, w, X, \varepsilon ) := V(k)-V_0
$
to be the \textit{cumulative trading gain-loss function} up to stage~$k$ where~$X:=\{X(i)\}_{i=0}^{k-1}$ is the sequence of returns~$X(0), \dots,X(k-1)$.
Then the expected cumulative gain-loss is given by
$$
\overline{\mathcal{G}}_k(\alpha, w, \mu, \varepsilon):=\mathbb{E}[\mathcal{G}_k(\alpha, w,  X, \varepsilon)].
$$
	The following lemma states the closed-form expression of the expected value and variance of the cumulative gain-loss function when there are transaction costs.

	\begin{lemmarep}[Expected Trading Gain-Loss and Variance] \label{lemma: expected cumulative gain or loss and variance}
		Fix~$\varepsilon \in [0,1]$. For stage $k=0,1,\dots$,
		the double linear policy with pair~$(\alpha, w) \in [0,1] \times \mathcal{W}$ yields expected cumulative gain-loss function
		\begin{align*}
			&\overline{\mathcal{G}}_k(\alpha, w, \mu, \varepsilon) \\
			&=  V_0 \left( \alpha \left( 1 + w (\mu- \varepsilon) \right)^k+ (1-\alpha)\left( 1 - w (\mu + \varepsilon)  \right)^k - 1 \right).
		\end{align*}
		Moreover, the corresponding variance is given by
		\begin{align*}
			&{\rm var}(\mathcal{G}_k(\alpha, w, X, \varepsilon))\\
			& = 	  \alpha^2   \left( (1 + w (\mu - \varepsilon) )^2 +  w^2 \sigma^2   \right)^k \\
			&\;\;\;\; + (1-\alpha)^2  \left( (1 - w (\mu + \varepsilon) )^2 + w^2 \sigma^2  \right)^k \\
			& \;\;\;\; + 2\alpha (1-\alpha)  ( 1 + 2 w \varepsilon - w^2 ( (\mu^2 + \sigma^2)- \varepsilon^2)  )^k  \\
			& \;\;\;\; - 2\alpha \left( 1 + w (\mu- \varepsilon) \right)^k  - 2(1-\alpha)\left( 1 - w (\mu+ \varepsilon) \right)^k  +1  \\
			& \;\;\;\;  -  \left( \alpha \left( 1 + w (\mu- \varepsilon) \right)^k+ (1-\alpha)\left( 1 - w (\mu + \varepsilon)  \right)^k - 1 \right)^2.
		\end{align*}
		Of course, the standard deviation of cumulative trading gain or loss is ${\rm std}(\mathcal{G}_k(\alpha, w, X, \varepsilon)) =\sqrt{{\rm var}(\mathcal{G}_k(\alpha, w, X, \varepsilon))}$.
	\end{lemmarep}

		\textit{Proof.} See Appendix.

	\begin{proof}
		Let $k$ be fixed. Set $R_L(k) := \prod_{j = 0}^{k - 1} \left( 1 + w \widetilde{X}_L(j) \right)$ and $R_S(k) := \prod_{j = 0}^{k - 1} \left( 1 - w \widetilde{X}_S(j)  \right).$
		Using the facts that the returns $X(k)$ are independent in $k$ and~$\mathbb{E}[X(k)] = \mu$ for all~$k$, we have
	{\small
		\begin{align} \label{eq: expected trading gain }
				& \overline{\mathcal{G}}_k(\alpha, w, \mu, \varepsilon) \nonumber  \\
				& = V_0 ( \alpha \mathbb{E}[ R_L(k)]  + (1-\alpha) \mathbb{E}[R_S(k)]  - 1  ) \nonumber  \\

				&= V_0 \left( \alpha \left( 1 + w (\mu- \varepsilon) \right)^k+ (1-\alpha)\left( 1 - w (\mu + \varepsilon)  \right)^k - 1 \right).
		\end{align}
	}
		Subsequently,	to obtain the variance, we use the fact
		$
		{\rm var}(\mathcal{G}_k(\alpha, w, X, \varepsilon) )=\mathbb{E}[\mathcal{G}_k^2(\alpha, w, X, \varepsilon)]-\overline{\mathcal{G}}_k^2 (\alpha, w, \mu, \varepsilon).
		$
		Since the second term can be obtained from Equation~(\ref{eq: expected trading gain }), it remains to calculate the second moment $\mathbb{E}[\mathcal{G}_k^2(\alpha, w, X, \varepsilon)]$.
		Without loss of generality, we set $V_0:=1$.
		Using the facts that~$X(k)$ are independent, $\mathbb{E}[X(k)] = \mu$, $\mathbb{E}[X^2(k)] = \sigma^2 + \mu^2$ for all $k$,  and the linearity of expected value, a straightforward calculation leads to
		\begin{align*}
			\mathbb{E}[\mathcal{G}_k^2(\alpha, w, X, \varepsilon)]
			&=  \mathbb{E}[ \left( \alpha R_L(k)  + (1-\alpha)R_S(k)  - 1 \right)^2 ]\\

			&=   \alpha^2 \mathbb{E}[  R_L^2(k)]  + (1-\alpha)^2 \mathbb{E}[  R_S^2(k)] \\
			&\;\;\;\; + 2(1-\alpha)\alpha \mathbb{E}[  R_L(k)R_S(k)]  \\
			&\;\;\;\; - 2\alpha \mathbb{E}[  R_L(k)  ] - 2(1-\alpha)\mathbb{E}[  R_S(k)]   +1
		\end{align*}
		where the three terms $\mathbb{E}[R_L^2(k)],$ $\mathbb{E}[R_S^2(k)]$, and $\mathbb{E}[R_L(k) R_S(k)]$ are obtained via lengthy but straightforward calculations. That is, we have
		\begin{align*}
			\mathbb{E}[R_L^2(k)]

			&=    \left( (1 + w (\mu - \varepsilon) )^2 +  w^2 \sigma^2   \right)^k,
		\end{align*}
and
$
			\mathbb{E}[R_S^2(k)]

			=   \left( (1 - w (\mu + \varepsilon) )^2 + w^2 \sigma^2  \right)^k,
$
		and the cross term
$
			\mathbb{E}[R_L(k)R_S(k)]

			=     ( 1 + 2 w \varepsilon - w^2 ( (\mu^2 + \sigma^2)- \varepsilon^2)  )^k.
$
		Therefore, in combination with the results above, the desired expression for the variance is obtained.
		To complete the proof, we note that the standard deviation ${\rm std}(\mathcal{G}_k(\alpha, w, X, \varepsilon))=\sqrt{{\rm var}(\mathcal{G}_k(\alpha, w, X, \varepsilon))}$, which is desired.
	\end{proof}

	\begin{remark} \rm
		$(i)$ As a sanity test, if one decides not to trade; i.e., $w=0$,  then
		$
		\overline{\mathcal{G}}_k(\alpha, w, \mu, \varepsilon) = {\rm var}(\mathcal{G}_k(\alpha, w, X, \varepsilon))=0.
		$
		$(ii)$~If there are no transaction costs and no uncertainty on the return; i.e., $\sigma = \varepsilon  :=0$, then  a straightforward calculation leads to
		$
		{\rm var}(\mathcal{G}_k(\alpha, w, X, \varepsilon)) =0.
		$
		$(iii)$~If the returns has no trend; i.e., $\mu=0$, then it follows that $\overline{\mathcal{G}}_k(\alpha, w,\mu,\varepsilon) =V_0 \left(  \left( 1 - w \varepsilon \right)^k - 1 \right) \leq 0$.
		Thus, with nonzero transaction costs~$\varepsilon>0$, one should invest with zero weight $w=0$ to avoid loss in expected gain-loss.

		$(iii)$ Lemma~\ref{lemma: expected cumulative gain or loss and variance} above can be viewed as a generalization of the result for the classical discrete-time SLS controller considered in \cite{deshpande2018generalization}.
	\end{remark}

	\subsection{Convexity and Positive Expectation} \label{SECTION: Convexity}
To study the expected positivity, the following  convexity result of the expected gain-loss function is useful.

	\begin{lemmarep} [Convexity] \label{lemma: convexity}
		Fix $\varepsilon \in [0,1]$. For $k > 1$,	if~$\alpha \in (0,1)$, then  the expected cumulative gain-loss function~$\overline{\mathcal{G}}_k(\alpha, w, \mu, \varepsilon)$ is strictly convex in $w \in \mathcal{W}$.

	\end{lemmarep}

		\textit{Proof.} See Appendix.

	\begin{proof}
		According to Lemma~\ref{lemma: expected cumulative gain or loss and variance}, we have $\overline{\mathcal{G}}_k(\alpha, w, \mu, \varepsilon) = V_0 f_k(\alpha, w,\mu,\varepsilon)$  where
		\begin{align*}
			f_k(\alpha, w,\mu,\varepsilon) &:=  \alpha \left( 1 + w (\mu- \varepsilon) \right)^k \\
			&+ (1-\alpha)\left( 1 - w (\mu + \varepsilon)  \right)^k - 1 .
		\end{align*}
		Note that any strict convex function multiplied by a strictly positive scalar is still strict convex.
		Fix $k \geq 2$.
		Since~$V_0>0$,  it suffices to show that $f_k(\alpha, w,\mu,\varepsilon)$ is strictly convex in $w \in \mathcal{W}$.
		Taking the derivative with respect to $w$ twice yields

		the second derivative
		{\small \begin{align*}
			\frac{\partial^2 f_k(\alpha, w, \mu, \varepsilon) }{\partial w^2}
			&= \alpha k(k-1)^2(1 + w (\mu - \varepsilon) )^{k-2} (\mu - \varepsilon)^2 \\
			&+ (1-\alpha)k(k-1)^2(1 - w (\mu + \varepsilon) )^{k-2}(\mu + \varepsilon)^2.
		\end{align*}
	 }To establish the desired convexity, it suffices to verify positivity of $(1 + w (\mu - \varepsilon) )^{k-2}$ and $(1 - w (\mu + \varepsilon) )^{k-2}$.
		Since $w \in \mathcal{W}$ and $\mu > -1$, we observe that
		$
		1+w (\mu - \varepsilon) > 1-w (1+\varepsilon)\geq 0.
		$
		Therefore, $(1+w (\mu - \varepsilon))^{k-2} >0$.
		On the other hand, note that
		$
		1 - w (\mu + \varepsilon)  \geq 1 - w(X_{\max} +\varepsilon)\geq 0,
		$ which implies that $(1 - w (\mu + \varepsilon) )^{k-2} \geq 0$.
		Therefore, in combination with the fact that $\alpha \in (0,1)$, it implies that $\frac{\partial^2}{\partial w^2} f_k(\alpha, w, \mu, \varepsilon) > 0$ for all~$w \in \mathcal{W}$. Hence, the strict convexity for  $\overline{\mathcal{G}}_k(\alpha, w, \mu, \varepsilon) $ is established.
	\end{proof}

	\begin{remark} \rm
	$(i)$	A similar proof used in Lemma~\ref{lemma: convexity} can be made to show that $\overline{\mathcal{G}}_k$ is strictly convex in $\mu.$
$(ii)$ For $\varepsilon >0$ and~$\alpha = 1/2$, the expected gain-loss function has the minimum at $\mu_0 = 0$. For $\alpha \neq 1/2,$ then the minimum, call it $\mu_0$, requires to solve a nonlinear equation
$
e^{\frac{1}{k} \log \frac{\alpha}{1-\alpha}} = \frac{1 + w(\mu-\varepsilon)}{ 1 - w(\mu +\varepsilon)}.
$
In either case, the associated gain-loss function~$\overline{\mathcal{G}}_k(\alpha, w, \mu_0, \varepsilon) < 0$; see also example in Section~\ref{subsection: Mote-Carlo Simulations}.
Using the fact that $\overline{\mathcal{G}}_k(\cdot)$ is continuous and strictly convex in $\mu$,  Intermediate Theorem; e.g., see \cite{rudin1964principles}, indicates that there exist two critical points~$\mu_{\pm}$ such that~$\overline{\mathcal{G}}_k(\alpha, w, \mu_{\pm}, \varepsilon) = 0$.
Therefore, strict convexity assures that for $\mu > \mu_+$ or $\mu < \mu_-$, $\overline{\mathcal{G}}_k(\alpha, w, \mu_{\pm}, \varepsilon) > 0$.
However, obtaining the two critical points $\mu_{\pm}$ analytically require solving high-order nonlinear functions, which is intractable in general. In the following theorem, we propose an asymptote approach for $\overline{\mathcal{G}}_k$ to determine a moderately conservative positive expectation property.
	\end{remark}

	\begin{theorem}[Positive Expectation] \label{theorem: Robust Positive Expectation}
		Consider the double linear policy with pair $(\alpha, w)$.  For all~$k>1$, the following statements hold true:

		$(i)$ If $\varepsilon  = 0$, then any pair $(\alpha, w)  \in \{1/2\} \times \mathcal{W} \setminus \{0\}$ guarantees the positive expected cumulative trading gain-loss; i.e.,
		$
		\overline{\mathcal{G}}_k(1/2, w, \mu, \varepsilon) > 0
		$
		for all $\mu \neq 0$. Moreover,  if $\mu =0$, we have $	\overline{\mathcal{G}}_k(1/2, w, \mu, \varepsilon) = 0.$

		$(ii)$ If $\varepsilon>0$, then any pair $(\alpha, w) \in (0,1) \times \mathcal{W} \setminus \{0\}$ guarantees a  positive expected cumulative trading gain-loss; i.e.,
		$
		\overline{\mathcal{G}}_k(\alpha, w, \mu, \varepsilon) > 0
		$
		for all $\mu > \mu_+$ or $\mu < \mu_-$ where
		$$
		\mu_+ = \frac{1}{w} \left( e^{\frac{1}{k}\log \frac{1}{\alpha}} - 1 \right) + \varepsilon
		$$
		and
		$$
		\mu_- = \frac{1}{w} \left( 1-e^{\frac{1}{k} \log \frac{1}{1-\alpha}} \right) -\varepsilon.
		$$
	\end{theorem}

	\begin{proof}
		To prove part~$(i)$, take $\varepsilon =0$ and fix $k>1$. Take the pair $(\alpha, w) \in \{1/2\} \times \mathcal{W} \setminus \{0\}$.  Lemma~\ref{lemma: expected cumulative gain or loss and variance} tells us that
		\[
		\overline{\mathcal{G}}_k(1/2, w, \mu, 0) = \frac{V_0}{2}\left( {{{\left( {1 + w\mu } \right)}^k} + {{\left( {1 - w\mu } \right)}^k} - 2} \right).
		\]
		If $\mu =0$, it is trivial to see that $\overline{\mathcal{G}}_k(1/2, w, 0, 0)=0.$
		On the other hand, if~$\mu \neq 0$, then $w\mu \neq 0$ for all $w \in \mathcal{W}\setminus \{0\}$.
		By virtue of the fact that $(1+x)^k + (1-x)^k > 2$ for all~$x \neq 0$ and~$k > 1$,
		the desired positivity is guaranteed by~$x := w \mu$.

		To prove part~$(ii)$, we take $\varepsilon >0 $ and
		fix $k>1$.
		Take a pair $(\alpha, w) \in  (0,1) \times \mathcal{W} \setminus \{0\}$.  Lemma~\ref{lemma: expected cumulative gain or loss and variance} tells us that
		$
		\overline{\mathcal{G}}_k(\alpha, w, \mu, \varepsilon) = V_0 f_k(\alpha, w,\mu, \varepsilon)
		$
		where
		\begin{align*}
			f_k(\alpha, w,\mu, \varepsilon)
			&:=\alpha \left( 1 + w (\mu- \varepsilon) \right)^k\\
			&+ (1-\alpha)\left( 1 - w (\mu + \varepsilon)  \right)^k - 1.
		\end{align*}
		Since $V_0>0$, without loss of generality, we set~$V_0:=1$.
		Note that
		$
		 f_k(\alpha, w,\mu, \varepsilon)  \geq \alpha \left( 1 + w (\mu- \varepsilon) \right)^k - 1
		 $
		  and  $f_k(\alpha, w, \mu, \varepsilon)$ is continuous in $\mu$.
		The asymptote~$\alpha \left( 1 + w (\mu- \varepsilon) \right)^k - 1$ is continuous and monotonically increasing in $\mu$.
		By solving the zero crossing root for the asymptote; i.e., solving~$\alpha \left( 1 + w (\mu- \varepsilon) \right)^k - 1:=0	$
		for~$\mu$ yields
		\[
		\mu  = \frac{1}{w} \left( e^{\frac{1}{k}\log \frac{1}{\alpha}} - 1 \right) + \varepsilon :=\mu_+
		\]
		and $\mu_+ >0.$
		Hence, for $\mu > \mu_+$, we have $f_k(\alpha, w, \mu, \varepsilon) >0.$
		Similarly, it is also readily verified that
		$$
		f_k(\alpha, w,\mu, \varepsilon)  \geq (1-\alpha)\left( 1 - w (\mu + \varepsilon)  \right)^k - 1
		$$
		and the asymptote  $(1-\alpha)\left( 1 - w (\mu + \varepsilon)  \right)^k - 1$ is again continuous and monotonically decreasing in $\mu$.
		Hence, solving~$(1-\alpha)\left( 1 - w (\mu + \varepsilon)  \right)^k - 1:=0$ yields
		\[
		\mu    =\frac{1}{w} \left( 1-e^{\frac{1}{k} \log \frac{1}{1-\alpha}} \right) -\varepsilon := \mu_-
		\]  and $\mu_- <0$.
		Therefore, for $\mu < \mu_-$, $f_k(\alpha, w, \mu, \varepsilon) >0$.
		It is also readily verified that $\mu_- < 0 <  \mu_+$.
		Hence, in combination with the results above, we conclude that the desired positivity~$
		f_k(\alpha, w, \mu, \varepsilon) >0$ holds
		for $ \mu >  \mu_+$ or~$ \mu <  \mu_- .$
	\end{proof}

	\begin{remark} \rm
		The two critical points $\mu_{\pm}$ obtained in Theorem~\ref{theorem: Robust Positive Expectation} relies on zero crossing roots of the asymptote of $\overline{\mathcal{G}}_k$. Hence, it gives a conservative criterion for assuring the positive expectation.
		In addition, if $k \to \infty$, then~$\mu_+, \mu_- \to \varepsilon, -\varepsilon$, respectively. When $k \to \infty$ and~$\varepsilon = 0$, part~$(ii)$ in Theorem~\ref{theorem: Robust Positive Expectation} reduces to part~$(i)$ in the same theorem.
	\end{remark}

	\subsection{Cash-Financing Lemma}
	As mentioned in Remark~\ref{remark: admissible set}, the double linear policy with pair $(\alpha, w) \in [0,1] \times \mathcal{W}$ assures a cash-financed trade.

	\begin{lemmarep}[Cash-Financing] \label{lemma: Cash-Financing}
		The double linear policy with a pair~$(\alpha, w) \in [0,1] \times \mathcal{W}$ satisfies the cash-financing condition. That is,
		$
		| \pi(k) |  \leq V(k)
		$ for all $k\geq 0$ with probability one.
	\end{lemmarep}

		\textit{Proof.} See Appendix.

	\begin{proof}
		With $V_0>0$, fix $(\alpha, w) \in [0,1] \times \mathcal{W}$. For $k=0$,
		we begin by noting that $\pi_L(0) = w V_L(0) = \alpha wV_0$ and $\pi_S(0) = -wV_S(0) = -(1-\alpha) wV_0$.
		Thus, observe that
		\begin{align*}
		|\pi(0)| &= |\pi_L(0) + \pi_S(0)| \\
		&\leq |\pi_L(0)| + |\pi_S(0)| = (2\alpha  - 1)wV_0 \leq V_0
		\end{align*}
	where the first inequality follows from the triangle inequality.
		Fix~$k \geq 1$. We have $\pi(k)  = \pi_L(k) +\pi_S(k)$ where $\pi_L(k) = wV_L(k)$ and $\pi_S(k) = -wV_S(k)$.
		Since~$w \in \mathcal{W}$, applying Lemma~\ref{lemma: survivability}, it is readily verified that~$V_L(k), V_S(k) \geq 0$ for all~$k\geq 1$ with probability one.
		\begin{align*}
			|\pi(k)|
			&= |\pi_L(k) + \pi_S(k)|\\
			& \leq w |V_L(k) + V_S(k)| \\
			&= w V(k) \leq V(k)
		\end{align*}
		where the last inequality holds since $w \in \mathcal{W} = [0, w_{\max}]$ and $w_{\max} \leq \frac{1}{1+\varepsilon} \leq 1$.
	\end{proof}

	\section{Illustrative Examples} \label{SECTION: Illustrative Examples}
	This section provides two examples to illustrate our double linear policy scheme. The first example examines the positive expectation by carrying large numbers of Monte-Carlo simulations for prices modeled by geometric Brownian motion with jumps. The second example is for illustrating the trading performance using cryptocurrency historical data.

\subsection{Monte-Carlo Simulations:  GBM with Jumps} \label{subsection: Mote-Carlo Simulations}
For $t \in [0, T]$,	consider a stock whose price $S(\cdot)$ follows a geometric Brownian motion (GBM) with jump model; see~\cite{etheridge2002course}; i.e.,
\[
S(t) = S_0 \exp \left(\left(\mu^* - \frac{1}{2}\sigma^{*2} \right)t + \sigma^* W(t)\right)(1-\delta)^{N(t)}
\]
where $\{W(t)\}_{t\ge 0}$ is a standard Wiener process and $\{N(t)\}_{t \ge 0}$ is  a standard Poisson process with
$P(N(t) = k) =\frac {(\lambda t)^k}{k!}e^{-\lambda t}$ that is independent of $W(t)$, $\mu^*$ is the {\it drift} constant, $\sigma^* > 0$ is the {\it volatility} constant, $\lambda$ is the average rate of the jump occurs for the process,  and~$\delta \in (0,1)$ denotes that for each jump reduces the stock price by a fraction $\delta$.

	Taking a basic period length $\Delta t:=1/252$ and $T:=1$, we simulate the stock behavior for one year with an annualized\footnote{For a one-year $252$ trading days,  annualized drift rate and annualized volatility can be approximated by the daily drift rate and daily volatility; i.e.,~$\mu^* = 252 \mu$ and $\sigma^* = \sqrt{252}\sigma$.} drift rate $\mu^* \in [-0.9, 0.9]$ and annualized volatility~$\sigma^* := 2 |\mu^*| Z$ with $Z$ being a uniformly distributed random variable on interval~$[0,1]$, jump intensity $\lambda = 0.1$ with jump size~$\delta = 0.05$.
	For each $\mu^* \in [-0.9 : 0.01 : 0.9]$ with incremental values $0.01$ and $\sigma^*$, we generate $10,000$ GBM with jump sample paths, hence, a total of $1,810,000$ paths;
	 see also \cite{luenberger2013investment} for details on this topic.

 	With $\varepsilon = 0.01\%$ and $V_0:=\$1$, we consider a double linear policy with pair $(\alpha, w) \in [0,1] \times \{1/(1+\varepsilon)\}$.
	Then we evaluate the associated expected trading gain-loss functions versus various~$\mu^*$; see Figure~\ref{fig:robustgainlossm10000}, where the red-colored dotted lines are obtained by Lemma~\ref{lemma: expected cumulative gain or loss and variance}, and the gray area indicates the negative expected gain-loss.
	From the figure, for each choice of $\alpha$, we see that the Monte-Carlo results are consistent with the theory. In addition, it indicates that the desired expected positivity occurs after $\mu >  \mu^*_+$ and $\mu < \mu^*_-$ for some $\mu_+^*$ and~$\mu_-^*$. Similar patterns are also found for different choices of $\varepsilon, \lambda, \delta,$ and  $\sigma$.

\begin{figure}[h!]
	\centering
	\includegraphics[width=0.8 \linewidth]{figs/robust_gain_loss_M=10000}
	\caption{Gain-Loss Function Versus $\mu^* \in [-0.9, 0.9]$}
	\label{fig:robustgainlossm10000}
\end{figure}

\subsection{Backtesting with Cryptocurrency} \label{subsection: trading with historical data}
In this example, we apply the double linear policy to trade Bitcoin USD (BTC-USD) within the period from January~02,~2020 to August  01, 2022 with a total of $952$ days; see Figure~\ref{fig:bitcoinprice} for the corresponding prices trajectory.
Compared to conventional stocks, Bitcoin prices are much more volatile and sensitive to regulatory and market events.
 In this case, maximum and minimum daily returns are~$X_{\max} \approx 0.1875$ and $X_{\min} \approx -0.3717$, respectively.
With initial account value of~$V_0 = \$100,000$, transaction costs\footnote{Generally, most cryptocurrency exchanges charge between 0\% and 1.5\% per trade, depending on the trading volume and whether it is buying or selling; e.g., see~\cite{kim2017transaction}. } $\varepsilon = 0.01\%$ and weight~$w= 1/4$, the trading performances in terms of gain-loss functions are shown in Figure~\ref{fig:bitcointrading}.
For higher transaction costs, say~$\varepsilon = 0.1\%$ with the same weight $w=1/4$, the trading performances  are shown in Figure~\ref{fig:bitcointradingk14}.
From the figures, we see the effect of transaction costs, and one should choose  a proper allocation constant $\alpha$ of the fund when $\varepsilon>0.$

\begin{figure}[h!]
	\centering
	\includegraphics[width=.8\linewidth]{figs/Bitcoin_price}
	\caption{BTC-USD Daily Prices (January 02, 2020 -- August 01, 2022).}
	\label{fig:bitcoinprice}
\end{figure}

\begin{figure}[h!]
	\centering
	\includegraphics[width=.8\linewidth]{figs/Bitcoin_trading}
	\caption{Trading Against BTC-USD with $\varepsilon = 0.01\%$.}
	\label{fig:bitcointrading}
\end{figure}

\begin{figure}[h!]
	\centering
	\includegraphics[width=0.8\linewidth]{figs/Bitcoin_trading_K_1_over_4}
	\caption{Trading Against BTC-USD with $\varepsilon = 0.1\%$.}
	\label{fig:bitcointradingk14}
\end{figure}

\section{Conclusion} \label{SECTION: Conclusion}
In this paper, we propose a new class of double linear policy schemes in an asset trading scenario with nonzero transaction costs.
We  indicated that the desired positive expectation result might disappear when the costs are involved.
 In addition, we quantify under what conditions the positive expectations can still be preserved.
We also validate our theory via heavy Monte-Carlo simulations using  GBM with jumps and provide a backtesting example using Bitcoin-USD historical data.

Regarding future research, it would be of interest to determine an ``optimal" pair $(\alpha, w)$ that also preserves desired positivity; see \cite{maroni2019robust} for some initial studies along this line.
The other interesting direction is to consider a multi-asset portfolio case; see \cite{primbs2018pairs, deshpande2020simultaneous} for a preliminary extension to pair tradings.

	\bibliographystyle{ieeetr}


\section{Introduction}
Fuelled by the Covid-19 pandemic, logistics has endured great stress in recent years and accelerated the adoption of new technologies. This has led to a surge in demand for intelligent robots able to operate in unstructured environment.
In this paper we focus on a sequential picking task for logistics setting (see Fig.~\ref{fig:task}). A robot manipulator needs to unload a stack of parcels having as unique source of input a RGB-D (Red, Green, Blue, Depth) image of the scene.

Classic systems address this type of pick-and-place problems by integrating objects' pose estimation \cite{yoon2003real, zhu2014single} with scripted planning and motion control \cite{frazzoli2005maneuver}. While these systems are robust in structured and deterministic setting such as manufacturing, they have shown to be unsuitable for the unstructured, noisy and stochastic setting typical of logistics.
This has renewed the interest in end-to-end models mapping pixels to action which can be trained from data and achieve better performance under uncertainties.
In this regard, Reinforcement Learning (RL) \cite{sutton2018reinforcement} turned out capable of training end-to-end models accomplishing good results in pick and place manipulation tasks \cite{levine2016end, kalashnikov2018scalable}. However, classic RL still requires copious amount of data to converge which makes it impracticable for real world scenarios.
In order to improve sample efficiency \cite{florence2019self}, recent works have integrated RL with object-centric assumptions such as keypoints \cite{manuelli2019kpam, kulkarni2019unsupervised}, embeddings \cite{jund2018optimization} or dense descriptors \cite{florence2018dense}. These representations are usually learned via supervised learning (SL) \cite{liu2011supervised} which often imposes tedious and expensive data collection burdens. Another line of research has recently focused on imitation learning (IL) or learning from demonstrations (LfD) \cite{zeng2020transporter, huang2022equivariant}. Despite the remarkable results and the need for a weaker form of supervision, IL and LfD remain in the SL spectrum and both observations and action/labels need to be provided to the learner.

In this work, we do another step towards successfully and efficiently learning picking strategies in a less supervised manner. We propose an algorithm based on Generative Adversarial Imitation Learning (GAIL) for unsupervised reward shaping in Deep RL which only requires expert observations. Note that the distinction between expert observations and demonstrations will be better formalized later in the paper; in simple terms we refer to demonstrations as state-action pairs; whereas, expert observations are only the states visited by the expert's policy.
To the best of our knowledge, this is the first work which solves a sequential picking task from visual observations without direct supervision. We compare our algorithm, Unsupervised Reward Shaping with Adversarial Imitation Learning from Observations (URSfO) with its RL-only counterpart and show improvement in asymptotic performance.

\paragraph{Outline:} The remainder of the paper is organized as follows. The Preliminaries section presents a brief formal introduction to the main algorithmic techniques used throughout the paper, i.e. RL, IL and Inverse RL. The Environment section provides details about the sequential picking task and the simulator used to recreate the task. The Methods section describes the tested algorithms and provides theoretical motivations for the URSfO formulation. The Implementations and Results section deals with implementation details and reports the obtained experimental results. Finally, in the Conclusions section we discuss our results and provide interesting directions for future work.

\paragraph{Notation:} Unless otherwise indicated, we use uppercase letters (e.g., $S_t$) for random variables, lowercase letters (e.g., $s_t$) for values of random variables, script letters (e.g., $\mathcal{S}$) for sets, and bold lowercase letters (e.g., ${\mathbf m}{\theta}$) for vectors. Let $[t_1 : t_2]$ be the set of integers $t$ such that $t_1 \leq t \leq t_2$; we write $S_t$ such that $t_1 \leq t \leq t_2$ as $S_{t_1 : t_2}$. Finally, $\mathbb{E}[\cdot]$ represents expectation, $\mathbb{P}(\cdot)$ probability and $\mathbb{D}(\cdot, \cdot)$ divergence between distributions.

\section{Preliminaries}

\paragraph*{Reinforcement Learning:}We consider an infinite-horizon discounted Markov Decision Process (MDP) defined by the tuple $(\mathcal{S}, \mathcal{A}, P, r, d_0, \gamma)$ where $\mathcal{S}$ is the set of states and $\mathcal{A}$ is the set of actions. $P:\mathcal{S}\times \mathcal{A} \rightarrow \Delta_{\mathcal{S}}$ is the transition probability function and $\Delta_{\mathcal{S}}$ denotes the space of probability distributions over $\mathcal{S}$. The function $r:\mathcal{S}\times \mathcal{A} \rightarrow \mathbb{R}$ maps state-action pairs to scalar rewards, $d_0\in\Delta_{\mathcal{S}}$ is the initial state distribution and $\gamma \in [0,1)$ the discount factor. We model the decision agent as a stationary policy $\pi:\mathcal{S}\rightarrow\Delta_{\mathcal{A}}$, where $\pi(a|s)$ is the probability of taking action $a$ in state $s$. The goal is to choose a policy that maximizes the expected total discounted reward $J(\pi)=\mathbb{E}_{\tau}[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)]$, where $\tau = (s_0,a_0,s_1,a_1,\dots)$ is a trajectory sampled according to $s_0 \sim d_0$, $a_t\sim\pi(\cdot|s_t)$ and $s_{t+1}\sim P(\cdot|s_t,a_t)$. To streamline the notation we write $\tau \sim \pi$. A policy $\pi$ induces a normalized discounted state visitation distribution $d_{\pi}$, where $d_{\pi}(s) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{P}(s_t=s | d_0,\pi,P)$. We define the corresponding normalized discounted state-action visitation distribution as $\mu_{\pi}(s,a)= d_{\pi}(s)\pi(a|s)$ and the normalized discounted state-transition visitation distribution as $\rho_{\pi}(s,s')=d_{\pi}(s)\int_{\mathcal{A}} P(s'|s,\Bar{a})\pi(\Bar{a}|s) d\Bar{a}$. We denote the state value function of $\pi$ as $V^{\pi}(s) = \mathbb{E}_{\tau \sim \pi}[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)|S_0=s]$, the state-action value function $Q^{\pi}(s,a) = \mathbb{E}_{\tau \sim \pi}[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)|S_0=s, A_0=a]$ and the advantage function as $A^{\pi}(s,a) = Q^{\pi}(s,a) - V^{\pi}(s)$. Finally, when a function is parameterized with parameters ${\mathbf m}{\theta} \in \varTheta \subset \mathbb{R}^k$ we write $\pi_{{\mathbf m}{\theta}}$.

\paragraph{Imitation Learning:}
Given a task and an expert, IL infers the underlying expert's policy distribution via a set of expert demonstrations (state-action pairs). Assuming the expert's policy is parameterized by parameters ${\mathbf m}{\theta}^*$, we refer to the process of estimating ${\mathbf m}{\theta}^*$ through a finite sequence of expert demonstrations $\tau_E = (s_{0:T},a_{0:T})$ with $2 \leq T < \infty$ as IL. One way to formulate this problem is through maximum likelihood estimation:
\begin{equation}
    \max_{{\mathbf m}{\theta}}  \mathcal{L}({\mathbf m}{\theta}),
    \label{eq:IL_max_likelihood}
\end{equation}
where $\mathcal{L}({\mathbf m}{\theta})$ denotes the log-likelihood and is equivalent to the logarithm of the joint probability of generating the expert demonstrations $\tau_E = \{s_0,a_0,s_1,a_1,\dots,s_T,a_T\}$, i.e.,
\begin{equation}
    \mathcal{L}({\mathbf m}{\theta}) = \log \mathbb{P}^{{\mathbf m}{\theta}}_{d_0}(\tau_E).
    \label{eq:log_likelihood}
\end{equation}
$\mathbb{P}^{{\mathbf m}{\theta}}_{d_0}(\tau_E)$ in \eqref{eq:log_likelihood} is defined as
\begin{align}
    \begin{split}
        &\mathbb{P}^{{\mathbf m}{\theta}}_{d_0}(\tau) =
        d_0(s_0)\bigg[\prod_{t=0}^{T} \pi_{{\mathbf m}{\theta}}(a_t|s_t) \bigg]\bigg[\prod_{t=0}^{T-1}P(s_{t+1}|s_t,a_t)\bigg].
    \end{split}
    \label{eq:Preliminary_joint_distribution}
\end{align}
Computing the logarithm of \eqref{eq:Preliminary_joint_distribution} and neglecting the elements not parameterized by ${\mathbf m}{\theta}$ we obtain the following maximization problem
\begin{equation}
    \max_{{\mathbf m}{\theta}}  \sum_{t=0}^{T}\log \big(\pi_{{\mathbf m}{\theta}}(a_t|s_t) \big).
    \label{eq:IL_max_prob}
\end{equation}

\paragraph{Inverse Reinforcement Learning:}
Given a task and an expert, inverse RL learns a reward function $r_{{\mathbf m}{\chi}}$ via a finite set of expert demonstrations $\tau_E = (s_{0:T},a_{0:T})$, which allows us to estimate the expert's policy distribution over state-action pairs.\\
One way to formulate this problem is by alternating a classification step with a RL step as in \cite{abbeel2004apprenticeship}. In this regard, GAIL \cite{ho2016generative} accomplishes important results formulating the IRL problem as a $\min$-$\max$ game between a discriminator network $D_{{\mathbf m}{\chi}}$ and the learner policy $\pi_{{\mathbf m}{\theta}}$ as similarly done in Generative Adversarial Netowrks (GANs) \cite{goodfellow2014generative}. \\
More specifically, assume we have a set of expert demonstrations $\tau_E$, a set of trajectories $\tau$ generated by $\pi_{{\mathbf m}{\theta}}$ and a discriminator network as $D_{{\mathbf m}{\chi}}: \mathcal{S}\times\mathcal{A} \to (0,1)$. GAIL alternatively optimizes the following two objectives:
\begin{align}
    \max_{{\mathbf m}{\chi}} \ \ \ &\mathbb{E}_{\tau \sim \pi_{{\mathbf m}{\theta}}}[\log(D_{{\mathbf m}{\chi}}(s,a))] + \mathbb{E}_{\tau_E}[\log(1 - D_{{\mathbf m}{\chi}}(s,a))], \label{eq:IRL_disc}\\
    \min_{{\mathbf m}{\theta}}\ \ \ &\mathbb{E}_{\tau \sim \pi_{{\mathbf m}{\theta}}}[\log(D_{{\mathbf m}{\chi}}(s,a))] \label{eq:IRL_policy}.
\end{align}
It has been shown that alternatively optimizing \eqref{eq:IRL_disc} and \eqref{eq:IRL_policy} is equivalent to minimizing the Jensen-Shannon divergence between $\mu_{\pi_{{\mathbf m}{\theta}}}(s,a)$ and $\mu_{\pi_E}(s,a)$, i.e. we are inferring the expert's state-action visitation distribution. Note that, \eqref{eq:IRL_policy} is minimized using the policy gradient theorem in \cite{sutton1999policy}.

\section{Environment}
In the following we provide a detailed description of our sequential picking task. The environment is based on PyBullet \cite{coumans2016pybullet} and all our code is freely accessible at our \href{https://github.com/VittorioGiammarino/Bootstrapping-RL-4-Sequential-Picking}{github repository}\footnote{https://github.com/VittorioGiammarino/Bootstrapping-RL-4-Sequential-Picking}.
\begin{figure}[ht]
    \centering
    \begin{subfigure}[t]{0.25\textwidth}
        \centering
        \includegraphics[width=3.5cm, height=3.5cm]{Figures/state.pdf}
        \caption{Visual Observation.}
        \label{fig:obs}
    \end{subfigure}
    ~
    \begin{subfigure}[t]{0.33\textwidth}
        \centering
        \includegraphics[width=6cm, height=3.5cm]{Figures/task.png}
        \caption{Environment.}
        \label{fig:task}
    \end{subfigure}
    ~
    \begin{subfigure}[t]{0.33\textwidth}
        \centering
        \includegraphics[width=3.6cm, height=3.6cm]{Figures/state_with_action.pdf}
        \caption{Visual Observation and Action.}
        \label{fig:obs+act}
    \end{subfigure}
    \caption{Overview of the Robotic setup. Video available \href{https://youtu.be/rcuXqNGOQRE}{here.}}
    \label{fig:task+obs}
\end{figure}

\paragraph{Task:}We are interested in the sequential picking of stacks of parcels arranged as in Fig~\ref{fig:task+obs}. Our picking agent is an industrial KUKA KR70 robot manipulator equipped with a suction gripper. We consider the problem of learning sequential picking end-effector poses from visual observations $s_t$:
\begin{equation*}
    f(s_t) \to \mathcal{T}^{\text{pick}}_t, \ \ \  t \in (0,T),
\end{equation*}
where $\mathcal{T}^{\text{pick}}_t$ is the pose of the end-effector used to pick an object in SE(3) at time $t$ and $T$ is the maximum number of steps in a episode which corresponds to the total number of parcels in the scene. The placing pose $\mathcal{T}^{\text{place}}_t$ is predefined and corresponding to the same conveyor position for each box. The complexity of the task is given by the sequentiality of the decision process: the robot is required to correctly sequencing 42 picking actions in order to complete the task. See for instance Fig.~\ref{fig:scene} (lower episode), where a single wrong decision compromises the scene and consequently the agent's final outcome and success rate. Finally, note that the methods and algorithms presented in this work can be easily generalized beyond sequential picking to all the tasks consisting of sequences of two-pose motion primitives.

\paragraph{Perception and Control:}The robot's perception of the scene is based on a high resolution 720x1280 RGBD camera placed on the upper-left side of the robot frame. The camera captures the scene as in Fig.~\ref{fig:obs}; the RGB image is then cropped and resized to $160\times160$ pixels which represents the actual dimension of our visual observations space $\mathcal{S}$. \\
Further, we model our action space $\mathcal{A}$ as a discrete grid of $160\times160$ pixels $\{(u,v)\}$. At each timestep $t$, a pixel in the image is selected as in Fig.~\ref{fig:obs+act}, where the red spot in the top-right corresponds to the selected $\{(u,v)\}$. Having a depth image of the scene and knowing intrinsic and extrinsic parameters of the camera \cite{szeliski2010computer} we transform $\{(u,v)\}$ into robot frame xyz-coordinates. The obtained xyz-coordinates determine the picking or target position of the end-effector, while the target orientation is selected by hard-coded policy. More on this is deferred to the Primitives paragraph. Target position and orientation comprise the picking pose $\mathcal{T}^{\text{pick}}_t$. Given $\mathcal{T}^{\text{pick}}_t$, we solve an inverse kinematics optimization problem and recover the target configuration of the robot in joint space. In order to reach this target configuration the robot's joints are controlled in position using a PD controller.

\paragraph{Primitives:}The primitives used by the robot consists of $5$ main waypoints all characterized by a specific end-effector pose. The waypoints are: out-of-camera pose, pre-pick, $\mathcal{T}^{\text{pick}}_t$, post-pick, pre-place, and $\mathcal{T}^{\text{place}}_t$. The robot starts in a out-of-camera pose at $t=0$, it receives an image of the scene from the camera, selects $\mathcal{T}^{\text{pick}}_t$ and then moves following the aforementioned waypoints. After placing the parcel, the robot reaches again the out-of-camera pose, we set $t\leftarrow t+1$ and the transition starts again. The suction gripper is activated at $\mathcal{T}^{\text{pick}}_t$ provided a collision with an object is detected.\\
Furthermore, the task comes with two main modes: one forces the robot to attempt only side-picks which means picking only on the surfaces visible in \ref{fig:obs} and which are never entirely occluded in an ideal scenario. The second mode assumes the robot will attempt side picks for the parcels placed in the upper part of the pile and top-picks below $0.5m$. Where with top-picks we mean picking the top surface of a parcel. As a result, according to the z-coordinate of the selected picking position the primitive and the end-effector orientation in $\mathcal{T}^{\text{pick}}_t$ will change to facilitate top or side picks. Note that, the type of attempted pick will necessarily influence the optimal picking point yielding therefore different optimal policies for the two task modes.

\paragraph{Scene:}We define the scene as the pile of pickable parcels placed in front of the robot. Each parcel is a cube of $0.25m$ edge with randomized color within the brown spectrum. At the beginning of each episode the scene appears as in fig.~\ref{fig:obs}. The robot interacts with scene influencing the future parcels pose during the episode as illustrated in Fig.~\ref{fig:scene}.
\begin{figure}
    \centering
    \includegraphics[width=\linewidth, height=4cm]{Figures/frames.pdf}
    \caption{Evolution of the scene during two different episodes. In the upper episode the agent acts optimally and is able to unload almost all the parcels. On the other hand, the lower episode shows how a single wrong decision, halfway during the episode, irreversibly compromises the scene conditioning the agent's final outcome.}
    \label{fig:scene}
\end{figure}

\section{Methods}
In the following, we formalize the end-to-end learning algorithms we tested on the sequential picking problem of the previous section. We present three main approaches which require different level of supervision, these are: Behavioral Cloning (BC) \cite{osa2018algorithmic}, Deep Q Learning (DQL) \cite{mnih2013playing} and our new algorithm called Unsupervised Reward Shaping using Adversarial Imitation Learning from observations (URSfO).

\paragraph{Behavioral Cloning:}
BC is a form of imitation learning from demonstrations which learns a direct mapping from states to actions. Given a set of demonstrations $\tau_E = \{s_0,a_0,s_1,a_1,\dots,s_T,a_T\}$ generated in a MDP, we define $\pi_{{\mathbf m}{\theta}}: \mathcal{S} \to \Delta_{\mathcal{A}}$, and train $\pi_{{\mathbf m}{\theta}}$ in a supervised fashion in order to clone the desired behavior. This approach treats the correlated state-action pairs generated in the MDP as identical and independent distributed (iid) leading to potential compounding of errors \cite{ross2010efficient}. However, being BC fundamentally a supervised learning algorithm, it turns quite effective when large amount of data are available and when associated with other techniques such as data augmentation and batch normalization.

\paragraph{Deep Q-Learning:}
DQL is a model-free RL algorithm introduced in the seminal work in \cite{mnih2013playing}. Recall the definition of state-action value function $Q^{\pi}(s,a) = \mathbb{E}_{\tau \sim \pi}[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)|S_0=s, A_0=a]$. We define a replay buffer $\mathcal{D}$ such that $(s,a,r,s')\sim\mathcal{D}$ and $Q_{{\mathbf m}{\theta}}: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ and use the following updating rule based on the Bellman equation:
\begin{align}
    V_j &= \mathbb{E}_{(s_j,a_j,r_j,s'_j)\sim\mathcal{D}}[r_j + \gamma \max_{a'}Q_{{\mathbf m}{\theta}}(s_j',a')], \label{eq:value} \\
    {\mathbf m}{\theta}' &\in \arg\max_{{\mathbf m}{\theta}} (V_j - Q_{{\mathbf m}{\theta}}(s_j,a_j))^2. \label{eq:MSE}
\end{align}
In DQL with discrete $\mathcal{A}$, the value update in \eqref{eq:value} uses a greedy target. When the target is susceptible to error $\epsilon$, which is unavoidable when function approximation is used, we have $\mathbb{E}_{\epsilon}[\max_{a'}(Q(s',a')+\epsilon)] \geq \max_{a'}(Q(s',a')$ \cite{thrun1993issues}. This overestimation bias often destabilizes the training in practice. We address this issue by using Double DQL with delayed target network updates as originally proposed in \cite{van2016deep}.\\
Unlike BC, DQL learns the task requiring only a reward function as unique form of supervision. Note that, in our task it is trivial to specify a simple reward function. However, specifying an "informative" reward function which takes into account the uncertainties in the scene, the primitives design and the sequential nature of the decision process is a much harder task.

\paragraph{Unsupervised reward shaping from expert's observations:}
As previously mentioned, BC assumes the existence of an expert which provides "ready-to-use" state-action pairs. On the other hand, DQL only requires designing a reward function. In the following we come up with an algorithm which merges the best of both worlds. We relax the assumption made for BC, requiring only the expert's observations $\tau_E^o=\{s_0, s_1, \dots, s_T\}$ rather than state-action pairs. Furthermore, we formalize an unsupervised reward shaping algorithm called URSfO which exploits $\tau_E^o$ in order to improve our reward function and make it more informative.

Recall the expected total discounted reward as $J(\pi_{{\mathbf m}{\theta}})=\mathbb{E}_{\tau}[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)]$, the normalized discounted state visitation distribution as $d_{\pi_{{\mathbf m}{\theta}}}(s) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\mathbb{P}(s_t=s | d_0,\pi_{{\mathbf m}{\theta}},P)$, the normalized discounted state-action visitation distribution as $\mu_{\pi_{{\mathbf m}{\theta}}}(s,a)= d_{\pi_{{\mathbf m}{\theta}}}(s)\pi_{{\mathbf m}{\theta}}(a|s)$ and the state-transition visitation distribution as $\rho_{\pi_{{\mathbf m}{\theta}}}(s,s')=d_{\pi_{{\mathbf m}{\theta}}}(s)\int_{\mathcal{A}} P(s'|s,\Bar{a})\pi_{{\mathbf m}{\theta}}(\Bar{a}|s) d\Bar{a}$. Given $\mathcal{A}$ as a discrete action space, we rewrite $\rho_{\pi_{{\mathbf m}{\theta}}}(s, s') = d_{\pi_{{\mathbf m}{\theta}}}(s)\sum_{\Bar{a}} P(s'|s,\Bar{a})\pi_{{\mathbf m}{\theta}}(\Bar{a}|s)$. We further define the normalized discounted joint distribution as $\rho_{\pi_{{\mathbf m}{\theta}}}(s, a, s') = \mu_{\pi_{{\mathbf m}{\theta}}}(s,a)P(s'|s,a)$ and the density function of the inverse dynamics model $\rho_{\pi_{{\mathbf m}{\theta}}}(a|s, s')$ as
\begin{equation}
    \rho_{\pi_{{\mathbf m}{\theta}}}(a|s, s') = \frac{P(s'|s,a)\pi_{{\mathbf m}{\theta}}(a|s)}{\sum_{\Bar{a}} P(s'|s,\Bar{a})\pi_{{\mathbf m}{\theta}}(\Bar{a}|s)}.
\end{equation}
In line with prior work \cite{ghasemipour2020divergence}, we cast the learning from expert problem in a divergence minimization problem
\begin{equation*}
    \min_{{\mathbf m}{\theta}} \ \ \ \mathbb{D}(\mu_{\pi_{{\mathbf m}{\theta}}}(s,a), \mu_{\pi_E}(s,a)).
\end{equation*}
We can now formulate our main theoretical result which is instrumental for our algorithm.
\begin{Theorem}
If the agent and the expert share the same transition $P(s'|s,a)$ then the following inequality holds
\begin{equation*}
    \big|J(\pi_{E}) - J(\pi_{{\mathbf m}{\theta}})\big| \leq \frac{\sqrt{2}R_{\max}}{1-\gamma}\Big(\sqrt{\mathbb{D}_{\chi^2}\big(\rho_{\pi_{{\mathbf m}{\theta}}}(a|s, s')||\rho_{\pi_{E}}(a|s, s')\big) + \mathbb{D}_{\chi^2}\big(\rho_{\pi_{{\mathbf m}{\theta}}}(s, s')||\rho_{\pi_{E}}(s, s')\big)}\Big)
\end{equation*}
where $\mathbb{D}_{\chi^2}$ is the Pearson $\chi^2$ divergence.

\begin{proof}
The proof follows from the definition of $J(\pi_{{\mathbf m}{\theta}})$ and exploits a series of results provided in \cite{yang2019imitation} and basic inequalities between the Total Variation distance ($\mathbb{D}_{\text{TV}}$), the Kullback–Leibler divergence ($\mathbb{D}_{\text{KL}}$) and the Pearson $\chi^2$ divergence ($\mathbb{D}_{\chi^2}$). We provide the full derivation in appendix.
\end{proof}
\label{Theo:1}
\end{Theorem}

\begin{remark}
Theorem~\ref{Theo:1} suggests that when only expert's observations are available we cannot fully minimize the suboptimality of $\pi_{{\mathbf m}{\theta}}$ by simply minimizing $\mathbb{D}_{\chi^2}\big(\rho_{\pi_{{\mathbf m}{\theta}}}(s, s')||\rho_{\pi_E}(s, s')\big)$. Note that, when $P(s'|s,\Bar{a})$ is injective with respect to $\Bar{a}$, i.e. $P(s'|s,a) = 0 \ \ \ \forall a \in \mathcal{A}-\{\Bar{a}\}$, then $\mathbb{D}_{\chi^2}\big(\rho_{\pi_{{\mathbf m}{\theta}}}(a|s, s')||\rho_{\pi_E}(a|s, s')\big)=0$ and therefore $\mathbb{D}_{\chi^2}\big(\rho_{\pi_{{\mathbf m}{\theta}}}(s, s')||\rho_{\pi_E}(s, s')\big) = \mathbb{D}_{\chi^2}\big(\mu_{\pi_{{\mathbf m}{\theta}}}(s, a)||\mu_{\pi_E}(s, a)\big)$ \cite{yang2019imitation}. This is not the case in our task since more than one picking point can potentially lead to the same transition.
\end{remark}

As a result, we propose an alternative algorithm which exploits $\tau_E^o=\{s_0, s_1, \dots, s_T\}$ only for reward shaping and not as the main objective. \\
We minimize $\mathbb{D}_{\chi^2}\big(\rho_{\pi_{{\mathbf m}{\theta}}}(s, s')||\rho_{\pi_E}(s, s')\big)$ in Theorem~\ref{Theo:1} using the Least-Square Generative Adversarial Network (LSGAN) optimization scheme \cite{mao2017least} which has been proved to minimize $\mathbb{D}_{\chi^2}$ (see appendix~\ref{app:B}). More precisely, we define a discriminator $D_{{\mathbf m}{\chi}}:\mathcal{S}\times \mathcal{S} \to \mathbb{R}$ and alternatively optimize the two following problems
\begin{align}
    \begin{split}
        \min_{\chi}& \ \ \ \frac{1}{2}\mathbb{E}_{\tau_E^o}\Big[\big(D_{{\mathbf m}{\chi}}(s,s')-1\big)^2\Big] + \frac{1}{2}\mathbb{E}_{(s,s')\sim\rho_{\pi_{{\mathbf m}{\theta}}}}\Big[\big(D_{{\mathbf m}{\chi}}(s,s')+1\big)^2\Big] \\
        &+ \frac{\lambda}{2}\mathbb{E}_{\tau_E^o}\Big[\big|\big|\nabla_{\chi}D_{{\mathbf m}{\chi}}(s,s')\big|\big|^2\Big],
        \label{eq:disc_LSGAN}
    \end{split} \\
    \min_{{\mathbf m}{\theta}}& \ \ \ \frac{1}{2}\mathbb{E}_{(s,s')\sim\rho_{\pi_{{\mathbf m}{\theta}}}}\Big[\big(D_{{\mathbf m}{\chi}}(s,s')-1\big)^2\Big],
    \label{eq:policyLSGAN}
\end{align}
where a discriminator gradient penalty is added for improving training stability \cite{mescheder2018training}.

We can now focus on $\mathbb{D}_{\chi^2}\big(\rho_{\pi_{{\mathbf m}{\theta}}}(a|s, s')||\rho_{\pi_E}(a|s, s')\big)$ in Theorem~\ref{Theo:1} which is intractable due to the unavailability of the expert's actions. We approximate this objective by using the following function:
\begin{equation}
    f(s,a) = \begin{cases}
    1-w \cdot \text{accuracy}, \ \ \ \text{if picking succeeds}, \\
    0, \ \ \ \text{otherwise},
    \end{cases}
    \label{eq:reward_func}
\end{equation}
where $w$ is an hyper-parameter and accuracy is defined as the euclidean distance between the picking point and center of mass of the parcel.
This function can be interpreted as an estimation of the reward function optimized by the expert and which is reasonable in our specific task of interest. Intuitively, \eqref{eq:reward_func} is likely maximized by an expert but it is hard to optimize in classic RL due to lack of feedback about the picking order. As a result, \eqref{eq:reward_func} will benefit from the information added by \eqref{eq:disc_LSGAN}-\eqref{eq:policyLSGAN}.
Overall, the estimated reward function $r_{\text{tot}}(s,a)$ have the following form
\begin{equation}
    r_{\text{tot}}(s,a) = \lambda_1 f(s,a) + \lambda_2 \max[0,1-0.25(D_{{\mathbf m}{\chi}}(s,s')-1)^2],
    \label{eq:reward}
\end{equation}
where we bound the second term to be in the range $[0,1]$ \cite{peng2021amp} and $\lambda_1$, $\lambda_2$ are tunable parameters. Note that, eq.~\ref{eq:reward} follows the additive reward shaping scheme as presented in \cite{ng1999policy}. URSfO is summarized in Algorithm~\ref{alg:URSfO}.

\begin{algorithm}[H]
\label{alg:URSfO}
\caption{URSfO}
Input: $\tau_E^o$, $\pi_{{\mathbf m}{\theta}}$, $D_{{\mathbf m}{\chi}}$, $\lambda$, $\lambda_1$, $\lambda_2$, $w$ \\
\For{$k=1,K$}{
$\tau \sim \pi_{{\mathbf m}{\theta}}$ \\
Optimize \eqref{eq:disc_LSGAN} \\
Use any RL algorithm to maximize $\mathbb{E}_{\tau}[\sum_{t=0}^{\infty}\gamma^t r_{\text{tot}}(s_t,a_t)]$ with $r_{\text{tot}}(s,a)$ in \eqref{eq:reward}
}
\Return $\pi_{{\mathbf m}{\theta}}$
\end{algorithm}

\section{Implementations and Results}
Recall we define $\mathcal{S}$ as a $160\times160$ RGB pixel space and $\mathcal{A}$ as a discrete action space of dimension $160\times160$. All our algorithms will focus on learning the critic network $Q_{{\mathbf m}{\theta}}:\mathcal{S}\times \mathcal{A} \to \mathbb{R}$  which is parameterized as $43$-layer encoder decoder residual network (ResNet) \cite{he2016deep} with $12$ residual blocks. We use three $2$-stride convolutions in the encoder and three bilinear-upsampling layers in the decoder as in \cite{zeng2020transporter}. Each layer, except the last, is interleaved with ReLU activation functions. Given $Q_{{\mathbf m}{\theta}}$, $\pi_{{\mathbf m}{\theta}}(s) \in \arg\max_{\tilde a} Q_{{\mathbf m}{\theta}}(s,\tilde a)$.\\
For the discriminator $D_{\chi}$, we use a 4-layer convolutional network which is then followed by a feed-forward network with a single hidden layer. More details and other important hyperparameters are available in the appendix~\ref{app:hyper}.

\begin{table}
\centering
\caption{Results for BC. We evaluate each policy trained for a different amount of training steps and with access to a different amount of expert's demonstrations. The evaluation occurs for $10$ randomized scenarios previously unseen by the agent and we report the percentage of successful picks over the entire evaluation, i.e. $420$ parcels. Side-picks-only and side-and-top-picks denote the task modes as described in the Environment section.}
\label{tab:table_offline}
\footnotesize
\begin{tabular}{c c c c c c c}\toprule
& \multicolumn{6}{c}{number of demonstrated episodes}\\
\cmidrule(lr){2-7}
& \multicolumn{3}{c}{side picks only} &  \multicolumn{3}{c}{side and top picks}\\
\cmidrule(lr){2-4} \cmidrule(lr){5-7}
training steps & 1 & 10 & 50 & 1 & 10 & 50\\
\bottomrule
100 & $100\%$ & $95\%$ & $100\%$ & $67\%$ & $95\%$ & $99\%$\\
1,000 & $98\%$ & $100\%$ & $100\%$ & $92\%$ & $90\%$ & $86\%$ \\
10,000 & $100\%$ & $97\%$ & $100\%$ & $94\%$ & $99\%$ & $100\%$\\
100,000& $100\%$ & $100\%$ & $99\%$ & $99\%$ & $96\%$ & $99\%$ \\
\bottomrule
\end{tabular}
\label{tab:BC}
\end{table}

\paragraph{Behavioral Cloning:} We start describing our results for BC. We use as training loss the cross-entropy between one-hot representations of the expert's actions and $Q_{{\mathbf m}{\theta}}(s,\cdot)$ with a softmax as output layer. In this context we rewrite it as $f_{{\mathbf m}{\theta}}(s):\mathcal{S}\to\Delta_{\mathcal{A}}$ and interpret it as a function that given a scene provides a distribution over picking points. We therefore minimize the following loss
\begin{equation*}
    \mathcal{L}({\mathbf m}{\theta}) = -\mathbb{E}_{\tau_E}[\log f_{{\mathbf m}{\theta}}(s)].
\end{equation*}
We train for a different amount of training steps and demonstrated episodes and then evaluate for $10$ episodes in unseen scenarios. The results, expressed in percentage of successful picks, are in Table~\ref{tab:BC} for both the task modes. Table~\ref{tab:BC} gives interesting insights about the complexity of the two modes: the side-picks-only is remarkably easier to solve in this setting than side-and-top-picks. This occurs due to a combination of factors such as camera pose, camera resolution, parcels dimension and overall ratio of top versus side picks in a demonstrated episode. More specifically, camera pose, resolution and parcels dimension influence the number of pixels for each box surface. Fewer pixels per surface greatly reduces the margin of picking error, thus increasing the probability of making mistakes. This is particularly relevant for top surfaces which, due to the camera pose, always appear smaller than side surfaces (Fig.~\ref{fig:scene}). At the same time, given we collect $\tau_E$ using a hard-coded policy based on the primitive described in the Modeling section, we have in the expert's data a higher number of demonstrated side picks rather than top picks. It follows that more training steps and/or demonstrations are needed when top picks are required.

\paragraph{Reinforcement Learning:} In the following we present our results for RL. We focus on the side-picks-only mode which has shown to be easily solvable assuming full expert supervision. We test both DQL and DQL+URSfO standardizing the two algorithms over the critic network, replay buffer and exploration strategy. The only source of difference is therefore the reward function: DQL will maximize the simple reward in \eqref{eq:reward_func}, whereas DQL+URSfO will maximize \eqref{eq:reward} with $\lambda_1=1$ and $\lambda_2=\delta$ where $\delta$ is equal to $1$ at $t=0$ and then decays linearly over time during the training process (see Table~\ref{tab:Hyper_1} in appendix~\ref{app:hyper}). The idea behind this choice is to exploit the expert knowledge during the first part of the training and then let slowly the agent on its own to avoid excessive and unnecessary bias towards the expert's behavior. Fig.~\ref{fig:RL} shows the final results. Interestingly, in this specific scenario, maximizing the reward in \eqref{eq:reward_func} as in DQL leads to faster but suboptimal convergence which quickly plateaus. On the other hand, DQL+URSfO with reward in \eqref{eq:reward} and the aforementioned $\lambda_1$, $\lambda_2$ schedule shows slower but constant improvement which eventually outperforms the DQL counterpart of roughly $19\%$ which means an additional $8$ parcels collected per scene.

\begin{figure}
    \centering
    \includegraphics[width=0.7\linewidth]{Figures/RL.pdf}
    \caption{Results for the RL experiment. On the $x$-axis we show the training steps and on the $y$-axis the reward in \eqref{eq:reward_func} which represents an objective metrics to evaluate the agents' picking rate. We train both DQL and DQL+URSfO for 200k steps and evaluate the algorithms every $3$ full scenes, i.e. $126$ steps, for a single randomized scene (see fig.~\ref{fig:obs} for a scene example). DQL shows a faster but less optimal convergence; while DQL+URSfO shows a slower but more monotonic improvement. At the end of the full training, DQL+URSfO can successfully pick $30$ parcels versus the $22$ of DQL, a $19\%$ increase.}
    \label{fig:RL}
\end{figure}

\section{Conclusions}

In this paper, we model a robotic sequential picking task, typical of the logistics sector, and open source a Pybullet \href{https://github.com/VittorioGiammarino/Bootstrapping-RL-4-Sequential-Picking}{simulator} \cite{coumans2016pybullet} for prototyping and testing sequential decision algorithms on this task. Furthermore, we theoretically motivate a reward shaping algorithm based on adversarial IL which requires only visual observations rather than state-action pairs. In the experimental part of the paper, we test three different approaches on our task: IL, DQL and our formulated URSfO together with DQL (URSfO+DQL). As Table~\ref{tab:BC} shows, IL achieves remarkable results robustly hitting $99\%$ industrial standard of collected parcels. On the other hand, DQL and DQL+URSfO are both unable to reach the performance level of IL and remain in the range of $55\%$ to $85\%$ of parcels successfully picked per scene. Our URSfO provides a $19\%$ improvement over the DQL only approach (roughly $8$ parcels per scene) despite resulting in a slower rate of convergence. Future work, will focus on developing and testing more principled tuning method for $\lambda_1$ and $\lambda_2$ in eq.~\ref{eq:reward} in an attempt to further improve both transient and asymptotic results. In addition to this, from an algorithmic point of view, we will search for alternatives to adversarial imitation learning and under the representation learning perspective we will explore alternatives to CNN based ResNets, such as attention based models \cite{dosovitskiy2020image}, for parameterizing $Q_{{\mathbf m}{\theta}}$.

\clearpage
\printbibliography

\clearpage