Under the supervision of the Residential House Manager, the Lead Caregiver for the Developmentally Disabled is the key figure in providing active treatment to persons served while helping to oversee the day-to-day operations and services provided to persons served in a group home setting. Provide active treatment and assistance with programming needs to all persons served in compliance with the State Board of Health regulations. Monitor director care staff to ensure assignment completion. Legibly document the major activities of the people served in daily notes. Administer medications responsibly according to procedures. Accurately document all medication passes in the medication log. Exhibit proactive leadership skills by suggesting ways to improve quality and effectiveness of services provided and leading staff in a fair and positive manner. Provide support to direct care staff during active treatment with persons served. Maintain physical condition of the group home. Provide and ensure a safe environment for the individuals served. Display a team-oriented, cooperative attitude and maintain a positive working relationship with co-workers, supervisory staff and persons served. Participate in IDT (Inter-Disciplinary Team) meetings as requested. Participate in staff meetings. Assist the Residential House Manager with conducting and documenting staff meetings as needed to review systems to ensure the best use of resources and assess quality of care for the persons served. Must be capable of meeting the physical demands required by caring for the persons served. Physical requirements may include, but not be limited to: bending; reaching; stretching; lifting 25-50 pounds (depending on person served) without assistance; grasping; visual acuity; running; walking; squatting, guiding; transporting; assisting with physical transfer; and manual dexterity to handle medications; and handling physical aggression as defined in CPI training. Provide services, as per Medicaid regulations. Must be capable of working scheduled shift. Must be able to accept direction from supervisor and other members of management. Must be able to function independently; follow and understand written and oral instruction and communication; and assess physical and behavioral situations in order to respond appropriately to the needs of the persons served. Must provide personal hygiene assistance to persons served. Remain awake and alert at all times during scheduled hours. Assist the Residential House Manager with scheduling staff and finding coverage for open shifts. Participate in the on-call rotation as assigned. Monitor performance and documentation of staff members. Communicate concerns to the Residential House Manager. Document any personnel issues and forward to the Residential House Manager. Follow all agency and residential division policies and procedures. Display patience in all situations involving persons receiving services, staff members, parents, and others in the community. Speak in an even, positive and personal tone of voice when conversing with persons served. Strictly maintain confidentiality of all people in compliance with agency policies and HIPAA requirements. Attend/request related training as authorized or required. Dress appropriately as defined by policies and procedures. Potential exposure to bodily fluids, pests, contagious pathogens, and/or bed bugs.
As per State regulatory requirements, must be at least 18 years old. Minimum of a High School Diploma or GED required. Must be capable of obtaining and maintaining certification in CPR, First Aid, CPI and Medication Administration. Maintain a valid Indiana driver's license, insurance on personal vehicle (minimum of PLPD with Medical liability coverage), and maintain insurability as defined by the agency liability insurance carrier. Must possess and maintain reliable transportation that meets state requirements for transporting persons served. Maintain a clear criminal record, especially with regard to neglect or abuse. Minimum 1 year experience working with MRDD population preferred.


text:
\section{Introduction}

\begin{wrapfigure}[19]{r}{0.5\textwidth}
    \centering
    \vspace{-1.2em}
    \includegraphics[width=\linewidth]{figs/results/pomdp/barplot_return_td3-gru-64-oa-separate.png}
    \vspace{-2.5em}
    \caption{{\footnotesize \textbf{Implementation Matters for Recurrent Model-Free RL.} This paper identifies critical design decisions for recurrent model-free RL that outperforms not only prior implementations (e.g. PPO-GRU and A2C-GRU from~\citet{kostrikov2018pytorch} in the figure), but also purpose-designed methods (e.g. VRM from~\citet{han2019variational} in the figure).
    }}
    \label{fig:trailer}
\end{wrapfigure}

While reinforcement learning (RL) is often cast as the problem of learning a single fully observable task, also known as MDP, training and testing on that same task, most real-world applications of RL demand some degree of transfer and partial observability. For example, visual navigation~\cite{zhu2017target} requires adaptation to unseen scenes with occlusion in observations, and human-robot collaboration requires that robots infer the intentions of human collaborators.~\cite{chen2018planning}.

Many subareas in RL study problems that are special cases of POMDPs, and we summarize them in Table~\ref{tab:problems}.
For example, meta RL
~\citep{schmidhuber1987evolutionary, thrun2012learning, duan2016rl,wang2016learning}
is a POMDP where certain aspects of the reward function or (less commonly) dynamics function are unobserved but held constant through one episode.
The robust RL problem~\citep{bagnell2001solving, rajeswaran2016epopt, pattanaik2017robust, pinto2017robust}
assumes that certain aspects of the dynamics or reward function are unknown, aiming at finding optimal policies that perform against adversarially-chosen perturbations.
Generalization in RL~\citep{whiteson2011protecting,zhang2018dissection,packer2018assessing,cobbe2019quantifying}
focuses on unobserved aspects of the dynamics or reward function that are novel during testing, using an average-case objective instead of a worst-case objective like robust RL.
Recent work has proposed efficient and performant algorithms for solving these \emph{specialized} problem settings. However, these algorithms often make assumptions that preclude their application to other classes of POMDPs. For example, methods for robust RL are rarely used for the meta RL setting due to objective mismatch; methods for meta RL are rarely used for general POMDPs due to the stationarity assumption in meta RL.

Nonetheless, many prior works have used a simple baseline that is applicable to all POMDPs~\cite{packer2018assessing, igl2018deep, rakelly2019efficient, yu2020meta}: model-free RL equipped with a recurrent policy and (sometimes) value function~\citep{duan2016rl,wang2016learning,fakoor2019meta}.
We will refer to this approach as \emph{recurrent model-free RL}. This baseline is simultaneously simple (requiring changing only a few lines of code from a model-free RL algorithm) and general.
However, prior work has consistently found that recurrent model-free RL performs poorly across a wide range of problem settings, including
meta RL~\citep{rakelly2019efficient,zintgraf2019varibad},
general POMDPs~\cite{igl2018deep,han2019variational},
robust RL~\citep{zhang2021robust},
and generalization in RL~\citep{packer2018assessing}.
One common explanation is that specialized algorithms that are tailored to specific types of POMDPs are very likely to outperform recurrent model-free RL because they (implicitly) encode inductive biases for solving these specific tasks.
For example, algorithms for meta RL may leverage the assumption that the underlying dynamics (while unknown) are fixed, and the underlying goals are fixed within one episode~\citep{rakelly2019efficient, zintgraf2019varibad}; algorithms for robust RL may assume that the dynamics parameters are known~\citep{rajeswaran2016epopt} and dynamics is Lipschitz continuous~\cite{jiang2021monotonic}.

This paper challenges this explanation. We argue that, contrary to popular belief, recurrent model-free RL is competitive with recent state-of-the-art algorithms across a range of different POMDP settings.
Similar to prior work in Markovian on-policy RL methods~\cite{engstrom2019implementation, andrychowicz2020matters}, our experiments show that implementation in recurrent model-free RL matters. Fig.~\ref{fig:trailer} shows a typical scenario in PyBullet occlusion environments~\cite{coumans2016pybullet} to support this argument.
Through extensive experiments, we show that the careful design and implementation of recurrent model-free RL is critical to its performance. Design decisions, such as the actor-critic architecture, conditioning on previous actions and/or rewards, the underlying model-free RL algorithms, and context length in RNNs, are especially crucial.

The main contribution of this paper is a performant implementation of recurrent-model free RL.
We demonstrate that simple yet important design decisions, such as the underlying RL algorithm and the context length, yield a recurrent model-free RL algorithm that performs on par with prior specialized POMDP algorithms \emph{on the environments those algorithms were designed to solve.}
Ablation experiments identify the importance of these design decisions. We also open-sourced our code that is easy to use and memory-efficient.

\section{Background}

\label{sec:prelim}
In this section, we introduce the notation for MDPs and POMDPs.  %

\paragraph{MDP.} A Markov decision process (MDP)~\cite{bellman1957markovian} is a tuple $({\mathcal{S}}, {\mathcal{A}}, T, T_0, R,  H, \gamma) $ , where ${\mathcal{S}}$ is the set of states, ${\mathcal{A}}$ is the set of actions, $T: {\mathcal{S}} \times {\mathcal{A}} \times {\mathcal{S}} \to [0,1]$ is the transition function (dynamics), $T_0:  {\mathcal{S}}  \to [0,1] $ is the initial state distribution, $R: {\mathcal{S}} \times {\mathcal{A}} \times {\mathcal{S}} \to \mathbb R $ is the reward function, $H \in \mathbb N$ is the time horizon, and $\gamma \in [0,1)$ is the discount factor.
Solving a MDP requires learning a memoryless policy $\pi: {\mathcal{S}} \times {\mathcal{A}} \to [0,1] $ that maximizes the expected discounted return: $\pi^* = \argmax_\pi \E{s_t,a_t,r_t\sim T,\pi}{\sum_{t=0}^{H-1} \gamma^t r_{t+1}\mid s_0}$.
For any MDP, there exists an optimal policy that is both memoryless and deterministic~\citep{puterman2014markov}.
MaxEnt RL algorithms~\cite{ziebart2010thesis}, such as the SAC~\cite{haarnoja2018soft}, augment the RL objective by adding an entropy bonus to the reward.

\paragraph{POMDP.}
A partially observable Markov decision process (POMDP)~\cite{aastrom1965optimal} %
is a tuple $(\mathcal S, \mathcal A, \mathcal O, T, T_0, O, O_0, R, H, \gamma)$, where the underlying process is an MDP $({\mathcal{S}}, {\mathcal{A}}, T, T_0, R,  H, \gamma)$. Let ${\mathcal{O}}$ be the set of observations and let $O: {\mathcal{S}} \times {\mathcal{A}} \times {\mathcal{O}} \to [0,1]$ be the emission function.
Let the observable trajectory up to time-step $t$ be  $\tau_{0:t} = (o_0, a_0, o_1, r_1,  \dots, a_{t-1},o_t, r_t)$, the memory-based policy in \textit{the most general} form is defined as $\pi(a_t \mid \tau_{0:t})$, conditioning on the whole history.
At the first time step $t=0$, an initial state $s_0\sim T_0(\cdot)$ and initial observation $o_0\sim O_0(\cdot\mid s_0)$ are sampled.
At any time-step $t \in \{0,\dots, H-1\}$, the policy emits the action $a_t \in {\mathcal{A}}$ to the system, the system updates the state following the dynamics, $s_{t+1} \sim T(\cdot \mid s_{t},a_{t})$, then the next observation is sampled $o_{t+1} \sim O(\cdot\mid s_{t+1}, a_t)$ and the reward is computed as $r_{t+1} = R(s_{t}, a_t, s_{t+1})$.

We refer to the part of the state $s_t$ at current time-step $t$ that can be directly unveiled from \textit{current} observation $o_t$ as the \emph{observable state} $s^o_t$, and the rest part of the state as the \emph{hidden state} $s^h_t$. Specifically, if the hidden state $s^h_t$ does not change w.r.t. the time-step $t$ for one trajectory in a POMDP, we call the hidden state \textit{stationary}. In this scenario, the policy objective can be rewritten as $\pi^* = \argmax_\pi \E{s^h\sim T_0}{\E{s_t,a_t,r_t\sim T, O, O_0, \pi}{\sum_{t=0}^{H-1} \gamma^t r_{t+1}\mid s^h}}$
for the average-case POMDP objective, or $\pi^* = \argmax_\pi \min_{s^h \in \supp(T_0)}\E{s_t,a_t,r_t\sim T, O, O_0, \pi}{\sum_{t=0}^{H-1} \gamma^t r_{t+1}\mid s^h}$ for the worst-case POMDP objective.

\section{Related Work}
\begin{table}[t]
\vspace{-4em}
    \footnotesize
    \centering
    \caption{{\footnotesize \textbf{The summary of selected POMDP subareas.} For each subarea, we list the information of the hidden state $s^h$, including its appearance in dynamics and reward function and its stationarity during one trajectory. We also list the policy input space that are connected with the hidden states, where \texttt{o}, \texttt{a}, \texttt{r}, and \texttt{d} refer to the sequence of observations, actions, rewards, and done signals, respectively. Finally, we list the RL objective in terms of average-case or worst-case, and whether there is domain shift between training and testing environments. We append the check (\cmark) or cross mark (\xmark) with * if it applies to some but not all the work in that subarea. The notation in this table will be covered in Sec.~\ref{sec:prelim}.}}
    \vspace{2mm}
\begingroup
\renewcommand{\arraystretch}{1.2} %
    \begin{tabular}{p{0.14\textwidth}|p{0.14\textwidth}|p{0.12\textwidth}|p{0.1\textwidth}|p{0.1\textwidth}|p{0.1\textwidth}|p{0.1\textwidth}}
    \toprule
      Subarea & $s^h$ in dynamics? &  $s^h$ in reward?  & Is  $s^h$ stationary? & Policy input space   & RL objective & Domain shift? \\
      \midrule
      ``Standard" POMDP & \cmark & \cmark & \xmark & \texttt{oar} & Avg & \xmark \\ \hline
      Meta RL & \xmark* & \cmark & \cmark & \texttt{oard} &  Avg & \xmark \\  \hline
        Robust RL & \cmark* & \xmark* & \cmark* & \texttt{oa} & Worst & \xmark \\  \hline
      Generalization in RL & \cmark* & \xmark* & \cmark* & \texttt{oa} & Avg & \cmark* \\
      \bottomrule
    \end{tabular}
    \vspace{-1em}
\endgroup
    \label{tab:problems}
\end{table}

In this section, we discuss several subareas of RL that explicitly and implicitly solve POMDPs, and algorithms proposed for these specialized settings. Table~\ref{tab:problems} summarizes these subareas.

\paragraph{RL for ``Standard" POMDPs.} We use the term ``standard'' to refer to prior work that explicitly labels the problems studied as POMDPs. Common tasks include scenarios where the states are partially occluded~\cite{heess2015memory}, different states correspond to the same observation (perceptual aliasing~\cite{whitehead1990active}),  random frames are dropped~\cite{hausknecht2015deep}, observations use egocentric images~\cite{zhu2017target}, or  the observations are perturbed with random noise~\cite{meng2021memory}.
These POMDPs often have hidden states that are non-stationary and affect both the rewards and the dynamics.
POMDPs are hard to solve~\cite{papadimitriou1987complexity, littman1996algorithms}
because of the curse of dimensionality: the size of the history grows linearly with the horizon length.
Many prior POMDP algorithms~\cite{sondik1978optimal, lovejoy1991survey, cassandra1994acting,littman1995learning,kaelbling1998planning,pineau2003point, roy2005finding, spaan2005perseus, smith2012heuristic,baxter2000reinforcement,Ng2000PEGASUSAP}
attempt to infer the state from the past sequence of observations, and then apply standard RL techniques to that inferred state.
However, the exact inference requires the knowledge of the dynamics, emission, and reward functions, and is intractable except the most simple settings.
A common strategy for solving these general POMDPs is to use recurrent policies, which take the entire history of past observations as inputs~\cite{schmidhuber1991reinforcement,bakker2001reinforcement,wierstra2007solving}.
This strategy is very simple and general, and can be applied to arbitrary tasks without knowledge of the task structure (e.g., whether the hidden states change within an episode) across long time horizons~\cite{duan2016rl}.
These recurrent strategies can be further subdivided into model-free methods~\cite{heess2015memory, hausknecht2015deep,mirowski2016learning,meng2021memory}
, where the single objective is to maximize the return, and model-based methods~\cite{watter2015embed,ha2018world,igl2018deep,han2019variational,Zhang2018SOLARDS, espeholt2018impala, lee2019stochastic, Gregor2019ShapingBS, Hafner2019LearningLD,Akuzawa2021EstimatingDB}
that have explicit objectives on modeling the belief states and use them as the inputs of memoryless policies.
The recurrent model-free RL that we focus on belongs to the class of model-free off-policy memory-based algorithms.

\paragraph{Meta RL.} Meta RL, also called ``learning to learn"~\cite{schmidhuber1987evolutionary, thrun2012learning}, focuses on POMDPs where some parameters in the rewards or (less commonly) dynamics are varied from episode to episode, but remain fixed within a single episode, which represent different \emph{tasks} with different values~\cite{humplik2019meta,Akuzawa2021EstimatingDB, chen2016pomdp}.
The meta-RL setting is almost the same as multi-task RL~\cite{wilson2007multi, yu2020meta}, but differs in that multi-task RL can observe the task parameters, making it an MDP instead of a POMDP.
Algorithms for meta RL can be roughly categorized based on how the adaptation step is performed. Gradient-based algorithms~\cite{hochreiter2001learning,finn2017model, fakoor2019meta,nichol2018first, rothfuss2018promp}
run a few gradient steps on the pre-trained models to adapt.
Memory or context-based algorithms use RNNs to implicitly adapt, which can be further subdivided into implicit and explicit task inference methods.
Implicit task inference methods~\cite{wang2016learning,duan2016rl} use RL objective only to learn recurrent policies.
Explicit task inference methods \cite{zintgraf2019varibad,rakelly2019efficient} train an extra inference model to explicitly estimate task embeddings (i.e., a representation of the unobserved parameters) by variational inference. The task embeddings are then used as additional inputs to memoryless policies.

\paragraph{Robust RL.}
The goal of robust RL is to find a policy that maximizes returns in the worst-case environments. Early work in the control and operations research community~\cite{khalil1996robust,nilim2005robust,iyengar2005robust, lim2013reinforcement}
and RL community~\cite{bagnell2001solving, morimoto2005robust, mannor2012lightning, tamar2014scaling, garcia2015comprehensive}
focused on linear or finite systems. %
Prior work designs deep RL algorithms that are robust against a variety of adversarial attacks, including attacks on the dynamics~\cite{rajeswaran2016epopt, derman2018soft, mankowitz2019robust, jiang2021monotonic}, %
observations~\cite{pattanaik2017robust, huang2017adversarial,lin2017tactics, behzadan2017vulnerability, russo2019optimal, zhang2020robust, zhang2021robust}, %
and actions~\cite{pinto2017robust, gleave2019adversarial, tessler2019action,tan2020robustifying}. %
Treating the robust RL problem as a POMDP, rather than an MDP (as done in most prior work), unlocks a key capability for RL agents, because agents can use their memory to identify the hidden states of the current adversarial environment, although previous work~\cite{rajeswaran2016epopt, jiang2021monotonic} only train Markovian policies on POMDPs.
While some work find memory-based policies are more robust to the adverarial attacks than Markovain policies~\cite{russo2019optimal, zhang2021robust}, they train these baselines in a single MDP without adversaries, which differs from our training setting where the recurrent model-free RL can have access to a set of MDPs. %

\paragraph{Generalization in RL.}
The goal of generalization in RL is to make RL algorithms perform well in test domains that are unseen during training, which emphasizes the average case on the novel test domains instead of the worse case in the possibly seen test domains as in robust RL.
Prior work have studied generalization to
initial states in the same MDP~\cite{whiteson2011protecting, rajeswaran2017towards, zhang2018study}, %
random disturbance in dynamics~\cite{rajeswaran2017towards}, states~\cite{stulp2011learning}, %
observations~\cite{zhang2018dissection, song2019observational}, %
and actions~\cite{srouji2018structured}, %
and different modes in procedurally generated games~\cite{justesen2018illuminating, farebrother2018generalization, cobbe2019quantifying,nichol2018gotta, juliani2019obstacle, cobbe2020leveraging}. %
Among them, \citet{packer2018assessing} provides a benchmark on both in-distribution (ID) and out-of-distribution (OOD) generalization to different dynamics parameters, and \citet{zhao2019investigating} extends the benchmark by introducing random noise in states, observations, and actions. %
Algorithms for improving generalization in RL can be roughly divided into
classic regularization methods such as weight decay, dropout, batch normalization, and entropy regularization~\cite{farebrother2018generalization, cobbe2020leveraging, igl2019generalization}, %
model architectures~\cite{srouji2018structured,raileanu2021decoupling},
data augmentation through randomization~\cite{tobin2017domain, lee2019network,laskin2020reinforcement, raileanu2020automatic, wang2020improving}, %
Although introducing observational noise and the change in dynamics parameters will transform MDPs to POMDPs, few work study memory-based policies such as model-free recurrent RL with mixed results. Same algorithm RL2~\cite{duan2016rl} was found to perform badly in~\citet{packer2018assessing} but relatively well in~\citet{yu2020meta}.

\section{Design Considerations for Recurrent Model-Free RL}
\label{sec:method}

Implementing a recurrent model-free RL algorithm requires making a number of design decisions. This section describes the decisions that we found most important to make recurrent model-free RL competitive with more complex, recent algorithms.
We will focus on continuous control problems with state-based inputs (i.e., not image-based inputs).
Importantly, we assume that the policy can observe the \textit{reward} and \textit{done} signals (the end of one episode during one trial~\cite{duan2016rl}) from the environment during \textit{evaluation}.
This assumption is common in prior work~\citep{zintgraf2019varibad, han2019variational}, but many recurrent model-free implementations do not provide the agent with information.
In the following paragraphs, we will describe the important decision factors in recurrent model-free RL.
Table~\ref{tab:baselines} summarizes how prior work and our method makes these design decisions when implementing recurrent model-free RL.

\begin{table}[t]
\vspace{-4em}
    \footnotesize
    \centering
    \setlength{\tabcolsep}{3pt}
    \caption{{\footnotesize \textbf{How the prior work and our method implement the recurrent model-free RL as their own method or baseline.}
    We can see that none of the prior work share the same set of decision variables, some of which have bad choices that may lead to the poor performance reported in the prior work.
    Our method covers a wide range of choices in these decision factors and finds the combinations shown in the last rows that lead to \textbf{the best performance in terms of the average performance} across the experimented environments in each subarea.}}
    \vspace{2mm}
    \begin{tabular}{cc|ccccc}
    \toprule
     Algorithm    &  Domain & Arch & Encoder & Inputs & Len & RL \\
     \midrule
     \citet{duan2016rl} & Meta RL  & separate & GRU & \texttt{oard} & 1000 & TRPO, PPO \\
     \citet{wang2016learning} & Meta RL & shared & LSTM & \texttt{oart} & 5-150 & A2C \\
     Baseline from~\citet{rakelly2019efficient} & Meta RL & separate & GRU & \texttt{oard} & 100 & PPO \\
     Baseline from~\citet{zintgraf2019varibad} & Meta RL &  separate & GRU & \texttt{oar} & Max & A2C \\
     Baseline from~\citet{fakoor2019meta} & Meta RL &  separate & GRU & \texttt{oar} & 10-25 & TD3 \\
     Baseline from~\citet{yu2020meta} & Meta RL & separate & GRU & \texttt{oard} & 500 & PPO \\
     \citet{kostrikov2018pytorch} & POMDP & shared & GRU & \texttt{o} & 5-2048 & PPO, A2C \\
     \citet{dong2020deep} & POMDP & separate & LSTM & \texttt{oa} & 150 & TD3, SAC \\
     \citet{meng2021memory} & POMDP & separate & LSTM & \texttt{oa} & 1-5 & TD3 \\
     Baseline from~\citet{igl2018deep} & POMDP & shared & GRU & \texttt{oa} & ~25 & A2C \\
     Baseline from~\citet{han2019variational} & POMDP & shared & LSTM & \texttt{o} & 64 & SAC \\
     Baseline from~\citet{zhang2021robust} & Robust RL & separate & LSTM & \texttt{o} & 100 & PPO \\
     Baseline1 from~\citet{packer2018assessing} & Generalization & shared & LSTM & \texttt{o} & 128-512  & PPO, A2C \\ %
     Baseline2 from~\citet{packer2018assessing} & Generalization & separate & LSTM & \texttt{oard} & 128-512   & PPO, A2C \\ %
     \midrule
     Our method & Meta RL & separate & LSTM & \texttt{oar} & 64 & TD3 \\
     Our method & POMDP & separate & GRU & \texttt{oa} &  64 & TD3 \\
     Our method & Robust RL & separate & LSTM & \texttt{o} & 64  & TD3 \\
     Our method & Generalization & separate & LSTM & \texttt{o} & 64 & TD3 \\
     \bottomrule
    \end{tabular}
\vspace{-1em}
    \label{tab:baselines}
\end{table}

\paragraph{Recurrent Actor-Critic Architecture.}
The first important design decision is whether the recurrent policy (actor) and the recurrent Q-value function (critic) use \textit{shared} RNN encoder (and embedders) or use \textit{separate} ones.
In the experiment section (Sec.~\ref{sec:ablation}) we will show that a shared encoder would cause a large gradient norm in the recurrent actor-critic and thus hinder learning, while separate encoders can significantly mitigate this issue and learn efficiently. This echoes prior work~\citep{fakoor2019meta, dong2020deep, meng2021memory} that also use separate encoders in their recurrent actor-critic.
To avoid running an inordinate number of experiments, we will use the \textit{separate} architecture in the rest of the paper.

\paragraph{Policy Input Space.} The next consideration is the input space of the model-free policy. The maximal input space of policy to emit an action $a_t$ at time $t$, should be the history of all quantities that the policy has observed, namely the past observations $o_{0:t}$, the past actions $a_{0:t-1}$, the past rewards $r_{0:t}$, and the past done signals $d_{0:t}$, which was already employed in the early work~\cite{duan2016rl}.
Generally, the input space of optimal policy should only depend on the quantities that have connections with \textit{hidden states} (defined in Sec.~\ref{sec:prelim})~\cite{poupart2003value,izadi2005using}.
We show the policy input spaces that are connected with the hidden states for the discussed subareas in the ``Inputs" column of Table~\ref{tab:problems}.
While prior work often only conditions the recurrent RL baseline on previous observations (and actions)~\cite{igl2018deep, kostrikov2018pytorch, han2019variational,  dong2020deep, meng2021memory}, our experiments in Sec.~\ref{sec:ablation} find that additionally conditioning on other previous information, such as previous rewards, can increase reward by up to $30\%$.

\paragraph{Model-free RL Algorithms.}
Recurrent model-free RL can be understood as applying an off-the-shelf model-free RL algorithm with an actor and a Q function parametrized to take sequences of inputs. As such, the choice of the underlying model-free RL algorithm is paramount.
Most prior work on continuous control POMDP problems used \textit{on-policy} algorithms, such as A2C~\cite{mnih2016asynchronous}, TRPO~\cite{schulman2015trust} or PPO~\cite{schulman2017proximal}.
While \textit{off-policy} algorithms such as TD3~\cite{fujimoto2018addressing} and SAC~\cite{haarnoja2018soft,haarnoja2018soft2} greatly improve the performance in continuous control MDP problems in terms of sample efficiency and asymptotic performance, these methods are rarely used in recurrent model-free RL baselines~\citep{rakelly2019efficient, zintgraf2019varibad,zhang2020robust}.
In the experiment section (Sec.~\ref{sec:compare-best}), we will show that using these off-policy algorithms for recurrent model-free RL provides results that are better than using on-policy algorithms and are comparable to their specialized methods in POMDP. This echoes the finding that model-free off-policy TD3-Context~\cite{fakoor2019meta} can be better than the specialized method PEARL~\cite{rakelly2019efficient} in meta RL.

\paragraph{RNN Variants and Context Length.}
RNN training is known to be unstable, especially with long sequences input~\cite{bengio1994learning}. The RNN variants like LSTM~\cite{hochreiter1997long} and GRU~\cite{chung2014empirical} mitigate the training issues, but still may fail to learn long-term dependencies~\cite{trinh2018learning}. In POMDP problems, these dependencies reflect the memory that an agent must have to solve a task. For example, a POMDP that hides velocities from observations theoretically requires a short memory length to infer velocities through consecutive positions~\cite{meng2021memory}.
Prior work in POMDPs choose a variety set of context lengths for RNNs from 1 to 2048 (see the ``Len" column of Table~\ref{tab:baselines}), and we select three representatives of short (5), medium (64), and long length (larger than 100) in the experiments (Sec.~\ref{sec:experiments}) for comparison. We also try both LSTM and GRU as RNN variants to compare their performance. We find that the optimal context length and RNN variant are task-specific (see Sec.~\ref{sec:ablation}).

\section{Experiments}
\label{sec:experiments}

Our experiments aim to answer two questions. First, how does a \emph{well-tuned} implementation of recurrent model-free RL compare to specialized POMDP methods, such as purpose-designed meta RL and robust RL algorithms?
To give these prior methods the strongest possible footing, we will compare prior methods on the specific problem types for which they were developed (i.e., meta RL algorithms were tested on meta RL tasks).
Our second question studies which design decisions are essential for recurrent model-free RL.
Due to the space limit, we put the environment details in Appendix~\ref{sec:env}.

\paragraph{Code Implementation.} We release a modular and highly-configurable implementation of recurrent (off-policy) model-free RL\footnote{Code is available: {\url{https://github.com/twni2016/pomdp-baselines}}}.
Our implementation is efficient in terms of computer memory compared to previous off-policy RL methods for POMDPs (200x less RAM than \citet{han2019variational} and 9x less GPU memory than \citet{dorfman2020offline}).
Please see the appendix~\ref{sec:code} for details, including an explanation of why our implementation is more memory-efficient than prior work.

\subsection{Recurrent Model-Free RL is Comparable with Prior Specialized Methods}
\label{sec:compare-best}

While prior work has studied a range of different POMDP settings (e.g., meta RL, occluded observations), recurrent model-free RL is a ubiquitous baseline~\citep{rakelly2019efficient,zintgraf2019varibad,humplik2019meta,igl2018deep,han2019variational}. However, prior work consistently reports that this baseline is reported to be unperformed to more specialized methods.
This section casts doubt on that claim, showing that a well-tuned implementation of recurrent model-free RL can perform \textit{at least} as well as more specialized methods.

We study four subareas of POMDPs: the ``standard" POMDP, meta RL, robust RL, and generalization in RL.
We tune a wide range of decision factors shown in Sec.~\ref{sec:method} in our implemented recurrent model-free RL. Appendix~\ref{sec:setting} shows the details of the tuning options.
For each subarea, we show the performance of \textbf{a single variant} that works best across the environments in that subarea, compared with the prior specialized methods in this subsection.
In other words, the following bar charts of each subarea reports \textbf{the same model-free recurrent RL algorithm} with the same hyperparameters. The exact configurations of each subarea can be found in the last four rows of Table~\ref{tab:baselines}.
Under this restricted setting, we find that our implementation can actually outperform prior (specialized) methods by a wide margin across the four subareas.
We show the full learning curves that generate the following the bar charts in Appendix~\ref{sec:learning_curves} due to space limit. The evaluation details on how to generate the bar charts can be seen in Appendix~\ref{sec:eval_details}.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\linewidth]{figs/results/pomdp/barplot_return_td3-gru-64-oa-separate_VRM-max_x500000.png}
    \vspace{-1em}
    \caption{{\footnotesize Final returns of our implemented recurrent model-free RL algorithm with the same hyperparameters, and the prior method VRM~\cite{han2019variational} across the eight environments in \textbf{``standard" POMDPs}, each of which trained in 0.5M simulation steps.}}
    \label{fig:pomdp_bar}
\end{figure}

\paragraph{``Standard" POMDP.} Our first experiments look at the ``standard" POMDPs that typically occlude some part of states in the environment. We will compare against \textbf{VRM}~\cite{han2019variational}, a recent state-of-the-art model-based POMDP algorithm.
We directly apply the environment design of VRM paper that occludes either positions\&angles or velocities of the simulated robot in PyBullet~\cite{coumans2016pybullet}.
There are 8 environments $\{$Hopper, Ant, Walker, Cheetah$\}$-$\{$P,V$\}$, where ``-P" stands for observing positions\&angles only, and ``-V" stands for observing velocities only.
Fig.~\ref{fig:pomdp_bar} shows that the best single variant of our model-free recurrent RL implementation outperform VRM in 7 out of 8 environments, especially in $\{$Cheetah,Hopper$\}$-$\{$P,V$\}$.
Our results suggest that, while the variational dynamics model used by VRM may be useful for some tasks, a simple recurrent model-free RL baseline can outperform VRM if properly tuned.
While we are primarily interested in sample complexity, but not compute, it is worth noting that our recurrent model-free RL implementation is substantially more efficient than the open-source VRM implementation, training $5\times$ faster and requiring at most $200\times$ less RAM usage (see Appendix~\ref{sec:code}).
Note that both Fig.~\ref{fig:trailer} and Fig.~\ref{fig:pomdp_bar} shows the final performance of the same single variant of our implementation, but the former shows our results with 1.5M simulation steps while the latter shows our results with 0.5M simulation steps to match with those of VRM due to the time budget.

\begin{wrapfigure}[18]{r}{0.4\textwidth}
    \centering
    \vspace{-1.2em}
    \includegraphics[width=\linewidth]{figs/results/meta/barplot_return_td3-lstm-64-separate.png}
    \vspace{-2.5em}
    \caption{{\footnotesize Final returns of our implemented recurrent model-free RL algorithm with the same hyperparameters, and the prior method VariBAD~\cite{zintgraf2019varibad,dorfman2020offline} across the three environments in \textbf{meta RL}. Our method and VariBAD are trained with the same number of simulation steps in each environment.}}
    \label{fig:metarl_bar}
\end{wrapfigure}

\paragraph{Meta RL.} We next compare the recurrent model-free RL to the meta RL setting, where some indicator of the task is unobserved. We compare our implementation of recurrent model-free RL to a specialized state-of-the-art method, \textbf{VariBAD}~\cite{zintgraf2019varibad} that explicitly learns the task embeddings by variational model-based objectives. We make VariBAD even stronger by replacing RL algorithm PPO with SAC as~\citet{dorfman2020offline}.
We adopt the three environments used in~\citet{dorfman2020offline} for experiments, including Semi-Circle and Cheetah-Vel, and we also adapt Wind to make it harder to solve.
Figure~\ref{fig:metarl_bar} shows that our best single variant outperforms VariBAD in the three meta RL environments, especially in Cheetah-Vel.
Prior work~\cite{rakelly2019efficient, zintgraf2019varibad} show that disentangling task inference and control can stabilize training. However, our experiments suggest that joint training of task inference and control could also have comparable performance if well implemented.
Additionally, because our method is trained end-to-end, without using pre-trained task representations saved in the replay buffer like the off-policy version of VariBAD~\cite{dorfman2020offline}, our implementation does not have the non-stationarity issue in task representations.

\begin{figure}
    \centering
    \includegraphics[width=0.38\linewidth]{figs/results/rmdp/barplot_return_avg_td3-lstm-64-o-separate.png}
    \includegraphics[width=0.38\linewidth]{figs/results/rmdp/barplot_return_worst_td3-lstm-64-o-separate.png}
    \caption{{\footnotesize Final \textbf{average returns} (left figure) and \textbf{worst returns} (right figure) of our implemented recurrent model-free RL algorithm with same hyperparameters, and prior method MRPO~\cite{jiang2021monotonic} across the three environments in \textbf{robust RL}.} Our method is trained with 3M simulation steps while MRPO is trained with over 15M steps.}
    \label{fig:rmdp_bar}
\end{figure}

\paragraph{Robust RL.} Thirdly, we focus on the robust RL that aims to maximize the worst returns over the tasks. We choose the recent specialized algorithm  \textbf{MRPO}~\cite{jiang2021monotonic} as the compared method, and adopt their used environments based on SunBlaze benchmark~\cite{packer2018assessing}.
These environments have hidden states that are fixed during one episode, including the density and the friction coefficients of the simulated robots, namely $\{$Cheetah, Hopper, Walker$\}$-Robust.
Fig.~\ref{fig:rmdp_bar} shows both the average return and worst return of our single best variant and MRPO on the three environments, where the worst return is measured by the average return in the worst $10\%$ testing tasks following the practice in ~\citet{jiang2021monotonic}.
The results are quite surprising: although our method, using average-case RL objective, is not expected to surpass MRPO in worst return, we found that our best variant vastly outperforms the specialized MRPO in both average return and worst return, with over $80\%$ fewer simulation steps.
Our method benefits from its memory and off-policy algorithms, while MRPO might suffer from its Markovian on-policy algorithm and a bit ideal Lipschitz assumption in dynamics.
Nevertheless, our method is around 17.5x slower than MRPO given the same simulation steps (see Appendix~\ref{sec:code}), so we only run it with 3M steps with a limited time budget.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.38\linewidth]{figs/results/generalize/barplot_Interpolation_td3-lstm-64-o-separate_EPOpt-PPO-FF.png}
    \includegraphics[width=0.38\linewidth]{figs/results/generalize/barplot_Extrapolation_td3-lstm-64-o-separate_EPOpt-PPO-FF.png}
    \vspace{-1em}
    \caption{{\footnotesize Final \textbf{success rates in interpolation setting} (left figure) and \textbf{success rates in extrapolation setting} (right figure) of our implemented recurrent model-free RL algorithm with same hyperparameters, and prior method EPOpt-PPO-FF~\cite{rajeswaran2016epopt} across the two environments in \textbf{generalization in RL}.} Our method is trained with 3M simulation steps while EPOpt-PPO-FF is trained to convergence.}
    \label{fig:genrl_bar}
\end{figure}

\paragraph{Generalization in RL.} Finally, we focus on the SunBlaze benchmark from ~\citet{packer2018assessing} for investigating generalization in RL, including $\{$Hopper, Cheetah$\}$-Generalize. We pick the best specialized method in the tables of final performance in~\citet{packer2018assessing}, Markovian on-policy robust RL method \textbf{EPOpt-PPO-FF}~\cite{rajeswaran2016epopt}.
Fig.~\ref{fig:genrl_bar} show the interpolation and extrapolation success rates, where in the interpolation the testing tasks have the same distribution of hidden states as that of the training tasks, while in the extrapolation the testing distribution is disjoint from that of training.
We can see that our model-free method is on par with the EPOpt-PPO-FF in the interpolation benchmark, while EPOpt-PPO-FF requires the access to the dynamics parameters but ours not.
In the extrapolation benchmark, our method greatly outperforms the previous method, although our objective does not consider extrapolation.

Overall, we can see that with careful tuning on recurrent model-free RL, it can at least perform as well as the specialized or more complicated methods, in various kinds of POMDPs.

\subsection{What Matters in Recurrent Model-Free RL Algorithms?}
\label{sec:ablation}
\label{sec:compare-all}

In the previous subsection, we showed that recurrent model-free RL can perform on par with the specialized (state-of-the-art) methods, then a natural question comes: Why our implementation of recurrent model-free RL outperforms the implementation used in prior work? %

Our analysis will focus on ablating the five important design decisions introduced in Sec.~\ref{sec:method}: the actor-critic architecture (\texttt{Arch}), the policy input space (\texttt{Inputs}), the underlying model-free RL algorithm (\texttt{RL}), the RNN encoder (\texttt{Encoder}), and the RNN context length (\texttt{Len}).
See Table.~\ref{tab:baselines} for a summary of how prior work made these design decisions.
Due to the space limit, we show the ablation results in some but not all the environments to compare the performance between the best single variant and the other variant that only differs in one decision factor.
We also provide ``single factor analysis" plots for each decision factor by averaging the performance over the other factors in Appendix~\ref{sec:single_factor}.

\begin{wrapfigure}[14]{R}{0.55\textwidth}
\centering
\vspace{-1.2em}
    \includegraphics[width=0.55\textwidth]{figs/results/meta/PointRobotSparse-v0/diagnosis.png}
    \vspace{-1em}
    \caption{\footnotesize{\textbf{Comparison between \textit{shared} and \textit{separate} recurrent actor-critic architecture} with all the other hyperparameters same, on Semi-Circle, a toy meta RL environment. We show the performance metric (left) and also gradient norm of the RNN encoder(s) (right, in \textbf{log-scale}).
    For the separate architecture, \texttt{:critic} and \texttt{:actor} refer to the separate RNN in critic and actor networks, respectively.}}
    \label{fig:separate_vs_shared_main}
\end{wrapfigure}

\paragraph{Recurrent Actor-Critic Architecture.} First, we ran some experiments with both shared and separate architectures on two toy POMDP environments.
Fig.~\ref{fig:separate_vs_shared_main} show the results in one of them (see Appendix~\ref{sec:results_arch} for the other). We can see that the shared architecture failed to learn, compared to the separate architecture.
The large RNN gradient norm in the shared architecture suggests that the actor and critic losses may cause conflicts in the shared RNN training.
Our results echo prior work~\citep{fakoor2019meta,meng2021memory} that only consider \textit{separate} RNN encoders that can achieve high asymptotic rewards, and also echo that ~\cite{han2019variational} shows poor results in the \textit{shared} architecture of SAC-LSTM.

\begin{table}[t]
    \centering
    \footnotesize
    \vspace{-4em}
\begingroup
\renewcommand{\arraystretch}{1.2} %
    \begin{tabular}{c|cccc}
    \toprule
     Change in one decision factor &  Subarea & Env   & Performance comparison \\
     \midrule
     \texttt{Inputs}: \texttt{oa} $\to$ \texttt{oar} & ``Standard" POMDP & Walker-P & 981.6 $\to$ 1345.0 ({\color{green}$1.3\times$})\\ %
     \texttt{RL}: TD3 $\to$ SAC & ``Standard" POMDP & Ant-P & 310.7 $\to$ 2123.5  ({\color{green}$6.8\times$}) \\ %
     \texttt{Encoder}: LSTM $\to$ GRU & Robust RL & Walker-Robust & 765.9   $\to$ 931.3 ({\color{green}$1.2\times$}) \\ %
     \texttt{Len}: 64 $\to$ 400 & Meta RL & Cheetah-Vel & -85.2  $\to$ -74.6  ({\color{green}+14\%}) \\  %
     \texttt{Len}: 64 $\to$ 5  & Generalization & Hopper-Generalize & 0.292  $\to$ 0.415 ({\color{green}$1.4\times$}) \\ %
     \texttt{Len}: 64 $\to$ 5  & ``Standard" POMDP & Walker-V & 121.4 $\to$ 264.3 ({\color{green}$2.2\times$}) \\ %
     \bottomrule
    \end{tabular}
\endgroup
    \caption{\footnotesize{\textbf{Ablation results in our implementation of recurrent model-free RL.} In this table, we show how a single change in one decision factor from the variant that is best on average in that subarea,  could significantly increase the performance. The first column shows how we change the single decision factor, and the last column shows the performance comparison between \textbf{the best variant in that subarea} (left) and \textbf{the ablated one} (right). For robust RL and generalization in RL, we show the performance metric in worst returns and extrapolation success rates, respectively.}}
    \vspace{-1em}
    \label{tab:ablation}
\end{table}

\paragraph{Policy Input Space.} The 1st row of Table~\ref{tab:ablation} shows the effect of policy input space in a POMDP environment Walker-P. The reward signals could help reveal the missing information of the velocity of the robot base, which is occluded in Walker-P. Therefore, it is reasonable that adding previous rewards into policy inputs can increase performance.

\paragraph{Model-free RL Algorithms.} The 2nd row of Table~\ref{tab:ablation} shows the effect of RL algorithm in a POMDP environment Ant-P. We can see that SAC is significantly better than TD3 (increase by 6.8$\times$, surpassing the PPO-GRU~\cite{kostrikov2018pytorch} in Fig.~\ref{fig:trailer}), possibly due to strategic exploration where the action noise conditions on the history instead of being independent. This is prominent mainly in Ant-P because Ant-P might be much more challenging than the other POMDP environments.

\paragraph{RNN Variants and Context Length.} The 3rd row of Table~\ref{tab:ablation} shows the effect of RNN encoder in a robust RL environment. We can see that replacing LSTM with GRU can increase the worst-case metric in Walker-Robust.
The remaining rows of Table~\ref{tab:ablation} show the mixed effects of context length in RNNs. As we can see, both increasing and decreasing the context length can boost the performance in different environments. Specifically, decreasing the length from 64 to 5 makes our method surpass VRM in Walker-V (increase by 2.2$\times$).
This might explain why the prior methods adopt a wide range of context lengths from 1 to 2048 (see Table~\ref{tab:baselines}). Therefore, the choice of context length is rather problem-specific that requires some extent of tuning.

\paragraph{Summary.}
We now summarize the main findings of our experiments:
\begin{enumerate}[itemsep=0pt, left=1em]
    \item Using separate weights for the recurrent actor and recurrent critic boosts performance, likely because it avoids gradient explosion (Fig.~\ref{fig:separate_vs_shared_main}).
    \item Using state-of-the-art off-policy RL algorithms as the backbone in recurrent model-free RL can improve asymptotic performance (Fig.~\ref{fig:trailer}).
    \item The context length for the recurrent actor and critic has a large influence on task performance, but the optimal length is task-specific. Reasonable values are 5 to 500, and 64 is a good start (Table~\ref{tab:ablation}, rows 4--6).
    \item It is important that the inputs to the recurrent actor and critic, such as past observations and past returns, contain enough information to infer the POMDP hidden states (Table~\ref{tab:ablation}, row 1).
\end{enumerate}
We believe that these findings may provide a useful initialization for researchers studying recurrent model-free RL.

\section{Conclusion}

In this paper, we show that a carefully-designed implementation of recurrent model-free RL can perform well across a range of POMDP domains, often on par with (if not significantly better than) prior methods that are specifically designed for specific types of POMDPs. Our ablation experiments demonstrate the importance of key design decisions, such as the underlying RL algorithm and RNN context length.
While the best choices for some decisions (such as using separate RNNs for the actor and the critic) are consistent across domains,
the best choices for other decisions (such as RNN context length) are problem-dependent.
We encourage future work to study automated mechanisms for selecting these crucial design decisions.
In releasing our code, we hope to aid future research into the design of stronger POMDP algorithms.

\subsection*{Acknowledgement}
The authors thank Pierre-Luc Bacon, Murtaza Dalal, Paul Pu Liang, Sergey Levine,  Evgenii Nikishin, and Maxime Wabartha for their feedback on the draft of this paper.

This material is supported by the Fannie and John Hertz Foundation and the NSF GRFP (DGE1745016).

\section{Code-level Details}
\label{sec:code}
In this section, we first introduce the outline of code design, especially the replay buffer for sequences, and then compare the system usage, including computing speed, RAM, and GPU memory with previous POMDP methods.

\subsection{Code Design}

\paragraph{Easy to use.} Our code can be either used an API to call the recurrent model-free RL class or a framework to tune the details in the class.
The recurrent model-free RL class takes the hyperparameters of RNN encoder type, shared or separate actor-critic architecture, and whether include previous observations, and/or actions, and/or rewards into the inputs, to generate different instances. The details of the hyperparameter tuning set is shown in Sec.~\ref{sec:setting}.

\paragraph{Memory-efficient replay buffer for sequences.} Moreover, we design an efficient replay buffer for off-policy RL methods to cope with sequential inputs. Previous methods~\cite{zintgraf2019varibad,dorfman2020offline,han2019variational} mainly use \textit{three-dimensional} replay buffer to store sequential inputs, with the dimensions of \texttt{(num episodes, max episode length, observation dimension)}, taking observation storage as example.
This kind of implementation becomes memory-inefficient if the actual episode length is far smaller than the max episode length~\cite{han2019variational}, which is common in locomotion tasks~\cite{coumans2016pybullet, schulman2017proximal}. Otherwise, it requires the ideal assumption that the episode has fixed length~\cite{zintgraf2019varibad,dorfman2020offline}.
Instead, we manage to implement a two-dimensional replay buffer of shape \texttt{(num transitions, observation dimension)} for observation storage, which also records the locations where each stored episode ends.
In case of actual episodes that are shorter than provided sampled sequence length, the buffer can also generate \textit{masks} to indicate if the corresponding transitions are valid. This enables the policy to receive a batch of previous experiences in a tensor-like data structure when sampling from the replay buffer.
To sum up, our replay buffer can support varying-length sequence inputs and subsequence sampling in a 2-dim manner.

\paragraph{Flexible training speed.} Finally, our code supports flexible training speed by controlling the ratio of the numbers of gradient updates in RL w.r.t. the environment rollout steps.
The training speed is approximately proportional to the ratio if the simulator speed is much faster than the policy gradient update.
Typically, the ratio is less than or equal to 1.0 to enjoy higher training speed.

\subsection{System Usage}
Table~\ref{tab:system} shows the typical system usage of our method and the compared specialized methods on different environments.
The time cost for our method and VariBAD depends on how many processes in parallel are run on a single GPU -- our method is run with 8 processes on a single GPU while VariBAD is run with one process due to large GPU memory usage.
From the results we can see that our method is memory-efficient in both RAM and GPU, and has an acceptable training speed with default hyperparameters.
The computer system we used during the experiments includes a GeForce RTX 2080 Ti Graphic Card (with 11GB memory) and Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz (with 250GB RAM and 80 cores).

\begin{table}[h]
    \centering
    \begin{tabular}{cc|ccc}
    \toprule
     Method    & Environment &  Time cost& RAM & GPU memory  \\
     \midrule
     \textbf{Ours} & Hopper-V & 22.5 h  & O(1) &  1 GB \\
     VRM~\cite{han2019variational} & Hopper-V &  102 h & O(200) & N/A \\
     \textbf{Ours} & Semi-Circle &  12 h  & O(1) & 1 GB \\
    VariBAD~\cite{dorfman2020offline} & Semi-Circle & 2.3 h  & O(1)* &  9.5 GB \\
    \textbf{Ours} & Cheetah-Robust &  7 h & O(1) & 1 GB \\
    MRPO~\cite{jiang2021monotonic} & Cheetah-Robust &  0.4 h & N/A & N/A \\
    \bottomrule
    \end{tabular}
    \caption{Comparison between our method and specialized methods in system usage.
    The time costs are evaluated within 1M environment steps.
    Both VRM and MRPO are run on CPUs and MRPO does not have a replay buffer (shown in N/A). VariBAD requires the assumption of fixed episode length for the RAM cost.}
    \label{tab:system}
\end{table}

\subsection{Our Hyperparameter Tuning Set}
\label{sec:setting}

Our proposed method has the following decision factors (introduced in Sec.~\ref{sec:method}) to tune in the experiments with the following options (the names in brackets are abbreviated ones):
\begin{itemize}
    \item Actor-Critic architecture (\textbf{Arch}): share the encoder weights between the recurrent actor and recurrent critic or not, namely \texttt{shared} and \texttt{separate}.
    \item Model-free RL algorithms (\textbf{RL}): \texttt{td3}~\cite{fujimoto2018addressing} or \texttt{sac}~\cite{haarnoja2018soft}
    \item Encoder architecture (\textbf{Encoder}): \texttt{lstm}~\cite{hochreiter1997long} or \texttt{gru}~\cite{cho2014learning}
    \item Policy input space (\textbf{Inputs}): \texttt{o}, \texttt{oa}, \texttt{or}, \texttt{oar}, \texttt{oard} (introduced in Sec.\ref{sec:method})
    \item Context length (\textbf{Len}): short (5), medium (64), long (larger than 100, depending on the POMDPs).
\end{itemize}

For each instance, we label it with the names of all the hyperparameters it used in lowercase as notation. For example, \texttt{td3-lstm-64-or-separate} in Fig.~\ref{fig:separate_vs_shared_main} refers to the instance that uses the separate actor-critic architecture, TD3 RL algorithm, LSTM encoder, the policy input space of previous observations and reward sequences, and RNN context length of 64.

\section{Evaluation Details}
\label{sec:eval_details}
The bar charts in Sec.~\ref{sec:experiments} and Table~\ref{tab:ablation} adopt the final performance of each method. We run each instance/variant in our method and each compared method with 4 random seeds. The final performance is calculated by the average performance of the last 20\% environment steps across the 4 seeds.

In terms of selecting the best variant in our method for each subarea, we first compute the final performance of each variant, then normalize the final performance into $[0,1]$, and finally select the best variant in the average of the normalized final performance across all the environments in each subarea.

\section{Environment Details}
\label{sec:env}

\subsection{Meta RL}
\label{sec:env_meta}
\paragraph{Semi-Circle.} We directly follow off-policy version of VariBAD~\cite{dorfman2020offline}. The observed state $s^o$ includes the agent's 2D position $p$, and the hidden state $s^h$ is referred to the goal state $p_g$. The goal state only appears in reward function: $R(s^o_{t}, s^o_{t+1}, a_t, s^h) \equiv R(p_{t+1},p_g) =  \mathbbm{1} (\|p_{t+1} - p_g\|_2 \le r)$.  The dynamic function $T$ is independent of the goal state.

\paragraph{Cheetah-Vel.} We directly follow \citet{dorfman2020offline} using MuJoCo~\cite{todorov2012mujoco} simulator of \texttt{HalfCheetah-v2}. The hidden state $s^h$ is the target velocity $v_{g}$ and observed state $s^o$ includes the velocity $v$. Reward function includes both the hidden state and action: $R(s^o_{t}, s^o_{t+1}, a_t, s^h) \equiv R(v_t,v_g,a_t) =  -\|v_t - v_g\|_1 - 0.05 \|a_t\|_2$. The dynamic function $T$ is independent of the goal state.

\paragraph{Wind.} We modified the parameters of Wind environment in \citet{dorfman2020offline} to make it harder to solve. The agent must navigate to a fixed (but unknown) goal $p_g$ within a distance of $D=1$ from its fixed initial state.
Similarly to Semi-Circle, the reward function is goal conditioned but without hidden state: $R(s^o_{t}, s^o_{t+1}, a_t, s^h) \equiv R(p_{t+1},p_g) =  \mathbbm{1} (\|p_{t+1} - p_g\|_2 \le r)$. The hidden state $s^h$ appear in the deterministic dynamics as a noise term, i.e.  $ s^o_{t+1} = s^o_t +  a_t + s^h $, where $s^h$ is sampled from $U[-0.08,0.08]$ at the initial time-step and then kept fixed.

\subsection{``Standard" POMDP}
\label{sec:env_pomdp}
Except for the classic Pendulum environment, we use PyBullet~\cite{coumans2016pybullet} as the simulator for ``standard" POMDP environments. As the practice in VRM~\cite{han2019variational}, we remove all the position/angle-related entries in the observation space for ``-V" environments and velocity-related entries for ``-P" environments, to transform the original MDP into POMDP.

\paragraph{$\{$Pendulum,Ant,Cheetah,Hopper,Walker$\}$-P.} The ``-P" stands for the environments that keep position-related entries by removal of velocity-related entries. Thus, the observed state $s^o$ includes positions $p$, while the hidden state $s^h$ is the velocities $v$.

\paragraph{$\{$Pendulum,Ant,Cheetah,Hopper,Walker$\}$-V.} The ``-V" stands for the environments that keep velocity-related entries by removal of position-related entries. Thus, the observed state $s^o$ includes positions $v$, while the hidden state $s^h$ is the velocities $p$.

\subsection{Robust RL}
\label{env:rmdp}
\paragraph{$\{$Hopper,Walker,Cheetah$\}$-Robust.} We directly adopt the environments used in MRPO~\cite{jiang2021monotonic}. In each environment, the hidden state is the dynamics parameters including the density and friction coefficients of the simulated robot in roboschool, adapted from the SunBlaze~\cite{packer2018assessing}. The exact ranges of the hidden states in each environment can be found in Table 1 of MRPO~\cite{jiang2021monotonic}. We evaluate the algorithms with 100 tasks in each environment, and use the average of them as average returns, and the average of the worst $10\%$ of them as worst returns, following MRPO paper.

\subsection{Generalization in RL}
\label{env:gen}
\paragraph{$\{$Hopper,Cheetah$\}$-Generalize.} We directly adopt the environments used in SunBlaze~\cite{packer2018assessing}. In each environment, the hidden state is the dynamics parameters including the density, friction coefficients, and the power of the simulated robot in roboschool. The exact ranges of both interpolation and extrapolation in the hidden state distribution for each environment can be found in Table 1 of SunBlaze~\cite{packer2018assessing}.
We follow the practice of SunBlaze to evaluate interpolation and extrapolation success rates.

\section{Full Experimental Results}
\label{sec:full_results}

\subsection{Learning Curves of All the Compared Methods}
\label{sec:learning_curves}
In this subsection, we show all the learning curves of all the compared methods in each subarea, namely ``Standard" POMDPs (Fig.~\ref{fig:pomdp_best_P} and Fig.~\ref{fig:pomdp_best_V}), meta RL (Fig.~\ref{fig:meta_best}), robust RL (Fig.~\ref{fig:rmdp_best}), and generalization in RL (Fig.~\ref{fig:generalize_best}). The final performance of these learning curves generate the bar charts in Sec.~\ref{sec:compare-best}.

\begin{figure}
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/antp/instance-td3-gru-64-oa-separate-max_x1500000-last0.8-window20.png} &
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/cheetahp/instance-td3-gru-64-oa-separate-max_x1500000-last0.8-window20.png} \\
    Ant-P &
    Cheetah-P \\
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/hopperp/instance-td3-gru-64-oa-separate-max_x1500000-last0.8-window20.png} &
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/walkerp/instance-td3-gru-64-oa-separate-max_x1500000-last0.8-window20.png} \\
    Hopper-P &
    Walker-P \\
    \end{tabular}
    \caption{\textbf{Learning curves on ``standard" POMDP environments that preserve positions \& angles  but occlude velocities  in the states in PyBullet~\cite{coumans2016pybullet} (namely ``-P").}
    We show the results from the \textbf{single best variant} of our implementation on recurrent model-free RL, the popular recurrent  model-free  on-policy implementation (PPO-GRU, A2C-GRU)~\cite{kostrikov2018pytorch}, and also model-based method VRM~\cite{han2019variational}. Note that VRM is around 5x slower than ours, so we have to run 0.5M environment steps for it.}
    \label{fig:pomdp_best_P}
\end{figure}

\begin{figure}
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/antv/instance-td3-gru-64-oa-separate-max_x1500000-last0.8-window20.png} &
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/cheetahv/instance-td3-gru-64-oa-separate-max_x1500000-last0.8-window20.png} \\
    Ant-V &
    Cheetah-V \\
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/hopperv/instance-td3-gru-64-oa-separate-max_x1500000-last0.8-window20.png} &
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/walkerv/instance-td3-gru-64-oa-separate-max_x1500000-last0.8-window20.png} \\
    Hopper-V &
    Walker-V \\
    \end{tabular}
    \caption{\textbf{Learning curves on ``standard" POMDP environments that preserve velocities but occlude positions \& angles  in the states in PyBullet~\cite{coumans2016pybullet} (namely ``-V").}
    We show the results from the \textbf{single best variant} of our implementation on recurrent model-free RL, the popular recurrent  model-free  on-policy implementation (PPO-GRU, A2C-GRU)~\cite{kostrikov2018pytorch}, and also model-based method VRM~\cite{han2019variational}. Note that VRM is around 5x slower than ours, so we have to run 0.5M environment steps for it.}
    \label{fig:pomdp_best_V}
\end{figure}

\begin{figure}
    \centering
    \begin{tabular}{c}
    \includegraphics[width=0.7\textwidth]{figs/results/meta/PointRobotSparse-v0/instance-td3-lstm-64-or-separate-max_x1500000-last0.8-window10.png} \\ PointRobotSparse-v0 \\
    \includegraphics[width=0.7\textwidth]{figs/results/meta/Wind-v0/instance-td3-lstm-64-oa-separate-max_xNone-last0.8-window10.png} \\ Wind-v0 \\
    \includegraphics[width=0.7\textwidth]{figs/results/meta/HalfCheetahVel-v0/instance-td3-lstm-64-oar-separate-max_x5000000-last0.8-window10.png} \\
    \end{tabular}
    \caption{\textbf{Learning curves on meta RL environments.}
    We show the results from the \textbf{single best variant} of our implementation on recurrent model-free RL, the specialized meta RL method VariBAD~\cite{dorfman2020offline}}
    \label{fig:meta_best}
\end{figure}

\begin{figure}
    \centering
    \begin{tabular}{c}
    \includegraphics[width=\textwidth]{figs/results/rmdp/hopper-robust/instance-td3-lstm-64-o-separate-max_x20000000-last0.8-window20.png} \\ Hopper-Robust \\
    \includegraphics[width=\textwidth]{figs/results/rmdp/cheetah-robust/instance-td3-lstm-64-o-separate-max_x15000000-last0.8-window20.png} \\ Cheetah-Robust \\
    \includegraphics[width=\textwidth]{figs/results/rmdp/walker-robust/instance-td3-lstm-64-o-separate-max_x20000000-last0.8-window20.png} \\ Walker-Robust \\
    \end{tabular}
    \caption{\textbf{Learning curves on robust RL environments.}
    We show the average returns (left figures) and worst returns (right figures) from the \textbf{single best variant} of our implementation on recurrent model-free RL, the specialized robust RL method MRPO~\cite{jiang2021monotonic}. Note that our method is much slower than MRPO, so we have to run our method within 3M environment steps. But the results show that our method have much better sample efficiency over MRPO.}
    \label{fig:rmdp_best}
\end{figure}

\begin{figure}
    \centering
    \begin{tabular}{c}
    \includegraphics[width=\textwidth]{figs/results/generalize/hopper-generalize/instance-td3-lstm-64-o-separate-max_xNone-last0.8-window20.png} \\ Hopper-Generalize \\
    \includegraphics[width=\textwidth]{figs/results/generalize/cheetah-generalize/instance-td3-lstm-64-o-separate-max_xNone-last0.8-window20.png} \\ Cheetah-Generalize \\
    \end{tabular}
    \caption{\textbf{Learning curves on generalization in RL environments.}
    We show the interpolation success rates (left figures) and extrapolation success rates (right figures) from the \textbf{single best variant} of our implementation on recurrent model-free RL. We also show the final performance of the specialized method EPOpt-PPO-FF~\cite{rajeswaran2016epopt} and another recurrent model-free (on-policy)  RL method (A2C-RC) copied from the Table 7 \& 8 in \citet{packer2018assessing}.}
    \label{fig:generalize_best}
\end{figure}

\subsection{Single Factor Analysis on Our Method}
\label{sec:single_factor}

Our analysis will focus on ablating the important design decisions: the actor-critic architecture (\texttt{Arch}), the policy input space (\texttt{Inputs}), the underlying model-free RL algorithm (\texttt{RL}), the RNN encoder (\texttt{Encoder}), and the RNN context length (\texttt{Len}).
As there are several decision variables in our method, we could only show the results of each variable in the plots by averaging the performance over the other variables, which we call \textbf{single factor analysis}. In this kind of analysis, we can only say one value is better than another in one factor in the average sense (not the maximal sense); therefore it can show the robustness of each factor, but cannot tell the best choices (showed in Sec.~\ref{sec:compare-best}).
We show single factor analysis plots in each subarea, namely ``Standard" POMDPs (Fig.~\ref{fig:pomdp_ablation_P} and Fig.~\ref{fig:pomdp_ablation_V}), meta RL (Fig.~\ref{fig:meta_all}), robust RL (Fig.~\ref{fig:rmdp_all}), and generalization in RL (Fig.~\ref{fig:generalize_all}).

From these plots, we can see that each decision factor can make a difference in some environments. For example, the choice of RL algorithm is crucial in Ant-P (Fig.~\ref{fig:pomdp_ablation_P}), Cheetah-V (Fig.~\ref{fig:pomdp_ablation_V}), Wind (Fig.~\ref{fig:meta_all}) and Hopper-Generalize (Fig.~\ref{fig:generalize_all}). The context length is essential in all the ``-P" environments (Fig.~\ref{fig:pomdp_ablation_P}), Cheetah-Vel (Fig.~\ref{fig:meta_all}),  and both the generalization environments (Fig.~\ref{fig:generalize_all}). The policy input space can make a difference in most ``-P" environments (Fig.~\ref{fig:pomdp_ablation_P}) possibly because \texttt{oar} contains the information of missing velocities.

\begin{figure}
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/antp/single_factor-RLEncoderLenInputs-window20.png} &
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/cheetahp/single_factor-RLEncoderLenInputs-window20.png}  \\
    Ant-P &
    Cheetah-P \\
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/hopperp/single_factor-RLEncoderLenInputs-window20.png} &
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/walkerp/single_factor-RLEncoderLenInputs-window20.png}  \\
    Hopper-P &
    Walker-P \\
    \end{tabular}
    \caption{\textbf{Ablation study of our implementation on ``standard" POMDP environments that preserve positions \& angles  but occlude velocities  in the states in pybullet~\cite{coumans2016pybullet} (namely ``-P").} We show the single factor analysis on the 4 decision factors including RL, Encoder, Len, and Inputs for each environment.}
    \label{fig:pomdp_ablation_P}
\end{figure}

\begin{figure}
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/antv/single_factor-RLEncoderLenInputs-window20.png} &
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/cheetahv/single_factor-RLEncoderLenInputs-window20.png}  \\
    Ant-V &
    Cheetah-V \\
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/hopperv/single_factor-RLEncoderLenInputs-window20.png} &
    \includegraphics[width=0.49\textwidth]{figs/results/pomdp/walkerv/single_factor-RLEncoderLenInputs-window20.png}  \\
    Hopper-V &
    Walker-V \\
    \end{tabular}
    \caption{\textbf{Ablation study of our implementation on ``standard" POMDP environments that preserve velocities but occlude positions \& angles  in the states in pybullet~\cite{coumans2016pybullet} (namely ``-V").} We show the single factor analysis on the 4 decision factors including RL, Encoder, Len, and Inputs for each environment.}
    \label{fig:pomdp_ablation_V}
\end{figure}

\begin{figure}
    \centering
    \begin{tabular}{c}
    \includegraphics[width=0.7\textwidth]{figs/results/meta/PointRobotSparse-v0/single_factor-ArchRLEncoderLen-window10.png} \\ Semi-Circle \\
    \includegraphics[width=0.7\textwidth]{figs/results/meta/Wind-v0/single_factor-RLEncoderLenInputs-window10.png} \\ Wind \\
    \includegraphics[width=0.7\textwidth]{figs/results/meta/HalfCheetahVel-v0/single_factor-RLEncoderLen-window10.png} \\  Cheetah-Vel \\
    \end{tabular}
    \caption{\textbf{Ablation study of our implementation on meta RL environments.} We show the single factor analysis on covering all the decision factors}
    \label{fig:meta_all}
\end{figure}

\begin{figure}
    \centering
    \begin{tabular}{c}
    \includegraphics[width=0.85\textwidth]{figs/results/rmdp/hopper-robust/single_factor-RLEncoderLenInputs-window20.png} \\ Hopper-Robust \\
    \includegraphics[width=0.85\textwidth]{figs/results/rmdp/cheetah-robust/single_factor-RLEncoderLenInputs-window20.png} \\ Cheetah-Robust \\
    \includegraphics[width=0.85\textwidth]{figs/results/rmdp/walker-robust/single_factor-RLEncoderLenInputs-window20.png} \\ Walker-Robust \\
    \end{tabular}
    \caption{\textbf{Ablation study of our implementation on robust RL environments.} We show the single factor analysis on the 4 decision factors including RL, Encoder, Len, and Inputs for each environment in both average returns and worst returns.}
    \label{fig:rmdp_all}
\end{figure}

\begin{figure}
    \centering
    \begin{tabular}{c}
    \includegraphics[width=\textwidth]{figs/results/generalize/hopper-generalize/single_factor-RLLenInputs-window20.png} \\ Hopper-Generalize \\
    \includegraphics[width=\textwidth]{figs/results/generalize/cheetah-generalize/single_factor-RLLenInputs-window20.png} \\ Cheetah-Generalize \\
    \end{tabular}
    \caption{\textbf{Ablation study of our implementation on generalization in RL environments.} We show the single factor analysis on the 3 decision factors including RL, Len, and Inputs for each environment in both interpolation and extraploation success rates.}
    \label{fig:generalize_all}
\end{figure}

\subsection{Additional Results on Separate vs Shared Recurrent Actor-Critic Architecture}
\label{sec:results_arch}
In Fig.~\ref{fig:separate_vs_shared_main} of Sec.~\ref{sec:ablation}, we show the importance of selecting a separate recurrent actor-critic architecture in a meta RL environment, Semi-Circle. Now we show the result in another POMDP environment, Pendulum-V, which occludes the positions and angles, in Fig.~\ref{fig:separate_vs_shared}. We can see that the shared encoder architecture is also worse than the separate one with a high gradient norm in the RNN.

\begin{figure}[h]
\centering
    \includegraphics[width=0.95\textwidth]{figs/results/pomdp/pendulumv/diagnosis.png}
    \caption{\footnotesize{\textbf{Comparison between \textit{shared} and \textit{separate} recurrent actor-critic architecture} with all the other hyperparameters same, on Pendulum-V, a toy ``standard" POMDP environment. We show the performance metric (left) and also the gradient norm of the RNN encoder(s) (right, in \textbf{log-scale}).
    For the separate architecture, \texttt{:critic} and \texttt{:actor} refer to the separate RNN in critic and actor networks, respectively.}}
    \label{fig:separate_vs_shared}
\end{figure}

redpajama_set_name: RedPajamaArXiv


text:
The oil droplet sign
P HEYWORTH,
J R O COLLIN
Moorfields Eye Hospital, London
Institute of Ophthalmology, London
P LUTHERT
Mr P Heyworth, Moorfields Eye Hospital, City Road, London EC1V 2PD susan.heyworth{at}virgin.net
http://dx.doi.org/10.1136/bjo.84.6.667f
Editor,—Over the past 22 months we have collected a series of five patients with basal cell carcinomas of the lid who exhibit the oil droplet sign. The following case report is a representative example.
A 27 year old Zimbabwean male presented with a 14 month history of a progressively enlarging mass in the right lower lid.
Clinically there appeared to be a nodular basal cell carcinoma involving the lateral part of the right lower lid measuring 3.5 × 3.5 mm and involving the lid margin (Fig 1). There was no regional node involvement. On everting the lid there were multiple transparent droplets beneath the tarsal conjunctiva measuring between 0.25 and 0.50 mm in diameter (Fig 2, see arrow). The droplets lay directly over the area of tumour involvement and extended beneath the forniceal conjunctiva. The droplets contained a clear or straw coloured fluid which had an oily appearance.
Photograph showing a 3.5 × 3.5 mm nodular basal cell carcinoma involving the right lower lid margin.
Photograph showing multiple lipid droplets (arrow) beneath the tarsal conjunctiva.
A full thickness wedge resection with 4 mm margins was performed. The lid was closed directly.
A light photomicrograph (Fig 3) of the full thickness lid resection shows a basal cell carcinoma at the lid margin (superiorly) and along the tarsal conjunctiva (left). Lipid droplets are associated with an intense chronic inflammatory cell infiltrate. Note one large droplet very close to the surface.
Photomicrograph of full thickness lid resection shows a basal cell carcinoma at the lid margin (superiorly) and along the conjunctiva (left). Lipid droplets are associated with an intense chronic inflammatory cell infiltrate (haematoxylin and eosin; original magnification ×27). The insert (from an adjacent section) shows the lipid droplets at higher magnification (×110).
We have noted this sign over many years and have collected five cases over the past 22 months. To our knowledge this sign does not appear in the literature. The sign has only been noted among cases of basal cell carcinoma and has not been associated with other tumours such as squamous cell carcinomas or meibomian gland carcinomas.
We propose that the lipid droplets represent trapped meibomian gland secretions which cannot drain through obstructed or infiltrated meibomian gland ducts. It is interesting to note that the droplets appear beneath the conjunctiva rather than forming collections within the tarsus as is seen with chalazion formation. The invasion and destruction of the tarsus by tumour (as opposed to benign obstruction) may offer a route of least resistance whereby the droplets appear beneath the conjunctiva.
There are insufficient numbers of cases to state whether this is exclusively a sign associated with basal cell carcinomas or whether it may appear among the rarer tumour types. The sign is however a good indicator of an infiltrative process and may assist the clinician in the diagnostic process.
Manuscripts from all countries except the UK and the Republic of Ireland should be sent to Professor C Hoyt, Editor, British Journal of Ophthalmology, University of California, Department of Ophthalmology, 10 Kirkham Street, K 301, San Francisco, CA 94143-0730, USA (tel: 001 415 502-6871; fax: 001 415 514-1512).
Manuscripts from the UK and the Republic of Ireland should be sent to Professor Andrew Dick, UK Editor,British Journal of Ophthalmology, Division of Ophthalmology, University of Bristol, Lower Maudlin Street, Bristol BS1 2LX (tel: 0117 928-4827; fax: 0117 925-1421).

redpajama_set_name: RedPajamaCommonCrawl


text:
Associate Professor Marketing
llau@uci.edu
PhD, UC Los Angeles
Professor Lau-Gesk's research interests fall into two areas:
the influence of affect or emotions in consumer decision making
the role of culture and self in consumer persuasion and judgment.
In both areas, she examines how consumers evaluate, interpret and manage complex conflicting experiences or decisions often encountered in everyday consumer life. Her research focuses on complex affective experiences where multiple emotions arise within a consumer context such as in advertising or actual consumption experiences. In the second area, Professor Lau-Gesk studies consumer self complexity due to cross and within cultural influences. Past research has looked at the impact of single factors on decision making despite the fact that many real life issues and experiences are much more complex, less straightforward and mixed. Professor Lau-Gesk's work in both areas illuminates the importance of studying more complex issues by resolving conflicts in the literature and offering new, oftentimes counterintuitive, insight to existing marketing theory and practice. Professor Lau-Gesk's research appears in Journal of Consumer Research, Journal of Marketing Research, Journal of Consumer Psychology and Marketing Letters, Journal of Behavioral Decision Making and Organizational Behavior and Human Decision Processes.
Selected Publications, News & Honors
Lau-Gesk, Loraine (2003), "Activating Culture Through Persuasion Appeals: An Examination of the Bicultural Consumer," Journal of Consumer Psychology, 13 (December), 301-315, Society for Consumer Psychology Young Contributor Award, runner-up.
Lau-Gesk, Loraine (2005), "Understanding Consumer Evaluations to Mixed Affective Experiences," Journal of Consumer Research, 32 (June), 23-28.
Drolet, Aimee, Patti Williams and Loraine Lau-Gesk (2007), "Age-Related Differences in Responses to Emotional vs. Rational Ads for Hedonic vs. Utilitarian Products," Marketing Letters, 18 (December), 211-221, lead article.
Basu-Monga, Alokparna and Loraine Lau-Gesk (2007), "Blending Co-brand Personalities: An Examination of the Complex Self," Journal of Marketing Research, 44 (3), 389-400.
Lau-Gesk, Loraine and Aimee Drolet (2008), "The Publicly Self Conscious Consumer: Prepared to be Embarrassed," Journal of Consumer Psychology, 18 (2), 127-36.
Kramer, Thomas, Loraine Lau-Gesk, and C.Y. Chiu (2009), "The interactive effects of duality expertise and coping frames on responses to ambivalent messages," Journal of Consumer Psychology, October, 661-672.
Lau-Gesk, Loraine and Joan Meyers-Levy (2009), "Emotional Persuasion: When the Valence Versus the Resource Demands of Emotions Influence Consumers' Attitudes," Journal of Consumer Research, December, 585-599.
Drolet, Aimee, Loraine Lau-Gesk, and Carol Scott (2011), "Aging Effects on Mixed Affective Experiences," Journal of Behavioral Decision Making, July, 293-314.
Kramer, Thomas, Ogze Yucel, and Loraine Lau-Gesk (2011), "The Effect of Schadenfreude on Choice of Conventional versus Unconventional Options," Organizational Behavior and Human Decision Processes, September, 140-147.
Mukerjee, Sayantani and Loraine Lau-Gesk (2016), "Retrospective Evaluations of Playful Experiences," Journal of Consumer Marketing, Vol 33, Issue 5, 387-395.
Mukherjee, Sayantani and Loraine Lau-Gesk (2016), "Coping with Sequential Conflicting Emotional Experiences," Journal of Business Research, forthcoming.


I wonder if y'all have been as anxiously waiting for the arrival of the new plush version of me as I have. Ever since my owner told me it was in the mail, I have been waiting and waiting and then waiting some more for the FedEx guy to deliver a package just for me.
As it turns out, the package wasn't addressed to me, it was addressed to my owner. Isn't that just the way of the world? It's hard for a capybara to be taken seriously these days. So then I had to wait even longer for my owner to get home from work and open it.
And I'll tell you what, I have never been so outraged! I've been called a peccary, a fish, a wombat and a kookaburra but I've never been called whatever that hideous thing is that she pulled out of that box! Do you see it? It's right there in the photo at the top of this post. That….thing…. is supposed to be a cute plush version of me. Maybe I'm blind, but I do not see the resemblance.
What happened to the cute version from the photo above? Exactly how does that hideous thing they sent resemble that cute photo of the prototype? I just don't know what to think or do!
Luckily my owner is more level-headed than I am. She says it must be some sort of mistake. Somewhere, someone opened a box containing that adorable stuffed me instead of that hideous monster. They are probably snuggling it right now! Probably they are not the least bit mad or disappointed. Probably they are planning on keeping plush me for themselves while all we can think about is getting that–I don't know what to call it–out of our house.
Oh, I so hope this doesn't mean a big delay in getting the real plush me! My little heart can't take much more of this.
Did you want to bite that ugly monster?
Not sure about biting it. It looks diseased.
I would bite that thing and eat all the little bits off. I would wipe my milk-mouth on it too.
My owner would not let me do that. She said we have to send it back in the hopes that whoever has the plush Caplin will do the same.
I would love to purchase that plush capybara. I hope I can get one. How much would one cost? It is so cute. It will have to do until I can convince my husband to get the real thing. ^__~!
YIPES! Is that supposed to be The Cat in the Hat? Any kid receiving that monstrosity will likely need YEARS of therapy to get over it.
Waiting with bated breath for the Caplin plushie!
No kids should be exposed to that thing.
True it looks like a terrible version of Cat in the Hat!
Poor Baby!!!! You look NOTHING like that horrible blue thing! Can't wait for a plush Caplin of my very own!!!!
Oh dear! He does not look very amused to be compared to that .. thing.
I wonder – and forgive me if this has been mentioned – if Caplin has ever "not been" a Coypu/Nutria?
is that some sort of narwhal??? ick… someone wasn't doing their job the day that box was packed!


text:
Tonaton.com
TonatonJOBS
Tonaton
Home/Electronics/"Budget" doesn't have to mean cheap: 5 good tablet picks for 2015
"Budget" doesn't have to mean cheap: 5 good tablet picks for 2015
Depending on where you are, you can't always tote a laptop around or even be able to afford one. The good news is that tablets can do most of what you need; you can watch movies, send an e-mail or write a letter.
But not all tablets are the same. Some, like the iPad, cost over $300 and for most people that's just too much to spend. Luckily, there are a few budget picks out there that are low-priced, but certainly not "cheap tablets".
Dell Venue 7 ($99.99)
This 7" tablet runs Android's Kit Kat operating system with a 1.66 GHz Intel Atom processor, has 16GB of internal memory, and a microSD card slot that will hold up to 64GB of whatever you choose. It has a no-fingerprint coating and comes in red or black, but there are other colored cases available. The battery lasts about eight hours while surfing the 'Net.
There aren't many pre-loaded apps on it, which is good; there's no sense in taking up space with stuff you don't need. It has a light sensor that adjusts the screen for brightness depending on where you are, and it has two cameras—one on the front, and one on the back. The front one isn't that good, but the camera on the back works just fine. It isn't the thinnest or the lightest tablet, but it does most things very well with only an occasional hiccup with large files or apps.
HP Slate 7 Plus ($99.99)
This is a pretty basic tablet, but it does have an ARM Cortex A7 quad-core processor that will run just about anything you can throw at it. It runs Google Android's Jelly Bean OS and has a microSD slot to expand its memory to up to 32GB—which is good because it only has 8GB of internal storage. Its battery lasts about 5.5 hours between charges.
The 7' screen has a resolution of 1024 x 600 pixels; not great, but not bad at all for a tablet under $100. It has 1GB of RAM, but this really isn't a gaming machine. For the basics plus watching a video now and again, it's excellent.
Dragon Touch A93 ($70.00 on Amazon new)
Dragon Touch's A93 tablet has a 9" screen and 1024 x 600 resolution, which is fine for almost anything you need to do. It also has an ARM Cortex A7 quad-core chip running things, so it's fast enough; if there's a problem, it's that it only has 512MB of RAM where most tablets have at least 1GB. But as long as you're only running one or two apps, you shouldn't notice a problem.
It has 8 GB of internal memory, but has a microSD slot for up to 32GB; it also weighs in at just over one pound. Keep in mind that big programs won't do well on this machine, though.
NeuTab N10 ($94.99)
This tablet has one of the largest screens out there with a 10.1" screen. The resolution is still a basic 1024 x 600, but it also comes with a 1.2 GHz quad-core processor, 16GB of internal storage with a microSD slot for more storage, and even Bluetooth.
The bad news? Both front-facing and rear cameras aren't that good, although most reviewers found Skype seems to work just fine with them. The battery also only lasts about 3-4 hours, and the plastic case could be stronger. Put it in a solid leather case and it should be just fine.
Dragon Touch A1X Plus ($109.99)
The second Dragon Touch entry on the list gets here because of a quad-core chip, 1GB of RAM, 16GB of internal storage, a microSD slot for up to 32GB, and even an FM radio option (although you have to plug headphones in to use it). It streams movies and games well, has a mini HDMI port, and runs Android Kit Kat 4.4.
As with many budget tablets, the cameras aren't so hot. But that's the price you pay when purchasing something for less than $200; on most budget machines, it's the cameras. Still, they do what they're supposed to, and Skype works just fine with them.
Not everyone can tote around a computer everywhere they go. But by picking carefully, a good budget tablet device can do almost everything a person needs to do until they get back to the office or home.
No spam guarantee.
The largest marketplace in Ghana!
Smartphone Wars: Is Samsung or Apple Best For You?
Top 10 Bestselling Mobile Phones of All Time (2020)
Top 6 Samsung Tablets For Work in 2020
iPhone 12 vs iPhone 11: What are the Differences?
Doorstep Delivery on Tonaton.com
Stay safe on Tonaton.com
Visit Tonaton.com | Tonaton Blog


Home » Here we go!
So. I bought a MediSpa.
And those that know me are saying, Whaaaaat the….??
Because although I have reinvented myself a few times in my life…ok, more than a few times, those that know me, and know me well, never saw it coming. Which is understandable, because not only did I not see it coming, I was so wrapped up in minutia of academia at the time, that it totally blindsided me.
I've been a student, a jewelry salesperson, a German cake café waitress, a lingerie salesperson, a camp counselor, a touring Shakespearean actor, a receptionist, a lab technician and truck fleet coordinator for a heat treating company, a summer theatre director and writer, an executive financial secretary on Bay Street for the likes of CIBC Wood Gundy, UBS, and Rothschild, a commercial actress, a theatre actress, an admin assistant for the Ontario Human Rights Commission, a mom, a substitute teacher at elementary and highschools, a mom again, a rental property manager, a mom again, a college instructor that migrated from Interior Design (how did that happen??) to Journalism (that was a stretch) to finally my penultimate job as an instructor of writing in the Design and Media Communications School at a major College, and now…..
I now own a MediSpa.
Thanks God I bought it with my amazingly talented sister, Dr. Mary Peirson, who handles all things clinical. I know all our patients are in her most excellent hands, and that is a great comfort to me as I navigate the business side of the world of lasers, aesthetic treatments, cosmetics and skincare.
I think I know why she asked me. Because of my (as my husband would call it) obsessional organizational skills and general fearlessness of things unknown. And the fact that I love marketing, and design, project management, talking to people, and that I'm not afraid to ask the awkward question if it will helps me learn something new.
I want to express my gratitude to the amazing staff at Laser Esthetiques(Now ArtMed) who have stayed the course with me over the last few months as I struggled with the reins of new business ownership. They have been my rock! It was amazing to watch them unfazed through the whole transition, and the outstanding care that they took in our clients. And I want to thank Dr. Swarsha, for building up such an amazing business and presenting us with this amazing opportunity which is now, happily, my career.
My job, it seems to me, outside of business management (which is a behemoth in itself) is to make sure you know that they are here for you. And that they, along with my tremendously talented sister, are awesome at their jobs. And I count myself lucky to be working alongside each of them.
I hope to share with you in this blog the amazing things happening in the wider world of medical aesthetics, and the things happening closer to home.
Stay tuned. This is going to be one wonderfully wild ride.


text:
You are here: Home / Public Health
Polluted air and rivers have acute local health effects and changing rainfall and temperature patterns cause global disruptions in supplies of food and clean water. Learn what is being done to mitigate the impact.
Project Green House
America is facing enormous social, economic and environmental costs associated with the 20 million vacant housing units and commercial space that is dragging down the national economy. These nonproductive real estate assets (liabilities) burden... Read More
Bringing Sunlight Indoors With 'Sundoliers'
Joel Goldberg
It's the heart of summer, your friends are off catching tans in the tropics, and yet you're stuck in a cold, dark office devoid of sunlight. Human beings are not adapted for such an injurious lighting dynamic. According to the U.S. Green Building... Read More
MN2020: Minnesota's Remedy for Medication Disposal
Minnesota 2020
When over-the-counter prescriptions or the most basic medications expire, an all too common reaction is to flush pills down the toilet. But even modern water treatment facilities aren't equipped to handle such chemicals, and it often ends up in... Read More
Dan Stelly
The torch has arrived, the athletes have settled into the village, and the opening ceremony has commenced. We're just as excited for the Summer Olympics as you are, but there's a lot more taking place besides the on-field events.
The Issue with Campground Campfires
Non Encrypted
Campfires pollute air so much that animals avoid the retching smells associated with smoke. Smoke interferes with animals' sense of smell for foraging as well as security. There are so many people using fires that animals are already cooped up or... Read More
Turning Gym Workouts into Electricity
Victoria Litherland
People sometimes don't realize just how much potential energy they possess. One great example concerns exercise, and the failure of gyms and health clubs to harness the energy of their patrons. As health club members use exercise machines such as... Read More
End World Hunger Through Meditation
Bill Douglas
World Healing Day
According to a 20 year study by Kaiser Permenente, 70 to 85% of all illness in America is caused by unmanaged stress. Our annual health costs are approaching $3 trillion per year. Mind-body practices such as meditation, yoga, tai chi, and qigong,... Read More
One Man's Waste is Another Man's Mushroom...?
Back to the Roots is committed to transforming one of the largest waste streams in America – the tons of coffee ground waste generated daily- into a highly-demanded, nutritious, and valuable food product: gourmet mushrooms. The Grow-it-at-Home... Read More
Seven Billion and Counting
UNSW Australia
With 7 billion mouths to feed--and many more to come--how will our cities, economies, health systems and environments cope? What are the challenges and opportunities? Some of UNSW's experts to canvass the big issues surrounding the global... Read More
SUNY Upstate Medical University: Sustainable Medical Care
Upstate Medical University
The State University of New York Upstate Medical University
Upstate Medical University set out to cut 75% of their campus' footprint. They were able to save $1 million over 20 months by re-processing polypropylene "blue wrap" to cut its use by 23%.
Congress may still seek new home for nation's nuclear waste
Medill News Service
Thirty-seven years ago, Congress decided the federal government should be responsible for disposing of the waste from the nation's nuclear power plants. But it's still in limbo.
Our country's food system is flawed. How can we fix it?
Rohan Agrawal
Mississippi is a contradiction of limited local food access in a state with deep agricultural roots. How can we solve this uniquely American problem, and what can we learn from other countries?
House panel OKs bills to rein in mining around Grand Canyon
Democratic lawmakers beat back a series of Republican amendments before advancing bills to restrict mining around the Grand Canyon and on other tribal lands, Miranda Faulkner reports.
House subcommittee looks toward the future of coal
Medill's Noah Broder reports from a House hearing, which looked at the value of coal to our economy. Dems called for more environmental controls, while the GOP stressed its importance to our energy infrastructure.
Burmese monk uses chant to fight climate change
Jake Meyers
Just a river?
Anna Ostrow
4 simple ways to reduce plastic waste on your next beach trip
Victoria Middleton
Planet Forward Digital Content Creator | George Washington University
Microplastics: A macro problem
Angelina Kossoff
Eckerd College
The Center for Innovative Media


text:
Indian farmers struggle with wheat export ban
Navtej Singh hoarded half of his wheat, hoping to sell it at a high price when there was a shortage, but the export ban prevented him from doing so.
India is the world's second-largest wheat producer, but New Delhi imposed an export ban on the project on May 14 to protect the country's food security, despite fears that the move could lead to severe global food shortages.
India's decision, combined with supply shortages in Russia and Ukraine, two of the world's top five wheat exporters, pushed wheat prices to record highs in Chicago and on the European Mercantile Exchange.
Indian farmers harvest wheat in a field on the outskirts of Amritsar, Punjab state, India, on April 12. picture: AFP
But in Asia's largest grain market in Khanna, Punjab state, considered India's "granary", wheat prices are heading in the other direction.
Thousands of farmers from wheat growing areas come here every year to sell. The market has dozens of huge warehouses, each the size of a football field.
Wheat used to sell for 2,300 rupees ($30) a tonne, but after the export ban, the price fell to 2,015 rupees, the government's floor price for grain purchases.
The situation has left India's hundreds of millions of wheat farmers struggling even more after being directly affected by the unusually hot weather.
This year, Navtej Singh has harvested 60 tonnes of wheat. He hoarded half of it, hoping to sell at a low price when the market was in short supply, but the export ban prevented him from reacting and hurriedly dumped the stock.
"This ban is appalling," Singh said. "The price is so low that it can't even cover the initial cost. I can't wait. I have to sell it right away."
He said authorities made "selfish" decisions without consulting farmers. "This year's harvest has been lost because of the reduced production. The ban has made our lives even more difficult," he said.
India produced 109 million tonnes of wheat in 2021 and exported 7 million tonnes before the heat hit India and Russia launched military operations in Ukraine. India is expected to increase mfy rice harvests and exports this year.
Prime Minister Narendra Modi even wanted to "provide wheat to the world" last month when the conflict in Ukraine caused food shortages. But extreme weather caused by climate change is affecting Indian crops more and more frequently. India's wheat output was 4 million tonnes lower than expected as heat hit Punjab.
In a food security plan during the pandemic, governments at all levels in India are reducing wheat purchases from the national distribution system, which provides free or subsidized food to nearly 800 million people.
However, the retail price of flour remains at a 12-year high. Manish Pajni, head of the Punjab government's grain procurement department in Khanna, supports the export ban on the grounds that wholesale prices could reach Rs 3,000 a tonne if the government does not intervene.
Workers load sacks of wheat onto a truck at a warehouse in Khana, Punjab state, on May 18. picture: AFP
Raj Sood, a wheat trader, said the government should monitor and analyse the situation rather than suddenly imposing an export ban and causing market volatility.
"The market has been under pressure from the crop failure crisis and the government has issued such a reckless ban," Sood said. "Of course, major wheat exporters like Cargill, ITC and Glencore will suffer a lot. But small traders, farmers will also suffer."
Many traders in Khanna believe that this measure is only temporary, as the market cannot avoid the law of supply and demand. Flour mill owner Divernder Verma usually buys ingredients from government warehouses, but says supplies are now low.
"Now we don't expect them to supply any more. Wheat supplies will be tight soon," Verma said. Private wholesalers will charge more, causing prices for bread and wheat products to continue to rise.
Hong Han (according to AFP)
Windows 10 license price is only $ 5.71, free upgrade to Windows 11
healthy beauty of Tay Do
UN launches fund to offset climate losses
Russia releases video of suicide drone ambushing Ukrainian artillery in Kherson
Kyiv prepares for evacuations in worst-case scenario
FIFA urges to stop 'promoting morality' at Qatar World Cup