text:
\section{Introduction}

    Reinforcement Learning (RL) has been proven to be successful at learning complex behaviors to solve a variety of robotic contact-rich tasks \cite{sutton2018reinforcement,kober2013reinforcement,wang2020deep}.
    However, RL solutions are still not widely adopted in real-world industrial tasks. One reason is that RL still requires an expensive and large amount of robot interaction with its environment to learn a successful policy. The more complex the target task is, the more interaction (samples) is required.

    To tackle this problem, domain transfer methods such as Domain Randomization (DR) and Curriculum Learning (CL) have been introduced.
    The concept of CL, where the learning process can be made more efficient by following a curriculum that defines an order in which tasks should be learned, has been introduced in previous works \cite{bengio2009curriculum, krueger2009flexible}. Additionally, DR of visual and physical properties of a task has been shown to improve the performance of tasks in novel domains \cite{tobin2017domain, andrychowicz2020learning}. However, most of these results have been only validated in simulated environments, from video games to robotic toy tasks, or in real-world toy environments. In this work, we tackle the problem of improving sample efficiency and performance when learning real-world complex industrial assembly tasks with rigid position-controlled robots. To this end, we seek to answer the question: Does the order in which the different environments (or tasks) are presented to the agent (through DR) affect the training sample efficiency and performance of the learned policy?

    We hypothesize that on top of DR, guiding a RL agent's training with a curriculum (presenting tasks in increasing order of difficulty) towards the desired behavior can increase sample efficiency. The reasoning is that the curriculum helps reduce the overall exploration needed to achieve the desired goal while DR enhances domain transferability.

    This paper presents a study of the combination of CL with DR. More specifically, we compare different curricula designs and different approaches at sampling values for DR. As a result, we propose a novel method that significantly outperforms our previous work \cite{beltran2020variable}. In \cite{beltran2020variable}, only DR is used to improve sim2real transferability without CL. Experimental results in simulation and real-world environments show that our novel method can be trained with only a fifth of the training samples required by our previous method and still successfully learn to solve the target insertion tasks. Furthermore, the learned policies transferred to the real world achieved high success rates (up to 86\%) on industrial level insertion tasks, with tolerances of $\pm 0.01~mm$, not seen during the training.

    This work's contributions are as follows:

    \begin{itemize}
        \item A study of the application of Curriculum Learning to a learning framework for rigid robots solving contact-rich manipulation tasks.
        \item A novel learning framework combining curriculum learning with domain randomization to accelerate learning and domain transfer.

        \item An improved reward function to guide the learning of force sensor-based contact-rich manipulation tasks. The reward perceived by the agent is dynamically discounted by the curriculum's level of difficulty. For our target domain, the idea is to encourage the agent to learn to solve the hardest tasks as discussed in \Cref{subsubsec:dynamic-reward}.
        \item An empirical study of the different methods considered in this paper was conducted. Novel tasks not seen during training were used to validate the performance of each method, both in simulation and in the real world, including complex industrial insertion tasks.
        Additionally, we study the impact of different components of our proposed method in the Appendix.

    \end{itemize}
    Additionally, this work's source-code\footnote{At https://github.com/cambel/robot-learning-cl-dr} will soon be open to the research community.

    The rest of this paper is organized as follows, related work is discussed in \Cref{sec:related-work}. The case study for this work and the proposed method are explained in \Cref{sec:materials-methods}. Experimental results and comparisons with alternative methods and our previous work are shown in \Cref{sec:experiments-results}. Ablation studies are described in the Appendix.

\section{Related Work} \label{sec:related-work}
    The related work to the topic of this paper, accelerating robot learning of contact-rich manipulation tasks, is introduced and discussed in this section.

\subsection{Reinforcement Learning for contact-rich manipulation tasks}
    Plenty of methods have been proposed to accelerate automation of robotic assembly tasks, such as peg-in-hole tasks. From search strategies to align the peg with the hole \cite{park2013intuitive,chhatpar2001search}, to learning-based methods \cite{luo2021learning, yasutomi2021peg}. Some researchers have explicitly focused on the domain transfer of assembly tasks from simulation to real-world environments. In \cite{schoettler2020meta}, a meta-RL technique is applied to transfer experiences and generalize better to the real world. In \cite{kaspar2020sim2real}, system identification of the real robot (KUKA LBR iiwa) with its simulated counterpart is performed to improve sim2real transferability. While RL-based policies have been proposed and proven to have the potential to solve assembly tasks in the real world, there is still a lack of adoption of such methods in real industrial assembly tasks. One reason for this gap between research and industry is the sample efficiency of such learning methods; a large amount of interaction of the agent with its environment is still necessary to learn robust policies. We aim to contribute to this area by proposing a more sample-efficient approach based on CL and DR, i.e., less time is required to train a successful policy without decreasing its transferability.

\subsection{Domain Randomization}
In the context of machine learning, DR has been proposed as a technique to improve domain transfer, such as going from one task to a harder one or moving from a simulated environment to a real-world environment, in particular for training vision-based models \cite{tobin2017domain} or sim2real models \cite{andrychowicz2020learning}. In \cite{mehta2020adr} an empirical study is presented to examine the effects of DR on agent generalization. Their results show that DR may lead to suboptimal, high-variance policies, which \cite{mehta2020adr} attributes to the uniform sampling of environment parameters. Following those results, this study proposed the use of DR based on CL and empirically studied the effect of the DR's sampling strategy.

\subsection{Curriculum Learning}
The concept of curriculum learning in the context of machine learning was first proposed by Bengio et al. \cite{bengio2009curriculum}. As mentioned before, CL can be understood as learning from easier to harder tasks, i.e., the order in which information is presented affects the policy's ability to learn. A comprehensive survey on Curriculum Learning applied to Reinforcement Learning has been presented in \cite{narvekar2020curriculum}.
Most CL approaches have been validated mainly on simulated environments, such as toy examples (e.g., grid worlds, cart-pole, and low-dimensional environments), video games, and simulated robotic environments. Few research works have focused on real-world robotic environments, such as in \cite{asada1996purposive} where a robot is trained to shoot a ball into a goal, \cite{baranes2013active, luo2020accelerating} where reaching tasks with a robot arm are tackled, and \cite{riedmiller2018learning} which focused on two tasks moving a cube to a target pose and cube stacking. Most recently, \cite{leyendecker2021deep} presented a CL method focused on a specific automotive production task, trained on simulation, and transferred to its real-world equivalent. Similarly, \cite{yasutomi2022curriculum} proposes a method to enable a robot to conduct anchor-bolt insertion, a peg-in-hole task for holes in concrete.
On the other hand, our study focuses on tackling various real-world complex industrial assembly insertion tasks, trained only on toy peg-in-hole simulated environments. Furthermore, we make an exhaustive study on the performance of several approaches to combine DR and CL. As a result, we propose a method to accelerate learning and domain transfer to real-world environments by an adaptive curriculum that affects how DR and the reward signal are considered during training.
\section{Materials and Methods}\label{sec:materials-methods}

\subsection{Problem Statement}\label{sec:problem-statement}

    In the present study, we consider the peg-in-hole assembly task that requires the mating of two components. One of the components is grasped and manipulated by the robot manipulator, while the second component has a fixed position via fixtures to a support surface. The proposed method is designed for position-controlled robot manipulators with access to force/torque information at the robot's end-effector (e.g., F/T sensor at the robot wrist), especially those robots where low-level torque control is not available. Thus, sensor-based force control is necessary to realize contact-rich manipulation tasks for such a type of robot.

\subsection{System Overview}\label{subsec:system-overview}

    Our propose method aims to improve the sample efficiency of the training phase.   \Cref{fig:system-overview} shows the overall system architecture which is based on our previous work \cite{beltran2020variable}. There are two control loops. The inner loop has an adaptive compliance controller; we choose to use a parallel position-force controller that was proven to work well for this kind of contact-rich manipulation tasks \cite{beltranhern2020learning}. The inner loop runs at a control frequency of 500 Hz, which is the maximum available in the Universal Robots e-series robotic arms\footnote{Robot details at https://www.universal-robots.com/e-series/}. The outer loop runs at a lower control frequency to account for the computation time required by the learning algorithm. Our system considers control of the 6 degrees of freedom of the Cartesian space at the robot's end-effector (position and orientation). To this previous learning control framework, we added the PID gains scheduling approach discussed in \Cref{subsubsec:pid-gains-scheduling}. Additionally, a new dense reward function is proposed and described in \Cref{subsubsec:cost-function}. Similarly, a DR method based on CL is implemented on top of this learning control framework, see \Cref{subsubsec:domain-randomization} and \Cref{subsubsec:curriculum-learning}.

\subsection{Reinforcement Learning}\label{subsec:rl-algo}
    The reinforcement learning framework followed in this work is modelled as an episodic Markov Decision Process (MDP), $M$, that has finite time steps with a limit of $T$ steps per episode. For a given task, the MDP consists of a state $\textbf{s} \in \mathscr{S}$, action space $\textbf{a} \in A$, state transition function $p(\textbf{s}(t+1) \,|\, \textbf{s}(t), \textbf{a}(t))$, which is the probability of transitioning to state $s'$ given that the action $a$ is taken while being in state $s$, and a reward function $r(s, a)$, which provides an immediate numerical reward for being in state $s$ and taking the action $a$. The goal of RL is to find a policy $\pi^*$ that maximizes the expected sum of discounted future rewards given by $R(t)=\sum_i^{\textbf{T}} \gamma r(s(t),a(t))$, where $\gamma$ is the discount factor \cite{sutton2018reinforcement}.

    We used a soft actor-critic (SAC) approach to learning the policy. SAC \cite{Haarnoja2018SoftAO} is an off-policy actor-critic deep RL algorithm based on maximal entropy. SAC aims to maximize the expected reward while also optimizing maximal entropy. The SAC agent optimizes a maximal-entropy objective, encouraging exploration according to a temperature parameter $\alpha$. The core idea of this method is to succeed at the task while acting as randomly as possible. Since SAC is an off-policy algorithm, it uses a replay buffer to reuse information from recent rollouts for sample-efficient training. Additionally, we used the distributed prioritized experience replay approach for further improvement \cite{horgan2018distributed}. Our implementation of the SAC algorithm was based on the TF2RL repository\footnote{TF2RL: Deep-reinforcement-learning library using TensorFlow 2.0. https://github.com/keiohta/tf2rl}.

    The agent's policy maps a multi-modal state, the robot's proprioception and the Force-Torque sensor data, to the robot's actions, detailed in \ref{subsec:parallel-control}, using a Time Convolutional Network (TCN); this policy representation and its improved performance over a simple neural network-policy were introduced in our previous work \cite{beltran2020variable}.

  \subsubsection{Reward function}\label{subsubsec:cost-function}
    On one hand, in our previous work \cite{beltran2020variable}, the reward function is defined only in terms of the contact force and distance between the current position of the robot's end-effector and the target position. The reason is to encourage the agent to get closer to the target position; the faster, the better, while discouraging any contact force. On the other hand, in this work, we propose the inclusion of the velocity of the robot's end-effector in the reward signal. While it is ideal that the agent achieves the task as fast as possible, high speeds are not desirable when the robot is close to the environment or in contact with it, as it can generate large contact forces. Thus, the proposed reward function has the following shape:

    \begin{equation}
    \begin{aligned}
        {r}(\textbf{s},\textbf{a}) = w_{1}r_{xv} + w_{2}r_F + w_3\rho
    \end{aligned}
    \label{eq:reward-function}
    \end{equation}
    where $r_{xv}$ is the component of the reward associated with the position and velocity of the robot's end-effector. $r_{xv}$ aims to encourage the agent to get closer and keep closer to the target position. Besides, the agent is encourage to move faster, if it is far from the target pose, or slower when it is close to the target pose. $r_{xv}$ is defined as:
    \begin{equation}
    \begin{aligned}
        r_{xv} = (1 - \tanh(5|x|)(1-|\dot{x}|) + (|\dot{x}|/2)^2
    \end{aligned}
    \label{eq:reward-position-velocity}
    \end{equation}
    Where $x$ is the distance between the robot's end-effector and the target position, and $\dot{x}$ is its velocity. A visualization of this reward component is shown in \Cref{fig:reward_xv}.
    The component of the reward \cref{eq:reward-function} associated with the contact force is defined as:
    \begin{equation}
    \begin{aligned}
        r_F = -1/(1+e^{-15|F_g-F_{ext}|+5})
    \end{aligned}
    \label{eq:reward-force}
    \end{equation}
    where $F_g$ is the desired insertion force, $F_{ext}$ is the contact force. The reward is always negative as a discount reward to encourage minimal contact force. However, due to the nature of the task, contact with the environment is unavoidable, so an $S$ shape function is proposed to allow small contact forces while strongly discouraging large ones. A visualization of this reward component is shown in \Cref{fig:reward_f}
    The position, velocity and contact force are normalized by the maximum value allowed for each one. Finally,  $\rho$ is defined as follows:
    \begin{equation}
    \rho = \left\{\begin{matrix}
     500,     & \textrm{Task completed}\\
     -200,     & \textrm{Collision} \\
     -1,     & \textrm{Otherwise}
    \end{matrix}\right.
    \label{eq:safety-reward}
    \end{equation}
    The task was considered completed if the Euclidean distance between the robot's end-effector and goal positions was less than 1 mm. The agent is encouraged to complete the task as quickly as possible by discounting the reward for every time step taken. Similar to our previous work \cite{beltranhern2020learning}, we imposed a collision constraint where the agent was penalized for colliding with the environment by giving it a large negative reward and immediately ending the episode. A \textit{collision} is defined as exceeding the force limit $F_{max}$. $F_{max}$ is a hyper-parameter that was defined as 50N in simulation or 30N in the real robot. Lastly, each component was weighted via $w$; all $w$s are hyperparameters. The performance of our new reward signal approach versus the reward signal proposed in our previous work \cite{beltranhern2020learning} is shown in \Cref{apx:reward-types}.

\begin{figure}[ht]
     \centering
     \includegraphics[width=0.7\columnwidth]{figures/reward_r_xv.png}
     \caption{Visualization of the position-velocity-based component of the reward function. $r_{xv}$ in \Cref{eq:reward-position-velocity}}
     \label{fig:reward_xv}
\end{figure}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.7\columnwidth]{figures/reward_r_f.png}
    \caption{Visualization of the contact-force-based component of the reward function. $r_F$ in \Cref{eq:reward-force}}
    \label{fig:reward_f}
\end{figure}

\subsection{Compliance Control in Task Space}\label{subsec:parallel-control}

    The agent's action space is based on our previous work \cite{beltranhern2020learning}, which consists of learning the force control parameters of a traditional sensor-based force controller. More specifically, we learn the parameters of a parallel position-force controller.

    The parallel position-force control requires the fine-tuning of three sets of parameters; gains of a PID for position tracking, gains of a PI for force tracking, and a selection matrix that defines the degree of control between force and position. The controller follows the control law shown in \cref{eq:force-law}

    \begin{equation}\label{eq:force-law}
        \begin{aligned}
            \textbf{x}_c ~=  S (K_p^x\textbf{x}_e &+ K_d^x\Dot{\textbf{x}_e} + \textbf{a}_x)  + (I-S)(K_p^fF_{e}+K_i^f\int F_{e} dt),
        \end{aligned}
    \end{equation}
    where $F_{e} = F_g - F_{ext}$ (goal contact force minus sensed contact force), $\textbf{x}_{e} = \textbf{x}_g - \textbf{x}$ (goal pose minus current pose), $\textbf{a}_x$ represents an arbitrary translation/rotation given by the agent, and $\textbf{x}_c$ is the commanded positions to the robot. The selection matrix is \[S = diag(s_1, ..., s_6),\quad s_j \in [0,1]\]
    The parameters to be learned are $K_p^x$, $K_p^f$, $S$, and $\textbf{a}_x$. One for each of the 6 Cartesian degrees of freedom. The remaining parameters $K_d^x$ and $K_i^f$ were defined proportionally to the $K_p^x$ and $K_p^f$ respectively. Therefore, the action space consists of 24 parameters. Each parameter is bounded to a continuous range of valid values. More details are provided in \cite{beltran2020variable}.

    \subsubsection{PID Gains Scheduling} \label{subsubsec:pid-gains-scheduling}
        An additional concept is explored in this paper, PID gains scheduling \cite{vesely2013gain}. Parallel force control for sub-millimeter tolerance insertion tasks tends to get stuck in the region very close to the alignment of the peg onto the hole. In the presence of very small position errors, the PID position controller barely generates any signal. On the other hand, the PI controller overcome the position PID controller due to small contact forces or noise coming from the F/T sensor. Therefore, on insertion tasks with sub-millimeter tolerances, the force controller does not move towards the target pose due to small resistance from the contact with the environment. To address this issue, we introduce PID gains scheduling, where once $\textbf{x}_e$ has been reduced to a position error of less than 1~cm, the PID gains are then scaled up based on the position error $K_p^x=K_p^x*(1/\textbf{x}_e)$. As $\textbf{x}_e$ approaches zero, the value of $K_p^x$ has a hyperbolic growth, thus the value of $K_p^x$ is bounded to be maximum $kn=10$ times its \textit{current} value (the agent's chosen value). \Cref{apx:pid-gains-scheduling} provides a comparison between the use of a traditional PID versus the PID gains scheduling on our proposed method.

    \subsection{Domain Randomization}\label{subsubsec:domain-randomization}

    Domain randomization (DR) \cite{tobin2017domain} is a popular method in robot learning to increase the generalization capabilities of policies trained in simulation, facilitating the transfer of the policy to a real-world robotic agent with minimal to no further refinement of the policy. In principle, the goal of DR is to provide enough variability to the simulated environment during training to generalize better to real-world conditions.
    In robot learning, DR randomizes a set of numerical parameters, $N_r$, of a physics simulator. With each parameter $\psi_i$ being sample from a randomization space $\Psi \in \mathbb{R}^{N_r}$. Each parameter is bounded on a close interval $\{[\psi_i^{low},\psi_i^{high}]\}_{i=1}^{N_r}$. For every episode, a new set of parameters is sample from the randomization space $\psi_i \in \Psi$. The most common approach is to draw sample uniformly from the randomization space.
    In this work, the randomized aspects of the peg-in-hole tasks are defined in \Cref{tab:randomize-values}.

    \begin{table*}[h!]
        \centering
        \begin{tabular}{|c|c|c|}
        \hline
        \multicolumn{2}{|c|}{\textbf{Condition}} & \textbf{Set} \\ \hline
        \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Initial position \\ (relative to goal)\end{tabular}} & Position (mm) & {[}-50, 50{]} \\ \cline{2-3}
         & Orientation (\degree) & {[}-30, 30{]} \\ \hline
        \multicolumn{2}{|c|}{\begin{tabular}[c]{@{}c@{}}Peg shape\end{tabular}} & {[}Cylinder, Cuboid, Hexagon prism, Triangular prism{]} \\ \hline
        \multicolumn{2}{|c|}{\begin{tabular}[c]{@{}c@{}}Hole Clearance (mm)\end{tabular}} & {[}\num{3.0}, \num{0.5}{]} \\ \hline
        \multicolumn{2}{|c|}{\begin{tabular}[c]{@{}c@{}} $\epsilon$: Distance from full insertion (mm)\end{tabular}} & {[}\num{15}, \num{1}{]} \\ \hline
        \multicolumn{2}{|c|}{\begin{tabular}[c]{@{}c@{}}Friction\\ (in Gazebo: \textit{surface/friction/ode/mu})\end{tabular}} & {[}\num{1}, \num{5}{]} \\ \hline
        \multicolumn{2}{|c|}{\begin{tabular}[c]{@{}c@{}}Stiffness\\ (in Gazebo: \textit{surface/friction/ode/kp})\end{tabular}} & {[}\num{5.0e-4}, \num{1.0e-6}{]} \\ \hline
        \end{tabular}
        \caption{Domain Randomization parameters and their maximum range of values}
        \label{tab:randomize-values}
    \end{table*}

    \subsection{Curriculum Learning}\label{subsubsec:curriculum-learning}

    Curriculum Learning comes from the notion that the order in which information is organized and presented to a learner impacts the learner's performance and training speed. This idea can be observed in the way humans learn, starting with simple concepts and gradually progressing to more complicated problems\cite{peterson2004day,krueger2009flexible}. CL can also be observed in the way we train animals \cite{skinner1958reinforcement}.

    In this work, we follow the notion that starting with easier tasks can help the agent learn better when presented with more difficult tasks later on. We consider the CL problem in the context of DR, where the goal is to reduce the training time by guiding the learning process without loosing domain transferability. Then, the problem becomes how to select parameters from the randomized space $\Psi$ to guide the agent's training.
    To this end, we consider four main approaches:

    \begin{itemize}
        \item Curriculum-based DR: The DR parameter's range of values is determined by the curriculum.
        \item The curriculum's evolution: a linear approach vs an adaptive approach.
        \item The DR sampling strategy: a Uniform distribution (UDR) vs a Gaussian distribution (GDR).
        \item A dynamic reward function based on the curriculum vs a standard reward function.
    \end{itemize}

    \subsubsection{Curriculum-based Domain Randomization}\label{subsubsec:dr-cl}
        We tackle the problem of defining a strategy to reduce the complexity of choosing a value for each randomization parameter. Though each parameter of the randomized space $\psi$ can be considered a degree of freedom that can be controlled to define the training tasks, adding new parameters would increases the difficulty of choosing the sequence of tasks to train the agent. Therefore, in order to simplify the problem while preserving the benefits of domain randomization, we propose the following approach: we represent the difficulty level $\mathbf{L}$ of a task as a numerical value in a close interval $[0, 1]$, from easiest to hardest. Then, a sub-set of each randomization parameter $\psi_i$ is defined based on the difficulty level $\mathbf{L_{ep}}$ at the beginning of each episode during training:

        \begin{equation}\label{eq:randomization-ranges}
            \psi_i: [\psi_i^{low}, \psi_i^{low} + \psi_i^{high}*\mathbf{L_{ep}}]
        \end{equation}

        where we assume that the parameter's set $\psi_i$ is defined in ascending order, such that, at $low$ and at $high$, the task is relatively the easiest and the hardest, respectively. The parameters considered in this work and their corresponding set are shown in \Cref{tab:randomize-values}.

    \subsubsection{Adaptive Curriculum Learning}\label{subsubsec:adaptive-cl}
        We consider two approaches to update the curriculum difficulty; on the one hand,  the naive approach is to monotonically increase the difficulty in a linear way, regardless of the agent's performance, i.e.,

        \begin{equation}
            \mathbf{L_{ep}} = ep/ep_{max}
        \end{equation}

        with $\mathbf{L_{ep}}$ being a constraint equal to 1 if the current episode number exceeds a defined maximum number of episodes. On the other hand, we propose an adaptive curriculum based on the agent's performance $\mathbf{P}$ during the last few episodes.
        The agent's performance is computed as the success rate of the last few episodes. Based on the agent's performance, the curriculum's level is updated by a defined step size $L_{step}$. Two thresholds are also defined. If the agent's performances surpass $L_{thld\_up}$ or fall below $L_{thld\_down}$ such thresholds, then the curriculum's level is increased or decreased respectively. \Cref{alg:cldr} describe our adaptive curriculum approach.

\begin{algorithm}
\caption{Adaptive Curriculum Learning Evolution}\label{alg:cldr}
\begin{algorithmic}
\State $\mathbf{P} = 0$
\For{Every episode $ep$}
    \State Update $\psi$ based on $\mathbf{L_{ep}}$
    \Comment{\cref{eq:randomization-ranges}}
    \State Sample task from $\psi$
    \State $\mathbin{/}\mathbin{/}$ \textit{+1: success, -1: failure}
    \State $\mathbf{P} \mathrel{{+}{=}} $ rollout current policy $\pi$ on task
    \State Update policy
    \If{$\mathbf{P} \ge L_{thld\_up}$}
        \State $L_{ep} \mathrel{{+}{=}}  L_{step}$
        \State $\mathbf{P} = 0$ \Comment{Consider newest rollouts}
    \ElsIf{$\mathbf{P} \le L_{thld\_down}$}
        \State $L_{ep} \mathrel{{-}{=}}  L_{step}$
        \State $\mathbf{P} = 0$ \Comment{Consider newest rollouts}
        \EndIf
\EndFor
\end{algorithmic}
\end{algorithm}

    \subsubsection{Domain Randomization Sampling Strategy}\label{subsubsec:dr-sampling-strategy}
        We consider the type of distribution from which the randomized parameters are sampled. Instead of the typical uniform distribution (UDR), we propose the use of a normal distribution (NDR), $\mathcal{N}(\mu,\sigma^2)$, with the mean being centered around the current curriculum's level $\mathbf{L}_{ep}$, and variance is a hyperparameter. The reason behind this choice is to keep increasing the general difficulty of the task with the increment of the difficulty level, but with a small probability, the curriculum can generate an easier task than the difficulty level to reduce the catastrophic forgetting problem \cite{mccloskey1989catastrophic, french1999catastrophic}.

    \subsubsection{Dynamic Reward Function}\label{subsubsec:dynamic-reward}
        Lastly, for our target task domain, we consider desirable for the RL agent to learn to handle the \textit{hardest} conditions to improve transferability to the real-world environment. To this end, we propose and evaluate a dynamically evolving reward with respect to the curriculum level difficulty. More specifically, the reward $r$, as defined in \Cref{subsubsec:cost-function}, is scaled by the current difficulty level $L_{ep}$; thus, the full reward would be obtained only when the agent reaches and maintains the hardest level.

    \begin{equation}
        \mathbf{r_t^d} = r*\mathbf{L_{ep}}
    \end{equation}
        where $r_t^d$ stands for the dynamic reward at time $t$.
        In other words, at each time step, the reward obtained by the agent is a fraction of the full possible reward for reaching such state.

\section{Experiments and results} \label{sec:experiments-results}

Through the following experiments, we aimed to understand the performance of our proposed method compared to alternative approaches, in terms of sample efficiency and generalization. To that end, experiments were performed with novel tasks not seen during training in simulation and in the real-world environment, using insertion tasks with medium grade industrial-level tolerances ($\pm 0.01~mm$).

The \textit{baseline} used throughout these experiments was based on our previous work \cite{beltran2020variable}, which mainly focused on the use of DR to enhance domain transferability. This experimental section focus on comparing the different curricula designs, sampling strategies for DR, and curriculum-based dynamic reward. As such, our previous work \cite{beltran2020variable} has been updated to include the new reward function and PID gain scheduling approach, proposed and described in \Cref{subsubsec:cost-function} and \ref{subsubsec:pid-gains-scheduling} respectively, which is used as the \textit{baseline} in this study. Ablation studies of these components of our proposed method are discussed in the Appendix.

\subsection{Experimental Setup} \label{subsec:experimental-setup}
    A simulated environment was used both for training and validation. The Gazebo simulator \cite{koenig2004design} version 9 was used.
    The choice of simulation environment is discussed in \Cref{sec:discussion}.
    Two real-world environments were used for validation purpose only; no further re-training was performed on the target domains\footnote{Despite Gazebo's simulation of the high-stiffness robot being accurate, the robot controllers respond faster than the real robot (maybe due to safety speed reduction on the side of the real robot controller, which we have not modify). Therefore, a minimal calibration is required. From our experience, scaling the reference trajectory or the command send to the controller by a factor of two worked well enough. A rough similarity between the simulation and real robot controller is enough to enable straightforward sim2real transfer.}. The components of the real-world setup is described in \Cref{fig:real-system}. Both environments consist of a Universal Robot 3 e-series robot arm with a control frequency up to 500 Hz. The robotic arm had a force/torque sensor mounted at its end-effector.
    In the simulation, the peg was considered as part of the robot's end-effector, as shown in \Cref{fig:sim_tasks}. The real-world robot simply used a Robotiq Hand-e parallel gripper. For the toy environments described in \Cref{subsubsec:toy-experiments}, this parallel gripper and a cuboid holder facilitate achieving a strong and stable grasp, similar to the simulation environment. However, for the industrial insertion tasks, we avoided the used of custom-made holders for the real-wold tasks, which increased the difficulty of the tasks as discussed in \Cref{subsubsec:wrs-experiments}.

    Our implementation of the RL agent that controlled both the simulated and real robot was developed on top of the Robot Operating System (ROS) \cite{quigley2009ros} with the Universal Robot ROS Driver\footnote{ROS driver for Universal Robot robotic arms developed in collaboration between Universal Robots and the FZI Research Center for Information Technology https://github.com/UniversalRobots/Universal\_Robots\_ROS\_Driver}. In both environments, training of the RL agent was performed on a computer with an Intel i9-10900X CPU and NVIDIA® Quadro RTX™ 8000 GPU.

    See the accompanying video \footnote{Graphical abstract and experimental results:
    \href{https://youtu.be/_FVQC5OcGjs}{https://youtu.be/\_FVQC5OcGjs}}

\begin{figure}[t]
    \centering
    \includegraphics[width=\columnwidth]{figures/real-world-system.png}
    \caption{Real experiment environment with a 6-degree-of-freedom UR3e robotic arm. WRS2020 Task board is shown, along side the three insertion tasks used for validation, motor pulley, bearing, and shaft. Each task has industrial level sub-mm tolerances.}
    \label{fig:real-system}
\end{figure}

\subsection{Training}

    The training phase consisted of the repeated execution of the insertion task using a variety of peg shapes and physical parameters of the simulator, as described in \Cref{tab:randomize-values}. An \textit{episode} was defined as a maximum of 1000 time steps, with each step being 50~ms. Early termination of an episode occurs under three conditions; 1) the target goal is reached, the peg inserted, and within $\epsilon$ distance from the full insertion, as described in \Cref{tab:randomize-values}. 2) the robot collides with the task board, i.e., a large contact force is sensed at any point during the task (more than 50~N in simulation or more than 30~N with the real-world robot). 3) the agent gets stuck, thus, the cumulative reward decreases to less than a set value $R_{min}$.

\begin{figure*}[t]
     \centering
     \hfill
     \begin{subfigure}[b]{0.16\textwidth}
         \centering
         \includegraphics[height=30mm]{figures/cylinder-peg.png}
         \caption{Cylinder.}
         \label{fig:cylinder}
     \end{subfigure}
     \hfill
      \begin{subfigure}[b]{0.16\textwidth}
         \centering
         \includegraphics[height=30mm]{figures/hexagonal-peg.png}
         \caption{Hexagonal}
         \label{fig:hexagon}
     \end{subfigure}
     \hfill
      \begin{subfigure}[b]{0.16\textwidth}
         \centering
         \includegraphics[height=30mm]{figures/cuboid-peg.png}
         \caption{Cuboid}
         \label{fig:cuboid}
     \end{subfigure}
     \hfill
      \begin{subfigure}[b]{0.16\textwidth}
         \centering
         \includegraphics[height=30mm]{figures/triangular-peg.png}
         \caption{Triangular}
         \label{fig:triangular}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.16\textwidth}
         \centering
         \includegraphics[height=30mm]{figures/trapezoid-peg.png}
         \caption{Trapezoid}
         \label{fig:trapezoid}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.16\textwidth}
         \centering
         \includegraphics[height=30mm]{figures/star-peg.png}
         \caption{Star}
         \label{fig:star}
     \end{subfigure}
     \hfill
        \caption{Simulated peg-in-hole environments. The cylinder, hexagonal prism, cuboid and triangular prism were used during training. The trapezoid prism and the star prism were used for testing.}
        \label{fig:sim_tasks}
\end{figure*}

\subsection{Learning performance}

    First, we compare the learning performance of the approaches presented in \Cref{subsubsec:dr-cl}, \ref{subsubsec:adaptive-cl}, and \ref{subsubsec:dr-sampling-strategy}:
    \begin{itemize}
        \item \textit{Baseline}: DR without curriculum learning (No Curriculum), as described in \Cref{sec:experiments-results}.
        \item Linear curriculum with Uniform distribution for DR (Linear Curriculum UDR).
        \item Linear curriculum with Gaussian distribution for DR (Linear Curriculum GDR).
        \item Adaptive curriculum with Uniform distribution for DR (Adp. Curriculum UDR).
        \item Adaptive curriculum with Gaussian distribution for DR (Adp. Curriculum GDR).
    \end{itemize}
    Each training session had a maximum of $100,000$ time steps, one-fifth of the training time required in our previous work \cite{beltran2020variable}.
    As described in \Cref{subsubsec:domain-randomization}, each episode is generated with a different set of values for the randomization parameters.
    \Cref{fig:Learning-comparison} shows the cumulative reward per method during a complete training session. Each training session was repeated with different random seeds. The average value and standard deviation are shown as the bold line and shadow region, respectively. The results are a preliminary highlight of the significant improvement of applying Curriculum Learning compared to the \textit{baseline}, which relied primarily on Domain Randomization alone. Furthermore, the adaptive curricula had a considerable performance above a simple linear increment of the curricula difficulty. Finally, using a Gaussian distribution instead of a Uniform distribution for the sampling of Domain Randomization parameters also significantly improves the agents' performance during learning. The dynamic reward (DyRe) approach discussed in \Cref{subsubsec:dynamic-reward} is not included here as the scale of the reward is different.

\begin{figure}[h!]
    \centering
    \includegraphics[width=\columnwidth]{figures/learning-comparison.png}
    \caption{Learning curve comparison using the cumulative reward of the overall training session. Each method was trained three times. The results are aggregated as the average cumulative reward and corresponding standard deviation, represented by the bold line and the shadow region.}
    \label{fig:Learning-comparison}
\end{figure}

\subsection{Evaluating learned Policies} \label{sucsec:eval-learning-curves}

    Next, we evaluated the performance of the learned policies on novel conditions not seen during training. Each policy was executed 100 times with different initial conditions and randomized parameters (with a fixed random seed for a fair comparison). More specifically, the peg shapes used for testing were a trapezoid prism and star prism, as shown in \Cref{fig:sim_tasks}. The trapezoid introduces a non-symmetric-shaped peg. The star prism peg is more challenging due to its sharp corners that make the peg prone to getting stuck during the aligning phase, making the overall insertion task harder to complete during the allowed time limit of 50 seconds (same time limit as during training).

    The results are shown in \Cref{tab:sim-results}. They include a comparison of the overall success rate and the average time needed to complete the task; failure cases are not included in the computation of the average time. Two main conclusions can be drawn from these results; 1) A curriculum may seem to have a better learning performance, but the resulting policy may not transfer well to novel environments, as is the case with the \textit{Linear Curricula} methods. Such linear curriculum approaches performed just slightly better than not using a curriculum at all. 2) The most successful methods are not necessarily the fastest. Our simulation environment did not handle very well friction between the peg and the task board, due to the high stiffness of the robot joints. In this almost friction-less world, the \textit{Adp. Curriculum UDR} method is able to solve the tasks between 20\% to 50\% faster than our best method \textit{Adp. Curriculum GDR DyRe}. However, the success rate of our method is at least 19\% higher.
    The main reason for such results is that since contact force and collision avoidance have higher priority than speed during learning, our proposed method moves slower when the peg gets closer to the task board so the contact force is reduced. This conclusion is further supported by the real-world experimental data described next in \Cref{subsubsec:learning-force-control}.

\begin{table}[h!]
\centering
\begin{tabular}{|c|cc|cc|}
\hline
\multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c|}{\textbf{Trapezoid   Prism}} & \multicolumn{2}{c|}{\textbf{Star Prism}} \\ \cline{2-5}
 & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Success\\ Rate\end{tabular}} & \begin{tabular}[c]{@{}c@{}}Avg.\\ Time(s)\end{tabular} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Success\\ Rate\end{tabular}} & \begin{tabular}[c]{@{}c@{}}Avg.\\ Time(s)\end{tabular} \\ \hline
No Curriculum & \multicolumn{1}{c|}{0.88} & 9.585 & \multicolumn{1}{c|}{0.627} & 11.304 \\ \hline
Linear Curriculum UDR & \multicolumn{1}{c|}{1.00} & 9.817 & \multicolumn{1}{c|}{0.696} & 9.152 \\ \hline
Linera Curriculum GDR & \multicolumn{1}{c|}{1.00} & 11.650 & \multicolumn{1}{c|}{0.775} & 11.780 \\ \hline
Adp. Curriculum UDR & \multicolumn{1}{c|}{1.00} & \textbf{6.881} & \multicolumn{1}{c|}{0.706} & \textbf{6.875} \\ \hline
Adp. Curriculum GDR & \multicolumn{1}{c|}{1.00} & 8.460 & \multicolumn{1}{c|}{0.794} & 8.013 \\ \hline
\begin{tabular}[c]{@{}c@{}}Adp. Curriculum \\ UDR DyRe\end{tabular} & \multicolumn{1}{c|}{1.00} & 8.429 & \multicolumn{1}{c|}{0.873} & 11.602 \\ \hline
\textbf{\begin{tabular}[c]{@{}c@{}}Adp. Curriculum \\ GDR DyRe\end{tabular}} & \multicolumn{1}{c|}{1.00} & 8.400 & \multicolumn{1}{c|}{\textbf{0.902}} & 11.544 \\ \hline
\end{tabular}
\caption{Evaluation of learned policies on novel conditions.}
\label{tab:sim-results}
\end{table}

\subsection{Real-world experiments} \label{subsec:real-world-experiments}

\begin{figure*}[t]
     \centering
     \begin{subfigure}[b]{0.39\textwidth}
         \centering
         \includegraphics[width=\columnwidth]{figures/toy-experiments.png}
         \caption{Trapezoid and star prism pegs}
         \label{fig:toy_tasks}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.59\textwidth}
         \centering
         \includegraphics[width=\columnwidth]{figures/wrs2020-experiments.png}
         \caption{Motor Pulley, Shaft, and Bearing.}
         \label{fig:wrs_tasks}
     \end{subfigure}
     \hfill
        \caption{Real-world experimental scenarios. \textbf{Left}: 3D printed primitive shape-pegs, different from the ones used for training in simulation. \textbf{Right}: Industrial level insertion tasks from the WRS2020 Robotics Assembly Challenge \cite{wrs2020_rulebook}.}
        \label{fig:toy_experiments}
\end{figure*}

    We performed two sets of experiments to evaluate the transferability of the learned policies to the real world and to novel tasks. The experiments were performed using the \textit{baseline} (No Curriculum) method and our newly proposed method (Adp. Curriculum DyRe GDR), which achieved the best results from the evaluation in simulation.
    The first set of tasks consisted of the same trapezoid and star prism-shaped pegs as the simulation experiments, which were not presented during training. A simplified peg was 3D printed using PLA material, as shown in \Cref{fig:toy_tasks}.
    The second set of tasks consisted of novel industrial level insertion tasks (See \Cref{fig:wrs_tasks}), similarly, these tasks were unseen during training in simulation. Both sets of tasks had sub-millimeter tolerances. Thirty trials were performed per method and task. The success rate and average completion time were measured where the initial position and orientation of the robot's end-effector at each trial were different and randomly sampled from a fixed random seed to fairly compare both methods. Each trial had a 500-time steps limit, i.e., 25~s.

\subsubsection{Primitive Shaped Pegs} \label{subsubsec:toy-experiments}

    The 3D printed pegs were designed with a cuboid holder, as shown in \Cref{fig:toy_tasks}, to increase the stability of the grasp. As a result, the stiffness of the contact is very high, as the task board was also firmly fixed to the workspace. Additionally, there is high friction due to the PLA material used for 3D printing and the imperfections on the printed surface. The high stiffness and friction made the task challenging. The results are shown in \Cref{tab:real-toy-results}. As mentioned before, for this test, the \textit{baseline} was also trained with only one-fifth of the samples shown to be needed \cite{beltran2020variable} to learn a successful policy. Thus, the learned policy's less refined force control tends to apply too much force to complete the task quickly, but the high friction and the corners of the star-shaped hole cause the peg to get stuck easily. Therefore, the \textit{baseline} method struggled to align the star prism peg and to get unstuck. On the other hand, our newly proposed approach successfully adapted to the real-world environment and succeeded at the novel tasks without further re-training the policies, just a straightforward sim-to-real transfer.

\begin{table}[h!]
\centering
\begin{tabular}{|c|cc|cc|}
\hline
\multirow{2}{*}{Method} & \multicolumn{2}{c|}{Trapezoid Prism Peg} & \multicolumn{2}{c|}{Star Prism Peg} \\ \cline{2-5}
 & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Success\\ Rate\end{tabular}} & \begin{tabular}[c]{@{}c@{}}Avg. \\ Time(s)\end{tabular} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Success\\ Rate\end{tabular}} & \begin{tabular}[c]{@{}c@{}}Avg. \\ Time(s)\end{tabular} \\ \hline
No Curriculum & \multicolumn{1}{c|}{1.000} & 6.500 & \multicolumn{1}{c|}{0.000} & - \\ \hline
\begin{tabular}[c]{@{}c@{}}Adp. Curriculum \\ DyRe GDR\end{tabular} & \multicolumn{1}{c|}{\textbf{1.000}} & \textbf{5.465} & \multicolumn{1}{c|}{\textbf{1.000}} & \textbf{6.023} \\ \hline
\end{tabular}
\caption{Evaluating learned policies on the real-world environment, using 2 toy scenarios not seen during training on simulation.}
\label{tab:real-toy-results}
\end{table}

\subsubsection{Industrial Level Insertion Tasks} \label{subsubsec:wrs-experiments}

    The second set of tasks used for evaluation consisted of 3 insertion tasks with industrial level tolerances. These were chosen from the assembly task used in the Industrial Robots Assembly Challenge of the World Robot Summit 2020 edition \cite{wrs2020_rulebook}. The tasks, as shown in \Cref{fig:wrs_tasks}, were the insertion of a pulley into a motor shaft, a shaft into a bearing, and a bearing into a plate. Similar to the previous tasks, the learned policies were directly transferred from the simulation environment without further training.
    These tasks are considerably more challenging as the grasp's stability significantly impacts the success. All three manipulated objects are round and grasped directly with a standard parallel gripper. Thus, torques applied along the direction of the grasp could easily change the object's orientation in the gripper. Small orientation changes significantly affect these very tight insertion tasks.

    The results are shown in \Cref{tab:real-wrs-results}.
    Our newly proposed method (Adp. Curriculum DyRe GDR) achieved a high success rate in all the tasks. For the motor pulley and the shaft tasks, our method also solves the task faster by finding the right fit faster. Our method is less likely to get stuck as it applies less contact force as shown in \Cref{fig:distance-force-comparison} and \ref{fig:distance-force-comparison-failure}
    In the case of the bearing task, the \textit{baseline} method, when successful, is slightly faster as it tends to apply higher contact force and move faster once the parts are aligned. However, the same high contact force makes it harder to find the proper alignment, thus resulting in a very low success rate. As a result, our newly proposed method outperforms the \textit{baseline} method, achieving a much higher success rate. These results are better appreciated in the supplemented video\footnote{Supplemental video:
    \href{https://youtu.be/_FVQC5OcGjs}{https://youtu.be/\_FVQC5OcGjs}}.

\begin{table*}[h!]
\centering
\begin{tabular}{|c|cc|cc|cc|}
\hline
\multirow{2}{*}{Method} & \multicolumn{2}{c|}{Motor   Pulley} & \multicolumn{2}{c|}{Shaft} & \multicolumn{2}{c|}{Bearing} \\ \cline{2-7}
 & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Success\\ Rate\end{tabular}} & \begin{tabular}[c]{@{}c@{}}Avg.\\ Time(s)\end{tabular} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Success\\ Rate\end{tabular}} & \begin{tabular}[c]{@{}c@{}}Avg.\\ Time(s)\end{tabular} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Success\\ Rate\end{tabular}} & \begin{tabular}[c]{@{}c@{}}Avg.\\ Time (s)\end{tabular} \\ \hline
No Curriculum & \multicolumn{1}{c|}{0.400} & 8.258 & \multicolumn{1}{c|}{0.667} & 9.199 & \multicolumn{1}{c|}{0.267} & 6.819 \\ \hline
\begin{tabular}[c]{@{}c@{}}Adp. Curriculum \\ DyRe GDR\end{tabular} & \multicolumn{1}{c|}{0.867} & 7.250 & \multicolumn{1}{c|}{0.833} & 7.015 & \multicolumn{1}{c|}{0.700} & 7.212 \\ \hline
\end{tabular}
\caption{Evaluating learned policies on the real-world environment, using 2 toy scenarios not seen during training on simulation.}
\label{tab:real-wrs-results}
\end{table*}

\subsubsection{Learning Force Control} \label{subsubsec:learning-force-control}

    In addition to the success rate and time to completion, we compare the detailed performance of the two methods. \Cref{fig:distance-force-comparison} shows the performance of both methods side by side for the three industrial insertion tasks.
    For simplicity, only the z-axis (i.e., the insertion direction), distance error (mm), and contact force (N) are displayed.
    As shown in \Cref{fig:distance-force-comparison}, our proposed method is more time-efficient and applies less contact force to the coupling part. Less contact force is desirable to avoid damage to either the assembly part or the robot.

    Similarly, \Cref{fig:distance-force-comparison-failure} shows the comparison of trials where both agents fail to complete the task on time. Though both agents failed, our method again shows a reduced exertion of contact force.
    In both cases, our method applies about 30\% less contact force.

\begin{figure*}[h!]
     \centering
     \begin{subfigure}[b]{0.32\textwidth}
         \centering
         \includegraphics[width=\columnwidth]{figures/success_pulley_df.png}
         \caption{Motor Pulley}
         \label{fig:motor_pulley_f}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.32\textwidth}
         \centering
         \includegraphics[width=\columnwidth]{figures/success_shaft_df.png}
         \caption{Shaft}
         \label{fig:shaft_f}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.32\textwidth}
         \centering
         \includegraphics[width=\columnwidth]{figures/success_bearing_df.png}
         \caption{Bearing}
         \label{fig:bearing_f}
     \end{subfigure}
        \caption{Agents performance on WRS2020 insertion tasks. For clarity, only the z-axis (Insertion direction) distance error and contact force are displayed. Comparison was made for each task when both methods successfully completed the task.}
        \label{fig:distance-force-comparison}
\end{figure*}

\begin{figure*}[h!]
     \centering
     \begin{subfigure}[b]{0.32\textwidth}
         \centering
         \includegraphics[width=\columnwidth]{figures/failure_pulley_df.png}
         \caption{Motor Pulley}
         \label{fig:motor_pulley}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.32\textwidth}
         \centering
         \includegraphics[width=\columnwidth]{figures/failure_shaft_df.png}
         \caption{Shaft}
         \label{fig:shaft}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.32\textwidth}
         \centering
         \includegraphics[width=\columnwidth]{figures/failure_bearing_df.png}
         \caption{Bearing}
         \label{fig:bearing}
     \end{subfigure}
        \caption{Performance of both methods where both failed to complete the task within the time limit.}
        \label{fig:distance-force-comparison-failure}
\end{figure*}

\section{Discussion}
\label{sec:discussion}

    Training a reinforcement learning agent with a curriculum that starts from easier tasks with reduced risk of encountering fatal states (e.g., a collision during a manipulation task) improves the learning sample efficiency and overall performance. In this case study, in particular, we integrate CL to contact-rich peg insertion tasks. A task is defined by various physical parameters as described in \Cref{tab:randomize-values}.
    We aim to allow the agent to carefully explore more states by presenting tasks in increasing order of difficulty, e.g., by reducing the stiffness of the contact between the peg and the board, the agent is less likely to apply excessive contact force to the environment (i.e., a collision).
    At the same time, starting with easier tasks, such as a shorter distance from the initial position to goal one, reduces the overall exploration. The curriculum is design around a domain randomization approach to preserve and enhance the domain transferability benefits from DR. Nevertheless, the type of curricula is very relevant for achieving better performance, both in terms of sample efficiency and success rate, as seen in the results in \cref{subsubsec:toy-experiments}.

    From our previous work \cite{beltran2020variable}, we demonstrated that with sufficient domain randomization-based training in simulation (at least $500,000$ time steps or about 8 hours) and further retraining in the real-robot environment, it is possible to learn policies that adapt to novel domains successfully. The present paper introduce a study on Curriculum Learning to tackle the problem of sample-efficiency, and the need to retrain in the target domain. The experimental results on the real-robot environment confirms that our newly proposed method is effective in learning contact-rich force control tasks. Despite that our proposed method was trained only in simulation with one-fifth of the training time used in our previous work \cite{beltran2020variable} and without further retraining, it achieves a high success rate on novel tasks, including challenging industrial insertion tasks with sub-millimeter tolerances.

    The main limitation of our proposed method is the assumption that the domain randomization parameter ranges are organized in order from \textit{easy} to \textit{difficult}. Prior knowledge is required to determine the \textit{difficulty} of a physical parameters on a given task. Such prior knowledge is task-specific and not necessarily easy to obtain or even impossible to determine manually. For example, considering only the peg-in-hole task and the stiffness between the peg and the task board, a low-stiffness can intuitively seem easier to handle for a high-stiffness robot arm, as less careful force control is required to achieve the task without generating large contact forces. However, depending on the material and how low the stiffness is, the peg may get stuck, or the task board may be deformed to such an extent that the task becomes impossible to solve.
    To tackle this problem, an interesting future avenue is to add a layer of learning, following works such as \cite{mehta2020adr,svetlik2017automatic}. The main idea of this line of research work is to train a neural network to define the RL agent's tasks. In other words, the network learns to choose the best values for the randomization parameters at each episode to increase the performance of the learned policy. The approaches differ in how the new network is trained, and how the agent's performance is defined, such as the cumulative reward, success rate, or another evaluation metric.
    Following these self-learned curricula approaches reduces the burden on prior knowledge, though as discussed in \cite{mehta2020adr}, a self-learned curriculum can provide an insight into incompatibilities between the task and randomization ranges. Therefore, such approaches may allow the use of many other parameters of the physics simulator for domain randomization, potentially increasing the transferability to novel domains. Nevertheless, the possible downside is the requirement of longer training sessions due to the added complexity.

    Another future avenue to further improve the presented work is the choice of a simulation environment. At the time of writing this paper, there are various physics engine simulators available that simulate the contact dynamics between bodies with different degrees of accuracy, among other capabilities. Our choice of the Gazebo simulator was motivated by its realistic simulation of rigid position-controlled robots. Additionally, the availability of ROS controllers that worked the same in the simulated and the real-world robot reduces the implementation burden and facilitates sim2real transfer. Nonetheless, other simulators provide better contact dynamics and are better adapted for Reinforcement Learning applications, such as Mujoco \cite{todorov2012mujoco}, or Nvidia Isaac Sim \cite{liang2018gpu}. Working with such simulators would be a significant improvement, as Domain Randomization is also easier to implement for vision-based learning methods and physical parameters of the simulated environment.

\section{Conclusions}
    This paper has studied the application of different approaches that combine Curriculum Learning with Domain Randomization to learn contact-rich manipulation tasks, particularly assembly tasks such as peg insertion. Based on such a study, we proposed to improve sample efficiency and generalization by training an agent purely in simulation, with the training being guided with CL and enhanced with DR. Additionally, this work introduced two enhancements to our learning framework, a new dense reward function and a PID gain scheduling approach, described in \Cref{subsubsec:cost-function} and \ref{subsubsec:pid-gains-scheduling} respectively, and validated in the Appendix.

    The learning framework proposed in this work is based on our previous work \cite{beltran2020variable} where a combination of sim2real with DR was proposed. Our previous methods still required a considerably large amount of agent's interaction with its environment and additional refinement in the real-world environment to learn a robust policy. On the contrary, our novel approach can be trained purely in simulation with only toy insertion tasks.
    Empirical results showed that with our proposed method a successful policy can be learned using only one-fifth of the samples needed in our previous work. Such policies can be straightforwardly transferred to real-world environments and still achieve a high success rate, up to 86\%, on novel complex industrial insertion tasks not seen during training.

\section{Patents}

\bibliographystyle{ieeetr}

redpajama_set_name: RedPajamaArXiv


text:
Socket::Socket(int sockfd)
  : sockfd_(sockfd)
{
  assert(sockfd_ >= 0);
}

Socket::~Socket()
{
  if (sockfd_ >= 0)
  {
    int ret = ::close(sockfd_);
    assert(ret == 0); (void)ret;
  }
}

void Socket::bindOrDie(const InetAddress& addr)
{
  int ret = ::bind(sockfd_, addr.get_sockaddr(), addr.length());
  if (ret)
  {
    perror("Socket::bindOrDie");
    abort();
  }
}

void Socket::listenOrDie()
{
  int ret = ::listen(sockfd_, SOMAXCONN);
  if (ret)
  {
    perror("Socket::listen");
    abort();
  }
}

int Socket::connect(const InetAddress& addr)
{
  return ::connect(sockfd_, addr.get_sockaddr(), addr.length());
}

void Socket::shutdownWrite()
{
  if (::shutdown(sockfd_, SHUT_WR) < 0)
  {
    perror("Socket::shutdownWrite");
  }
}

void Socket::setReuseAddr(bool on)
{
  int optval = on ? 1 : 0;
  if (::setsockopt(sockfd_, SOL_SOCKET, SO_REUSEADDR,
                   &optval, sizeof optval) < 0)
  {
    perror("Socket::setReuseAddr");
  }
}

void Socket::setTcpNoDelay(bool on)
{
  int optval = on ? 1 : 0;
  if (::setsockopt(sockfd_, IPPROTO_TCP, TCP_NODELAY,
               &optval, static_cast<socklen_t>(sizeof optval)) < 0)
  {
    perror("Socket::setTcpNoDelay");
  }
}

InetAddress Socket::getLocalAddr() const
{
  struct sockaddr_storage localaddr;
  socklen_t addrlen = sizeof localaddr;
  struct sockaddr* addr = reinterpret_cast<struct sockaddr*>(&localaddr);
  if (::getsockname(sockfd_, addr, &addrlen) < 0)
  {
    perror("Socket::getLocalAddr");
  }
  return InetAddress(*addr);
}

InetAddress Socket::getPeerAddr() const
{
  struct sockaddr_storage peeraddr;
  socklen_t addrlen = sizeof peeraddr;
  struct sockaddr* addr = reinterpret_cast<struct sockaddr*>(&peeraddr);
  if (::getpeername(sockfd_, addr, &addrlen) < 0)
  {
    perror("Socket::getPeerAddr");
  }
  return InetAddress(*addr);
}

int Socket::recv(void* buf, int len)
{
#ifdef TEMP_FAILURE_RETRY
  return TEMP_FAILURE_RETRY(::recv(sockfd_, buf, len, 0));
#else
  return ::recv(sockfd_, buf, len, 0);
#endif
}

int Socket::send(const void* buf, int len)
{
#ifdef TEMP_FAILURE_RETRY
  return TEMP_FAILURE_RETRY(::send(sockfd_, buf, len, 0));
#else
  return ::send(sockfd_, buf, len, 0);
#endif
}

Socket Socket::createTCP(sa_family_t family)
{
  int sockfd = ::socket(family, SOCK_STREAM | SOCK_CLOEXEC, IPPROTO_TCP);
  assert(sockfd >= 0);
  return Socket(sockfd);
}

Socket Socket::createUDP(sa_family_t family)
{
  int sockfd = ::socket(family, SOCK_DGRAM | SOCK_CLOEXEC, IPPROTO_UDP);
  assert(sockfd >= 0);
  return Socket(sockfd);
}

redpajama_set_name: RedPajamaGithub


text:
Auroras Nascem Tranquilas (, 1972) é um longa-metragem dirigido por Stanislav Rostotsky baseado na novela de Boris Vasilyev com o mesmo nome. Em 1973 o filme foi nomeado para o Oscar de "Melhor filme estrangeiro".

Filmes da União Soviética de 1972
Filmes em língua russa
Filmes baseados em obras de autores da Rússia

redpajama_set_name: RedPajamaWikipedia


This look is so fun and surprisingly easy! Trust me, watch the video below and try for yourself. Please tag me on Instagram or send me a DM if you try it! I would love to see you recreating this.
Again, I hope ya'll are having a great week. My husband, Jake, has a huge fishing tournament this weekend so it'll be Aniston and me getting into trouble alone ;). I think we may go to Nebraska Furniture Mart to see what they have for her big girl room. Also, I have moved into my new salon. I love it so much! We are having a grand opening August 25th if you can make it I'd love to meet you!


text:
Current mode logic (CML), or source-coupled logic (SCL), is a digital design style used both for logic gates and for board-level digital signaling of digital data.

The basic principle of CML is that current from a constant current generator is steered between two alternate paths depending on whether a logic zero or logic one is being represented. Typically, the generator is connected to the two sources of a pair of differential FETs, with the two paths being their two drains. Bipolar equivalents operate similarly, with the output being taken from the collectors of the BJT transistors.

As a differential PCB-level interconnect, it is intended to transmit data at speeds between 312.5 Mbit/s and 3.125 Gbit/s across standard printed circuit boards.

The transmission is point-to-point, unidirectional, and is usually terminated at the destination with 50 Ω resistors to Vcc on both differential lines. CML is frequently used in interfaces to fiber optic components. The principle difference between CML and ECL as a link technology is the output impedance of the driver stage: the emitter follower of ECL has a low resistance of around 5 Ω whereas CML connects to the drains of the driving transistors, that have a high impedance, and so the impedance of the pull up/down network (typically 50 Ω resistive) is the effective output impedance. Matching this drive impedance close to the driven transmission line's characteristic impedance greatly reduces undesirable ringing.

CML signals have also been found useful for connections between modules. CML is the physical layer used in DVI, HDMI and FPD-Link III video links, the interfaces between a display controller and a monitor.

In addition, CML has been widely used in high-speed integrated systems, such as for serial data transceivers and frequency synthesizers in telecommunication systems.

Operation
The fast operation of CML circuits is mainly due to their lower output voltage swing compared to the static CMOS circuits, as well as the very fast current switching taking place at the input differential pair transistors. One of the primary requirements of a current-mode logic circuit is that the current bias transistor must remain in the saturation region to maintain a constant current.

Ultra low power
Recently, CML has been used in ultra-low power applications. Studies show that while the leakage current in conventional static CMOS circuits is becoming a major challenge in lowering the energy dissipation, good control of CML current consumption makes them a very good candidate for extremely low power use. Called subthreshold CML or subthreshold source coupled logic (STSCL), the current consumption of each gate can be reduced down to a few tens of picoamps.

See also
 Low-voltage differential signaling (LVDS) A differential standard used primarily for signals between modules.
 Positive-referenced emitter-coupled logic, a differential signaling standard for high speed inter-module communications

References

 System Interface Level 5 (SxI-5): Common Electrical Characteristics for 2.488 – 3.125 Gbit/s Parallel Interfaces. OIF, October 2002.
 TFI-5: TDM Fabric to Framer Interface Implementation Agreement. OIF, September 16, 2003
 Introduction to LVDS, PECL, and CML, Maxim, https://pdfserv.maximintegrated.com/en/an/AN291.pdf
 http://www.ee.iitm.ac.in/~nagendra/videolectures/doku.php?id=ee685:start
 Interfacing between LVPECL, VML, cml and LVDS Levels, http://focus.ti.com/lit/an/slla120/slla120.pdf
 For more details on design automation and low power design of CML circuits, see: http://lsm.epfl.ch

External links
 JESD204B - a JEDEC Standard for serial data interfacing - Analog Devices
 JESD204B Overview (slides) - Texas Instruments

Logic families


text:
Israel's consulate in New York honors 9/11 victims on 20th anniversary of attacks
Acting Consul General of Israel in New York Israel Nitzan and the Port Authority Police Department gathered at the National September 11 Memorial & Museum in Lower Manhattan on Tuesday to mark the Jewish calendar's 20th anniversary of the Sept. 11 terrorist attacks and its victims.
"That tragic and devastating day was an attack on the free world, democracy and the values we all hold so dear," said Nitzan at the event. "As terror and suffering continue to grow in this world, Israel and the United States must continue to join forces, and fight hate and evil from our midst. We have come a long way and yet have long to go."
Others who attended the service included Port Authority Police Superintendent Ed Cetnar; Rabbi Mendy Carlebach, chaplain of the Port Authority of New York & New Jersey, who blew a shofar at the event; and UJA-Federation of New York president Amy Bressman.
Read More: JNS

redpajama_set_name: RedPajamaCommonCrawl


text:
SLHS Sax Player to Take National Stage at All-American Bowl
Dave Emke January 6, 2017 at 2:30pm
David Clark (back row, third from right) at the Alamo this week.
Previous Image 1/3 Next Image
One South Lakes High School student will remember the Alamo — and the Alamodome — for a long time after this week.
David Clark, a saxophone player in the SLHS band, is representing the school, Fairfax County and the state this weekend as part of the U.S. Army All-American Marching Band. The band, made up of 125 top high school marching band performers from across the nation, will play at halftime of the 2017 U.S. Army All-American Bowl football game Saturday at the Alamodome in San Antonio, Texas.
Clark told Reston Now this week that he is ecstatic for the opportunity to play on the big stage.
"I can't wait," he said. "It's going to be a lot of fun."
The member of the SLHS Class of 2017 has been in San Antonio all week preparing for the performance. He said Army Field Band members are acting as mentors for the high schoolers. His mentor is Staff Sgt. Daniel Goff, a saxophone player from Morgantown, West Virginia. (Clark said Reston native Staff Sgt. Pamela Daniels – a flute player in the Army Field Band – is also at the event working with another performer.)
Clark is the son of John and Mary Nell Clark of Reston. His participation in the event makes him the third South Lakes senior in the past three years to be so honored. In 2015, Samantha Gifford (trombone) was part of the show; in 2016, Brandon Coplen (percussion) made the trip.
Clark, who has been playing the sax since the fifth grade, is involved in a number of SLHS musical groups, including the wind ensemble, jazz band and pit orchestra, in addition to the marching band.
"I've (always) just really liked playing saxophone," he said. "It's a really awesome instrument; I think you can do a lot with it."
According to information provided by the South Lakes Band Boosters, members of the band are "selected by the National Association for Music Education and the US Army from more than 2,000 applicants [and] the rigorous application process includes essays, recommendation letters, audition videos and academic records." Clark said he was honored to be nominated by SLHS Band Director Grayson Fore.
"I wanted to take advantage of such an amazing opportunity where I could play with such elite, talented and motivated players," Clark said. "I love taking advantage of all these opportunities."
It hasn't been all hard work in San Antonio this week for Clark and the rest of the band. He said they've had the chance to explore the city's well-known attractions including the Riverwalk and the Alamo.
"My family is from Texas, so I've been (here) a few times," Clark said. "We saw the Alamo (on Wednesday); I took a bunch of pictures there."
The football game, showcasing top high school players from across the country, will air at 1 p.m. Saturday on NBC. The halftime show, featuring Clark and the other members of the All-American Marching Band, will be shown live and in its entirety at www.banddirector.com.
Photos courtesy David Clark and the South Lake Band Boosters
South Lakes High School
Registration for Reston Sprint Triathlon Fills Up Quickly
New Pet Grooming Salon Coming to Lake Anne Plaza


text:
Gun-maker can be sued over Sandy Hook shooting for how rifle was advertised, court rules
Connecticut Supreme Court issued the split decision
Daniel Acker/Bloomberg via Getty Images
A divided Connecticut Supreme Court ruled Thursday that gun manufacturer Remington can be sued for alleged wrongful marketing of its Bushmaster AR-15 rifle, which was used in the mass murder of children and teachers at Newtown's Sandy Hook Elementary School in 2012.
A lower court judge previously threw out a wrongful death lawsuit against the company, saying it violated a 2005 federal law called the Protection of Lawful Commerce Act. Justices reversed that ruling in a 4-3 decision, with the majority agreeing that under Connecticut's advertising laws, firearm companies may be held liable, The Associated Press reported.
Remington is being sued by the relatives of nine victims who were murdered in the massacre, and by one survivor. On Dec. 14, 2012, 20 children and six educators were killed at Sandy Hook Elementary School.
Attorney Joshua Koskoff, who represents the plaintiffs, told the AP, "The families' goal has always been to shed light on Remington's calculated and profit-driven strategy to expand the AR-15 market and court high-risk users, all at the expense of Americans' safety.
"Today's decision is a critical step toward achieving that goal," he added.
The lawsuit alleges that Remington has "for years sold AR-15s in a manner that foreseeably leads to the use of those weapons by unauthorized and unsafe users" by marketing them as weapons used by soldiers in battle.
The plaintiffs argue that the gun-maker "attract[ed] buyers by extolling the militaristic and assaultive qualities of their AR-15 rifles" by "advertising that the most elite branches of the military — including Special Forces, SEALs, Green Berets, and Army Rangers — have used them.
"A Bushmaster product catalog shows soldiers moving on patrol through jungles, armed with Bushmaster rifles," the lawsuit notes. "Superimposed over the silhouette of a soldier holding his helmet against the backdrop of an American flag is text that reads: 'When you need to perform under pressure, Bushmaster delivers.'"
A majority of Connecticut's high court agreed, citing the state's Unfair Trade Practices Act. In defending the ruling, Justice Richard Palmer wrote, "The regulation of advertising that threatens the public's health, safety, and morals has long been considered a core exercise of the states' police powers."
The dissenting justices argued that the majority was misreading the federal shield law in this "national question of first impression," the Washington Post reported.
Remington filed for Chapter 11 bankruptcy last year amid falling sales, heavy debts, and lawsuits against the firm. The company did not comment on Thursday's ruling.
Is gun maker liable? Court takes up Newtown shooting case ... ›
Remington Cites Sandy Hook Among Reasons for Bankruptcy Filing ... ›
Newtown families seek to hold gun maker accountable in ... ›
Connecticut Supreme Court Hears Newtown Families' Appeal ... ›
Remington's Bankruptcy Stalls Ruling in Sandy Hook Families' Suit ... ›


text:
package org.tuckey.web.filters.urlrewrite.utils;

import junit.framework.TestCase;
import org.tuckey.web.filters.urlrewrite.substitution.ChainedSubstitutionFilters;
import org.tuckey.web.filters.urlrewrite.substitution.SubstitutionContext;
import org.tuckey.web.filters.urlrewrite.substitution.SubstitutionFilterChain;

/**
 * @author Paul Tuckey
 * @version $Revision: 36 $ $Date: 2006-09-19 18:32:39 +1200 (Tue, 19 Sep 2006) $
 */
public class StringMatchingMatcherTest extends TestCase {

    public static String replaceAll(StringMatchingMatcher matcher, String from, String replacement) {
        SubstitutionContext substitutionContext = new SubstitutionContext(null, matcher, null, replacement);
        SubstitutionFilterChain substitutionFilter = ChainedSubstitutionFilters.getDefaultSubstitutionChain(true, false, false, false);
        return substitutionFilter.substitute(from, substitutionContext);
    }

    public void testPatterns() throws StringMatchingPatternSyntaxException {
        RegexPattern pat1 = new RegexPattern("a(a)", false);
        StringMatchingMatcher mat1 = pat1.matcher("aaf");
        assertEquals("$1f", replaceAll(mat1, "aaf", "\\$1"));

        RegexPattern pat2 = new RegexPattern("aa", false);
        StringMatchingMatcher mat2 = pat2.matcher("aaf");
        assertEquals("$1f", replaceAll(mat2, "aaf", "\\$1"));

        WildcardPattern pat3 = new WildcardPattern("/*/*/*/");
        StringMatchingMatcher mat3 = pat3.matcher("/aa/bb/cc/");
        assertTrue(mat3.find());
        assertEquals(3, mat3.groupCount());
        assertEquals("aabbcc$3$2", replaceAll(mat3, "/aa/bb/cc/", "$1$2$3\\$3\\$2"));

        // try again with no matches
        WildcardPattern pat4 = new WildcardPattern("aa");
        StringMatchingMatcher mat4 = pat4.matcher("aaf");
        assertEquals("aaf", replaceAll(mat4, "aaf", "a\\$1b"));

        WildcardPattern pat5 = new WildcardPattern("/**");
        StringMatchingMatcher mat5 = pat5.matcher("/aa/bb/cc/?aa&bb#cc");
        assertTrue(mat5.find());
        assertEquals(1, mat5.groupCount());
        assertEquals("aa/bb/cc/?aa&bb#cc$3$2", replaceAll(mat5, "/aa/bb/cc/?aa&bb#cc", "$1$2$3\\$3\\$2"));

        WildcardPattern pat6 = new WildcardPattern("/aa/**");
        StringMatchingMatcher mat6 = pat6.matcher("/aa/bb/cc/?aa&bb#cc");
        assertTrue(mat6.find());
        assertEquals(1, mat6.groupCount());
        assertEquals("bb/cc/?aa&bb#cc$3$2", replaceAll(mat6, "/aa/bb/cc/?aa&bb#cc", "$1$2$3\\$3\\$2"));

        WildcardPattern pat7 = new WildcardPattern("/ee/**");
        StringMatchingMatcher mat7 = pat7.matcher("/aa/bb/cc/?aa&bb#cc");
        assertFalse(mat7.find());

    }

    /**
     * this was throwing an exception... it shouldn't.
     */
    public void testStrangeProblem1() throws StringMatchingPatternSyntaxException {
        RegexPattern pat = new RegexPattern("^(/.*)$", false);
        StringMatchingMatcher mat = pat.matcher("/tester/one-level-sub/two-leel-sub/");
        mat.find();
        mat.groupCount();
        mat.groupCount();
        mat.groupCount();
        mat.group(1);

    }

}


text:
Q: Changing app name broke ability to run tests in xcode After I changed the product name for my app, I could still run the app in the simulator, but when I try to run tests, I get this error:
ld: file not found: /Users/stefankendall/Library/Developer/Xcode/DerivedData/ProjectName-fcxmoirjyqcvalgdndrznrmnvtdw/Build/Products/Debug-iphonesimulator/OldAppName.app/ProjectRootDirectory
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Why is the test action still trying to read from the old app name, even though I've changed the Product Name build property and xcode no longer builds to that location?

A: you need to change the "bundle loader" property of your unit test target.
Check the other properties of both the test and the app for the "OldAppName" by typing it in the search field on the "Build Settings" page for both targets.
Here is a screen shot to illustrate:

A: How you changed your appname? from the Project navigation pane? or from the build settings packaging -> product name of the target ?However i also want to tell that second way is ideal way to do it.

A: I think your problem is that you have to change the name of the folder in which your project is stored to your new name.

A: Change the product name to previous one. Build the project. Now update the product name to newer one. This works for me.

redpajama_set_name: RedPajamaStackExchange


So there I was coming into the fourth phase of my revision process, and a thought hit me — what next?
With all my attention wrapped up in completing my first polished manuscript for submission, thoughts of 'after' were far from my mind. Until the 'end' of my revisions peeked around the corner and I started to freak out that I had no other projects on the table.
That's not to say I didn't have story ideas noted, or drafts floating about, but none of them inspired me to start fleshing them out or start revising.
What was I going to write? What if I never come up with another idea worth writing? What the heck do I do now?
Since my 'new idea' breakdown, I've encountered other writers experiencing the same phenomenon. They too had approached a 'dead end' in their writing where getting a new idea off the ground seemed impossible.
These momentary setbacks have a way of causing us to question if we're truly writers, and I am here to tell you — yes you are.
Inspiration comes to those who seek it. In many cases, writing success comes down to you being intentional with your actions. Some days the muse will turn up on a white horse, and on other days it's up to you to grab the reigns of your creativity.
"Grabbing the reigns" for me equals research, and I found the key to unlocking a shiny new idea from one of my go-to sources, Well-Storied.com.
Up until I read Kristen's blog, I had one method for coming up with new ideas — letting them pop up at random.
Of course, this worked out fine while in the middle of an active project. There wasn't any urgency for these random plot bunnies to transform into full-fledged stories. They came; I jotted them down and continued on with my WIP.
It feels strange saying this but — I'd never sat down and intentionally brainstormed a new idea from scratch. Most of my stories developed from plot bunnies and fragments of dreams. In the past three years since I've taken up writing seriously, I've discovered curious nuggets about myself, and this was one of them.
Being a passive creator, however, no longer brought me ideas I connected with enough to expand on. And if I planned on writing full time I couldn't take a "passenger" approach either.
So here's where I became proactive about hunting down my next big idea.
Some people may see this and think "gasp, plagiarizing" but it's nothing like that.
This she notes, is the "power of influence" or as I call it inspired creativity.
We all do it whether consciously or not. After analyzing the plot bunnies I'd collated plus my WIP. and my deserted drafts, I could clearly see elements from my favorite books and television shows and movies. Historical events I studied during my time at University bled into the creation of my epic fantasy world.
Writers make comparisons to known works in their pitches, and others revamp old stories into fresh, fantastical retellings.
One thing I often do these days is research the creative influences of my favorite authors. I'm always intrigued by how they weave the elements of their stories together and from where they gather their inspiration.
Armed with all the necessary information, I blocked out time in my schedule to get intentional with my brainstorming.
As a History major, I have a thing for research which spills over into my absolute love for worldbuilding. But you can't build a world without a concept, and you can't put together a concept without elements for inspiration.
These influences have no limits whether they be artistic (books, movies, television shows, etc.) or real world (settings, locations, news stories, histories, etc.).
Writer tip: I am always collecting interesting pictures and facts on my Pinterest boards and on Instagram. Always be on the lookout for creative inspiration and hoard them like the writer dragon you are.
Write out the elements of each that captured your attention. Themes from your artistic influences can also be noted.
Take your time with this step and create as detailed a list as possible, but don't fuss over it. No matter how out of place all the elements may seem, write them down. We'll organize in later steps.
Again — no second guessing! No over thinking! Trust your creative instincts.
In this step, we'll test the elements for compatibility. Note the definition of 'compatibility' is at your creative discretion.
From the list you created in Step #2, jot down individual elements on your index cards or post-it notes. Think of them as puzzle pieces you'll shuffle to create your Frankenstein. Combine at least three items, each from a different influencer and see which fit best together.
I say at least three because I found below that threshold didn't allow sufficient wiggle room for splicing. What you don't want at the end of this exercise is a carbon copy of your artistic influences. Consider mixing in some real-world influences if you haven't selected any.
Once you've whittled down your list and have decided on your combo, you can move on to Step #4.
Writer tip: While you're going through the steps note all the ideas they shake loose. Don't be the "I don't need to write it down I'll remember it later, but then you don't" writer. Every nugget is the beginning of a magnificent new world.
With your initial (always allow for changes) combination decided, take some time to learn more about each element.
Researching will open the door to new knowledge and inspiration. Take notes during your research, and by the end, you'll have the beginnings of a writing notebook for your new story.
Writer tip: I'd suggest you include in your research the creative processes of your chosen influencers. I always walk away feeling inspired by the stories of my favorite creators and this spark of inspiration will help you flesh out your new idea.
It's time for the fun part — sewing your ideas into the beginnings of a new story.
Write out a synopsis for a story that includes all of the elements you've chosen and researched.
You may find that additional influences come to you while you're in this phase, let them. Note them.
The influencers you started with may not be the only ones you end with, and that's okay. You're building on your world and plot one layer at a time.
Your creativity is limited only by the walls you build around it.
There you have it, the steps that helped me shake loose a new idea for my next WIP.
During your brainstorming phase, you're gathering some of the pieces you'll need to create your story. As you write and get a better feel of where all the pieces fall, you'll add and subtract until it all fits. So have fun and dive in with an open mind.
From here you'll need to consider who your Protagonist and Antagonist are, their motives and the core themes and plot of your story. These elements will give your Frankenstein personality and heart and bring it to life!
New ideas don't always flow as smooth as we want them, but that's okay. Trust your process and yourself as a writer. Be patient with yourself. Quit comparing yourself and your pace with others. Having a slower pace doesn't mean you're a lousy writer.
Go forth and conquer my friends. May the words forever be in your favor!
Interested in free resources and monthly updates about what's happening at The Writerly Owl? Then sign up for my newsletter. Head on over to the Owl Lounge and enter the Owl Code provided at the bottom of your welcome email for instant access.


New Hope is located in Pennsylvania. New Hope, Pennsylvania 18938 has a population of 2,510.
The median household income in New Hope, Pennsylvania 18938 is $83,313. The median household income for the surrounding county is $76,824 compared to the national median of $53,482. The median age of people living in New Hope 18938 is 47.1 years.
This content last refreshed on 4/21/2019 11:31 PM. Some properties which appear for sale on this web site may subsequently have sold or may no longer be available.
I am interested in 4846 Toll Gate Road, New Hope, PA 18938.


Is there any advice you or anyone else can offer?
Food may be a lot less appealing if youre sick or taking certain medications. But its still important to eat a balanced diet.
Obtains prior authorization for Suboxone and or Vivitrol.
Native Americans, blacks and Hispanics also have a higher prevalence of the disease than whites.
In addition, elevated white blood cells and glucose levels in the bursa fluid can be an indication of infection.
Patients experiencing poor tolerability with donepezil or rivastigmine should wait until side effects subside or allow a 7day washout period prior to beginning galantamine.
However, the difference in formulation has not reduced overall AE rates.
If you smoke cigarettes, now is the time to quit.
He was one of 9 deaths in the small northern California county of Shasta who lost their lives from this dangerous drug.
Screening for diabetes needs to be targeted Screening for diabetes solely on the basis of age uncovers few cases and so is better used for patients with multiple risk factors, researchers say.
Do something about this epidemic!
Yet only 11 percent of addicted inmates receive any treatment.
Especially the huge companies that dispense these drugs.
WEDNESDAY, Aug. 5, 2015 HealthDay News The spines of boys and girls arent the same size at birth, a new study shows.
This drug came into my life after a major car accident and I battled this monster and other opiates for years after.
Or I ask Alexa to play the song.
At every treatment center that he has been to I see bright, sensitive young men(mainly) all 20 something whose lives have been derailed or destroyed by this drug.
I'm amazed at how easily this drug is prescribed.
Younger children are often placed in a fulltorso cast that is changed every few months for up to 18 months.
Since then I can't even work or do anything because I am afaird of the withdrawls.
TMJ problem that is related to the disc inside that joint. This is called disc displacement with reduction.
He was a wonderful man when not on the drug which sadly was not often.
Pinna GS, Kafetzis DA, Tselkas OI, Skevaki CL. Kawasaki disease: an overview. Curr Opin Infect Dis.
Not that I can find. It should work, but should is not will.
FDA approved for treatment of opioid dependence.
As far as I am concerned first make the criminals that make and market this drug give it to their wives and children and relatives to take when they "hurt" a bit (it's safe isn't it?
David Bradford. Hes the chair of public policy with the University of Georgias Department of Public Administration and Policy.
Wear moisturewicking clothes to keep skin dry and prevent germ growth.
De esos adolescentes, se encontr que 150 tenan hipertrofia del ventrculo izquierdo, que puede llevar a cardiomiopata hipertrfica, la causa ms comn de la muerte cardiaca repentina.
Eibner said. Without the individual mandate, the government would have to spend more overall to insure a lot fewer people.
Watch this brief report on VIVITROL by Lindsey Davis on ABC News.
The Nurse Care Manager is responsible for performing care management within the scope of licensure for members with complex and chronic care needs by assessing, developing, implementing, coordinating, monitoring, and evaluating care plans designed to optimize member health care across the care continuum and ensuring member access to services appropriate to their health needs.
If an IUD is going to slip out of place, it will most likely happen in the first few months of use or during your period.
In this example, breast cancer would be the primary cancer and the lungs would be the site of metastasis.
Commissioner Davis surmised there may have been a sediment blockage in the valve that could have caused the problem.
Although Blayne was hit by a car, he was a product of the pain clinics and fought addiction the last 8 years of his life.
VIVITROL does not appear to cause liver damage or hepatitis at the recommended dose.
The consequences are no longer confined to unplanned pregnancies and treatable sexually transmitted infections STIs.
Is Your State Preventing Drivethrough Mastectomies.
Vivitrol comes in two forms.
There is also typically a recovery period after angioplasty, though it is frequently less extensive than it would be for an open heart surgery.
You hustle like no other and work really hard to support your family and you use to save so much money.
Too many are lost to this drug.
Depression is a serious medical illness that affects millions of Americans each year.
I'm raising 3 teenage boys and are scared for them because of oxiconton, it weighs on my heart dearly for those who have lost someone.
This drug is an absolute nightmare!
After almost 12months of not using the drug, he had a extremely sore foot one night, and took his normal dose of 1 or 2 tablets.
SOURCE: Scripps Research Institute, news release, Oct.
Atomoxetine has demonstrated levels of safety equivalent to amphetamine and methylphenidate.
The most important thing to be done now, said Richard J.
After a year of sobriety, he seldom had the dreams.
He was no longer experiencing physical withdrawal but wanted to use.
Research also shows that the Vivitrol shot is more effective than medications that require the patient to comply with a daily treatment regimen.
But we didn't learn about Naloxone during one of his several rehab attempts, even as other patients left rehab and immediately died from overdose.
Beside when I got pregnant I didn't take any but after this last surgery in Jan I promised myself and for my kids to do anything in my power to quit.
San Diego Health Alliance provides medically supervised methadone maintenance and Suboxone (buprenorphine) detox treatment to individuals who are attempting to overcome an.
Friedly: Department of Rehabilitation Medicine, University of Washington, 325 Ninth Avenue, Box 359612, Seattle, WA 98104.
When you are born, you have T and B cells that have receptors on them.
Recent federal and state laws have also banned ingredients used in making synthetic cannabinoids.
Suboxone is a cash cow for many doctors.
For starters: If I take allergy meds all the time, am I preventing my body from building up a tolerance to allergens.
But scientists are worried that the virus could infect people due to its similarity to the viruses responsible for the outbreaks of SARS severe acute respiratory syndrome and MERS Middle East respiratory syndrome.
OxyContin has almost killed me and my doctor is more than happy to increase it any time I want.
Big Pharmacy, the FDA, and thousands of doctors are all accessory to murder, distribution of a dangerous intoxicant (often to minors who can't even legally buy cigarettes, through street drug trade!
He joins panels at churches to teach about addiction disease.
The metaanalysis showed patients taking the medications experienced more nausea, diarrhea, vomiting and headaches than those taking placebos.
If you have pes anserine bursitis or tendonitis, your physical therapist is a great resource to help you manage your pain and to implement strategies to keep the pain away.
Digital magnifying devices use video or pictures to enlarge certain items for viewing.
Ensure compliance through the administration of random urine drug tests at the facility.
Take care of yourselves (and I never, ever deal with medications now without wearing my reading glasses).
This is a very, very, very additive drug and in alot of cases a legal alternative to heroin.
We need to stand up for what is right and force these manufacturers to be more responsible.
It is used by the Department of Veterans Affairs and is being utilized by numerous justice agencies across the country including the Departments of Corrections in California, Colorado, Florida, Missouri, and Pennsylvania and numerous jail systems including Los Angeles County and New York City at Riker's Island.
People with underlying lung problems such as asthma or or other chronic illnesses may be more prone to bacterial infections and should seek a professional opinion sooner.
Cox2 is best known for its role in triggering inflammatory responses that, if misdirected, can lead to headache or arthritic pain.
Please discontinue the distribution of this dangerous drug.
Shame on them and the drug companys for selling out our next generation of promise!
Signs and symptoms of druginduced interstitial lung disease or pulmonary fibrosis may include exertional dyspnea, which may develop gradually with progressive worsening, accompanied by a nonproductive cough that does not respond to traditional antitussive medications.
Vivitrol also can be used for people who are incarcerated, he says.
Because tramadol was not scheduled, it could be prescribed by physicians who had not registered for the designation.
Two of our plow contractors are asking if we could go out to bid shortly, by the second week of July.
Before taking Vivitrol, a person has to have cleared opioids from the body, by not taking any for seven to 10 days.
In the context of SBM, SEID is a gateway disease into the world of pseudomedicine.


6 What is the Numerology of Rubi ?
10 Is there Rubi name in the Bible/Torah/Quran?
11 Is Rubi name fit for baby name ?
What Does Rubi Mean and History? From the Latin ruber, meaning "red" or "ruddy".
What is the Numerology of Rubi ?
Is there Rubi name in the Bible/Torah/Quran?
Is Rubi name fit for baby name ?
Our research results for the name of Rubi is fit name.