text:
\section{Introduction}
In the past decades, researchers have devoted great efforts to making computers
more autonomous, such that they could take care of more and more tasks in our
society. Today, algorithms make a great number of decisions which were made by
humans previously: They decide on the best way to get from one place to another,
what movie we may want to watch next and what articles we might be interested
in. They take strategic decisions for companies, place advertisements and invest
large sums of money. They play and win against us in video games
\cite{mnih2013playing}, chess \cite{campbell2002deep}, and Go
\cite{silver2017mastering}. Yet, the way we construct buildings, clean, cook,
dispose of our trash, plant and harvest food, search and rescue and assist
people with physical disabilities has remained largely unaffected. Where does
this striking contrast come from?

A difference which catches the eye is that the first class of tasks is primarily
concerned with manipulation of information, whereas the second class is
concerned with manipulation of matter, achieved through physical contact
between the agent and the external world. We believe that a key issue which has
hindered progress in the second class of problems is the high cost of
experimentation. Robotic manipulators are typically very expensive and can break
easily when entering into contact with the external world. Furthermore, they are
usually operated only in the presence of human supervisors, ready to hit the
emergency stop in case something goes wrong. These factors have largely
prohibited systematic large scale experimentation on physical manipulation
systems thus far. A large part of the robotic-reinforcement-learning (RL) community has
hence focused on simulation experiments
\citep[e.g.][]{Haarnoja2018-ox, Fujimoto2018-ur, Popov2017-hh, Mnih2016-mn,
Heess2017-vd, duan2016benchmarking, henderson2018deep}. However, results
obtained in simulation experiments do often not translate to real systems
\citep[see e.g.][]{tobin2017domain, james2019sim}. The physics of contact
interaction are nonsmooth and the outcome is highly sensitive to parameters and
initial conditions (e.g. a slight difference in the shape of objects in
contact can lead to very different motions).

Therefore, we believe that an experimental platform, capable of generating large
amounts of data from a wide range of possible contact interactions with external
objects at a low cost, could greatly support progress in autonomous robotic
manipulation.
The goal of this paper is to take a step in this direction. We present an
open-source robotic platform called TriFinger. Its hardware and software design
provide it with the following key strengths:
\begin{description}
	\setlength\itemsep{0.1em}
	\item[\namedlabel{req:sufficient_capabilities}{Dexterity}:] The robot design
	consists of three fingers and has the mechanical and sensorial capabilities
	necessary for complex object manipulation beyond grasping.
	\item[\namedlabel{req:unsupervised_execution}{Safe Unsupervised Operation}:] The
	combination of robust hardware and safety checks in the software allows users to
	run even unpredictable algorithms without supervision. This enables, for
	instance, training of deep neural networks directly on the real robot.
	\item[\namedlabel{req:ease_of_use}{Ease of Use}:]The C++ and Python interfaces
	are very simple and well-suited for reinforcement learning as well as optimal control
	at rates up to \SI{1}{kHz}. For convenience, we also provide a simulation
	(PyBullet) environment of the robot.
	\item[\namedlabel{req:viability}{Viability}:]The hardware design, based on
	\cite{Grimminger2020-tl}, is very simple and inexpensive (about
	\SI{5000}[\$]{} for the complete system), such that as many researchers as
	possible will be able to build their own platforms. All the information
	necessary for reproducing and controlling the platform is
	open-source\footnote{\label{google_site}\url{https://sites.google.com/view/trifinger}}.
\end{description}
An important point to note is that most of the software framework
is robot-agnostic and new systems can be integrated easily.
Hence, it is a contribution in its own right and can be used
independently of the robotic system proposed here.

In the remainder of the paper, we discuss related work and then describe the
design of the hardware and software in detail. Finally, we present experiments
in learning and optimal control to illustrate the aforementioned capabilities.

\section{Related Work}
In the past years, a large part of the RL community has focused on simulation
benchmarks, such as the deepmind control suite \cite{tassa2018deepmind} or
OpenAI gym \cite{brockman2016openai} and extensions thereof
\cite{zamora2016extending}. These benchmarks internally use physics simulators,
typically Mujoco \cite{todorov2012mujoco} or pyBullet
\cite{coumans2016pybullet}.

These commonly accepted benchmarks allowed researchers from different labs to
compare their methods, reproduce results, and hence build on each other's work.
Very impressive results have been obtained through this coordinated effort
\citep[see e.g.][]{Haarnoja2018-ox, Fujimoto2018-ur, Popov2017-hh, Mnih2016-mn,
Heess2017-vd, duan2016benchmarking, henderson2018deep}.

In contrast, no such coordinated effort was possible on real robotic systems,
since there is no shared benchmark. There have been isolated successes on real
systems \cite{levine2018learning, zhu2020the,
pinto2016supersizing,andrychowicz2020learning}, but these results cannot be
compared or reproduced, since each lab has their own robotic setup.

\subsection{Robot Hardware}
This lack of standardized real-world benchmarks has been recognized by the
robotics and reinforcement learning community \citep{behnke2006robot,
Bonsignorio2015-lg, Calli2015-vu, Calli2015-zj, Amigoni2015-gh, Murali2019-rg}.
Recently, there have been renewed efforts in this direction: For instance,
\citet{pickem2017robotarium} propose the \textsc{Robotarium}, a real-world
benchmark for mobile robots, and \citet{Grimminger2020-tl} propose an
open-source quadruped.

For manipulation, there are a number of affordable robots, such as Franka Emika,
Baxter, and Sawyer. \citet{yang2019replab} propose Replab, a simple manipulation
platform which has an even lower cost. Similarly, CMU proposed LoCoBot, a
low-cost, open-source platform for mobile manipulation.
\citet{dieter} propose a design using pneumatic actuators and show
its suitability for robotic learning.
However, all of these
platforms have very simple end-effectors, typically 1-D grippers, which limit
the possibilities of interaction with the environment.

For dexterous manipulation, there are a number of robotic hands on the market
(e.g. BarrettHand, Shadow Hand, Schunk Hand) which cost typically at least
50\,000 Euro per hand (an affordable exception is the Allegro hand).
In addition, \citet{dollar2010highly,she2015design,
xu2016design} proposed some innovative, exploratory hand designs. However, none
of these hands are designed for long-term unsupervised operation. In addition, to
have a sufficient workspace for manipulation, they have to be mounted on a robot
arm, which increases the complexity and risk of damage even further.

The two setups which are most similar to the TriFinger are the D'Claw, a
three-fingered robotic hand \cite{ahn2019robel} and the Phantom Manipulation
Platform, consisting of three Phantom Haptic Devices
\cite{lowrey2018reinforcement}, see \cref{fig:competitors}. As the TriFinger,
both of these setups consist of three manipulators with 3 DoF each. However, the
workspace, where these manipulators can interact with objects, is much larger
for the TriFinger (see \cref{fig:workspace}). In addition, the Phantom
Manipulation Platform is at 30\,000\$ about six times more expensive than the
proposed setup. The D'Claw robot, at 3\,500\$, is in a similar price range as
the proposed setup, but its actuators allow for far less dynamic motion and are
less robust to impacts, as they are only backdrivable with substantial
force, i.e. they do not give in as easily. We provide more details on these points in
\cref{appendix:comparison_d_claw}.
\begin{figure}
	\centering
	\includegraphics[height=0.26\textwidth]{figures/phantom.png}
	\includegraphics[height=0.26\textwidth]{figures/dclaw.png}
	\caption{Phantom Manipulator (left, image from
		\cite{lowrey2018reinforcement})   and D'Claw (right, image from
		\cite{ahn2019robel}). As the TriFinger, both platforms consist of three
	manipulators with 3 DoF each.}
	\label{fig:competitors}
\end{figure}%
\subsection{Robot Software}
Robot software can roughly be divided into two classes:
\begin{itemize}
	\item Low-level software allows to communicate with the robot in real-time.
	      Essentially, it sends motor commands to the robot and retrieves sensory
	      measurements.
	\item High-level software which does not need to run in real-time and often
	      operates on a more abstract level. The most widely-used high-level software
	      is by far ROS. More recently  \citet{murali2019pyrobot} proposed PyRobot, a
	      python interface for motion generation and learning.
\end{itemize}%
Here we are mainly concerned with the low-level robot control software.
Unfortunately, there is no framework which is widely used across labs, instead
there is a large number of diverse solutions. These solutions typically rely on
programming languages close to machine language, in order to ensure that
control loops run in real-time (i.e. at a constant rate, e.g. \SI{1}{kHz}).
For example SL \cite{Schaal2009} uses C while ROS-control
\cite{ros_control}, LAAS-CNRS Stack-of-Tasks \cite{mansard:icar:09}, and
ETH control-toolbox \cite{adrlCT} are implemented in C++. A particular
case is the IHMC Robotics software \cite{ihmc} which uses a real-time Java with
a modified garbage collector. These frameworks differ in
how end-users integrate their controllers.
SL and control-toolbox provide static methods or classes to
fill-in. ROS control uses the concept of ros-services to load and change
controllers online. Stack-of-Tasks relies on Python bindings to
dynamically interact with a control graph.

Unfortunately, these frameworks are only accessible to experienced users who
spent substantial amounts of time getting used to the particular approach at
hand. Implementation of controllers must usually follow a strict structure which
does not easily accommodate e.g. neural networks. In contrast, we designed the
user interface to be simple enough that even inexperienced users are able to
write controllers easily. It essentially consists of only two
functions: appending actions to a queue and accessing a history of observations.
Both functions are exposed in C++ as well as Python. The user may employ these
functions in any desired manner, for real-time or non-real-time control (using
e.g. a neural network). An additional benefit of the proposed design is that it
makes integration into high-level frameworks very simple.%
\section{Hardware Design}
The hardware design is loosely inspired by thumb, index and middle finger of
a human hand, see \cref{fig:human_hand}.
\begin{figure}[ht]
	%
	\begin{center}
		\includegraphics[width=0.8\columnwidth]{figures/robot_images/trifinger_edu/hand.JPG}
		\caption{Analogy between the human hand and the proposed platform.}
		\label{fig:human_hand}
	\end{center}
	%
\end{figure}%
We will therefore refer to the individual manipulators as fingers and to the
whole setup as TriFinger. In \cref{fig:catching} we can see  the main components
of the robotic platform, including the fingers, the frame and the three cameras.
In the following we will describe each of these components. All the details
necessary for building an instance of the proposed platform are open source
(\cref{google_site}).
\subsection{Finger Mechanics and Electronics}
The mechanics and electronics of the proposed robot are based on a recently
published open-source quadruped \cite{Grimminger2020-tl} consisting of
inexpensive high-performance motors, off-the-shelf parts, and 3D printed shells
(see \cref{fig:actuator_module}).
\begin{figure}[ht]
	%
	\begin{center}
		\includegraphics[width=0.75\columnwidth]{figures/robot_images/actuator_module.png}%
		\caption{Figure from \cite{Grimminger2020-tl} showing the actuator module which
		is the main building block of their quadruped legs and our fingers.}
		\label{fig:actuator_module}
	\end{center}
	%
\end{figure}%
\begin{figure}[ht]
	%
	\begin{center}
		\includegraphics[width=0.8\columnwidth]{figures/robot_images/trifinger_edu/technical_drawing.png}
		\caption{Technical drawing of a single finger, measurements are in mm.}
		\label{fig:technical_drawing}
	\end{center}
	%
\end{figure}%
\citet{Grimminger2020-tl} originally proposed a 2 DoF leg, which they extended to
3 DoF in the meanwhile (see
\footnote{\url{https://github.com/open-dynamic-robot-initiative/open_robot_actuator_hardware/tree/master/mechanics}}
for an overview of the designs based on \cite{Grimminger2020-tl}). The fingers
of the proposed platform are identical to the 3 DoF version of the quadruped
leg, apart from some slight modifications to the mounting, the 3D-printed shells, and
the end-effector.
We hence inherit all the favorable properties of the design from
\citet{Grimminger2020-tl}:
\begin{itemize}
	\item The high-performance brushless DC motors provide \textbf{high-torque}
	      actuation while having a \textbf{low weight}, which allows for dynamic
	      manipulation of objects up to a few hundred grams in weight.
	\item Transparency of the transmission enables \textbf{force control and
		      sensing} and \textbf{robustness to impacts}, both of which are crucial for
		robotic manipulation. Transparency means that forces applied at the
		end-effector directly translate to torques at the motors, rather than being
		absorbed by a high-gear-ratio transmission. This implies that end-effector
		forces can be obtained by measuring the motor currents and that impacts will
		not break the transmission.
		\item The motors can be \textbf{controlled at high frequency (1 kHz)} from a
		      consumer computer with a realtime-patched Ubuntu. This allows the robot to
		      sense external forces and react to them extremely quickly.
		\item The \textbf{design is very simple} and conists of \textbf{inexpensive}
		      off-the-shelf parts and 3D printed shells. This will allow other researchers
		      to build their own platforms.
	\end{itemize}
	A small but important addition in our design is a soft tip which mimics the
	human finger tip. This increases the stability of interaction with external
	objects greatly, as impacts are damped and contact extends to a surface rather
	than a single point.
	%
	\subsection{Kinematics}
	%
	\begin{figure}[ht]
		%
		\begin{center}
			\includegraphics[width=0.8\columnwidth]{figures/robot_images/trifinger_edu/workspace.png}
			\caption{An overlay of different robot configurations to illustrate the workspace.}
			\label{fig:workspace}
		\end{center}
		%
	\end{figure}%
	%
	The kinematics were designed to maximize the workspace where all three fingers
	can interact with an object simultaneously, see \cref{fig:workspace}.

	Each of the manipulators has 3 DoF, which implies that the finger tip can move
	in any direction (see \cref{fig:technical_drawing} for a technical drawing). This is
	important for dexterity and it makes the finger robust to impacts, as it can
	give in to forces from any direction.
	%
	\subsection{Frame and Boundary}
	\begin{figure}[ht]
		%
		\begin{center}
			\includegraphics[width=0.8\columnwidth]{figures/robot_images/trifinger_edu/disassembled.jpg}
			\caption{The platform can be disassembled into three modules for  transportation and storage.}
			\label{fig:disassembled}
		\end{center}
		%
	\end{figure}%
	%
	We use an aluminium frame to attach the fingers and cameras. The height of the
	fingers can be adjusted easily according to the requirements of a specific task.
	In addition, the entire platform can simply be flipped (see \cref{fig:catching})
	for tasks such as throwing and catching.

	For manipulation on the table, we designed an optional boundary to confine
	objects to the workspace of the platform, see \cref{fig:high_arena_border}. This
	is essential for learning during extended periods of time without human
	supervision.

	Finally, the platform can be disassembled easily into three compact modules, see
	\cref{fig:disassembled}.
	%
	\begin{figure}[ht]
		%
		\begin{center}
			\centerline{\includegraphics[width=0.8\columnwidth]{figures/robot_images/trifinger_edu/boundary.JPG}}
			\caption{A boundary to prevent objects from leaving the workspace.}
			\label{fig:high_arena_border}
		\end{center}
		%
	\end{figure}
	%
	%
	%
	\subsection{Cameras}
	As there are three fingers interacting closely with the target object, there
	is a lot of potential for occlusion. Therefore we place three cameras around
	the platform (see \cref{fig:catching}), ensuring that the object will at all
	times be visible from at least one camera. We use Basler acA720-520uc
	cameras with Basler C125-0418-5M-2000034830 lenses, as they have a high rate
	of up to 525 fps using global shutter, low latency, and an appropriate field
	of view (see \cref{fig:cameras}).
	%
	\begin{figure}[ht]
		%
		\begin{center}
			\centerline{
				\includegraphics[width=0.32\columnwidth]{figures/robot_images/trifinger_edu/robot_camera_0.png}
				\includegraphics[width=0.32\columnwidth]{figures/robot_images/trifinger_edu/robot_camera_120.png}
				\includegraphics[width=0.32\columnwidth]{figures/robot_images/trifinger_edu/robot_camera_240.png}
			}
			\caption{Images taken from each of the three cameras. Having three cameras ensures that the object will always be visible from at least one camera.}
			\label{fig:cameras}
		\end{center}
		%
	\end{figure}
	%
	%
	%
	%
	\subsection{Measurements and Control Signals}
	As in \cite{Grimminger2020-tl}, the signals are transferred between the control
	computer (standard consumer PC) and the motors through CAN at 1 kHz. In
	accordance with RL terminology, we will call the control signal action and the
	measurements observation.\\
	\textbf{Action:} The input of this platform is a
	nine-dimensional vector: A desired torque (which is proportional to the
	current) for each joint. The robot expects this signal to be sent at a rate
	of 1 kHz.\\
	\textbf{Observation:} The output consists of proprioceptive measurements
	(joint angles, joint velocities, joint torques), acquired at 1 kHz, and
	images from the three cameras, obtained typically at 100 Hz.

	%
	%
	\section{Software Design}
	The key strengths of our software framework are:
	\begin{itemize}
		\item The user interface in C++ and Python is very \textbf{simple}, yet
		      well-suited for real-time (1 kHz) optimal control and reinforcement
		      learning.
		\item It performs \textbf{safety checks} to prevent the robot from breaking.
		      This frees the user from this burden and allows them to execute even complex and
		      unpredictable algorithms without surveilling the platform. This opens, for
		      instance, the possibility of training a deep neural network policy during
		      several days directly on the robot.
		\item A synchronized \textbf{history of all the inputs and outputs} of the
		      robot is available to the user and can be logged.
		\item The software is designed such that new robots and simulators can
		      easily be integrated. This may facilitate reuse of algorithms across robots.
	\end{itemize}
	%
	%
	\subsection{Control Modes}
	There are two modes of control supported by our software:
	\begin{definition}[Real-time control]\label{def:real_time_control} By real-time
		control we mean that actions have to be sent to the system at a fixed rate of
		$\Delta$ seconds. This is necessary for real-world dynamical systems, as they
		evolve in time and can hence not wait for the next control to arrive. For
		instance, a falling humanoid robot cannot stand still in the air to wait for
		computation of the next action to be completed.
	\end{definition}%
	%
	%
	\begin{definition}[Non-real-time control]\label{def:non_real_time_control} By
		non-real-time control we mean that a change in the time at which actions are
		applied does not change the outcome, and that actions may take varying amounts
		of time. An example of such a system is a simulator: it will wait until the next
		action is provided and simulation of an action may take varying amounts of time
		for computational reasons. Another example is a a mobile robot which is
		controlled through highlevel actions, such as moving to a specified goal
		location. In between actions the robot will stand still, and execution time of
		an action will vary according to the distance to the goal etc.
	\end{definition}%
	%
	%
	An important feature of our software design is that both modes are supported
	through the same interface, which makes it easy to run the same code in
	simulation and on the real robot.
	%
	\subsection{Overview}
	\begin{figure}[ht]
		\begin{center}
			\centerline{\includegraphics[width=0.9\columnwidth]{figures/software_graphs/robot_interfaces_architecture.pdf}}
			\caption{Software Architecture.  The modules communicate through the RobotData and do not have any direct connections.}
			\label{fig:software_architecture}
		\end{center}
	\end{figure}
	%
	%
	The software framework has three main components (see
	\cref{fig:software_architecture}):
	\begin{itemize}
		\setlength\itemsep{0.1em}
		\item The \textbf{back-end} communicates with the robot through the driver,
		\item the \textbf{front-end} allows the user to control the robot through
		      C++ or Python3,
		\item the \textbf{logger} logs all the inputs and outputs of the robot.
	\end{itemize}
	%
	Each of these components can run in a separate process, which is advantageous
	for computational reasons and it separates the back-end from the user code.
	Since all the commands sent to the robot flow through the back-end, we can
	implement checks which ensure safety no matter what happens in the user code.

	In the following we will describe the front-end and the back-end. The back-end
	section is relevant for readers interested in setting up their own robot, users
	only need to know about the front-end.
	%
	%
	\subsection{Front-end}
	%
	%
	In our design, the only objects that exist from the user-perspective are
	\begin{itemize}
		\setlength\itemsep{0.1em}
		\item a time-series of desired actions $a$ computed by the user,
		\item a time-series of actions actually applied to the robot $a'$, which is
		      identical to $a$ except for potential modifications to satisfy safety
		      constraints,
		\item and a time-series of observations $y$.
	\end{itemize}
	\Cref{fig:variables_sequence} shows the temporal relations of these variables.
	%
	\begin{figure}[ht]
		\begin{center}
			\centerline{\includegraphics[width=0.9\columnwidth]{figures/software_graphs/variables_sequence_with_applied_actions.pdf}}
			\caption{This figure shows the temporal sequence of variables. In real-time mode
				(\ref{def:real_time_control}) there is a fixed control and observation rate
				$\Delta$. Each observation $y_t$ corresponds to a single instant in time
				$t\Delta$, while each action $a_t$ corresponds to a time interval $[t \Delta ,
					(t+1)\Delta)$.}
				\label{fig:variables_sequence}
			\end{center}
		\end{figure}
		%
		The user may perform two operations: They may append actions to the
		desired-action time-series $a$ and they may read from any of the three
		time-series $a, a', y$, where $a',y$ are filled-in by the back-end.

		The user interface implementation is equivalent to the following pseudo code:
		{\small
			\setlength{\abovedisplayskip}{6pt}
			\setlength{\belowdisplayskip}{\abovedisplayskip}
			\setlength{\abovedisplayshortskip}{0pt}
			\setlength{\belowdisplayshortskip}{3pt}
			\begin{align*}
				\texttt{def } & \texttt{append\_desired\_action(\ensuremath{x})}:               \\
				              & \texttt{\ensuremath{a\leftarrow(a,x)}}                          \\
				              & \texttt{\texttt{return len(\ensuremath{a})}-1}                  \\
				\texttt{def } & \texttt{get\_observation(\ensuremath{t})}:                      \\
				              & \texttt{wait until len(\ensuremath{y})>\ensuremath{t}}          \\
				              & \texttt{return }y_{t}                                           \\
				\texttt{def } & \texttt{get\_desired\_action(\ensuremath{t})}:\texttt{as above} \\
				\texttt{def } & \texttt{get\_applied\_action(\ensuremath{t})}:\texttt{as above}
			\end{align*}
		}
		%
		This interface provides access to a synchronized history of all the inputs and
		outputs of the robot. The user has complete freedom how to use this data and
		when to append actions, as long as they make sure to do so on time, before the
		action is needed by the robot. For instance, they may choose to run a typical
		real-time control loop where a new action $a_t$ is computed periodically, or
		they may choose to compute entire action sequences at a lower rate and then
		append them in a burst by repeatedly calling \texttt{append\_desired\_action}.

		If the user attempts to access a future observation through
		\texttt{get\_observation($t$)}, this function will wait and return as soon as
		this observations is acquired. For instance, in the case of real-time control
		(\cref{def:real_time_control}), if the call \texttt{get\_observation($2$)} is
		made at time $<2\Delta$ seconds, the function will wait until the observation
		$y_2$ is received at time $2\Delta$ seconds and then return. This feature allows
		for synchronization with the real system.

		The function \texttt{append\_desired\_action($x$)} will append action $x$ to the
		action time-series. For convenience it returns the timeindex of the appended
		action, but if the user keeps track of timeindices externally, the return value
		can be ignored. The instant of the first call to
		\texttt{append\_desired\_action} marks time $0$, after which the back-end will
		start filling in the $a', y$ timeseries and expect the action timeseries $a$ to
		be filled in by the user.

		A basic control loop can be written as follows:
		\begin{example}[Basic control loop]\label{ex:basic_control_loop} {\small
				\setlength{\abovedisplayskip}{6pt}
				\setlength{\belowdisplayskip}{\abovedisplayskip}
				\setlength{\abovedisplayshortskip}{0pt}
				\setlength{\belowdisplayshortskip}{3pt}
				\begin{align*}
					  & \texttt{robot.append\_desired\_action(\ensuremath{a_{0}})}                 \\
					  & \texttt{for \ensuremath{t} in \ensuremath{(0,...,T)}:}                     \\
					  & \qquad\texttt{\ensuremath{y_{t}} = robot.get\_observation(\ensuremath{t})} \\
					  & \qquad\texttt{\ensuremath{a_{t+1}} = some\_policy(\ensuremath{y_{t}})}     \\
					  & \qquad\texttt{robot.append\_desired\_action(\ensuremath{a_{t+1}})}
				\end{align*}}
			%
			No explicit wait is necessary, synchronization with the back-end is ensured
			through the call to \texttt{robot.get\_observation(\ensuremath{t})}, which will
			wait until the back-end has appended $y_t$.

			This control loop is valid for both real-time (\ref{def:real_time_control}) and
			non-real-time (\ref{def:non_real_time_control}) control:\\
			If the back-end is running in \textbf{real-time mode}, it will add observations
			$y_t$ periodically and the loop above will hence run at a fixed rate, unless the
			call \texttt{some\_policy} is too slow. If that is the case, the back-end will
			detect that the user did not append the next action on time and it will shut
			down the robot and raise an error. Alternatively, the back-end can be configured
			to simply repeat the previous action if the next action has not been appended on
			time.\\
			In contrast, if the back-end is running in \textbf{non-real-time
			mode}, it will wait for the next action, and the function
			\texttt{append\_desired\_action} may be called with arbitrary delay.
			Similarly, the execution of actions may take varying amounts of
			time, and hence the call to \texttt{get\_observation} will not
			return at a predetermined time. This mode makes sense e.g. for
			simulation, where the simulator and the policy add actions and
			observations whenever they are done with their respective
			computations.
		\end{example}

		If we wish to control the robot at a lower rate, this can also be implemented
		very easily:
		\begin{example}[Basic control loop at a reduced
			rate]\label{ex:basic_control_loop_at_lower_rate} {\small
				\setlength{\abovedisplayskip}{6pt}
				\setlength{\belowdisplayskip}{\abovedisplayskip}
				\setlength{\abovedisplayshortskip}{0pt}
				\setlength{\belowdisplayshortskip}{3pt}
				\begin{align*} & \texttt{robot.append\_desired\_action(\ensuremath{a_{0}})}\\
					  & \texttt{for \ensuremath{t} in \ensuremath{(0,...,T)}:}                                         \\
					  & \qquad\texttt{\ensuremath{y_{t}} = robot.get\_observation(\ensuremath{t}\ensuremath{\cdot k})} \\
					  & \qquad\texttt{\ensuremath{a_{t+1}} = some\_policy(\ensuremath{y_{t}})}                         \\
					  & \qquad\texttt{for \_ in range(\ensuremath{k})}                                                 \\
					  & \qquad\texttt{\ensuremath{\qquad}robot.append\_desired\_action(\ensuremath{a_{t+1}})}
				\end{align*}}
			Here, the control frequency is reduced by a factor of $k$. Note that the
			index $t$ refers to the control cycle here, not the robot cycle.
		\end{example}
		%
		%
		%
		\subsection{Relation to the Standard Reinforcement Learning Framework}
		As we shall see, there is a gap between the standard Markov decision process
		(MDP) formulation of reinforcement learning and the presented framework. This is
		due to the realtime constraints we face when working with real robots. Here we
		show show to close this gap, which allows us to e.g. wrap our interface into
		the OpeanAI Gym interface.

		\paragraph{Some Simplifications:} For simplicity of exposition, we will only
		consider the case where the observations $y_t$ are Markovian, the extension to
		the non-Markovian case is straightforward. In addition, we will not treat the
		applied actions $a'_t$ explicitly anymore, as they do not affect the independence
		structure between the desired actions $a_t$ (henceforth referred to simply as
		actions) and observations $y_t$. They can either be simply ignored or they can
		be added to the observations if one wishes to give the RL algorithm access to
		this information.

		\subsubsection{Dependence Structure of a Real-Time System}
		We can deduce the dependences between variables from their temporal ordering
		represented in \cref{fig:variables_sequence}. Naturally, any variable can only
		causally depend on variables which precede it in time. However, in real-time
		systems there is the additional constraint that the controller needs some time
		to compute actions, hence they may not depend on observations they coincide with
		in time. More precisely, action $a_t$ cannot depend on $y_t$, since $a_t$ starts
		at precisely the time when observation $y_t$ is acquired. In fact, we already
		took this into account in \cref{ex:basic_control_loop}, where $a_{t+1}$ is a
		function of $y_t$ rather than $y_{t+1}$.
		%
		\begin{figure}[tb]
			%
			\begin{center}
				\centerline{\includegraphics[width=0.5\columnwidth]{figures/software_graphs/variables_dependencies_real.pdf}}
				\vspace{-0.1cm}\caption{\vspace{-0.1cm}The Bayes net showing the dependence
				between variables in a real-time system.}
				\label{fig:variables_dependencies_real}
			\end{center}
			%
		\end{figure}
		%
		These considerations lead to the Bayes net in
		\cref{fig:variables_dependencies_real}. By comparing with the Bayes net of an
		MDP in \cref{fig:variables_dependencies_gym} we see that we cannot simply
		identify the state $s_t$ with $y_t$, even in the Markovian case. In the
		following, we propose two ways in which this structure can be mapped to a
		standard MDP.
		%
		\begin{figure}[tb]
			%
			\begin{center}
				\centerline{\includegraphics[width=0.5\columnwidth]{figures/software_graphs/variables_dependencies_gym.pdf}}
				\caption{The dependence structure (Bayes net) of an MDP.}
				\label{fig:variables_dependencies_gym}
			\end{center}
			%
		\end{figure}
		%
		%
		\subsubsection{State Augmentation}
		By defining the state in the slightly counter-intuitive way $s_{t+1} = (y_{t},
		a_{t})$, we retrieve the dependence structure of the MDP, see
		\cref{fig:variables_dependencies_compromise}.
		%
		\begin{figure}[tb]
			%
			\begin{center}
				\centerline{\includegraphics[width=0.5\columnwidth]{figures/software_graphs/variables_dependencies_compromise.pdf}}
				\caption{A Bayes net showing the dependence between variables obtained from
					\cref{fig:variables_dependencies_real} by defining the state as
					$s_{t+1}=(y_t , a_t)$. This yields the same dependency structure as in
					\cref{fig:variables_dependencies_gym}.}
				\label{fig:variables_dependencies_compromise}
			\end{center}
			%
		\end{figure}
		%
		This allows us to apply any of the RL algorithms which have been formulated for
		standard MDPs to our system. \citet{ramstedt2019real} use a similar trick and
		propose a method to take the resulting structure into account when performing
		RL.

		Given these considerations, we can now define an OpenAI Gym environment for our
		robot interface. For our discussion here, only the step function is relevant:
		\begin{example}[Gym Env - State
			Augmentation]\label{para:gym_state_augmentation}
			%
			%
			{\small
				\setlength{\abovedisplayskip}{6pt}
				\setlength{\belowdisplayskip}{\abovedisplayskip}
				\setlength{\abovedisplayshortskip}{0pt}
				\setlength{\belowdisplayshortskip}{3pt}
				\begin{align*}\texttt{def } & \texttt{step(\ensuremath{a}):}\\
					  & \texttt{\ensuremath{t} = robot.append\_desired\_action(\ensuremath{a})} \\
					  & \texttt{\ensuremath{y} = robot.get\_observation(\ensuremath{t})}        \\
					  & \texttt{\ensuremath{s'} = \ensuremath{(y,a)}}                           \\
					  & \texttt{return \ensuremath{s'}, some\_other\_stuff}
				\end{align*}
			} Where the $'$ in $s'$ denotes the subsequent time index.
		\end{example}
		%
		%
		This opens up the possibility of applying numerous implementations of RL
		algorithms out-of-the-box to our platform. For instance, we can directly apply
		the algorithms from \cite{stable-baselines}, \cite{garage} and \cite{baselines}, as we will demonstrate in the experimental section.

		It is often the case that we want to operate at a control rate which is lower
		than the communication rate of the robot. For instance, in many tasks it is not
		necessary to control the proposed system at $1000$ Hz, we may want to control it
		at e.g. $100$ Hz. We can define an according OpenAI Gym environment.

		\begin{example}[Gym Env - State Augmentation at Reduced
			Rate]\label{para:gym_state_augmentation_reduced} The logic here is exactly the
			same as in \cref{para:gym_state_augmentation}, except that each action is
			applied $k$ times, where $k$ is the rate reduction factor. {\small
				\setlength{\abovedisplayskip}{6pt}
				\setlength{\belowdisplayskip}{\abovedisplayskip}
				\setlength{\abovedisplayshortskip}{0pt}
				\setlength{\belowdisplayshortskip}{3pt}
				\begin{align*}\texttt{def } & \texttt{step(\ensuremath{a}):}\\
					  & \texttt{\ensuremath{t} = robot.append\_desired\_action(\ensuremath{a})}   \\
					  & \texttt{for \_ in range(\ensuremath{k-1\text{):}}}                        \\
					  & \texttt{\ensuremath{\qquad}robot.append\_desired\_action(\ensuremath{a})} \\
					  & \texttt{\ensuremath{y} = robot.get\_observation(\ensuremath{t})}          \\
					  & \ensuremath{s'}\texttt{ = }\ensuremath{(\ensuremath{y},\ensuremath{a})}   \\
					  & \texttt{return \ensuremath{s'}, some\_other\_stuff}
				\end{align*}}
		\end{example}
		%
		%
		\subsubsection{Approximate Mapping for Control at Lower Rates}
		%
		%
		%
		In the case of low-rate control, we can also define an approximate
		mapping of the real-time system to
		a standard MDP which does not
		require state augmentation.
		This may in some cases be preferable, because it is more intuitive
		and it does not lead to an observation space of increased
		dimensionality.
		To see how this can be done, consider
		\cref{fig:variables_dependencies_approximation}.
		%
		\begin{figure}[tb]
			%
			\begin{center}
				\centerline{\includegraphics[width=0.95\columnwidth]{figures/software_graphs/variables_dependencies_approximation.pdf}}
				\caption{When we control at a reduced rate each control action
				$a_t$ is applied $k$ times and hence produces $k$ robot actions
				$a^r_{(t-1)k+1:tk}$. To approximately map this to the dependence
				structure of a standard MDP
				(\cref{fig:variables_dependencies_gym}), we can ignore the
				light-blue arrows.}
				\label{fig:variables_dependencies_approximation}
			\end{center}
			%
		\end{figure}
		%
		If we control at a rate reduced by a factor $k$, each control action $a_t$ is
		applied $k$ times and hence affects $k$ observations. For $k$ sufficiently
		large, a reasonable approximation is to ignore the dependence of the last of
		these $k$ observations on $a_t$, see
		\cref{fig:variables_dependencies_approximation}. Doing so leads again to
		standard MDP dependencies, see
		\cref{fig:variables_dependencies_approximation_condensed}.
		%
		\begin{figure}[tb]
			%
			\begin{center}
				\centerline{\includegraphics[width=0.35\columnwidth]{figures/software_graphs/variables_dependencies_approximation_condensed.pdf}}
				\caption{This is the dependence structure of the states $s$ and actions
					$\bar{a}$ implied by \cref{fig:variables_dependencies_approximation},
				ignoring the light-blue arrows.}
				\label{fig:variables_dependencies_approximation_condensed}
			\end{center}
			%
		\end{figure}
		%
		We can now define again an according OpenAI Gym environment:

		\begin{example}[Gym Env - Approximation at Reduced
			Rate]\label{para:gym_approximate_reduced} {
				\small
				\setlength{\abovedisplayskip}{6pt}
				\setlength{\belowdisplayskip}{\abovedisplayskip}
				\setlength{\abovedisplayshortskip}{0pt}
				\setlength{\belowdisplayshortskip}{3pt}
				\begin{align*}\texttt{def } & \texttt{step(\ensuremath{a}):}\\
					  & \texttt{for \_ in range(\ensuremath{k\text{):}}}                                           \\
					  & \texttt{\ensuremath{\qquad}\ensuremath{t} = robot.append\_desired\_action(\ensuremath{a})} \\
					  & \texttt{\ensuremath{s'} = robot.get\_observation(\ensuremath{t})}                          \\
					  & \texttt{return \ensuremath{s'}, some\_other\_stuff}
				\end{align*}
			}
		\end{example}
		Here, the time index of the last appended robot action is $t$, and we return
		$s'=y_t$. Hence, all robot actions up to time index $t-1$ have to be executed
		before we can retrieve this observation, and only then the step function will
		return. Nevertheless, the step function will have to be called again before
		robot action $t+1$ can start. This means that the controller will only have one
		robot cycle to compute the next action, despite the lower control rate.

		Summarizing, we can say that this approximation is a good solution if the
		computation of the next action is substantially faster than the control rate.
		Otherwise, the solution in \cref{para:gym_state_augmentation_reduced} is
		preferable, since there the controller may use the entire control cycle to
		compute the next action.
		%
		%
		\subsection{Back-end and Driver}
		To integrate a new robot into our software framework, the only necessary steps
		are to define an \texttt{Action}, an \texttt{Observation}, and a
		\texttt{RobotDriver} class in C++ with the following functions: {\small
			\setlength{\abovedisplayskip}{6pt}
			\setlength{\belowdisplayskip}{\abovedisplayskip}
			\setlength{\abovedisplayshortskip}{0pt}
			\setlength{\belowdisplayshortskip}{3pt}
			\begin{align*}
				  & \texttt{RobotDriver}                                            \\
				  & \texttt{\{}                                                     \\
				  & \qquad\texttt{Observation get\_latest\_observation();}          \\
				  & \qquad\texttt{Action apply\_action(Action \ensuremath{a_{t}});} \\
				  & \texttt{\}}
			\end{align*}
		} The first function is self-explanatory. The second function takes a desired
		action as input, it may perform safety checks, and then returns the applied
		action.

		The back-end takes care of calling the right functions at the right times. It
		will read the desired actions, appended by the user to the timeseries $a$, and
		pass them to the \texttt{RobotDriver}, and it will fill the observation $y$ and
		applied action $a'$ timeseries with the outputs from the \texttt{RobotDriver}.
		For details we refer the interested reader to the documentation in the code.
		%
		%
		\subsubsection{Implementation of the TriFinger Robot}
		\label{sec:trifinger_implementation}
		The implementations of the classes above for the TriFinger robot are:
		\begin{itemize}
			\item \texttt{Observation}: Joint position, velocity and torque of each
			      joint. The camera images are retrieved through a separate
			      interface with the same structure, since they arrive at a different rate.
			\item \texttt{Action}: Torque to be applied at each joint and
			      optionally a joint position along with the gains of a PD
			      controller. The driver will then sum the torque with the
			      feedback from the user-specified position controller.
			\item \texttt{RobotDriver}: Here, the implementation of \texttt{get\_latest\_observation}
			      returns the latest measurements which were received from the motorboards
			      through CAN, see \cite{Grimminger2020-tl} for details. The
			      \texttt{apply\_action} makes sure that the joint velocity and torques do not
			      become too large, since otherwise the robot may break or the motors motors
			      may overheat. It then sends the modified action to the motorboards and
			      returns it.
		\end{itemize}
		%
		%
		%
		%
		\subsection{Safety Checks}
		While the robot hardware is very robust, some additional software safety
		checks are necessary for ensuring that the user code cannot break the
		robot. There are five main checks we perform:
		\begin{itemize}
			\item The backend continuously monitors the timing of received actions
			in a real-time loop. If the expected rate of 0.001s is exceeded
			substantially, the robot is shut down.
			\item There is an additional time-out on the motor board. For instance, in
			case the computer crashes and the motor board does not receive any messages
			for some time it will shut down the motors.
			\item If a joint exceeds a predefined angle, it is brought back into the
			admissible range using a PD controller. This prevents e.g. collision of the
			fingers with the electronics.
			\item We determined the maximum admissible current to prevent overheating of
			the motors, and we ensure that this current is not exceeded by clipping the
			desired torque if necessary.
			\item We simulate joint damping (D-gain) to ensure that the fingers do not
			reach excessive velocities.
		\end{itemize}

		Note that we do not prevent collisions (except with the electronics), since the
		hardware design is robust to collisions. In addition, the software design is
		such that users may easily implement their own robot or  even task-specific
		safety checks.
		%
		\subsection{Relation to Robot Operating System (ROS)}
		The core software described above is independent of ROS. We use catkin (which
		can be installed without ROS) for compilation. Further, we use ROS in some of
		the robot-specific packages for peripheral purposes, such as locating other
		packages. Finally, we use Xacro (which is part of ROS) for defining the URDF
		robot model of the TriFinger.

		%

		\section{Experiments}
		\begin{figure*}[h]
			\centering
			\includegraphics[width=0.98\textwidth]{figures/trifinger_edu_upward/horizontal_merged_upward.jpg}
			\centering
			\caption{TriFinger performing the pickup task.}
			\label{fig:upward_motion}
		\end{figure*}

		The purpose of this section is not to improve the state-of-the art in robotic
		manipulation, but rather to illustrate the capabilities of the proposed hardware
		and software. Each experiment highlights different aspects:
		\begin{itemize}
			\item \Cref{sec:exp:optimal_control} demonstrates the \SI{1}{kHz} real-time
			torque-control abilities and the ease-of-use of the software interface for
			classical control loops.
			\item \Cref{sec:exp:reinforcement_learning} shows that the backend safety
			features allow for deep reinforcement learning from scratch, without any
			safety checks on the user side. This experiment also shows that the hardware
			is robust against collisions, which will necessarily occur during the
			learning of manipulation tasks. In addition, this use case shows that the
			software interface allows for application of out-of-the-box implementations
			of deep RL methods.
			\item In \cref{sec:exp:throwing}, we perform a throwing experiment to show
			that the actuators allow for highly-dynamic motions.
			\item In \cref{sec:exp:fine_manipulation} we show through
			demonstration experiments that the platform is capable of fine-manipulation.
			\item Finally in \cref{sec:exp:endurance} we discuss some experiments assessing
			the durability of the design.
		\end{itemize}
		Videos of the experiments are available at \cref{google_site}.
		%
		%
		\subsection{Optimal Control}\label{sec:exp:optimal_control} Controlling robot
		interactions with the environment is challenging due to the unilateral nature of
		the contact constraints and the stiff behavior of contact forces. We tackle this
		problem, for picking up and moving a cube, with the control loop depicted in
		\cref{fig:control_loop_qp}.
		\begin{figure}[H]
			\centering
			\includegraphics[width=\linewidth]{figures/optimal_control/flow.pdf}
			\caption{Control loop at 1kHz with force optimization.}
			\label{fig:control_loop_qp}
		\end{figure}
		%
		\paragraph{Center of Mass Wrench:}
		The first step is to compute the wrench (force and moment) which need to be
		applied to the object to maintain it on the desired trajectory. We do so using a
		simple PD law
		\begin{align}
			F_{\text{com}} & = P \delta x_{\text{com}} + D \delta \dot{x}_{\text{com}} - m \Vec{g}\\
			M_{\text{com}} & = P \delta q +  D \delta \omega
		\end{align}
		which will compute a force $F_{\text{com}}$ and moment $M_{\text{com}}$
		at the object center-of-mass, to correct for errors in position ($\delta
		x_{\text{com}}$), linear velocity ($\delta \dot{x}_{\text{com}}$),
		orientation ($\delta q$) and angular velocity ($\delta \omega$). $P,D$
		are the controller gains, $\Vec{g}$ is the gravity vector and $m$ is the object mass.
		%
		%
		\paragraph{Force Optimization:} The next question is what forces the finger tips
		must apply to the object in order to achieve the desired wrench calculated
		above. Therefore, we formulate a quadratic program to find optimal distribution
		of contact forces:
		\begin{align}
			F_\text{tip}=\arg\min_{y}\quad & \frac{1}{2}y^{\top}y       \\
			\text{subject to}\quad         & G y\le h                   \\
			                               & A y=\left(\begin{array}{c}
			F_{\text{com}}\\
			M_{\text{com}}
			\end{array}\right)
		\end{align}
		%
		where the optimization variable $y$ is the stack of contact forces at the three
		finger tips expressed in the
		local frame of the object. The equality constraint ensures that the tip forces
		produce the desired wrench on the object ($A$ transforms tip forces to
		center-of-mass wrench given known contact locations, see e.g.
		\cite{murray2017mathematical}).
		%
		\begin{figure}[H]
			\centering
			\includegraphics[width=.6\linewidth]{figures/optimal_control/friction_cone.png}
			\caption{The friction cone ($ \lvert f_p \rvert \leq
				\frac{\mu}{\sqrt{2}} f_n$) can be approximated using linear
				inequalities such that the approximation lies inside the actual cone.
				$\mu$ is the static
				friction coefficient and $f_n, f_p$ are the force components normal
			and parallel to the contact surface, respectively.}
			\label{fig:friction_cone}
		\end{figure}
		%
		The inequality constraint enforces that the tips can push but not pull, and it
		ensures that the object does not slip. It is a linear approximation to the
		friction cone (see \cref{fig:friction_cone} and \cite{murray2017mathematical}
		for details).
		%
		%
		\paragraph{Impedance Control:}
		The final step is to compute the torques $\tau$ to be applied at the robot
		joints to produce the desired tip forces $F_\text{tip}$ computed in the previous
		step. This can easily be achieved via Jacobian ($J$) transpose control. For
		stabilization, we also add a feedback on the fingertip position and velocity
		errors $\delta x_\text{tip}, \delta\dot{x}_\text{tip}$ (the desired tip
		trajectories are obtained given the desired object trajectory and known contact
		locations). This yields a simplified version of the impedance controller
		introduced by \citet{hogan1984impedance}
		%
		\begin{align}
			\tau = J^{\top} \left(F_\text{tip} + P'\delta x_\text{tip} +  D' \delta \dot{x}_\text{tip}\right)
			\label{eq:finger_impedance}
		\end{align}
		%
		where $P'$ and $D'$ are hand-tuned controller gains.
		%
		%
		\subsubsection{Results}
		We apply this methodology to two tasks: Lifting a cube \SI{20}{cm} along a
		vertical line (see \cref{fig:upward_motion}) and sliding that same cube in a
		circle along the table. As can be seen in the videos at \cref{google_site}, the
		forces applied to the object are appropriate for moving it along the desired
		trajectory without slippage.
		%
		%
		%
		\subsection{Reinforcement Learning}\label{sec:exp:reinforcement_learning}
		\begin{figure}[t!]
			\centering

			\centering
			\includegraphics[width=0.8\columnwidth]{figures/plot_tip_position/start.pdf}
			\caption{Beginning of training}
			\includegraphics[width=0.8\columnwidth]{figures/plot_tip_position/mid_ellipse.pdf}
			\caption{After 200 episodes}
			\centering
			\includegraphics[width=0.8\columnwidth]{figures/plot_tip_position/end_ellipse.pdf}
			\caption{End of training}
			\caption{Trajectories of the finger tip relative to the goal position in the xy-plane. Each trajectory corresponds to one episode (start: cyan cross, end: pink dot). The yellow ellipse marks 2\,cm position error on each axis.}
			\label{fig:plot_learning_tip_position}
		\end{figure}
		%
		%
		%
		We illustrate the suitability of the platform for real-time reinforcement
		learning by training a DDPG \cite{ddpg} agent from scratch on a reaching task,
		using the DDPG implementation from stable-baselines \cite{stable-baselines}.

		In this task, the goal is for each finger-tip to reach a randomly-sampled target
		location as accurately as possible.
		The episode length is set to 2 seconds. The observation space of the policy
		consists of joint positions, joint velocities, and target positions
		for each finger specified in task space. The action is the desired joint
		configuration of the fingers (the software back-end provides a PD controller,
		see \cref{sec:trifinger_implementation}).
		The reward at each timestep is the negative Euclidean distance between the
		end-effector and the target. We train the system for 700 episodes, corresponding
		to 23 minutes of execution on the robot, plus a few minutes of computation time
		used by DDPG.

		At the beginning of training, the fingers' motions are jerky, and they often
		collide, see the video at \cref{google_site}. As the training progresses, the
		motion becomes smoother and more accurate, see
		\cref{fig:plot_learning_tip_position}. At the end of training, the fingers are
		able to reach the target positions consistently within an error of about 2cm.
		%
		%
		%
		\subsection{Throwing}\label{sec:exp:throwing} To showcase the ability of
		performing highly dynamic tasks, we execute throwing motions recorded through
		kinesthetic teaching (i.e. the motion was demonstrated by guiding the robot
		fingers). The videos at \cref{google_site} show that the TriFinger is able to
		throw light objects several meters. We expect that using appropriate
		controllers, instead of kinesthetic teaching, one could improve considerably on
		these results.
		%
		%
		%
		\balance
		\subsection{Fine Manipulation}\label{sec:exp:fine_manipulation} Finally, to
		illustrate the dexterity of the platform, we perform several fine manipulation
		motions, which, as above, were demonstrated through kinesthetic teaching. As can
		be seen in the videos at \cref{google_site}, the experiments include flipping a
		cube, turning it with one finger while the others hold it, balancing a flat
		cuboid on its side, and picking up a pen and drawing.
		%
		%
		%
		\subsection{Durability Experiments}\label{sec:exp:endurance}
		We ran durability experiments on a single finger, executing a fast motion in
		free space. In the first experiment a timing belt broke after 79 days of
		continuous operation. This may be partially due to the nonuniform stress put
		on the timing belt by such repetitive motions, which would be less of an
		issue in realistic operation. In the second experiment the shell of the
		center link broke after 72 days of continuous operation, the design has been
		improved since to avoid such breakage.

		In addition, we performed some tests on a TriFinger version used in a robot
		competition hosted at our
		institute\footnote{\url{https://real-robot-challenge.com/}}. That version,
		called TriFingerPro, was developed for internal use and is too complex for
		open-sourcing. Nevertheless, it is essentially identical in terms of
		kinematics and actuation, and we would expect its durability to be
		indicative of the open-source version. We executed random motions, including
		collisions between fingers and with external objects, on two of those
		platforms for one week continuously without breakage.

		These results are naturally not statistically significant, and the
		durability most likely depends on the material used for the 3D printing.
		Nevertheless, they are promising and we believe that we can further improve
		durability by fixing weak points in the design as they emerge.
		%
		%
		\section{Conclusion}
		We presented an open-source robotic platform with novel hardware and
		software. We have shown its i) capabilities for dexterous manipulation,
		ii) suitability for deep RL from scratch thanks to the robustness of the
		hardware and safety checks of the software and iii) ease of use for both
		real-time optimal control as well as deep RL. We hope that these
		factors, in combination with the simple and inexpensive hardware design,
		will motivate many researcher to adopt this platform as a shared
		benchmark for real-world dexterous-manipulation. This could lead to a
		more coordinated effort among different labs and the generation of
		orders of magnitude more real-robot data than was possible thus far.

		\balance

		\newpage

redpajama_set_name: RedPajamaArXiv


text:
Publish Her Story
The Power of Hashtags: Two Sisters and Their Company, Kids for Culture, Land a Dream Retail Deal
By Kelly Westhoff
When Kristen O'Meally and Lamia Haley launched their toy company, Kids for Culture, they never imagined that one year later they would be collaborating with Walmart.
"This is a story about the power of hashtags," Kristen said. "A buyer from Walmart saw our products on Instagram and sent us an email through the contact page of our website."
Kids for Culture hadn't even been in business six months. Before she replied to that message, Kristen Googled the buyer's name and email address to see if it was real.
Kids for Culture started with four products: a deck of ABC affirmation flashcards, a memory game, stickers, and a deck of emotional literacy cards that feature faces displaying different feelings. All the products are specifically designed for preschool-aged children. Flashcards and game pieces are large and thick, making them easy for small hands to grab and difficult to rip. Not only do the products showcase kids with a variety of skin tones, but they also make use of bold and soothing colors to attract children.
Before launching their products, Kristen and Lamia—sisters raised in Kentucky—tested them in the real world. They gave the flashcards and game pieces to their nieces and nephews. Lamia had a young child at home with whom she shared the cards. And Kristen, the director of a preschool, brought the items into her classrooms. There, she said, "kids, educators and parents all gravitated towards them."
Among the Kids for Culture products, Walmart was most interested in the ABC affirmation flashcards. Each card prominently features one letter of the alphabet along with a positive affirmation: "A is for Amazing—I am an amazing person," "B is for Believe—I believe in myself," "C is for Change—I can change the world." Kids for Culture started selling the affirmation cards on its own website. Just a few months later, the cards were also available on Walmart's website and earning positive customer reviews.
"Walmart was a fan of the affirmation cards, and they asked us to put a spin on the product," Lamia said. "They wanted a product pair that included a plush toy. They really left it up to us as to what this would look like. We did get some feedback, but it was minimal."
Kids for Culture created two new products for Walmart, which are packaged as a gift set—a 16-page ABC board book that echoes its ABC flashcards and is filled with positive affirmations for young children, and a line of plush baby dolls in three different skin tones called Positive Pals. Each doll bears a positive message on its soft belly: "I am Amazing," "I am Kind," "I am Loved."
Many small business owners dream of collaborating with big retailers to sell more products, and Kristen and Lamia's partnership with Walmart has enabled them to get their books and toys into the hands of more kids and families. But beyond that, it has provided an opportunity to introduce a broader audience to the Kids for Culture mission, which is ultimately diversity in literature, media, games and toys.
"We really want children to see themselves reflected in our products," Kristen said. "But we also want them to be exposed to others. We don't just highlight one race or one demographic or one culture. Our products are an opportunity for children to be introduced to diversity."
To learn more about Kristen and Lamia's company, visit KidsForCulture.com and follow @kidsforculture on Instagram.
Photo credit: Kids for Culture
Kelly Westhoff is a freelance writer who always enjoys talking to women business owners. She finds their energy contagious and their stories fun to share.
Feature Story: Joy McBrien, Fair Anita
Publish Her Among 40 Women-Owned Businesses to Receive National 'Lifted Up by HerStory' Honor
Publish Her Announces the Release of Children's Book 'The Mitten Thief' by Outdoor Enthusiast and Entrepreneur Jill Kaufenberg
Join Our Growing Team! Publish Her Seeks Freelance Book Editing and Design Experts
We Sparkle's 'Sparkle Assistant' Uses Technology to Save Small Business Owners Time and Money
Business Owner and Author Mikaila Ulmer's Social Enterprise and Guidebook Inspire Entrepreneurs of All Ages to Pursue Their Purpose

redpajama_set_name: RedPajamaCommonCrawl


At Wester Ross, we are part of our salmon's journey from egg to processing.
The salmon is an anadromous animal meaning that its juvenile phase is spent in freshwater and then it undergoes physiological changes to allow it to spend the majority of its life in seawater. It then returns to freshwater to breed.
After hatching from ova, the first phase of the life cycle of a salmon is the alevin stage. At this stage, the fish uses the nutrients in its yolk sack for growth. Once the yolk sac is exhausted, the young salmon reaches the 'free-swimming' fry stage; at which point they begin to eat other food.
We nuture our fry at our Elphin Freshwater Site in water from the hills, often snow melt, until they are ready to smolt.
Eventually, dark, oval parr marks develop along its flank as it reaches the final freshwater stage. This is when the fry becomes a parr. During the spring the increasing day length triggers a physiological change to allow the fish to survive in salt water. The parr loses its parr-marks and starts swimming with the current. This change is known as smoltification, and at this point the salmon can be transferred to seawater to grow.
At smoltation, our salmon are moved to be stocked in either one of two seawater sites — Ardmair Seawater Site in even years, Corry Seawater Site in odd years. Alternating the sites provides a fallow period, allowing natural recovery and regeneration of the site and preserving the environment.
Our salmon spend the rest of their time at our Ardessie Seawater Site. This is our grow-out site where, once the fish have been graded, the smaller fish are moved to grow to selling weight.
Our salmon are hand-fed a responsibly-sourced diet and a high-quality, sustainable fishmeal, developed specifically for Wester Ross. The fishmeal is made from offcuts of the filleting process of fish bound for human consumption originating from the North-east Atlantic only.
The welfare of our salmon is critical. That is why each of our salmon is entirely hand reared.
Careful, humane harvesting methods in accordance with RSPCA Assured guidelines avoid stress to the fish and ensure the best quality salmon.
Our processing facility is in Dingwall, where our salmon are cleaned and filleted by hand to customer specifications. Despatch from the processing plant is within 24 hours of harvest.


text:
Alienware and Tobii Continue to Bring Eye Tracking to Gamers with the Launch of the 2018 Alienware 17 Gaming Laptop
Apr 09, 2018 Tobii Tech
At the game show PAX East, Alienware announced their updated 17-inch gaming laptop with cutting edge hardware and software. The 2018 Alienware 17 will retain industry-leading features, including integrated Tobii eye tracking technology, while refreshing other components to increase system performance and improve user experience.
"Alienware is known for innovation and for adding the most advanced technologies and capabilities to its gaming notebook and desktop products," said Oscar Werner, President of Tobii Tech. "Today we are thrilled to confirm that Alienware remains committed to providing its customers with the latest technology-- including Tobii eye tracking hardware and software."
Tobii and Alienware lead the way introducing eye tracking for the award-winning Alienware 17 gaming laptop in late 2016. Today, over 100 gaming titles include eye-tracking support, including many top-tier titles like Ubisoft's latest smash hit Far Cry 5. Recently, the companies have collaborated closely to bring the benefits of eye tracking to a wide range of gamers, pro teams, and esports audiences.
In January 2018, the finals of the Eleague Major for CS:GO broke the record for the most concurrent viewers of a single Twitch stream with over 1,130,000 viewers. During the stream, Eleague used technology from Alienware and Tobii to give viewers insight into the minds of professional gamers, showing incredible reaction times and top-level tactics. In March of 2018, leading professional esports organization Team Liquid shared that they have added Tobii eye tracking hardware and software to their new Alienware Training Facility, upping the standards for esports training.
"At Alienware we are focused on bringing our customers the most sophisticated capabilities and technologies available to make sure they can compete at the highest levels," said Christopher Sutphen, global marketing director at Alienware. "We are pleased to continue our work with Tobii, the world's leader in eye tracking technology, to bring the powerful advantage of eye tracking to our community and to create a world-class gaming experience."
Read more about how professional teams, broadcasters, and leagues are using eye tracking here.
Alienware and Tobii Continue to Bring Eye Tracking to Gamers - press release - 9 April 2018
Alienware och Tobii fortsätter att ta eyetracking till spelare - press release - 9 April 2018
Ben Conrad, Vice President of Media and Influencers, Tobii Tech, Phone +1 (650) 224-6261, email: ben.conrad@tobii.com
About Tobii
Tobii is the global leader in eye tracking. Our vision is a world where all technology works in harmony with natural human behavior. Tobii operates through three business units: Tobii Dynavox makes specially designed computers that are controlled by eye movement or touch screens for use by people with special needs due to spinal cord injuries, CP, ALS or other medical conditions. Tobii Pro develops and sells eye-tracking equipment and services used today by more than 3,000 companies and 2,000 research institutions, including all of the world's 50 highest ranked universities. Tobii Tech further develops Tobii's technology for new volume markets, such as computer games, personal computers, virtual reality and smartphones. Tobii is headquartered in Sweden and is listed on Nasdaq Stockholm (TOBII). The group has over 900 employees. For more information, please visit www.tobii.com.


text:
Home Popular India vs England: 'We deserved to lose' – Virat Kohli after Lord's...
India vs England: 'We deserved to lose' – Virat Kohli after Lord's Test defeat
Hindustan Times – Virat Kohli conceded that the Indian cricket team deserved to lose the second Test match against England at Lord's after the visitors were beaten by an innings and 159 runs.
Virat Kohli conceded that the Indian cricket team deserved to lose the second Test match against England at Lord's after the visitors were beaten by an innings and 159 runs.
Having conceded the first-innings lead of 289 runs, India's batting line-up could register only 130 in their second innings to add to their first innings total of 107.
ALSO READ: Virat Kohli's captaincy still 'a work in progress' – Clive LlyodWhen asked about the rain-affected conditions, Kohli said that they should not blame the conditions and England deserved to win the game because of their relentless approach on the field.
Kohli did not take the field on Day 4 due to back issues and was forced to bat at No.
When asked about it, the Indian cricket team skipper said that the issue will need a bit of rehabilitation but the back issue will not be posing a bigger threat.
Previous articleFloundering India battle to avoid innings defeat to England – live!
Next articleAnderson's haul
England bowler Jofra Archer 'grieved during cricket World Cup' after cousin was shot dead
New Zealand coach: ICC should consider trophy share in future after final drama
Three bookies arrested from Gurugram hotel
O'Keeffe announces return to cricket TV
Australia Cric Editor - July 12, 2018
India match sells out after emotional plea by captain
Patel spins New Zealand to thrilling win in first Test over Pakistan
West Indies all-rounder Carlos Brathwaite to play for Kent in first four matches of T20 competition- Firstcricket News, Firstpost
Can cricket become a global game? Here's Tendulkar's opinion – Rediff.com Cricket
India Cric Editor - June 11, 2018
Richie Benaud to be honoured as 'Legend' of Australian Sport
Popular Cric Editor - October 6, 2018
England 'put us to the sword' says Australian captain Tim Paine after record-breaking defeat
Sporting News Cric Editor - June 20, 2018
Pak blind women team for Nepal series named
Popular Cric Editor - January 1, 2019


text:
What it's like to work in digital health
Careers with our NHS
#datasaveslives
data saves lives
Digital Health or Health Tech is a rapidly developing area and we need young people to consider working in this exciting field. This Careers Zone will provide more information to students, teachers and parents about careers in this field and how young people can pursue them.
DHS Careers Zone
Everything you need to know about careers in Digital Health
Here you'll find resources to find out more and work towards a fantastic career in Digital Health
Look at some career profiles to see if some of these roles would be of interest to you.
Check out Live Jobs with NHS Digital to see what roles are being recruited for and what the salaries are
Stay up-to-date with the latest developments in digital health and news from ORCHA.
Select user-type Health Professional Commissioner Individual Developer Teacher
© ORCHA
© Copyright Orcha 2021


Coury suspected that what was reported as gunshots may have been popping sounds from an electrical box.
ZANESVILLE - An apartment building on Putnam Avenue that police investigated for a possible active shooter on Friday afternoon was condemned shortly after officers cleared the scene.
No shooter was found, said Zanesville Police Chief Tony Coury. A 911 caller had reported that several gunshots were fired in the attic of the building, located at 543 Putnam Ave., and officers evacuated the building.
Coury suspected that what was reported as gunshots may have been popping sounds from an electrical box. The building was in bad shape, he said, with unsafe electrical wiring, trash outside and holes in the roof and ceiling.
The power was cut from the building late Friday afternoon, he added. And officials were trying to find a place for the building's tenants, approximately a dozen in number, to spend the night and possibly the weekend.
"Right now, the people who live there are just out for the night," he said. "Most don't have a place to go."
Coury said the building had been condemned once before for an exterior issue, which the landlord fixed. But until Friday, the city's code enforcement had been unable to enter the building without an invitation.
The building's owner, Sally Tompkins, said she didn't know what was going on with the building. She heard there were electrical problems and trash left outside.
When asked what she was going to do about her tenants being evicted because the building was condemned, she said, "We're gonna take care of the problem."
In response to a question about whether her building has ever been condemned before, she hung up.
Diana Wickham was sleeping in her apartment in the building, when she heard what sounded to her like gunshots coming from the attic above.
She heard six pops, she said, and she called 911 and ran outside with her roommate. After law enforcement arrived, she heard two more pops.
Other neighbors said they didn't hear anything and didn't know anything was amiss until police instructed them to evacuate.
Coury said officers found some ammunition in the attic, but no gun and no person.
"We have swept every inch of this building," he said.
Muskingum County Sheriff's Office K9 unit, Zanesville Police Department SWAT and Ohio State Patrol responded to the scene. Zanesville Fire Chief Eric Waltemire, Public Safety Director Fred Buck and Zanesville Mayor Jeff Tilton were also at the scene.


Painting has taken up the majority of my time, lately, which is why I'm probably single again. You see, I grew up always being denied my dreams of becoming an artist. Growing up with Roman Catholic parents who dreamed of their famous soccer player every single day made it difficult to ever express my dreams, or myself for that matter. When I did, I would be shot down with the typical, "there's no money in that!" "You need to get a scholarship by playing soccer!" They hammered it into my head so much so that when it came time for college I didn't know who else to 'go become". And anytime I considered an art degree I'd run away afraid of committing to the debt. So I packed my bags and joined the Navy, and after my first enlistment, I still decided to stay in the service because I was comfortable. I believed that I could never make it as an artist or photographer even though I had my G.I. Bill at my disposal. Now ten years later I am ready get out of the service. I did my time, and my heart just isn't in it anymore. My whole life up until this point has revolved around putting my fears and others before me.
My heart now lies with the dreams my soul conceived when I first put a crayon to paper. Unfortunately, the ambition and dedication it takes to become someone and fill the shoes you were meant to fill is not conducive to the beginning of a new love story. With happy ever after love comes dedication and prioritizing someone else's needs before yours. So I recently hit an impasse.... a combination of work, therapy, chasing my dream, going for my BFA next year, getting out of the service, and starting a new relationship boiled over inside of me. I realized I couldn't be that gay family with the white picket fence, dog, little ones running around, dinner ready on the table, and household chores, because at the moment my priorities are me, myself, and I (Thanks Beyoncé).... Not yet at least, because I do want those things along with a flat in NYC, but I don't want my needs to be a burden or strain on someone else.
Successes: I didn't want the resin to go over the edges for the 2nd piece, so I used a technique called "Damming". Which from all the videos I saw, artists would wrap metal tape all around the edges, which I did, but I modified a bit. I did not want a thick Lucite layer of resin, so I wrapped painters tape around all of the edges and went over that with metal tape. Knowing that I was going to be as careful as possible not to let the resin run over the edges. If it did, the metal tape would come off first and whatever minor run off I had would just peel off with the painters tape.
2) Do a single pour. Don't sit there working with the resin you poured then decide you want a little more and add bits here and there. This resulted in dimpling in random spots.
3) If you are working with magazine clippings like I was, seal your work. After pouring the resin, I noticed it bled through on a few of the clippings I had put down.
4) If you are doing a collage, make sure EVERYTHING is tacked down so the resin doesn't seep underneath. I didn't have a glue stick, so I made my own modge-podge with Elmer's glue; and because I didn't want the rippling to happen, I avoided slathering it on and only doing the corners of my clippings. Rippling still happened and a couple corners rose to up and out of the completely cured resin.
5) If you like to sign your paintings on the front, DON'T FORGET TO SIGN IT BEFORE YOU POUR. I was so excited to work with it that I forgot to sign my pieces. Fortunately, I'm going to have to pour over both pieces anyways, so I'll have a chance to sign the frames at least.
6) Line your work area with newspapers, wax paper, or parchment paper. "You're going to be careful and not let it run off anywhere"....yah that's what I said....now I have a foam poster board sealed onto an old mantel.
7) Don't NOT read the what not to do's online, because chances are you will do something that you think won't harm the end result, and well, it does. A good resource I found for all your resin questions is at www.ArtResin.com. Now this is not me endorsing them lol. I swear I watched every video I could on pouring resin, and it's like they reached out to all the resin artists and asked them to feature their product because "Everyone loved Art Resin and you couldn't use anything else!" Well, I did, not because I didn't want to, but my art store didn't have the large size, so I went with Envirotex Lite.
Check out my youtube video for a more in depth explanation of my process!


What do I need to do once I earn a letter jacket?
How do I get patches for my letter jacket?
Student athlete must earn one varsity letter.
Pick up a letter jacket form and your athletic letter from the Athletic office. Ask the athletic director to sign the form. Take the letter jacket form to The Athlete.
Pick up a patch form from the Athletic office, have your coach complete the form, then take the form to The Athlete.


text:
Squaw/Alpine ready to roll out $24 million in improvements
Courtesy Matt Palmer Crews are hard at work on the new Big Blue Express at Squaw Valley.
OLYMPIC VALLEY, Calif. and#8212; Squaw Valley and Alpine Meadows on Tuesday announced nearly $24 million in capital improvements for the 2012-13 winter season. This yearand#8217;s investment follows the $15 million in base area and on-mountain improvements from last winter (2011-12).
New projects for the upcoming ski and snowboard season include significant snowmaking improvements at both resorts; a new high-speed, six-passenger chairlift and improved High Camp beginner experience at Squaw; and a mile-long terrain park at Alpine Meadows.
and#8220;Snowmaking improvements, in particular, for this season and in the future, will be substantial and will vastly improve the consistency and quality of our snow surface and afford us an excellent opportunity for top-to-bottom skiing quite early in the winter season,and#8221; said Andy Wirth, president and CEO of Squaw Valley and Alpine Meadows.
The 2012-13 winter season marks year two of the Renaissance, originally billed as the five-year, $50 million capital improvement plan bringing dramatic improvements to Squaw. Now, the Renaissance will total nearly $70 million in improvements to both Squaw and Alpine by winter 2015-16.
Year one of the Renaissance brought a renovated base village with new restaurants, bars and fire pits to Squaw Valley, along with revamped terrain parks in partnership with Snow Park Technologies, the new SnoVentures activity zone with tubing and mini-snowmobiles and the worldand#8217;s first ski-in/ski-out Starbucks.
Below is a breakdown of some of the larger changes in store. For detailed information, visit http://www.squaw.com.
For the 2012-13 winter season, Squaw will install the Big Blue Express, a high-speed six-pack chairlift, to replace the High Camp chairlift. The base terminal of Big Blue Express will start where the base of the High Camp lift was formerly located and will extend to the top of the Shirley Lake ridgeline.
This alignment will not only provide quicker and easier travel to Shirley Lake and beyond, it will also give skiers and riders access to beginner and intermediate terrain that was previously inaccessible via the old High Camp chairlift. The new Park Pulley telecord, a modern day version of a tow lift, will be also installed to provide easy access to the Belmont terrain park and to eliminate traversing across the mountaintop meadow.
Further, the Links double chairlift will be removed and replaced with a triple chairlift named Mountain Meadow, which will be realigned to bring skiers and riders up to the ridgeline near the top terminals of the Solitude and Silverado lifts.
As to snowmaking, Squaw Valley will pursue a variety of system-improvements including the installation of high-technology air/water guns; replacement of manual fan guns with fully automated versions; rebuilding of pumps and compressors; and installing state-of-the-art monitoring and control equipment. Together, these improvements will enhance the efficiency and effectiveness throughout the entire system, from Squaw Creek and Red Dog to Gold Coast and the Mountain Run, ensuring the best possible snow surface from top-to-bottom.
Snowmaking upgrades planned for Alpine Meadows will significantly improve efficiency, making it possible to generate more snow faster and#8212; delivering better surface conditions earlier in the season.
Upgrades will include computer-driven automation to control the snowmaking operation. The automated system monitors a range of data (e.g. temperature, humidity, flow, etc.) making it possible to configure the operation of specific snowmaking units based on conditions in that area. This greatly improves snowmaking efficiency, resulting in a larger quantity of higher quality snow in a shorter period of time.
With the recent partnership between Alpine Meadows and Snow Park Technologies, skiers and riders can look forward to continued terrain park improvements and an enhanced park experience throughout the winter season. Snow Park Technologies is renowned in the terrain park business with 15 years of experience designing parks for major resorts and events nationwide including the Dew Tour and Winter X Games.
One of the most exciting changes to the terrain park experience is the expansion of the medium and large park. The combined parks, from Terryand#8217;s Return to Dancefloor, will offer an entire mile-long park run, making it the longest terrain park run in the Tahoe region.
Also new this year, there will be a beginner progression park on Subway where lessons will be available. Guests eating out on the sundeck can enjoy the view of skiers and riders testing out their skills at a new base area jib arena. New jibs will be added this season, including four new beginner boxes, two intermediate boxes and four new snowmaking pipe jibs, increasing the park's fleet to more than 60 jib features.


The story centers around the employees of a legal document firm. Tamura Katsuhiro is a young man with a strong sense of justice. Although he believes that the law isn't always right, he becomes an administrative scrivener with the goal of manipulating the law in order to protect the weak. On the other hand, Sumiyoshi Misuzu, a former delinquent who hates to lose and who is now a capable scrivener. Katsuhiro and Misuzu are always at odds, but they gradually come to understand each other.


A dinner meeting Saturday between US President Donald Trump and his Chinese counterpart Xi Jinping to discuss their growing trade war went "very well," White House top economic advisor Larry Kudlow said.
Kudlow's comment to journalists, as Trump and his delegation were about to take off for Washington from Buenos Aires, was the first public reaction from the high-stakes dinner lasting more than two hours.
White House spokeswoman Sarah Sanders said she expected to release a statement soon. There was no immediate word from China's government.
Trump and Xi, who were in the Argentinian capital over the last two days for a summit of G20 countries, had gone into the meeting in a bid to pull the world's top two economies from the brink of a full-fledged trade war.
Both leaders expressed optimism as they and top aides sat down at a long hotel table adorned with flowers.
"We will probably end up ending up getting something that will be good for China and the United States," Trump said.
Xi stated that they shared tremendous responsibility to find a solution. "Only with cooperation between us can we serve the interest of both peace and prosperity," he said.
The meeting, featuring a menu of sirloin steak, caramel rolled pancakes and Argentinian wine, went on longer than scheduled. And while it may have been tacked on to the end of two days of G20 diplomacy, it was in many ways the main event of the weekend.
Trump has already imposed tariffs on more than $250 billion in Chinese goods -- about half of the total imported into the United States each year -- in an attempt to pressure Beijing to change its trade rules.
Duties of 10 percent currently on most of those goods are set to rise on January 1 to a whopping 25 percent if a deal isn't reached.
As Kudlow said in Washington ahead of the dinner: "If things don't work out in this US-China summit meeting, he will invoke some 267 billion dollars in additional tariffs."
Navarro's inclusion in particular was a surprise as he has harshly criticized China, accusing its leadership of duplicity. However, keen White House watchers noted that Trump was flanked at the table by Mnuchin and his secretary of state, Mike Pompeo.
They include attempts to secure the denuclearization of North Korea and China's increasingly self-confident expansion of military power, especially in the South China Sea.