text:
\section{Introduction}
\label{sec:introduction}
Since DeepMind's 2015 Nature paper, the Arcade Learning Environment (ALE) has become the \emph{de facto} deep RL benchmark for new training algorithms~\cite{bellemare13arcade, mnih2015nature, machado17arcade}. ALE has several appealing qualities: humans learn to play Atari and become more skilled with experience, it is a ``real-world'' environment that was not originally constructed to evaluate RL methods, and it has greater complexity than prior environments (e.g., GridWorld, mountain car).

ALE has been used in several ways to evaluate the performance of deep RL agents. The vast majority of evaluations follow a version of the approach described by \citet{bellemare2012investigating}: first researchers choose network architectures and tune hyperparameters on small set of Atari games; then they train agents using those hyperparameters on new games, reporting the learning curves (or a statistic of a collection of those curves)~\cite{mnih2015nature, van2016deep, mnih2016asynchronous, hessel2018rainbow}.

While ALE has enabled demonstration and evaluation of much more complex behaviors of deep RL agents, it presents challenges as a suite of evaluation environments for topics on the frontier of deep RL.

\textit{Challenge: Limited variation within games.} Very little about individual games can be systematically altered, so ALE is poorly suited to testing how changes in the environment affect training and performance. New benchmarks such as OpenAI's \textit{Sonic the Hedgehog} emulator and \textit{CoinRun} inject environmental variation into the training schedule, while introducing train/test splits~\cite{nichol2018gotta, cobbe2018quantifying}. Similarly, \citet{zhang2018natural} suggest benchmarks that incorporate the kind of non-random noise found in nature. \citet{kansky2017schema} implemented Breakout variants in order to achieve variation for generalization.

\textit{Challenge: No counterfactual evaluation.} Meanwhile, assertions about intelligent agent behavior remain untestable in the face of black-box evaluation environments. For example, ALE does not enable testing the conjecture that agents trained on Breakout learn to build tunnels \cite{mnih2015nature} or that they enter a tunneling mode \cite{greydanus2017visualizing}. No system currently permits experiments to answer counterfactual questions about agent behavior.

\textbf{Contribution.} We propose \textsc{ToyBox}{}, a suite of high-performance and highly parameterized
Atari-like environments designed for the purpose of experimentation. We demonstrate that the \textsc{ToyBox}{} implementations of three Atari 2600 games achieve similar performance to their ALE counterparts across three deep RL algorithms.  We demonstrate that \textsc{ToyBox}{} enables a range of post-training analyses not previously possible, and we show that \textsc{ToyBox}{} is orthogonal to concurrent efforts in deep RL to address issues of robustness, generalization, and reproducible evaluation.

\textbf{Organization.} The rest of the paper is organized as follows: Section~\ref{sec:toybox} introduces the \textsc{ToyBox}{} design and its functional capabilities. Section~\ref{sec:evaluation} describes our evaluation, including performance and fidelity testing against the ALE. Section~\ref{sec:case-studies} describes four behavioral tests we present as case studies. Related work not otherwise addressed can be found in Section~\ref{sec:related}.  Section~\ref{sec:discussion} discusses \textsc{ToyBox}{} applications beyond the scope of this paper. We conclude in Section~\ref{sec:conclusion}.

\section{\textsc{ToyBox}{}: System Design}
\label{sec:toybox}

\textsc{ToyBox}{} is a high-performance, highly parameterized suite of Atari-like games implemented in Rust, with bindings to Python. The suite currently contains three games:  Breakout, Amidar, and Space Invaders.  We chose these games for diversity of genre (paddle-based, maze, and shooter, respectively) and likely familiarity to readers.\footnote{Although Amidar may not be familiar, it is very similar to PacMan, but with simpler rules.}

\textbf{Software requirements.} Atari 2600 games were designed for human players. For \textsc{ToyBox}{}, the primary user is a reinforcement learning algorithm, and we expect machine learning researchers to be able to customize gameplay. To that end, we developed \textsc{ToyBox}{} to meet the following set of software requirements
\begin{enumerate}[label=\textbf{R\arabic*}, leftmargin=*]
    \item\label{req:cpu}\textsc{ToyBox}{} should only leverage the CPU, even for graphical tasks. Although modern games leverage the GPU for faster rendering, we expect the machine learning libraries to be using the GPU, and so we wish to create our screen images using CPU-only.

    \item\label{req:efficiency}\textsc{ToyBox}{} should be at least as efficient as the Stella-emulated version of the game. Since reinforcement learning algorithms require millions of frames of training data, we must be able to simulate and render millions of frames in order to enable efficient use of computation resources for learning.

    \item\label{req:data-driven}\textsc{ToyBox}{} should provide for data-driven user customization. Changing the bricks in Breakout, the board in Amidar, or the alien configuration in Space Invaders should not require re-compilation of the core game code, nor should it require the ability to write Rust.

    \item\label{req:gym}\textsc{ToyBox}{} should be accessible through OpenAI Gym, which is a Python API. Furthermore, \textsc{ToyBox}{} should be usable as a drop-in replacement for the analogous ALE environment.

\end{enumerate}
\begin{figure}[!h]
    \centering
    \includegraphics[width=\linewidth]{toybox-system}
    \caption{The \textsc{ToyBox}{} architecture.}
    \label{fig:toybox-system}
\end{figure}

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.8\columnwidth]{RL-diagram.pdf}
    \caption{A traditional RL diagram, augmented with features required for counterfactual reasoning (shaded above). There is a physical system that governs behavior within the environment that may only be partially known. The researcher has a model of the environment's physics and, when attempting to explain agent behavior, must implicitly form a model of the agent's model. Arrows emanating from the researcher point to elements of the system where the researcher may intervene. The dotted line from the researcher to the environment is possible in \textsc{ToyBox}{}, but is not typically considered.}
    \label{fig:rl-system}
\end{figure}

\textbf{Architecture.} Figure~\ref{fig:toybox-system} depicts the \textsc{ToyBox}{} architecture. The game logic is written in Rust. Every game implements two core structs: \texttt{Config} and \texttt{State}. The \texttt{Config} struct contains data that we would generally only expect to be initialized at the start of an episode (i.e., a game, which may include multiple lives). The \texttt{State} struct contains data that may change between frames. At any point during execution, a \textsc{ToyBox}{} game can be paused, state exported and modified, and resumed with the new state.

\textbf{Aside: Why Atari?} Given the range of open problems in deep RL, creating a new ALE-like system may not seem like an effective way to facilitate cutting-edge deep RL research. However, there are still many poorly understood properties of Atari games and agent behavior on them.

There are many axes of complexity in the reinforcement learning environment: required planning horizon, reward function assignment, number of actions available, environment stochasticity, and  state representation are all different components affecting the total complexity of the environment, each presenting unique challenges for policy learning:

\pagebreak
\begin{itemize}[leftmargin=*]
  \item \textbf{Planning Horizon} Atari environments requiring temporally extended planning strategies for success, as in Frostbite, still present challenges to deep reinforcement learning algorithms ~\cite{jiang2015dependence, lake2017building, farquhariclr218}.
    \item \textbf{Reward Function} Environments with sparse reward functions are another sticking point for RL, having only recently seen significant progress on games in this category~\cite{goexplore, salimans2018,burda2018exploration}. Atari examples include Pitfall and Montezuma's Revenge.
    \item \textbf{State Representation} Operating over pixel representations like Atari games means the agent experiences a huge state space, but the underlying objects defining those sensory representations can often be represented more compactly~\cite{guestrin2003generalizing, littman2008oomdp, kansky2017schema}. Models trained on a compact representation of state can train more quickly~\cite{brunskill2018strategic, melnik2018modularization}, but the underlying environment dynamics have not changed.
    \item \textbf{Action Space Complexity}  Increasing the size of the action space can quickly lead to intractable computation for Q-value or policy function approximation, especially when that approximation is computationally expensive as in deep RL ~\cite{dulac2015deep}. Atari games available in ALE allow 4-18 actions ~\cite{mnih2013playing}. While this may seem small, even environments with as few as 10 actions can present challenges to efficient learning~\cite{dulac2012fast}.
    \item \textbf{Environment Stochasticity} Previous work has shown that environments with higher stochasticity can be more difficult for some types of RL algorithms to learn~\cite{henderson2017deep}. Atari games are deterministic, but \textsc{ToyBox}{} enables Atari-like environments with parameterized stochasticity.

\end{itemize}

Despite these open challenges, it has been argued that Atari's environments are not sufficiently complex to evaluate reinforcement learning agents because the source code is small~\cite{zhang2018natural}. However, source code size as minimum description length is a poor proxy for environment complexity. As ~\citet{raghu2017can} have shown, Erd\H{o}s-Selfridge-Spencer games can be represented quite compactly, simply requiring the assignment of two parameters, but represent a large combinatorial space of potential games for evaluating RL algorithms.

Ultimately, each of these axes of complexity is observed through the agent's interaction with the environment. Figure~\ref{fig:rl-system} depicts the traditional RL diagram, overlaid with the interventions a researcher may make, as well as the models that inform both the researcher's and the agent's decision-making. The solid arrows represent current avenues for experimentation used by deep RL researchers to evaluate models: sensory perception of state, action selection manipulations, and reward function definition.

Without intervention or introspection of the environment, researchers must use observational data of agent behavior to reason about experimental results. \textsc{ToyBox}{} enables new methodology for experimentation in deep RL. Ultimately, we mimic Atari games because there is a font of untapped research questions related to testing and explaining the behavior of deep RL agents. We chose our initial set of games to establish that there are surprising results on even seemingly ``solved'' games such as Breakout and Space Invaders.

 %

\section{Evaluation}
\label{sec:evaluation}
\textbf{Performance.}
We achieved \ref{req:cpu} by designing a simple CPU graphics library; we demonstrate \textsc{ToyBox}{} efficiency (\ref{req:efficiency}) in Table~\ref{tab:performance}. Note that \textsc{ToyBox}{} permits researchers to process games entirely in grayscale and achieve substantial additional performance gains.  However, since this is not a feature offered in ALE, we only compared against the \textsc{ToyBox}{} RGB(A) rendering.

\begin{table}[t!]
    \caption{\textbf{\textsc{ToyBox}{} vs ALE performance}. Thousand frames per second (kFPS) on a MacBook Air (OSX Version 10.13.6) with a 1.6 GHz Intel Core i5 processor having 4 logical cores. Rates are averaged over 30 trials of 10~000 steps and reported to two significant digits, with standard error. We consistently observed an approximately 95\% slowdown when interacting with both ALE (C++) and \textsc{ToyBox}{} (Rust) via OpenAI Gym. All benchmarks are run from CPython 3.5 and include FFI overhead (via \texttt{atari-py} for ALE).}
    \centering
    \begin{tabular}{p{0.18\linewidth}p{0.15\linewidth}*{2}{p{0.22\linewidth}}}
&& Raw kFPS & Gym kFPS\\
        \cline{3-4}
        \multirow{2}{*}{{Breakout}} & {ALE} & 52 (1.3) & 3.4 (0.065) \\
        &{\textsc{ToyBox}{}} & 230 (5.4) & 7.2 (0.23)\\
        \cline{2-4}
        \multirow{2}{*}{{Amidar}} & ALE & 61 (2.9) & 3.0 (0.083)\\
        & \textsc{ToyBox}{} & 250 (2.3) & 6.0 (0.112)\\
        \cline{2-4}
        \multirow{2}{*}{\parbox[t]{4cm}{Space\\Invaders}} & ALE & 55 (1.3) & 3.9 (0.072)\\
        & \textsc{ToyBox}{} & 120 (3.4) & 5.2 (0.082)
     \end{tabular}
    \label{tab:performance}
\end{table}

\begin{figure*}[t!]
  \begin{minipage}[b]{.27\linewidth}
    \centering
    \includegraphics[width=0.9\linewidth]{ale_breakout_frame100.png}
    \includegraphics[width=0.9\linewidth]{tb_breakout_frame100.png}
    \subcaption{\small Upper: ALE; lower: \textsc{ToyBox}{}.}
  \end{minipage}\quad%
  \begin{minipage}[b]{0.4\linewidth}
    \centering
      \begin{minipage}[b]{0.49\linewidth}
        \includegraphics[width=\linewidth]{ale_amidar_frame0359.png}
      \end{minipage}~%
      \begin{minipage}[b]{0.49\linewidth}
        \includegraphics[width=\linewidth]{tb_amidar_frame_0101.png}
      \end{minipage}
      \subcaption{\small  Left: ALE; right: \textsc{ToyBox}{}.}
  \end{minipage}\quad%
  \begin{minipage}[b]{.27\linewidth}
    \centering
    \includegraphics[width=0.9\linewidth]{ale_spaceinvaders_frame0499.png}
    \includegraphics[width=0.9\linewidth]{tb_spaceinvaders_frame0499.png}
    \subcaption{\small Upper: ALE; lower: \textsc{ToyBox}{}.}
    \label{fig:space-invaders}
\end{minipage}
  \caption{Side-by-side comparisons of screen shots from ALE and \textsc{ToyBox}{} Atari games. Each game represents a different deterministic action trace, but traces are the same between ALE and \textsc{ToyBox}{}. \textsc{ToyBox}{} implementations of Breakout and Space Invaders have nondeterministic elements. Amidar is deterministic in both ALE and \textsc{ToyBox}{}. Due to idiosyncrasies at the start of ALE Amidar gameplay, the frames are not from identical points in the action trace; frames for Breakout and Space Invaders are. Note that for Amidar ALE appears to be missing an enemy, which is due to Atari only rendering a subset of sprites each frame due to computational constraints.}
  \label{fig:screenshots}
\end{figure*}

\textbf{Fidelity.} Figure~\ref{fig:screenshots} depicts frames at roughly equivalent points in the execution of a fixed action trace. If \textsc{ToyBox}{} perfectly reproduced the games in ALE, the frames would be exactly the same. Three factors prevent exact replication of games: (1) Atari 2600 game source code is not available, (2) there are no formal specifications and few informal specifications of games,\footnote{Some informal specifications contain errors (e.g., the Atari manual for Breakout refers to a row of bricks that does not exist).} and (3) inferring arbitrarily complex programs from data is extremely challenging~\cite{raychev2016learning}.
Therefore, comparing frames of program traces is not a sufficient measure of how closely we have approximated ALE games.

A human study could help assess whether the two implementations are are sufficiently alike. While we did solicit feedback from Atari aficionados during development (via a playable interface),
we did not view human perception of equivalence as a sufficient measure of fidelity. Human players rely on unique problem-solving capabilities that deep RL agents have not yet achieved, while deep networks are undeterred by the kind of noise that can confuse humans  ~\cite{adversarial,dubey2018investigating}. Instead, we focused on tuning our environments to produce comparable post-training agent performance.

\textbf{Methodology.} We used three off-the-shelf implementations of training algorithms with default parameter settings for 5e7 steps from OpenAI Baselines~\cite{baselines}: \texttt{a2c}~\cite{mnih2016asynchronous},
\texttt{acktr}~\cite{wu2017scalable}, and
\texttt{ppo2}~\cite{schulman2017proximal}. Due to issues with variability across agents and environments~\cite{henderson2017deep, clary2018variability, jordan2018benchmarks}, we trained ten replicates for each of these training algorithms, differentiated by their random seed. Since there are various other uncontrolled sources of randomness, we evaluated each of these thirty agents per game using thirty unique random seeds over thirty games. Figure~\ref{fig:ranking} depicts our results: we find that agents achieve sufficiently similar performance in each analogous environment, and have roughly equivalent rankings (idiosyncrasies discussed in the in Fig.~\ref{fig:ranking} caption).

\textbf{Findings.} \textsc{ToyBox}{} is much faster than ALE and our ranking results in Fig.~\ref{fig:ranking} show that our current implementations are comparable to their ALE counterparts. We will continue to strive for fidelity over the course of game development.

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{ranking.png}
    \caption{Rankings across 300 model replicates per model, per game, with standard error. Each horizontal bar is the average performance for a trained model, evaluated over 30 games. Stridations should be similar in both environments. \textbf{Breakout}: One training seed led an agent trained using \texttt{acktr} to have abysmally poor performance in every trial (i.e., all average scores below 2). \textbf{Amidar}: The lack of within-game variation is reflected in the short error bars and similar rankings between backends. \textbf{Space Invaders}: \textsc{ToyBox}{} only implements the first level of Space Invaders. Since there is currently no way to detect levels within ALE, we let both agents play indefinitely. However, because Space Invaders levels vary considerably, performance is not comparable. For this reason we use different scales for the score (x-axis).}
    \label{fig:ranking}
\end{figure}

\section{Case Studies}
\label{sec:case-studies}
We demonstrate~(\ref{req:data-driven}: Customization) with four case studies, in which we test the post-training performance\footnote{The analyses in this paper focus on evaluations of post-training performance, but \textsc{ToyBox}{} interventions can be applied at any time--including during training.} of agents according to some hypothesis about behavior. This sort of testing is useful for evaluating a single agent prior to deployment (i.e., acceptance testing) or as existential proof for behavior under counterfactual conditions. None of these tests are currently possible in ALE because they all rely on resuming gameplay from an arbitrary modified state.
Furthermore, all experiments in this section can be expressed in fewer than 200 lines of code, in addition to the code required for loading up the OpenAI baselines models. We include a code snippet from one of our experiments in Figure~\ref{fig:snippet}.

\begin{figure}
    \centering
    \begin{minipage}[c]{.33\linewidth}
        \centering
        \includegraphics[trim={50 50 50 50}, clip, height=1.25in]{polarstart50.png}
        \subcaption{Polar starts}
        \label{fig:polar-starts}
    \end{minipage}~%
    \begin{minipage}[c]{0.66\linewidth}
        \centering
        \includegraphics[height=1.25in]{ezchannel_50.png}
        \subcaption{Tunneling}
        \label{fig:tunneling}
    \end{minipage}
    \label{fig:breakout}
    \caption{\textbf{Breakout case study.} Results from counterfactual queries for starting angles and tunnels. Both tests were run on an agent trained with the default OpenAI Baselines parameters for \texttt{ppo} for 5e7 steps, one life, and a 4 min. timeout. \textbf{(a)}:~The black lines indicate the starting angles seen during training, the light gray area the maximum score achieved from this starting angle, and the dark gray area represents the mean score achieved across trials. \textbf{(b)}:~Brick hue scaled according to the inverse of the median number of steps required to clear the particular brick in that test; bright-yellow represents fewer steps and red represents more steps (17-400).}
\end{figure}
\begin{figure*}[!h]
    \centering
    \begin{minipage}[c]{0.55\linewidth}
        \centering
    \includegraphics[width=\columnwidth]{amidar_test.png}
        \subcaption{Accumulated game score of a single \texttt{a2c}-trained agent.}
        \label{fig:amidar-test-single}
        \end{minipage}\quad%
    \begin{minipage}[c]{0.4\linewidth}
        \centering
        \includegraphics[width=\linewidth, height=2.5in]{movement_protocol_test.png}
        \subcaption{Ranking of 30 model replicates.}
    \label{fig:amidar-test-rank}
    \end{minipage}
\caption{\textbf{Amidar case study.} for four enemy movement protocols (``Control'' is a lookup table, ``Amidar'' is the ``Amidar movement,'' ''Random'' enemies move in a random direction at every junction, and ``Target'' causes enemies to pursue the player when it is in line of sight). \textbf{(a)}: \textit{Upper left}: baseline performance of the agent on each of the four protocols. \textit{Lower left}: baseline performance of the agent on each protocol without the ability to jump over enemies. \textit{Upper right}: ``Ganging up'' test, where all agents start close to the player. \textit{Lower right}: ``Ganging up'' test with no jump. \textbf{(b)}: Score ranking of 30 model replicates for the baseline condition with jumps (i.e., the upper left corner of the left graph) for each movement protocol.}
\end{figure*}

\paragraph{Breakout: Polar Angles.} An agent that has learned to play Breakout must have at least learned to hit the ball. Our first test manipulates the starting angle of the ball.

We modified the start state to change the initial launch angle of the ball in 5$^\circ$ increments (72 configurations). Figure~\ref{fig:polar-starts} depicts the results. Note that the agent fails to achieve any score with horizontal ball angles: since Breakout has no gravity, balls simply bounce horizontally forever, never hitting any bricks or threatening the paddle. The agent also sometimes struggled with vertical angles. When we observed this behavior, the agent would keep the ball aligned perfectly in the center of the board, hitting it precisely in the center of the paddle, failing to make progress. This is an unexpected behavior that is entirely unlike human gameplay. In all, we found the agent to be resilient to starting angles, albeit with high variance. This suggests that an agent can be successful even with balls traveling at angles it may never have observed in training, a powerful recommendation for the training algorithms that produced such robust RL agents.

\paragraph{Breakout: Tunneling.} One of the most promising behaviors observed in deep RL has been the apparent ability to learn higher-level strategies. Perhaps no high-level strategy has been written about more than ``tunneling'' in the game of Breakout, which happens when the player clears a column of bricks, causing the ball to bounce through the hole and onto the ceiling, clearing many bricks rapidly~\cite{mnih2015nature, greydanus2017visualizing}. One way to test whether an agent intentionally exploits tunneling is to give it a board with a nearly complete tunnel, save for a single brick, and test whether the agent can prioritize aiming at that single brick.

For every brick, we removed all other bricks in the column, creating a nearly-completed tunnel. Figure~\ref{fig:tunneling} depicts the results: the value for each brick is the reciprocal of the median number of time steps before that brick was removed.

If an agent were able to build tunnels, we would expect, for example, symmetry along one or both axes. Instead, we see that the agent clears one column in the center very quickly, the left adjacent column and some bricks in the upper left region a bit more slowly, and the remaining bricks take all about the same time. Observing agent gameplay, we saw the agents hit the ball to predictable locations, regardless of the board configuration.

\begin{figure}
    \centering
    \footnotesize
    \begin{lstlisting}
# Fourth test: Target Player
# Delete Existing Enemies
config = toybox.get_config_json()
config['enemies'] = []
# Add Enemies that chase the player:
for i in range(5):
    config['enemies'].append({
        'EnemyTargetPlayer' : {
            'start' : starts[i],
            'start_dir': 'Right',
            'dir': 'Right',
            'player_seen': None
    }})
# Update game configuration
toybox.write_config_json(config)
# Run 30 trials with OpenAI Gym API
obs = env.reset()
for trial in range(n_trials):
    n_steps = 0
    done = False
    # until death or max_steps
    while n_steps < max_steps and not done:
        action = agent.step(obs)
        obs, _, death, info = env.step(action)
        done = death and not toybox.game_over()
        log_step(toybox)
        n_steps += 1
    log_after_episode(toybox)
    \end{lstlisting}
    \caption{Code snippet from our Amidar Protocol test. We build JSON config in Python and run it in \textsc{ToyBox}{}.}
    \label{fig:snippet}
\end{figure}
\paragraph{Amidar: Enemy Protocol.} Suppose we would like to test whether an agent has learned to avoid adversarial elements of a game: e.g., the enemies in Amidar. To test this, we might drop the agent around the corner from an enemy, or position the enemies to ``gang up'' on the player, forcing the agent to move in a particular direction.

This kind of intervention is only meaningful if enemy position is a function of current location. Observation led us to conclude that enemies move in fixed loops, likely implemented as lookup tables. This contrasts with ``Amidar movement'' is believed to dictate enemy behavior.\footnote{An enemy moves with a diagonal velocity, flipping the vertical direction when encountering the top or bottom of the board and horizontal direction when encountering the left or right edge (\url{https://en.wikipedia.org/wiki/Amidar})} The protocol matters for intervention because, for a lookup table, moving enemies will have no effect: enemies will simply ``teleport'' to the next location in the lookup table.

The upper left plot in Fig.~\ref{fig:amidar-test-single} shows a baseline test for how an individual trained agent performs under each of four different enemy movement protocols: (1) a lookup table, on which the model was trained, (2) the ``Amidar movement'' protocol, (3) a random protocol, where at each junction the enemy chooses a random direction, and (4) an adversarial protocol, where enemies explore via random turns until the player is within line of sight, at which time they move toward the player's location. Note that, since enemies start far away from the player, the agent can (and does) easily make progress at the start of the game, regardless of enemy position. However, as the game progresses, the enemies close in and there are fewer opportunities for rewards.

The upper right plot in Fig.~\ref{fig:amidar-test-single} shows a test in which enemies ``gang up'' on the player: the enemies start position is modified to be close to the player. We were at first surprised to see how well the agent did; however, upon examination, we found we found that the agent was using up the jump button, which allows the player to bypass enemies, at beginning of the game. The lower half of Fig.~\ref{fig:amidar-test-single} depicts the results of running the baseline and test for no jumps: while the baseline performs similar, the player dies quickly for all non-lookup table enemy protocols.

\begin{figure}
    \centering
    \includegraphics[width=0.9\columnwidth]{space_invaders_test.png}
    \caption{\label{fig:space-invaders-test}\textbf{Space Invaders case study.} The top three bands depict the tests of the solitary first, second, and third shields respectively. The bars depict the total number of steps spent in the corresponding horizontal location, while the boxplots depict the scores. The fourth band shows the behavior for all shields present (green; score boxplots on the left) and no shields present (purple; score boxplots on the right). The bottom band shows the Space Invaders terrain and default positions of the shields.
    }
\end{figure}

\paragraph{Space Invaders: Shield Usage.}
In Space Invaders, the player can seek refuge under three shields from the frontier of alien ships shooting down. We ran a test to see whether removing two of the three shields would cause an agent to use the remaining one more often. We also ran two baseline comparisons for a fixed amount of time: one where all sheilds are present (the default setting) and one where no shields were present.

Figure~\ref{fig:space-invaders-test} shows the results under test. Since score provides an incomplete picture of agent behavior, we also tracked the agent's location (a simple query in \textsc{ToyBox}{}).  We observe that the player does not appear to change its preferred locations under any of the tests.

\emph{A Note on Negative Results:} Space Invaders, as we have implemented it, has turned out to be a fairly uninteresting game. Randomly selecting from the trimmed action set that OpenAI Gym allows can lead to fairly good performance. Furthermore, our implementation, which included both random and adversarial enemy behavior, led the agent's behavior to be invariant to randomness in enemy behavior.

\textbf{Findings.} We have shown a range of interventions and queries possible with \textsc{ToyBox}{}, all of which would be impossible to conduct using ALE. The interventions we demonstrated were designed to demonstrate the power of \textsc{ToyBox}{}'s design and implementation, rather than to satisfy any particular RL research agenda. We were able to rapidly iterate on all of our experiments due to \textsc{ToyBox}{}'s fast performance and its simple API for editing state. In addition to highlighting \textsc{ToyBox}{}'s capacity for evaluating a single agent, we have shown how \textsc{ToyBox}{} may be used to evaluate models, by comparing the post-training performance ranking under test.

\section{Related Work}
\label{sec:related}
We are hardly the first to suggest new or different benchmarks for deep RL~\cite{kansky2017schema, zhang2018natural, wang2019paired}. Four major qualities differentiate \textsc{ToyBox}{} from prior work: (1) it is based on a widely used and accepted community standard (ALE); (2) results on ALE can be replicated and compared in \textsc{ToyBox}{}, providing continuity to individual research trajectories; (3) a wide array of features of \textsc{ToyBox}{} environments are intervenable; furthermore, a particular configuration is easily exported and can be shared as part to further replication efforts; and (4) an individual game may be modified to produce a family of games, leading to a potentially infinite number of environments per-game; for example, the injection of real-world images into the background of Breakout described in~\cite{zhang2018natural} would be trivial to implement in \textsc{ToyBox}{}.

Recall the available interventions in the traditional RL research environment shown in Figure \ref{fig:rl-system}. Most existing work manipulates the state input to the agent (i.e., the agent's \emph{perception} of state), the reward function, or the agent's actions, e.g.:
\begin{itemize}[leftmargin=*]
    \item\textit{State input}: Injecting real-world data into the background of Atari 2600 games simulates non-random noise~\cite{zhang2018natural}. Skipping frames periodically is a critically important hyperparameter for tuning algorithms to play Atari~\cite{mnih2013playing, braylan2015frame, frameskip}.
    \item\textit{Reward function}: Hybrid reward structures decompose the reward function, making it easier for some agents to learn particularly difficult games in the Atari suite~\cite{van2017hybrid}.
    \item\textit{Agent actions}: Sticky actions, human starts, and random starts are all methods for intervening on the agent's actions outside the normal parameters of $\epsilon$-greedy exploration~\cite{sutton1998reinforcement, mnih2013playing, bellemare13arcade, nair2015gorila}.
\end{itemize}
These efforts can help combat overfitting, learning spurious correlations, or generally failing to make progress on a task. \textsc{ToyBox}{} is orthogonal to such efforts.

We have introduced relevant citations throughout the paper. Here we highlight critical work that was not otherwise mentioned.

\textbf{Evaluation and Replication.} Recent investigations into in how the community handles the evaluation and replication of agent performance has exposed some serious challenges that the community needs to address~\cite{henderson2017deep, balduzzi2018reevaluating, clary2018variability, jordan2018benchmarks}. Environments such as \textsc{ToyBox}{} and evaluations of the style presented in Section~\ref{sec:case-studies} are one possible way to ameliorate issues surrounding replication, robustness, and variability.

\textbf{Adversarial RL.} Much work on adversarial RL focuses on exploiting decision boundaries~\cite{mandlekar2017adversarially}, adding nonrandom noise to state input for the purpose of altering or misdirecting policies~\cite{huang2017adversarial}, or introducing additional agents to apply adversarial force during training to produce agents with more robust policies in physics simulations~\cite{pinto2017robust}.

\textbf{Saliency maps.} Saliency maps were developed as an insight into model behavior~\cite{simonyan2013deep}, but more recently have been put forth as tool for explainability~\cite{ greydanus2017visualizing}. We show that in at least one case, saliency maps can be misleading, due in part to bias exemplified by the researcher's models of the agent and the environment as shown in Fig.~\ref{fig:rl-system}. Experiments enabled by \textsc{ToyBox}{} provide much more specific information and can disambiguate competing hypotheses about agent behavior.

\section{Discussion}
\label{sec:discussion}
This paper is a proof-of-concept for experimentation about the behavior of deep RL agents. However, there are many possible applications beyond this type of post-training testing:

\textbf{Rejection sampling/dynamic analysis.} One of the biggest strengths offered in \textsc{ToyBox}{} is the ability to answer arbitrary questions about the environment structures and code at any time. Agents may encounter local minima during training that are not representative of the target deployment distribution, due to factors such as random seeds~\cite{rlblogpost}. Training replicates with many random seeds is a costly solution to this problem. Instead, researchers could use \textsc{ToyBox}{} to monitor environmental features to test whether an agent is spending too much time in an undesirable state.
Similar types of model monitoring could be used to identify ``detachment,'' a condition of the agent-environment interaction that induces catastrophic forgetting~\cite{kirkpatrick2017overcoming,goexplore}.

\textbf{Datasets from game families.} The ability to generate a family of games with similar but different mechanics provides a convenient dataset which can be used in a variety of ways. For example, with \textsc{ToyBox}, a researcher can define a family of Breakout-style games with slightly different movement physics (e.g., ball velocity and acceleration, paddle-bounce mechanics) sampled from some real-valued parameter domain. This can be used to create a train/test split over environments~\cite{cobbe2018quantifying}, for supporting transfer learning experiments from one game physics to another~\cite{taylor2009transfer}, or for testing generalization across multiple environments~\cite{guestrin2003generalizing}.

\textbf{Adversarial testing.} With total control over environment dynamics, trained agents can be stress-tested by running the agent on progressively more difficult versions of the game. Tests of this form can serve to disambiguate agent behavior that can be explained in multiple ways---much as \textsc{ToyBox}{}'s more advanced Amidar movement protocols revealed agents had not necessarily learned to avoid enemies so much as memorize their observed paths. Difficulty can be increased by increasing stochasticity in the environment, or increase the speed or accuracy of adversarial game elements. Similar methods could be used to create a suite of curriculum learning environments.

\section{Conclusions}
\label{sec:conclusion}
We have shown that \textsc{ToyBox}{} unlocks novel and important capabilities for evaluating deep reinforcement learning agents. We introduce a new paradigm for thinking about evaluating agents, in the style of acceptance testing. We demonstrate \textsc{ToyBox}{} capabilities with four case studies and outline a variety of other applications.


text:
Kerala » Media » Newspapers » Mathrubhumi
Mathrubhumi a malayalama language newspaper was conceived as the mouthpiece of India's freedom movement is one of the front-runners among the Malayalam newspapers. Mathrubhumi was founded by K. P. Kesava Menon, an active volunteer in the Indian freedom struggle against the British. Matrabhumi means "Mother land" . It is the second most read newspaper of Kerala state of India after Malayala Manorama. Today Mathrubhumi Group is not only a fast growing publishing house of repute but also a corporate entity in the mass communication industry, and has struck deep roots in the hearts of Malayalees wherever they are.
Type Daily Newspaper
Format Broadsheet
Owner The Mathrubhumi Printing and Publishing Company Ltd.
Editor in Chief M Kesava menon
Language Malayalam
Headquarters Kozhikode
Officiak Website http://www.mathrubhumi.com
History of Mathrubhumi
Mathrubhumi was founded in 1923 in the aftermath of Gandhi's non-cooperation movement as a public limited company. This status makes it rare among newspapers, which tend to be closely held private companies owned by a single family.The newspaper's founders were members of the Indian National Congress led by K.P. Kesava Menon , its shareholders included about 350 men and women of Kerala.
Mathrubhumi was Kerala's leading daily with an estimated circulation of 19,000 at independence in 1947. M. P. Veerendra Kumar, former minister for state, Govt. of Indiais the Chairman and Managing Director of Mathrubhumi.
Mathrubhumi Publications
Mathrubhumi Illustrated Weekly- Malayalam Literature for the readers.
Grihalakshmi-Magazine covering almost all aspect of feminine psyche.
Chitrabhumi-A magazine for cinema lovers.
Thozilvartha-State's informative cell in service sector or employment paper.
Mathrubhumi Sports masika– Monthly Sports magazine
Balabhumi– Comprehensive children's magazine.
Arogya Masika– Health Magazine in Malayalama
Mathrubhumi Yathra– Traveller Magazine
Minnaminni – A bi-weekly in Malayalam for kids
Cartoon plus– An exclusive cartoon bi-weekly for kids launched in November 2010.
Mathrubhumi Yearbook Plus
Mathrubhumi Azhchappathippu (Illustrated Weekly)
Contact Details of Mathrubhumi
PB No 46
K.P Kesava menon Road
Kozhikode-673001
Phone- 2366655, 2366544, 2366
Fax-2366656
Official Website– http://www.mathrubhumi.com
Chairman and Managing Director
Phone-2366423
One Response to "Mathrubhumi"
Mathrubhumi discussions
ettirankandath krishnadas Says:
This has reference to the news item under the caption "Palakkad-Pollachi gauge conversion loses steam" published in the national daily 'The Hindu' dated 10th August, 2014. It is a wake up call to all the people from Palakkad and its surrounding region to act. Hitherto fund constraint was made as a villain for delay in completion of the gauge conversion work in the Palakkad – Pollachi segment and acquisition of land and fund constraint as stumbling block for completion of gauge conversion work in Pollachi-Podhanur segment. Now sufficient fund allocation has been earmarked in the current budget for completion of the work and land acquisition has also been completed amicably, but surprising to note the cynicism about meeting the deadline of 31.12.2014. While presenting the Budget, the Union Railway Minister has made a point blank assertion that don't crib over new projects not being announced and that the States and the MPs should follow up completion of numerous pending ones. Therefore, any further delay in completion of on going work will deprive the regions for consideration of fresh initiatives/projects by the Railway Ministry in the near future. Mere raising the issue in the house by the elected representatives is not sufficient. The elected representatives should take more proactive massive action in an apolitical manner by involving all sections among the society from the region. While the undue hardship due to delay in completion of the rail connectivity is on daily commuters, tourists, people on pilgrimage, the major beneficiary due to this inordinate delay in gauge conversion is the powerful bus operators lobby. Unfortunate that the politicians in Kerala who agitate on Palestine and Israel issues and behaves like champions of global issues are not bothered and ready to consider this development issue of their region confronting the common as an issue.
Add Mathrubhumi Feedback, Comments, Reviews
Janmabhumi Newspaper

redpajama_set_name: RedPajamaCommonCrawl


British Invasion Week continues..... Stella McCartney, daughter of Paul & Linda McCartney, is a shining star in her own right. She is the creative force behind her highly regarded & successful fashion label - Stella McCartney. Recently, Vogue magazine featured the grounds & gardens of her English country home getaway. Check out the photos below & click here for more.


text:
10 ways other than Uber and Lyft to make money using your car
| November 19th, 2018
Work at Home & Telecommuting
Let's face it: A 9 to 5 is not for everyone, and that's OK. Advances in technology and the way work gets done means that, today, the sharing economy is a viable means of making a living for untold numbers of Americans.
One thing that means is that if you have a vehicle — whether it's paid off or not — you're in the driver's seat when it comes to being able to generate a considerable amount of income.
10 ways (besides Uber & Lyft) to make money driving your car
In this article, 'm going to show you some viable ways you can make money driving your car.
Much has been made about the most well-known ridesharing services, Uber and Lyft, and how they allow drivers to get paid for ferrying people around town. But the truth is that many of those drivers are making less than $10 an hour, according to NPR.
But take heart, there are more profitable ways to earn income behind the wheel. Here are several that you could consider, if they're available in your area:
Unlike Uber and Lyft, DoorDash doesn't care about the condition of your car, as long as you have a license, insurance and a clean driving record. DoorDash does offer supplemental insurance "up to $1,000,000 in bodily injury and/or property damage to third parties" if the crash occurs while on a "Dash" (delivery).
DoorDash gets food to people wherever they are, whether it's at a picnic, a family reunion, their home or place of business. You'll have to pass a background check, which is initiated during the application process on the site.
What about the pay? The company advertises that you can "Get your first check this week." According to the site, you get paid on a weekly basis for all deliveries completed between Monday to Sunday of the previous week (ending Sunday at midnight PST). Direct deposits usually take a few days, so expect to see the fruit of your labor on Wednesday nights, the service says in the website's FAQ section.
For "DoorDash Orders," delivery pay is calculated as $1 + 100% of tip + pay boost. The pay boost amount will vary based on order complexity, distance and other factors. For "Driver Orders," it's $5 per delivery and 100% of tip paid to DoorDash by the customer or merchant. Click here to their Dasher Pay Model FAQ.
Caviar couriers can earn up to $25 an hour driving or riding their bike and delivering food to their neighbors. Through its app, you can tell Caviar when you're available to work and orders will appear on your smartphone. From there, all you have to do is choose to accept or not.
What about the pay? "Money is deposited directly to your bank account each week," the service says on its website.
Like DoorDash, Roadie doesn't care how old or what condition your car is in, as long as you can get from point A to point B. The site matches people who need items delivered with drivers already headed in that direction. To join as a driver, you just sign up, create a "Gigs Alert," make offers and wait for them to be accepted. Two of the perks include discounted roadside assistance and free Waffle House!
What about the pay? Roadies get paid via direct deposit. "Each direct payment should appear in your designated bank account within 4 – 6 business days of Gig completion," it says on the site.
One of the largest service economy food-delivery platforms, Grubhub promises its drivers competitive pay and a flexible schedule. One benefit of driving for Grubhub is that the network is so vast, you can stay quite busy if you want to.
What about the pay? Many areas have guaranteed hourly minimums, so once you sign up, the company wants you to refer to your individual "Grubhub Payment Terms" to find out what your local rate will be. As for the formula, it says on Grubhub's support website that "You will be paid per order + 0.50 cents per mile from the restaurant to the customer + 100% of your tips." On online forums, many drivers said they made around $10 an hour.
Part of the Grubhub network, Seamless lets people order food for delivery or takeout from their favorite local restaurant. This means you get to drive for Seamless and Grubhub, giving you the opportunity to earn more money. Seamless has the cooler slogan, though, which includes the line, "No menus, no phone calls, no repeating yourself."
What about the pay? It's the same as Grubhub.
Wrapify
Wrapify will pay you for advertising a particular company by covering your vehicle's exterior in a branded wrap. Clients get to select daily mileage caps but the drivers get to approve campaigns before going forward. Drivers get paid by the mile along with a visibility formula that includes time of day and areas of heavy traffic.
What about the pay? For a full wrap, you can make between $264 and $452 a month, according to Wrapify's website. For a partial wrap, the earnings are $196 – $280 a month. And for just a panel, you can make around $84 – $140 every 30 days. Carvertise is a similar service, but they usually pay only about $100 a month.
Amazon Flex is the program Amazon uses, outside of its courier staff, for regular people to deliver packages to customers. Promoted as a "be your own boss" type of gig, Amazon Flex offers people who don't want to deal face-to-face with customers an easy way to make money. The flip side of that is that you have to be Johnny on the Spot at those fulfillment centers.
What about the pay? Amazon says that drivers can make $18 to $25 an hour, but you'd better not be late with those deliveries: The company's one-to-two-hour Prime Now promise is at stake.
Shipt
One of the most popular grocery-delivery services, Shipt was bought by Target late last year. If you sign up to drive for Shipt you're also more or less a shopper as well. That means you'll be responsible for orders as small as a stick of butter and as large as a whole shopping cart full of groceries.
What about the pay? Shipt says that most of its shoppers earn between $15 and $25 per hour, but that depends on the size of the orders, your ability to shop quickly and the distance you have to drive to complete the delivery. For an in-depth look at the service, See our 5 things to know about Shipt story.
Instacart promises some deliveries in as little as one hour. Once catering to a more exclusive crowd (Whole Foods), Instacart in recent years has expanded to major retailers such as Kroger, Food Lion and the like. The Instacart app is only available on the App Store for iOS devices (sorry, Android users).
What about the pay? Shoppers' wages are based on a per-item commission as well as per-order — and they vary depending on what city you're in. Some make more, some make less. This woman makes $17 an hour shopping.
If you happen to have a large vehicle like a van, truck or semi, you can make some decent money by becoming a carrier with uShip, a shipping marketplace that helps you bid on customers' shipments. You might surprised how many people need help hauling large items on a recurring basis — and cross-country. The site charges a fee for drivers, but that is typically built into your bid price.
What about the pay? It depends on what you bid. Like most of the other apps, there's no hand-to-hand payment. The customer pays the entire shipping cost at booking and releases it to the carrier at the time of delivery. Drivers get paid via payment code, available within their booking confirmation email.
Getting paid for driving your car looks like it's here to stay. The more sophisticated the sharing economy grows, the more prevalent services like these will become. And that's a good thing for people who want to have more flexibility in their lives and earn some cash at the same time.
RELATED: These apps let you earn extra money


In many patients suffering from Jock's Itch- a Tinea infection affecting the groin area- the skin in the groin and on the thighs turns patchy, dark, or, in some cases, bluish black. Itchiness is also another symptom here, although the ringworm has been cured.Naturally, many patients do not wish to see a doctor for tinea versicolor, as most are under the impression that they would be... 29/03/2008 · Best Answer: Tea tree oil, or 100% cocoa butter stick.both are good for scars and sometimes help to get the color back too.
Get Rid of Your Jock Itch for Good You can mix and match these treatments as you like. For example, soak in an Epsom salt bath, dry the skin, and then apply Sudocrem.
5 years ago, I contracted jock itch and it spread in my groin area, with both inner thighs affected. It was cured eventually, however the discoloration had been there for already 4 years and it has not faded as much as I thought it would.


text:
Three Great Arcade Apps to Try in 2017
Rahul Asthana 7 days ago
Evelyn Johnson 4 weeks ago
Socialnomics Trends February 24, 2017 0 0
Although there's always an endless series of great social media apps to try out, sometimes we just want to keep it old-school and kick back with some classic arcade games.
So here's a look at some great updates of classic arcade styles in 2017 that could reintroduce you with the fun of platforming, the thrill of a casino win, and with the excitement of the racetrack.
Super Mario Run: Platforming Fun
One of the biggest gaming hits at the end of last year was the arrival of Super Mario Run to our mobile devices. This was Nintendo's first major entry to gaming apps, and it quickly achieved some remarkable sales figures, as gamers flocked to try out the fun of a Mario platform title on a smartphone.
What really made Super Mario Run succeed was the clever way that the instantly recognizable arcade platforming format was cleverly updated for the touchscreen, thanks to some simple swipe-able controls. Although the app is currently only available for iOS devices, it's expected to get its Android release in March and will become one of the biggest mobile games of all time.
Mr. Smith Casino slots and table games
Casino games have been one of the surprise success stories of mobile gaming in recent years. And if you're looking to see how this has happened, then simply head over to MrSmithCasino and check out the jackpots on offer on their slots games.
With jackpots frequently running into seven figures and a massive range of slots styles and looks to choose from, it shows just how diverse the casino gaming scene has become recently. And what's best is that both the website and app feature some classic table live games, like roulette and blackjack that can put a real casino experience in the palm of your hand.
Horizon Chase: World Tour Racing Excellence
And finally, for anyone who grew up on classic racing arcade games like Out Run, then Horizon Chase: World Tour offers a great way to revisit this much-loved genre.
Thankfully, the developers have stayed true to the original arcade inspiration as the stylish retro graphics and 2D scrolling backgrounds are matched by the simple controls of steering, gas and nitro to ensure that things never get too tedious.
And while it lacks the multiplayer options found at some of the live casino games, its got just enough retro charm to ensure that it becomes one of the great modern gaming apps.
Socialnomics Trends
The Socialnomics Team is always looking for the latest global trending news around the Biz & Buzz of Tech. Socialnomics Trends is our collaborative team, ensuring you are in the know.
Previous ArticleThings To Consider When Starting A New Financial Services Company
Next ArticleHow to Optimize Instagram as A Small Business


TORTILLA SOUP is the story of Martin Naranjo (Hector Elizondo) - a retired, widowed, Mexican-American chef who cannot taste anything and lives in LA with his three beautiful daughters. He passionately loves to cook meals for his family and uphold the traditional sit-down meal at dinner times, which equally brings all the family together to enjoy his cooking and discuss family matters.
However his daughters are all searching for greater fulfilment outside of their family. Maribel (Tamara Mello), studying at college is fed up with the singles scene and wants to have a long-term relationship with someone. Carmen (Jacqueline Obradors) is a forthright businesswoman but is becoming increasingly annoyed by the way her boyfriend is easily distracted by other women. Lastly Leticia (Elizabeth Pena) a school teacher, who has suppressed her own romantic feelings, now longs for a meaningful, loving relationship.
Then their father meets an outgoing, vivacious, divorced woman, which starts a trend in the family as each of his daughters start to find someone else who will give them the fulfillment they are missing and spice up their lives.