text:
\section{Introduction}

Deep Q-Learning (DQL), a family of reinforcement learning algorithms that includes Deep Q-Network (DQN) \cite{Mnih2013, Mnih2015} and its continuous-action variants \cite{Lillicrap2016, Fujimoto2018, Haarnoja2018}, is often successful at training deep neural networks for control. In DQL, a function approximator (a deep neural network) learns to estimate the value of each state-action pair under the optimal policy (the $Q$-function), and a control policy selects actions with the highest values according to the current $Q$-function approximator. DQL algorithms have been applied fruitfully in video games \cite{Mnih2015}, robotics \cite{Kalashnikov2018, Haarnoja2018a}, and user interactions on social media \cite{Gauci2018}.

However, despite the high-profile empirical successes, these algorithms possess failure modes that are poorly understood and arise frequently in practice. The most common failure mode is divergence, where the $Q$-function approximator learns to ascribe unrealistically high values to state-action pairs, in turn destroying the quality of the greedy control policy derived from $Q$ \cite{VanHasselt2018}. Divergence in DQL is often attributed to three components common to all DQL algorithms, which are collectively considered the `deadly triad' of reinforcement learning \cite{Sutton1988, Sutton2018}:
\begin{itemize}
\item function approximation, in this case the use of deep neural networks,
\item off-policy learning, the use of data collected on one policy to estimate the value of another policy,
\item and bootstrapping, where the $Q$-function estimator is regressed towards a function of itself.
\end{itemize}
Well-known examples \cite{Baird1995, Tsitsiklis1997} demonstrate the potential of the deadly triad to cause divergence in approximate $Q$-learning. However, actionable descriptions of divergence in the general case remain elusive, prompting algorithm designers to attack the triad with an increasingly wide variety of heuristic solutions. These include target networks \cite{Mnih2015}, entropy regularization \cite{Fox2016, Haarnoja2018}, n-step learning \cite{Hessel2017}, and approximate double-Q learning \cite{VanHasselt2016}.

The absence of theoretical characterization for divergence in DQL makes it challenging to reliably deploy DQL on new problems. To make progress toward such a characterization, we give an analysis inspired by \citet{Gordon1995}, who studied the behavior of approximate value learning algorithms in value space. We examine how $Q$-values change under a standard DQL update, and derive the leading order approximation to the DQL update operator. The approximate update turns out to have a simple form that disentangles and clarifies the contributions from the components of the deadly triad, and allows us to identify the important role played by the neural tangent kernel (NTK) \cite{Jacot2018} of the $Q$ approximator. We consider conditions under which the approximate update is or isn't a contraction map in the sup norm, based on the intuition that when it is a contraction DQL should behave stably, and when it is an expansion we should expect divergence.

Based on our analysis, we design an algorithm which is intended to approximately ensure that the $Q$-function update is non-expansive. Our algorithm, which we call Preconditioned Q-Networks (PreQN) is computationally expensive but theoretically simple: it works by preconditioning the TD-errors in minibatch gradient updates, using the inverse of a matrix of inner products of $Q$-function gradients. We demonstrate that PreQN is stable and performant on a standard slate of MuJoCo benchmarks from the OpenAI Gym \cite{Brockman2016}, despite using none of the tricks typically associated with DQL. We also find a neat connection between PreQN and natural gradient \cite{Amari1998} methods, where under some slightly restricted conditions, the PreQN update is equivalent to a natural gradient Q-learning update. This connection explains a result noted by \citet{Knight2018}: that natural gradient Q-learning appeared to be stable without target networks.

\section{Preliminaries}

\subsection{Contraction Maps and Fixed Points}

We begin with a brief mathematical review. Let $X$ be a vector space with norm $\|\cdot\|$, and $f$ a function from $X$ to $X$. If $\forall x, y \in X$, $f$ satisfies
\begin{equation}
\|f(x) - f(y)\| \leq \beta \|x - y\| \label{contraction}
\end{equation}
with $\beta \in [0,1)$, then $f$ is called a contraction map with modulus $\beta$. If $f$ satisfies Eq \ref{contraction} but with $\beta=1$, then $f$ is said to be a non-expansion.

By the Banach fixed-point theorem, if $f$ is a contraction, there is a unique fixed-point $x$ such that $f(x)=x$, and it can be obtained by the repeated application of $f$: for any point $x_0 \in X$, if we define a sequence of points $\{x_n\}$ such that $x_n = f(x_{n-1})$, $\lim_{n\to\infty} x_n = x$.

\subsection{Q Functions and TD-Learning}
DQL algorithms learn control policies in the reinforcement learning (RL) setting with the infinite-horizon discounted return objective. They attempt to learn an approximator to the optimal action-value function $Q^*$, which is known to satisfy the optimal Bellman equation:
\begin{equation}
Q^*(s,a) = \underE{s' \sim P}{R(s,a,s') + \gamma \max_{a'} Q^* (s', a')}. \label{bellman}
\end{equation}
Here, $s$ and $s'$ are states, $a$ and $a'$ are actions, $P$ is the transition kernel for the environment, $R$ is the reward function, and $\gamma \in (0,1)$ is the discount factor. The optimal policy $\pi^*$ can be obtained as the policy which is greedy with respect to $Q^*$: $\pi^*(s) = \arg \max_{a} Q^*(s,a)$. Thus, DQL algorithms approximate the optimal policy as the policy which is greedy with respect to the $Q^*$-approximator.

Let ${\mathcal T}^* : {\mathcal Q} \to {\mathcal Q}$ be the operator on Q functions with ${\mathcal T}^*Q (s,a)$ given by the RHS of Eq \ref{bellman}; then Eq \ref{bellman} can be written as $Q^* = {\mathcal T}^* Q^*$.  The operator ${\mathcal T}^*$ is called the optimal Bellman operator, and it is a contraction in the sup norm with modulus $\gamma$. When the $Q$-function can be represented as a finite table and the reward function and transition kernel are fully known, ${\mathcal T}^*$ can be computed exactly and $Q^*$ can be obtained by \textit{value iteration}: $Q_{k+1} = {\mathcal T}^*Q_k$. The convergence of value iteration from any initial point $Q_0$ is guaranteed by the Banach fixed-point theorem.

When the reward function and transition kernel are not fully known, it is still possible to learn $Q^*$ in the tabular setting via $Q$-learning \cite{Watkins1992}. $Q$-learning updates the $Q$-values of state-action pairs as they are visited by some exploration policy according to:
\begin{equation}
Q_{k+1}(s,a) = Q_k (s,a) + \alpha_k \left( \hat{{\mathcal T}}^*Q_k (s,a) - Q_k(s,a)\right), \label{watkins}
\end{equation}
where $\hat{{\mathcal T}}^* Q_k(s,a) = r + \gamma \max_{a'} Q_k(s',a')$ is a sample estimate for ${\mathcal T}^* Q_k (s,a)$ using the reward and next state obtained from the environment while exploring. Under mild conditions on state-action visitation (all pairs must be visited often enough) and learning rates $\alpha_k$ (they must all gradually go to zero and always lie in $[0,1)$), $Q$-learning converges to $Q^*$. $Q$-learning is called a \textit{temporal difference} (TD) algorithm because the updates to $Q$-values are based on the temporal difference error:
\begin{align*}
\delta_t &= \hat{{\mathcal T}}^* Q(s_t,a_t) - Q(s_t,a_t) \\
&= r_t + \gamma \max_{a'} Q(s_{t+1},a') - Q(s_t,a_t).
\end{align*}

\section{Towards Characterizing Divergence in Deep Q-Learning}

Deep Q-Learning (DQL) algorithms are based on the generalization of Eq \ref{watkins} to the function approximation setting:
\begin{equation}
\theta' = \theta + \alpha \left(\hat{{\mathcal T}}^*Q_{\theta}(s,a) - Q_{\theta}(s,a)\right) \nabla_{\theta}Q_{\theta}(s,a), \label{approxqlearning}
\end{equation}
where $Q_{\theta}$ is a differentiable function approximator with parameters $\theta$, and $\theta'$ are the parameters after an update. Note that when $Q_{\theta}$ is a table, Eq \ref{approxqlearning} reduces exactly to Eq \ref{watkins}.

Typically, DQL algorithms make use of experience replay \cite{Lin1992} and minibatch gradient descent \cite{Mnih2013}, resulting in an expected update:
\begin{equation}
\theta' = \theta + \alpha \underE{s,a \sim \rho}{ \left({\mathcal T}^*Q_{\theta}(s,a) - Q_{\theta}(s,a)\right) \nabla_{\theta}Q_{\theta}(s,a)}, \label{deepq}
\end{equation}
where $\rho$ is the distribution of experience in the replay buffer at the time of the update. For stability, it is conventional to replace the bootstrap, the ${\mathcal T}^* Q_{\theta}$ term, with one based on a slowly-updated target network: ${\mathcal T}^*Q_{\psi}$, where the parameters $\psi$ are either infrequently copied from $\theta$ \cite{Mnih2013} or obtained by Polyak averaging $\theta$ \cite{Lillicrap2016}. However, we will omit target networks from our analysis.

Unlike Q-learning, DQL in its standard form currently has no known convergence guarantees, although some convergence results \cite{Yang2019} have been obtained for a closely-related algorithm called Neural-Fitted Q Iteration \cite{Riedmiller2005} when used with deep ReLU networks.

\subsection{Taylor Expansion Analysis of Q}

To gain a deeper understanding of the behavior of DQL, we study the change in Q-values following an update based on Eq \ref{deepq}, by examining the Taylor expansion of $Q$ around $\theta$ at a state-action pair $(\bar{s},\bar{a})$. The Taylor expansion is
\begin{equation}
Q_{\theta'}(\bar{s},\bar{a}) = Q_{\theta}(\bar{s}, \bar{a}) + \nabla_{\theta} Q_{\theta}(\bar{s},\bar{a})^T (\theta' - \theta) + {\mathcal O}\left(\|\theta'-\theta\|^2\right), \label{taylor}
\end{equation}
and by plugging Eq \ref{deepq} into Eq \ref{taylor}, we obtain:
\begin{align}
Q_{\theta'}(\bar{s},&\bar{a}) = \\
& Q_{\theta}(\bar{s}, \bar{a}) \nonumber \\
& + \alpha \underE{s,a\sim\rho}{k_{\theta}(\bar{s},\bar{a},s,a) \left({\mathcal T}^*Q_{\theta}(s,a) - Q_{\theta}(s,a)\right)} \nonumber \\
& + {\mathcal O}\left(\|\alpha g\|^2\right), \label{deepqtaylor}
\end{align}
where
\begin{equation}
k_{\theta}(\bar{s},\bar{a},s,a)\doteq \nabla_{\theta} Q_{\theta}(\bar{s}, \bar{a})^T \nabla_{\theta} Q_{\theta}(s,a) \label{kernelcomponent}
\end{equation}
is the neural tangent kernel (NTK) \cite{Jacot2018}, and $\alpha g = \theta' - \theta$. It is instructive to look at Eq \ref{deepqtaylor} for the case of finite state-action spaces, where we can consider its matrix-vector form.

\begin{theorem} For Q-learning with nonlinear function approximation based on the update in Eq \ref{deepq}, when the state-action space is finite and the $Q$ function is represented as a vector in $\Real{|S||A|}$, the $Q$-values before and after an update are related by:
\begin{equation}
Q_{\theta'} = Q_{\theta} + \alpha K_{\theta} D_{\rho} \left({\mathcal T}^*Q_{\theta} - Q_{\theta} \right) + {\mathcal O}(\|\alpha g\|^2), \label{deadlytriad}
\end{equation}
where $K_{\theta}$ is the $|S||A|\times|S||A|$ matrix of entries given by Eq \ref{kernelcomponent}, and $D_{\rho}$ is a diagonal matrix with entries given by $\rho(s,a)$, the distribution from the replay buffer.
\end{theorem}

Although extremely simple, we believe that Eq \ref{deadlytriad} is illuminating because it shows the connection between the deadly triad and the thing we really care about: the $Q$-values themselves. At leading order,
\begin{itemize}
\item the $K_{\theta}$ term is the contribution from function approximation, with its off-diagonal terms creating generalization across state-action pairs,
\item the $D_{\rho}$ term is the contribution from the off-policy data distribution,
\item the ${\mathcal T}^*Q_{\theta}$ term is the contribution from bootstrapping,
\end{itemize}
and these terms interact by multiplication. The form of Eq \ref{deadlytriad} suggests that a useful way to think about the stability and convergence of deep Q-learning is to reason about whether the leading-order update operator ${\mathcal U} : {\mathcal Q} \to {\mathcal Q}$ with values
\begin{equation}
{\mathcal U} Q_{\theta} = Q_{\theta} + \alpha K_{\theta} D_{\rho} \left({\mathcal T}^*Q_{\theta} - Q_{\theta} \right) \label{dqlupdate}
\end{equation}
is or is not a contraction on ${\mathcal Q}$. In what follows, we will develop an intuition for such conditions by considering a sequence of operators that incrementally introduce the components of ${\mathcal U}$. After building intuition for failure modes, we will consider how prior methods try to repair or mitigate them. We contend that prior work in DQL predominantly focuses on the data distribution or the bootstrap, with limited exploration of the contribution from $K_{\theta}$ to instability. This analysis inspires PreQN, our new algorithm which attempts to repair divergence issues by cancelling out within-batch generalization errors created by $K_{\theta}$.

\subsection{Building Intuition for Divergence}

The aim of this section is to understand how the update ${\mathcal U} : {\mathcal Q} \to {\mathcal Q}$ given by Eq \ref{dqlupdate} can give rise to instability in deep Q-learning, and how that instability might be repaired. To begin with, consider the operator ${\mathcal U}_1$ given by
\begin{equation}
{\mathcal U}_1 Q = Q + \alpha \left({\mathcal T}^*Q - Q\right). \label{update1}
\end{equation}

\begin{restatable}{lemma}{lemuone} For $\alpha \in (0,1)$, ${\mathcal U}_1$ given by Eq \ref{update1} is a contraction on ${\mathcal Q}$ in the sup norm, and its fixed-point is $Q^*$.
\end{restatable}
Proof for this and all other results in appendix. With sampling, ${\mathcal U}_1$ would be essentially the same operator as used in tabular Q-learning \cite{Watkins1992}, and it would benefit from similar performance guarantees. This gives us intuition point 1:

\textbf{Intuition 1:} When ${\mathcal U}$ more closely resembles ${\mathcal U}_1$, we should expect deep Q-learning to behave more stably.

Next, we consider the operator ${\mathcal U}_2$ given by
\begin{equation}
{\mathcal U}_2 Q = Q + \alpha D_{\rho}\left({\mathcal T}^*Q - Q\right), \label{update2}
\end{equation}
where $D_{\rho}$ is a diagonal matrix with entries $\rho(s,a)$, a probability mass function on state-action pairs.

\begin{restatable}{lemma}{lemutwo} If $\rho(s,a) > 0$ for all $s,a$ and $\alpha \in (0, 1/\rho_{max})$ where $\rho_{max} = \max_{s,a} \rho(s,a)$, then ${\mathcal U}_2$ given by Eq \ref{update2} is a contraction in the sup norm and its fixed-point is $Q^*$. If there are any $s,a$ such that $\rho(s,a)=0$  and $\alpha \in (0,1/\rho_{max})$, however, it is a non-expansion in $Q$ and not a contraction.
\end{restatable}

By considering ${\mathcal U}_2$, we see how the data distribution can have an impact: as long as the exploration policy touches all state-action pairs often enough, ${\mathcal U}_2$ behaves well, but missing data poses a problem. The $Q$-values for missing state-action pairs will never change from their initial values, and bootstrapping will cause those erroneous values to influence the $Q$-values for `downstream' state-action pairs. This leads us to our second point of intuition:

\textbf{Intuition 2:} When data is scarce, deep Q-learning may struggle, and initial conditions will matter more.

Data is scarcest at the beginning of training; empirical results from \citet{VanHasselt2018} suggest that this is when DQL is most susceptible to divergence.

Next, we consider the operator ${\mathcal U}_3$ given by
\begin{equation}
{\mathcal U}_3 Q = Q + \alpha K D_{\rho}\left({\mathcal T}^*Q - Q\right), \label{update3}
\end{equation}
where $K$ is a constant symmetric, positive-definite matrix. Interestingly, ${\mathcal U}_3$ corresponds exactly to the expected update for the case of linear function approximation, which we make precise in the next lemma:

\begin{restatable}{lemma}{lemuthree} For $Q$-learning with linear function approximators of the form  $Q_{\theta}(s,a) = \theta^T \phi(s,a)$ and updates based on Eq. \ref{deepq}, under the same conditions as Theorem 1, the $Q$-values before and after an update are related by
\begin{equation}
Q_{\theta'} = {\mathcal U}_3 Q_{\theta}, \label{lindeadlytriad}
\end{equation}
where $K(\bar{s},\bar{a},s,a) = \phi(\bar{s},\bar{a})^T \phi(s,a)$. Eq. \ref{lindeadlytriad} differs from Eq. \ref{deadlytriad} in that there are no higher-order terms, and $K$ is constant with respect to $\theta$.
\end{restatable}

We now consider when ${\mathcal U}_3$ is a contraction in the sup norm.
\begin{restatable}{theorem}{thmtwo}
Let indices $i$, $j$ refer to state-action pairs. Suppose that $K$ and $\rho$ satisfy the conditions:
\begin{align}
\forall i, & \;\;\;\;\; \alpha K_{ii} \rho_i < 1, \label{overshoot}\\
\forall i, & \;\;\;\;\; (1+\gamma) \sum_{j\neq i} |K_{ij}| \rho_j \leq (1-\gamma) K_{ii}\rho_i. \label{diagdom}
\end{align}
Then ${\mathcal U}_3$ is a contraction on ${\mathcal Q}$ in the sup norm, with fixed-point $Q^*$.
\end{restatable}
To frame this discussion, we'll note that the condition in Eq. \ref{diagdom} is extremely restrictive. It requires that $\rho > 0$ everywhere, and that the off-diagonal terms of $K$ are very small relative to the on-diagonal terms (for typical choices of $\gamma$, eg $\gamma=0.99$). As a result, an analysis of this kind may not suffice to explain the success of linear TD-learning in typical use cases. But we nonetheless find this useful in motivating the following point of intuition:

\textbf{Intuition 3:} The stability of $Q$-learning is tied to the generalization properties of the $Q$-approximator. Approximators with more aggressive generalization (larger off-diagonal terms in $K_{\theta}$) are less likely to demonstrate stable learning.

So far, we have reasoned about several individual update operators, but we have not made explicit reference to the full dynamics of training with nonlinear function approximators. In deep Q-learning, both the kernel $K_{\theta}$ and the data distribution $\rho$ change between update steps. Thus, each step can be viewed as applying a different update operator. It is important to ask if our intuitions so far have any bearing in this setting; this is the subject of our next result.

\begin{restatable}{theorem}{thmthree} Consider a sequence of updates $\{{\mathcal U}_0, {\mathcal U}_1, ...\}$, with each ${\mathcal U}_i : {\mathcal Q} \to {\mathcal Q}$ Lipschitz continuous, with Lipschitz constant $\beta_i$, with respect to a norm $\|\cdot\|$. Furthermore, suppose all ${\mathcal U}_i$ share a common fixed-point, $\tilde{Q}$. Then for any initial point $Q_0$, the sequence of iterates produced by $Q_{i+1} = {\mathcal U}_i Q_i$ satisfies:
\begin{equation}
\|\tilde{Q} - Q_i\| \leq \left(\prod_{k=0}^{i-1} \beta_k\right) \|\tilde{Q} - Q_0\|. \label{qconverge}
\end{equation}
Furthermore, if there is an iterate $j$ such that $\forall k \geq j, \; \beta_k \in [0,1)$, the sequence $\{Q_0, Q_1, ...\}$ converges to $\tilde{Q}$.
\end{restatable}

Roughly speaking, this theorem says that if you sequentially apply different contraction maps with the same fixed-point, you will attain that fixed-point.

In DQL, the common fixed-point between all update operators based on Eq. \ref{deepq} is $Q^*$. For neural network approximators commonly used in practice, such update operators are unlikely to be contractions and convergence to $Q^*$ is out of reach (especially considering that $Q^*$ may not even be expressible in the approximator class). Nonetheless, we view Theorem 3 as motivating our final point of intuition:

\textbf{Intuition 4:} Although the DQL update operator varies between steps, intuitions from the constant-update setting can provide useful guidance for understanding and repairing divergence issues in DQL.

To sum up, we enumerate and discuss the failure modes for DQL that appear likely based on our analysis so far.

\textbf{Failure Mode 1: Linearization breaks.} The learning rate $\alpha$ is too high, second-order terms in Eq. \ref{deadlytriad} are large, and updates do not correlate with Bellman updates in any meaningful way. (Based on Theorem 1.)

\textbf{Failure Mode 2: Overshooting.} The learning rate $\alpha$ is small enough for the linearization to approximately hold, but is large enough that ${\mathcal U}$ from Eq. \ref{dqlupdate} is sometimes an expansion. (Based on Theorem 2, Eq \ref{overshoot}.)

\textbf{Failure Mode 3: Over-generalization.} The kernel matrix $K_{\theta}$ has large off-diagonal terms, causing the $Q$ function to generalize too aggressively and making ${\mathcal U}$ sometimes an expansion. (Based on Theorem 2, Eq \ref{diagdom}.)

\textbf{Failure Mode 4: Extrapolation error.} The data distribution used for the update is inadequate. $Q$-values for missing (or under-represented) state-action pairs are adjusted solely or primarily by generalization, which sometimes produces errors. Bootstrapping then propagates those errors through the $Q$-values for all other state-action pairs. (Based on Lemma 2.) This failure mode was previously identified, named, and studied empirically by \cite{Fujimoto2018a}.

It is important to note that these failure modes may not present in clearly-distinguishable ways: indeed, they can cascade into each other, creating feedback loops that lead to divergence. For instance, consider the interaction between over-generalization and extrapolation error. A network with limited generalization would keep the $Q$-values for missing state-action pairs close to their initial values. This would lead to inaccurate, but not necessarily divergent, downstream $Q$-values. On the other hand, a network that over-generalizes will significantly alter the $Q$-values for missing state-action pairs. A slight positive bias (where all of those $Q$-values increase) will get propagated to downstream $Q$-values due to extrapolation error, making them optimistic. But this reinforces the positive bias in the generalization to missing state-action pair $Q$-values---creating a feedback loop, and ultimately divergence.

\subsection{Interpreting Prior Work}

A substantial body of prior work on stabilizing DQL focuses on modifying either the data distribution or the TD-errors.

Data distribution-based methods include massive-scale experience collection, as in Gorila-DQN \cite{Nair2015}, Ape-X \cite{Horgan2018}, and R2D2 \cite{Kapturowski2019}, and methods for improved exploration, like entropy regularization \cite{Haarnoja2018}. We speculate that such methods improve stability in DQL primarily by mitigating extrapolation error, by reducing the number of missing state-action pairs. As an alternative to improved data collection, BCQ \cite{Fujimoto2018a} mitigates extrapolation error by simply preventing missing state-action pairs from being used to form the Bellman backup.

TD error-based methods include target networks \cite{Mnih2015}, clipped TD errors \cite{Mnih2015} (commonly implemented via the Huber loss function \cite{Sidor2017}), double DQN \cite{VanHasselt2016}, n-step backups \cite{Sutton1988, Hessel2017}, transformed Bellman operators \cite{Pohlen2018}, and clipped double-Q learning \cite{Fujimoto2018}. These methods do not directly attack specific failure modes, but we speculate that they interfere with error propagation by preventing bad updates from quickly spreading to downstream $Q$-values. This allows more time for bad updates to get averaged out, or for missing data to be collected.

Relatively little work focuses on over-generalization. Ape-X DQfD \cite{Pohlen2018} uses an auxilliary temporal consistency (TC) loss to prevent the $Q$-values of target state-action pairs from changing; ablation analysis suggested that the TC loss was critical to performance. Similarly, \cite{Durugkar2017} proposed Constrained Q-Learning, which uses a constraint to prevent the average target value from changing after an update; however, \cite{Pohlen2018} did not find evidence that this technique worked on complex problems.

To the best of our knowledge, no prior work addresses the root cause of overshooting or over-generalization failures in DQL: the neural tangent kernel, $K_{\theta}$. Work in this direction would either modify network architecture to result in a $K_{\theta}$ more favorable to stability, or modify the update rule in a way which controls the influence of $K_{\theta}$ on generalization. Dueling DQN \cite{Wang2016} does modify network architecture in a way which is known to improve training on Atari games, but there is currently no known theoretical explanation for its benefits. We speculate that an analysis based on $K_{\theta}$ may prove insightful, though we have not yet found a clear result on this despite preliminary effort. The general absence of work on DQL stability via $K_{\theta}$ is the inspiration for our algorithmic contributions.

\section{Preconditioned Q-Networks}

In this section, we will introduce Preconditioned Q-Networks (PreQN), an algorithm which is intended to approximately ensure that the $Q$-function update is a non-expansion. The core idea is to alter the DQL update so that it behaves as much as possible like Eq. \ref{update1} in $Q$-value space.

Concretely, let $\Phi_{\theta} \in \Real{d \times |S||A|}$ denote the matrix whose columns are $\nabla_{\theta} Q_{\theta}(s,a)$. To first order, what we have is
\begin{equation}
Q_{\theta'} \approx Q_{\theta} + \Phi_{\theta}^T (\theta' - \theta), \label{have}
\end{equation}
and what we want is an update which results in
\begin{equation}
Q_{\theta'} \approx Q_{\theta} + \alpha \left( {\mathcal T}^* Q_{\theta} - Q_{\theta}\right), \label{want}
\end{equation}
for some $\alpha \in (0,1)$. If $K_{\theta} = \Phi_{\theta}^T \Phi_{\theta}$ were invertible, then the update
\begin{equation}
\theta' = \theta + \alpha \Phi_{\theta} K_{\theta}^{-1} \left( {\mathcal T}^* Q_{\theta} - Q_{\theta}\right) \label{theoryupdate}
\end{equation}
would attain Eq \ref{want}. This update is like a normal DQL update where the TD-errors have been replaced with \textit{preconditioned} TD-errors, where $K_{\theta}^{-1}$ is the preconditioner. In practice, there are three obstacles to directly implementing Eq \ref{theoryupdate}:
\begin{itemize}
\item For large or continuous state or action spaces (as in many problems of interest), it would be intractable or impossible to form and invert $K_{\theta}$.
\item If the number of state-action pairs is greater than the number of parameters, $K_{\theta}$ will be rank deficient and thus not invertible.
\item For nonlinear function approximators (as in DQL), step sizes must be selected to keep higher-order terms small.
\end{itemize}
To handle these issues, we propose a minibatch-based approximation to the algorithm in Eq \ref{theoryupdate}. Like in standard DQL, we maintain a replay buffer filled with past experiences. Each time we sample a minibatch $B$ from the replay buffer to compute an update, we form $K_{\theta}$ for the minibatch, find the least-squares solution $Z$ to
\begin{equation}
K_{\theta} Z = {\mathcal T}^* Q_{\theta} - Q_{\theta} \label{preqn1}
\end{equation}
for the minibatch, and then compute a proposed update
\begin{equation}
\theta' = \theta + \alpha \sum_{(s,a) \in B} Z(s,a) \nabla_{\theta} Q_{\theta}(s,a). \label{preqnupdate}
\end{equation}
Finally, to ensure that the higher-order terms are small, we use a linesearch that starts at Eq \ref{preqnupdate} and backtracks (by exponential decay) to $\theta$. The acceptance criterion for the linesearch is
\begin{equation}
\cos\left( Q_{\theta'} - Q_{\theta}, {\mathcal T}^*Q_{\theta} - Q_{\theta}\right) \geq \eta,
\end{equation}
where $\eta$ is a hyperparameter (close to, but less than, $1$). That is, a proposed update is only accepted if the resulting change in $Q$-values for the minibatch is well-aligned with its TD-errors. We refer to this algorithm as Preconditioned Q-Networks (PreQN).

In our experiments, we consider the variant of PreQN styled after DDPG \cite{Lillicrap2016}, where a separate actor network is trained to allow efficient computation of $\max_{a} Q_{\theta}(s,a)$. We give the complete pseudocode as Algorithm \ref{alg1}. Note the omission of target networks: we hypothesize that the design of the algorithm makes instability less likely and thus makes target networks unnecessary.

\begin{algorithm}[t]
\small
   \caption{PreQN (in style of DDPG)}
   \label{alg1}
\begin{algorithmic}[1]
     \STATE Given: initial parameters $\theta, \phi$ for $Q, \mu$, empty replay buffer ${\mathcal D}$
	 \STATE Receive observation $s_0$ from environment
	 \FOR{$t = 0,1,2,...$}
	 \STATE Select action $a_t = \mu_{\phi}(s_t) + {\mathcal N}_t$
	 \STATE Step environment to get $s_{t+1}, r_t$ and terminal signal $d_t$
	 \STATE Store $(s_t, a_t, r_t, s_{t+1}, d_t) \to {\mathcal D}$
	 \IF{it's time to update}
	 \FOR{however many updates}
	 \STATE Sample minibatch $B = \{(s_i,a_i,r_i,s'_i,d_i)\}$ from ${\mathcal D}$
	 \STATE For each transition in $B$, compute TD errors:
	 \begin{equation*}
	 \Delta_i = r_i + \gamma (1-d_i) Q_{\theta}(s'_i, \mu_{\phi}(s'_i)) - Q_{\theta}(s_i,a_i)
	 \end{equation*}
	 \STATE Compute minibatch $K_{\theta}$ matrix and find least-squares solution $Z$ to $K_{\theta} Z = \Delta$
	 \STATE Compute proposed update for $Q$ with:
	 \begin{equation*}
	 \theta' = \theta + \alpha_q \sum_{(s,a) \in B} Z(s,a) \nabla_{\theta} Q_{\theta}(s,a)
	 \end{equation*}
	 \STATE Exponentially decay $\theta'$ towards $\theta$ until
	 %
	 \begin{equation*}
	 \cos\left( Q_{\theta'} - Q_{\theta}, {\mathcal T}^*Q_{\theta} - Q_{\theta}\right) \geq \eta,
	 \end{equation*}
	 %
	 then set $\theta \leftarrow \theta'$.
	 \STATE Update $\mu$ with:
	 \begin{equation*}
	 \phi \leftarrow \phi + \alpha_{\mu} \frac{1}{|B|}\sum_{s \in B} \nabla_{\phi} Q_{\theta}(s, \mu_{\phi}(s))
	 \end{equation*}
	 \ENDFOR
	 \ENDIF
	\ENDFOR
\end{algorithmic}
\end{algorithm}

\subsection{Connection to Natural Gradients}

As it turns out, PreQN is equivalent to natural gradient Q-learning (NGQL) when the same samples are used to form both the gradient and the Fisher information matrix. To recap, the update for NGQL is
\begin{equation}
\theta' = \theta + \alpha F^{-1}_{\theta} g, \label{ngql}
\end{equation}
where $g$ is the gradient from Eq \ref{deepq} and
\begin{equation}
F_{\theta} = \underE{s,a \sim \rho}{\nabla_{\theta} Q_{\theta}(s,a) \nabla_{\theta}Q_{\theta}(s,a)^T} \label{fisher}
\end{equation}
is the Fisher information matrix for a gaussian distribution, ${\mathcal N}(Q_{\theta},I)$. When using sample estimates of the expectations, we can write the NGQL update in terms of the matrix $\Phi_{\theta}$ (the $d \times |S||A|$ matrix with columns $\nabla_{\theta}Q_{\theta}(s,a)$) and the vector of TD-errors $\Delta = {\mathcal T}^*Q_{\theta} - Q_{\theta}$ as:
\begin{equation}
\theta' = \theta + \alpha (\Phi_{\theta} \Phi_{\theta}^T)^\dagger \Phi_{\theta} \Delta, \label{ngql1}
\end{equation}
where $(\Phi_{\theta} \Phi_{\theta}^T)^\dagger$ is the pseudoinverse of $\Phi_{\theta} \Phi_{\theta}^T$. Similarly, the PreQN update as described by Eqs \ref{preqn1} and \ref{preqnupdate} can be written as
\begin{equation}
\theta' = \theta + \alpha \Phi_{\theta} (\Phi_{\theta}^T \Phi_{\theta})^\dagger \Delta. \label{preqn2}
\end{equation}
By the following lemma, the two updates in Eqs \ref{ngql1} and \ref{preqn2} are equivalent:
\begin{restatable}{lemma}{lemufour} $(\Phi_{\theta} \Phi_{\theta}^T)^\dagger \Phi_{\theta} = \Phi_{\theta} (\Phi_{\theta}^T \Phi_{\theta})^\dagger$.
\end{restatable}
The connection between NGQL and approximately non-expansive $Q$-update operators may explain the finding by \cite{Knight2018} that NGQL did not require target networks to remain stable. A related observation was made by \cite{Schulman2017}, who showed that a natural policy gradient could be viewed as approximately applying a version of Eq \ref{update1} for entropy-regularized Q-learning. They also demonstrated a version of DQL that could learn stably without target networks.

\section{Experiments}

In our experiments, we investigated the following questions:
\begin{enumerate}
\item What insights can we obtain about the neural tangent kernel $K_{\theta}$ in the context of RL? Can we exploit empirical analyses of $K_{\theta}$ to make better decisions about neural network architecture?
\item How does PreQN behave? To evaluate performance, we compare to TD3 \cite{Fujimoto2018} and SAC \cite{Haarnoja2018} on various OpenAI Gym \cite{Brockman2016} environments.
\item To what degree does a standard DQL update push $Q$-values towards their targets? How does this change with architecture? How should we interpret PreQN results in light of this?
\end{enumerate}

\subsection{Neural Tangent Kernel Analysis}

Based on Theorem 2, we are motivated to empirically evaluate two properties of the neural tangent kernel that appear relevant to stability in DQL: the magnitudes of diagonal elements, and the degree of off-diagonalness. To measure the latter, we consider the ratio of the average off-diagonal row entry to the on-diagonal entry, $R_i$:
\begin{align*}
R_i (K) &\doteq \frac{1}{N}\frac{\sum_{j\neq i} |K_{ij}|}{K_{ii}},
\end{align*}
where $N$ is the size of the square matrix $K$. We refer to this quantity as the `row ratio.'

We evaluate the standard neural networks used in DQL for continuous control: namely, feedforward multi-layer perceptrons with between 1 and 4 hidden layers, and between 32 and 256 hidden units, with either tanh, relu, or sin activations. (See, eg, \cite{Lillicrap2016, Fujimoto2018, Haarnoja2018, Rajeswaran2017} for examples where networks in this range have previously been used.) Because divergence typically occurs near the beginning of training \cite{VanHasselt2018} and the NTK is known to converge to a constant in the infinite-width limit \cite{Jacot2018}, we focus only on the properties of $K_{\theta}$ at initialization in these experiments.

For each of three Gym environments (HalfCheetah-v2, Walker2d-v2, and Ant-v2), we sampled a dataset ${\mathcal D}$ of 1000 state-action pairs using a ``rails-random" policy: $a = \mbox {\rm sgn}(u), u \sim \text{Unif}({\mathcal A})$. We then randomly initialized neural networks of various sizes and activation functions, computed $K_{\theta}$ for each using the state-action pairs in ${\mathcal D}$, and evaluated their on-diagonal elements and average row ratios. We show partial results in Figure \ref{ntkfig}, complete results in Appendix \ref{completentk}, and summarize findings here:
\begin{itemize}
\item Diagonal elements tend to increase with width and decrease with depth, across activation functions.
\item Row ratios tend to increase with depth across activation functions, and do not clearly correlate with width.
\item Relu nets commonly have the largest on-diagonal elements and row ratios (so they should learn quickly and generalize aggressively).
\item Sin networks appear to be in a ``sweet spot" of high on-diagonal elements and low off-diagonal elements. This analysis suggests sin activations may be more useful for DQL than has been previously realized.
\end{itemize}

Based on these results, as we will detail in subsequent sections, we experimented with using sin activations for TD3, SAC, and PreQN.

\begin{figure}
\centering
\includegraphics[width=0.4\textwidth]{figures/ntk_width/walker2d-v2_r_means}
\caption{Average row ratio for networks with 2 hidden layers of size 32 (small), 64 (med), 128 (large), and 256 (exlarge), using data from Walker2d-v2. Error bars are standard deviations from 3 random network initializations (with fixed data).}
\label{ntkfig}
\end{figure}

\subsection{Benchmarking PreQN}

\begin{figure*}[t]
\centering
\includegraphics[width=0.3\textwidth]{figures/bench/preqn_half_v2.pdf}
\includegraphics[width=0.3\textwidth]{figures/bench/preqn_walk_v2.pdf}
\includegraphics[width=0.3\textwidth]{figures/bench/preqn_hop_v2.pdf}

\includegraphics[width=0.3\textwidth]{figures/bench/preqn_swim_v2.pdf}
\includegraphics[width=0.3\textwidth]{figures/bench/preqn_ant_v2.pdf}
\caption{Benchmarking PreQN against TD3 and SAC on standard OpenAI Gym MuJoCo environments. Curves are averaged over 7 random seeds. PreQN is stable and performant, despite not using target networks. The PreQN experiments used sin activations; the TD3 and SAC experiments used relu activations.}
\label{bench}
\end{figure*}

We benchmarked PreQN on five environments from the OpenAI Gym, comparing to TD3 and fixed-temperature SAC, and we present results in Figure \ref{bench}. For each algorithm, we experimented with using relu and sin activations, and we found that PreQN performed best with sin activations, while TD3 and SAC performed best with relu activations. (As a result, we report results in our benchmark for PreQN-sin, TD3-relu, and SAC-relu.) However, we did not do any hyperparameter tuning for TD3 and SAC to specifically accomodate the sin activations, and instead relied on hyperparameters based on the literature which were well-tuned for relus. Hyperparameters for all experiments are given in Appendix \ref{benchmethods}.

In general, PreQN is stable and performant, comparing favorably with the baseline algorithms. In some cases it outperforms (eg Swimmer and Ant) and in some cases it underperforms (eg Hopper). We find this outcome interesting and exciting because PreQN represents a different development path for DQL algorithms than is currently standard: it lacks target networks, only uses a single Q-function instead of multiple, makes no modifications to the bootstrap, and uses vanilla gradient steps for Q-learning instead of adaptive or momentum-based optimizers like Adam \cite{Kingma2015}. However (as we will shortly discuss), we found that it did not fully avert divergence when combined with relu networks.

\begin{figure}
\centering
\includegraphics[width=0.23\textwidth]{figures/align/preqn_align_half_v3.pdf}
\includegraphics[width=0.23\textwidth]{figures/align/preqn_align_half_v3_cos.pdf}

\caption{Examining the cosine alignment of actual $Q$-value change with intended $Q$-value change ($\cos(Q'-Q, y -Q)$) for PreQN and TD3 with relu and sin activations. Curves are averaged over 3 random seeds.}
\label{align}
\end{figure}

To measure how well DQL updates push $Q$-values towards their targets, we evaluated an alignment measure given by $\cos(Q'-Q, y - Q)$, where $y$ is the target for the algorithm (${\mathcal T}^* Q_{\theta}$ in PreQN, and ${\mathcal T}^* Q_{\psi}$ in TD3, where $\psi$ are the parameters of the slowly-changing target network). We show partial results in Figure \ref{align} and complete results in Appendix \ref{alignfull}. We compared PreQN to TD3, because these two algorithms are equivalent except for the $Q$-learning component. While PreQN produces high alignment regardless of architecture by design, TD3 with the sin function (TD3-sin) produces updates that are better-aligned with their targets than TD3 with the relu function (TD3-relu). This accords well with our empirical analysis of the NTK: for sin networks, the NTK is closer to diagonal, so $Q' - Q \approx \alpha K (y-Q)$ is closer to $\alpha (y-Q)$. Perhaps surprisingly, performance for TD3-sin was generally weaker than performance for TD3-relu, but we did not retune any of the hyperparameters from TD3-relu for TD3-sin; we speculate that better performance with TD3-sin may be achievable with a target network that updates more quickly. Performance for PreQN-relu was generally weaker than PreQN-sin, primarily due to occasional divergence; this result suggests that cancelling within-batch generalization is not a universal solution to divergence issues, and favorable architecture choices may be useful. However, in experiments not included in this report, we found that divergence issues with PreQN-relu were straightforwardly resolved by decreasing the learning rate (at a nontrivial cost to performance).

We are intrigued by the fact that empirical analysis of the NTK successfully predicts how the cosine alignment of a DQL update changes with architecture in the TD3 experiments. It has been observed that architecture changes can have a significant effect on performance in deep RL (for example, see \mbox{\citet{Henderson2018}}), but to the best of our knowledge, no one has previously proposed any method for predicting how the behavior of a given algorithm might change with architecture. Based on our results, we are cautiously optimistic that the NTK is the correct object of study for such predictions, and we recommend a more rigorous empirical analysis relating NTK measurements, architectures, and hyperparameter choices in DQL to performance.

\subsection{Remarks on Computational Cost}

Our implementation of PreQN was significantly slower than our implementation of SAC (by more than 50\%), due to the requirement of calculating backward passes separately for each state-action pair in the batch, and solving the system of equations $K_{\theta} Z = \Delta$. However, we consider it plausible that many adjustments could be made to reduce computational cost from our basic code. (For instance: we did not reuse the gradients from computing $K_{\theta}$ for forming the update, $\sum_{s,a} Z(s,a) \nabla_{\theta} Q_{\theta}(s,a)$, and this redundancy can be eliminated.)

\section{Other Related Work}

Previously, \citet{Melo2008} proved sufficient conditions for the convergence of $Q$-learning with linear function approximators. Their conditions were fairly restrictive, essentially requiring that the algorithm behave as if it were on-policy---removing one of the components of the triad. We see an interesting parallel to our results for the linear approximation case (Theorem 2), which also effectively remove a component of the triad by requiring the algorithm to behave as if it were tabular.

Concurrently to our work, \cite{Bhatt2019} developed CrossNorm, a variant of DDPG that uses a careful application of BatchNorm \cite{Ioffe2015} to achieve stable learning without target networks. Also concurrently, \citet{Fu2019} performed a rigorous empirical study of Fitted Q-Iteration (FQI) \cite{Riedmiller2005} to gain insight into divergence issues in DQL, and ultimately proposed an algorithm based on data distribution modifications to improve performance.

\section{Conclusions}

In this work, we examined how $Q$-values change under a DQL update in order to understand how divergence might arise. We used our insights to develop a practical algorithm, called PreQN, which attacks one of the root causes of divergence: the generalization properties of the $Q$-function approximator, as quantified by the neural tangent kernel \cite{Jacot2018}. Our experiments show that PreQN, with appropriate design choices, is stable and performant on various high-dimensional continuous control tasks.

Intriguingly, theoretical and empirical work shows that the NTK converges to a constant (independent of network parameters) in the limit of wide networks \cite{Jacot2018}; this result makes it possible to study the evolution of neural network functions through their linearization around the starting point \cite{Lee2019}. In this regime, through the correspondence in Lemma 3, DQL should behave quite closely to linear TD-learning. We consider the detailed analysis of this connection to be an interesting avenue for potential future work.

redpajama_set_name: RedPajamaArXiv


Bowie: Album by Album examines every one of Bowie's studio albums in fine detail, right up to 2016's critically acclaimed Blackstar. Each album is placed within the context of the time in which it was recorded and all the albums' subsequent influence and legacy are charted. Also included is commentary from the musicians, engineers and producers who worked on the recordings - such as Brian Eno and Tony Visconti - and Bowie's own quotes provide a fascinating insight into his restlessly creative mind.
Paolo Hewitt is a journalist and the author of over 20 books. He has written for Melody Maker and NME, as well as Vogue Germany, Fare Musica in Italy and The Sunday Times. Among other books, he is the author of The Jam: A Beat Concerto, The Small Faces: The Young Mod's Forgotten Story, Getting High: The Adventure of Oasis, and Alan McGee and the Story of Creation Records, as well as several works of fiction. Robert Elms is a writer and broadcaster and radio show host for BBC London. A former columnist for NME and The Face, he was an early champion of the New Romantics. His last book, The Way We Wore: A Life in Threads (2006), documented his self-confessed obsession with fashion's relationship with music and youth culture. Famously, he isn't a fan of The Beatles, but he loves David Bowie.


As cops with 70 years of service between them, Pete and Don know who they are. They can joke about needing to sit in the last row at the conference, with their backs to the wall, as they check out the other participants. They can express how hard it is to trust anyone who isn't a cop.
I met Pete and Don at a conference on treating survivors after a sudden, violent death. The conference site was a Christian retreat center in Memphis.
Pete and Don know who they are in another way, a profound way. They know that repeated exposure in their careers to traumas of violence and death has been very damaging to them and to their fellow officers. They know that some of the values of their officer community, including stoic silence, have contributed to the hurt. Pete and Don want to help their brother and sister cops, and they know that as members of the police subculture they are in the best position to do that.
And another thing that Pete and Don know, and share, is that their own inability to process trauma brought them to terrible low points in their lives. To give me context, they talked about the fact that every 55 hours one cop is killed and three commit suicide. They talked about the fact that officer suicide rates are comparable to those of Marines. They said that police have twice the national average of divorce. They said that police officers suffer diseases of adaptation, such as diabetes, digestive illnesses, and skin disorders. They said that 30% of cops have diagnosable PTSD.
Pete told me the story of a cop who was called to the SIDS death of an infant – an infant that was the same age as the officer's own child. The officer tried unsuccessfully to revive the baby and later accompanied the ambulance with the body to the hospital. Then, immediately afterward, the officer had to contain his grief to deal with a false alarm and an irate homeowner.
Pete and Don are cops with 70 years of service between them. They have learned to value emotions, including tears, to deal with traumatic death. They have the exceptional goal to share this knowledge with their brother and sister officers.
Peter Cove, formerly of South Boston, is training director at Tennessee Public Safety Network, which supports officers after a critical incident. Donald Benson is Assistant Chief of Internal Affairs and Training in the Blount County Sheriff's Office in Maryville, Tennessee.


Currently, GOF has filed a lawsuit against the ATF and DOJ to stop their ban on bump stocks. A successful challenge in this case will prevent a future anti-gun president from using these regs to ban AR-15s and other semi-automatic firearms. Please help support this lawsuit to preserve our rights by contributing here.
Gun Owners of America -- and its foundation (GOF) -- have filed an amicus brief opposing a request for rehearing by the State of Hawaii of a Ninth Circuit decision which overturned Hawaii's virtual ban on citizens bearing weapons.
Gun Owners of America -- and its Foundation (GOF) -- have submitted an amicus brief before the Supreme Court in NYSRP v. NY City. Although the case pertains to gun control in the Big Apple, it has huge ramifications for our gun-owning freedoms nationwide.
The Gun Owners brief in Kolbe v. Maryland can be viewed here.
GOA and GOF filed an earlier amicus brief in this case on November 12, 2014, when the case was known as Kolbe v. O'Malley.
On July 24, Gun Owners and Gun Owners Foundation filed a brief in the U.S. Supreme Court in the case Robinson v. United States.
This brief urges the High Court to review a court of appeals decision that authorizes the police to search and disarm a gun owner at a traffic stop — even if the firearm is being lawfully carried under state law.
On May 3, 2018, GOA filed the only amicus brief supporting a 20-state challenge to Obamacare being led by Texas Attorney General Ken Paxton.
On June 26, 2017, the Supreme Court denied the petition for certiorari in Peruta v. California. We filed amicus briefs on your behalf in both the U.S. Court of Appeals for the Ninth Circuit and on petition to the Supreme Court.
In our most recent filing with the Supreme Court, we had taken the controversial position that our "friends" litigating the case in California had bartered away Second Amendment rights in an attempt to appear reasonable to judges. We had gone back and forth about whether we should even file a brief and, if we did, whether it in fact should oppose the granting of the petition.


League's development leads answer your most pressing and… intriguing questions.
Meddler works with the League teams on balance, champ design, items, updates, and annual Ryze reworks.
Safelocked drives product development with the gameplay teams on LoL. It's her birthday today.
SuperCakes is a Product Manager on League of Legends. When not working or gaming, she is obsessed with baking...mostly cakes...super cakes.


text:
The College Study
An educational blog published in your interest to brighten your future and to make you lead and succeed
MCQs ▼
B.Ed MCQs
Notes ▼
FSc Notes
Home / Articles / Effects of Violence in Media on Society
Effects of Violence in Media on Society
Last updated on Wednesday, January 13th, 2021 - Leave a comment
Is societies violence the media's fault? This is the question that has been asked since before television was in every house. Of course there are the 1 different types of media today ranging from newspapers, to on-line reports and stories. There have been arguments upon arguments about this issue, and over 3,000 studies conducted in the world. Unfortunately there isn't one single result; there is only an array of supposed answers to this undying question.
University of Michigan (USA) psychologists Dr. Leonard Eron and Dr. Rowell Huesmann conducted a study, which continued for decades. This was conducted beginning in 1960. They took 800 eight-year-olds and found that children who watched many hours of violent television tended to be more aggressive in the playground and the classroom. They checked back with these kids 11 and 22 years later. They found the aggressive eight-year-olds grew up to become even more aggressive. They testified before congress in 1992 stating, "Television violence affects youngsters of all ages, of both genders, at all socioeconomic levels and all levels of intelligence. The effect is not limited to Essays on Communication Media children who are already disposed to being aggressive and is not restricted to this country.
David Phillips, a scientist at the University of California in San Diego conducted a study on prize-fights on television. He thought of this topic, because he felt there wasn't enough research being conducted on the copycat violence. He found that after prize fights on television, there would be about a 10 percent increase in murders for a few days afterwards.
He quoted,
It also seems to be the case that the kind of person killed just after the prize-fight is similar to the person beaten in the prize fight.
There are four major theories of television violence. The "arousa." theory, the "social learning" theory, the "disinhibition hypothesis," and the "catharsis hypothesis." These four hypothesis/theories are old and new conclusions to the question at hand. It is notable to see that some of these theories were stated as early as 1961. Most would have to disagree with these theories just because of the age of their births, but to most people's surprise they still hold in the 21st century.
The arousal theory is basically self-explanatory. This was theorized by P.H. Tannenbaum in 1975. He said exposure to television violence increases aggression because violence increases excitation, or "arouses" viewers. This is also being found in the recent studies, which shows the progression in the media's will to change.
The "social learning' theory was described by Dr. Bandura. This theory says ways of behaving are learned by observing others, and that this is a major means by which children acquire unfamiliar behaviour, although performance of acquired behaviour will depend at least in part on factors other than acquisition. A perfect example of this theory was when the murders occurred after the prizefights.
The "disinhibition hypothesis" was L. Berkowitz's investigation. This hypothesis explains that television violence in certain circumstances will result in increased interpersonal aggression because it weakens inhibitions against such behaviour.
The final theory, "catharsis hypothesis" was written by S. Feshbach. This theory explains that under certain conditions exposure to television violence will reduce subsequent aggression. What this is saying is that if someone sees a fantasy on TV, or now with technology, entertains themselves with virtual reality, that fantasy is fulfilled, which makes them not feel they have to do that in real life.
So many people have discussed the topic of media affecting society, from Aristotle to the present day researchers. It has always been a question, but never as needy for an answer as now. Hopefully the government has some say in this soon, so the drama of centuries will finally be over. But that probably won't occur anytime soon. Aristotle was a big supporter of "catharsis." He believed that the audience became psychologically involved with the story on stage, even though they knew it was 100% fiction. He felt when aggression climaxed with the actors, there was a "catharsis" in the audience, which was pleasurable to experience and left the audience "cleansed, uplifted, and less likely to act violently among themselves."
All Sigmund Freud also felt as Aristotle did by saying, "Unless people were allowed to express themselves aggressively, the aggressive energy would be dammed up, pressure would build, and the aggressive energy would seek an outlet, either exploding into acts of extreme violence or be manifesting itself as symptoms of mental illness But there is no direct evidence for this conclusion. But for people who have never been taught to understand the consequences of their action these things can unintentionally set forth a chain reaction of ever more impulsive behavior.
Hollywood figures of the 21st century blame factors such as poverty, drugs and alcohol, poor schooling, lax gun control and a general breakdown of families but not screen violence. University of Iowa professor of Journalism and Mass Communication Albert Talbott said, "In the '30s, when I was a toddler, one of the things that concerned parents were comic books and the violence in them. As soon as the modern media started to develop, we have all kinds of things on how we are affecting people." Technology today isn't helping everyone to feel better about this dilemma. It is actually going to get worse before it gets better. There isn't only movies or news reports someone can watch to see violence, but also the new video game craze. Video games have become an enormous industry in the past decade. People from 4 years old to 70 years old own their own video games.
So many studies have been done on the affects of media violence on children. Most are concerned with the results, especially parents. If the government, parents or others are so concerned with the effect of their child seeing violence on the television, maybe they should practice what they preach. They should think twice before buying video games based on violence for their children. This is where it gets sticky. Parents need to draw the line between appropriate and not appropriate. It would be a nice convenience to have a rating system on the television, but parents should be aware enough of what their children are doing and watching that they are the rating system themselves. What is the effect of violence in media with children compared to with adults ? Children model behaviour they see in the media.
If they don't see the consequences of violence, it will teach them that violence doesn't cause serious harm. Adults see more violence in the media than actually exists in real life. That's because producers believe that they have to include extraordinary violence in order to keep the viewer. When heroes use violence, children think that violence is an appropriate way to respond to problems, Children are younger, so they see things and apply that to their lives, because they are learning everything at that age. Adults look at it as the "mean world syndrome" in which they see how society is portrayed on TV, and they think that every neighbourhood is dangerous, like shown. When in fact most neighbourhoods are nothing like they are portrayed on TV. The writers and producers are exaggerating, to make it all interesting. There is also discussion of violence on TV not having any affect at all. People have seen so much, that they don't really think about the actual act occurring on screen. Hanno Hardt, a professor at J-MC School said, "It's lost it's shock value. Maybe 20 to 30 years ago we would have been shocked. Now, a generation later, we know that this is a violent society. And when we read about violence, it only reinforces what we know." People have become used to seeing violence on television, but this has become somewhat surreal to them. They don't think of it as reality until it happens to them.
We have covered a huge amount of information about the effect of violence in media on society. Did we answer the question though? I don't think we did, but I do think that the answer is making progress. We are also a lot more informed now of what exactly is in the media right now, and what studies have shown to be happening. There has always been an issue of something effecting society, and there will always be a plentitude of scapegoats. What is the actual answer though? No one seems to have it. There is a lot of gray area, but society seems to be making this more of a black and white issue. Will the government ever really take action? Does action need to be taken? Hopefully after reading this, one is more educated on the difficulty in answering these questions.
[PDF Download]
Subscribe to brighten your future
An email was just sent to confirm your subscription. Please find the email and click 'Confirm Follow' to start subscribing.
About Editorial Staff
It is an educational blog and intended to serve as complete and self-contained work on essays, paragraph, speeches, articles, history, letters, stories, quotes.
View all posts by Editorial Staff →
About / Contact / Downloads / Privacy policies / Copyrights / Advertisements / Cookie policy / Terms of use

redpajama_set_name: RedPajamaCommonCrawl


This innovation course is a hands-on learning experience that will contribute to a mindset change, inspire you to innovate in all functions and give you the practical tools to develop your ideas into a viable business proposition. The main objective of the programme is to understand how to become entrepreneurial within your own organisation and to inspire your teams and colleagues to innovate together.
On the From Idea to Innovation programme we take participants on an Innovation Safari – exposing delegates to different innovations and aiming to change mindsets from static to proactive and forward thinking.


text:
Q: ASP.NET MVC: Output validation I am using the following method inside my ASP.NET MVC project to get some XML-data from another web service:
[HttpPost]
[ValidateInput(false)]
public ActionResult MyAction()
{
    try
    {
        byte[] reqContent = Helper.GetBytes(Request.Unvalidated.Form["xml"]);

        WebRequest request = WebRequest.Create("url");
        request.Method = "POST";
        request.ContentType = "text/xml";
        request.ContentLength = reqContent.Length;
        request.GetRequestStream().Write(reqContent, 0, reqContent.Length);

        string responseXml = null;

        using (WebResponse response = request.GetResponse())
        {
            using (StreamReader reader = new StreamReader(response.GetResponseStream()))
            {
                responseXml = reader.ReadToEnd();
            }
        }

        return Content(responseXml, "text/xml");
    }

    catch(Exception)
    {
        return Json(new { Error = true });
    }
}

The request inside the action works perfect and I get the right response when I debug the code. But unfortunately when I look at the Chrome Debug tools, the response code from my Action (not the request sent using WebRequest) is 500 with the error: "A potentially dangerous Request.Form value was detected from the client (xml=somexml).".
Is there some sort of output validation or do I miss something else here? Also the body of the POST-Request to the MyAction controller method consists of XML data, but using the ValidateInput(false)-attribute and the Unvalidated-property of the Request object, I get no exception and all works fine inside the method.
EDIT: SOLUTION
Thanks to the answer which I marked as accepted, I do not only changed the input validation on up to date standards, I also dug deeper into possible causes and realized that the problem was the global OutputCacheAttribute. This post finally solved the problem.

A: MVC is still validating the POST request before you hit your Action. The new way to go is to attribute the property that shall hold the XML with [AllowHtml]. [ValidateInput(false)] is deprecated.
See Securing Your ASP.NET Applications.
public class PostXmlModel {
    [AllowHtml]
    public string Xml {get; set;}
}

[HttpPost]
public ActionResult MyAction(PostXmlModel postData) {
    string xml = postData.Xml;
    // ...
}

PS: to make [ValidateInput(false)] work, you would also need to set <httpRuntime requestValidationMode="2.0" /> in the web.config (not recommended). See Allow user to input html in asp net mvc validateinput or allowhtml.

redpajama_set_name: RedPajamaStackExchange


Let us pray: Dear God, as I go to work today let your wisdom direct me along the way. Let me act upon your will. Teach me and guide me in dealing with my colleagues and the people working under me. May I always treat them with utmost respect no matter what positions they take. I pray that I may always be careful with my actions and speech and avoid being discourteous and rude. Amen.


text:
<?php
namespace TYPO3\CMS\Extbase\Scheduler;

/**
 * Scheduler task to execute CommandController commands
 */
class Task extends \TYPO3\CMS\Scheduler\Task\AbstractTask
{
    /**
     * @var string
     */
    protected $commandIdentifier;

    /**
     * @var array
     */
    protected $arguments = [];

    /**
     * @var array
     */
    protected $defaults = [];

    /**
     * @var \TYPO3\CMS\Extbase\Object\ObjectManagerInterface
     */
    protected $objectManager;

    /**
     * @var \TYPO3\CMS\Extbase\Mvc\Cli\CommandManager
     */
    protected $commandManager;

    /**
     * @var \TYPO3\CMS\Extbase\Scheduler\TaskExecutor
     */
    protected $taskExecutor;

    /**
     * Instantiates the Object Manager
     */
    public function __construct()
    {
        parent::__construct();
        $this->objectManager = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Extbase\Object\ObjectManager::class);
        $this->commandManager = $this->objectManager->get(\TYPO3\CMS\Extbase\Mvc\Cli\CommandManager::class);
        $this->taskExecutor = $this->objectManager->get(\TYPO3\CMS\Extbase\Scheduler\TaskExecutor::class);
    }

    /**
     * Sleep
     *
     * @return array Properties to serialize
     */
    public function __sleep()
    {
        $properties = get_object_vars($this);
        unset($properties['commandManager']);
        unset($properties['objectManager']);
        unset($properties['taskExecutor']);
        return array_keys($properties);
    }

    /**
     * Wakeup
     *
     * @return void
     */
    public function __wakeup()
    {
        $this->objectManager = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance(\TYPO3\CMS\Extbase\Object\ObjectManager::class);
        $this->commandManager = $this->objectManager->get(\TYPO3\CMS\Extbase\Mvc\Cli\CommandManager::class);
        $this->taskExecutor = $this->objectManager->get(\TYPO3\CMS\Extbase\Scheduler\TaskExecutor::class);
    }

    /**
     * Function execute from the Scheduler
     *
     * @return bool TRUE on successful execution
     * @throws \Exception If an error occurs
     */
    public function execute()
    {
        try {
            $this->taskExecutor->execute($this);
        } catch (\Exception $e) {
            $this->logException($e);
            // Make sure the Scheduler gets exception details
            throw $e;
        }
        return true;
    }

    /**
     * @param string $commandIdentifier
     */
    public function setCommandIdentifier($commandIdentifier)
    {
        $this->commandIdentifier = $commandIdentifier;
    }

    /**
     * @return string
     */
    public function getCommandIdentifier()
    {
        return $this->commandIdentifier;
    }

    /**
     * @param array $arguments
     */
    public function setArguments($arguments)
    {
        $this->arguments = $arguments;
    }

    /**
     * @return array
     */
    public function getArguments()
    {
        return $this->arguments;
    }

    /**
     * @param array $defaults
     */
    public function setDefaults(array $defaults)
    {
        $this->defaults = $defaults;
    }

    /**
     * @return array
     */
    public function getDefaults()
    {
        return $this->defaults;
    }

    /**
     * @param string $argumentName
     * @param mixed $argumentValue
     */
    public function addDefaultValue($argumentName, $argumentValue)
    {
        if (is_bool($argumentValue)) {
            $argumentValue = (int)$argumentValue;
        }
        $this->defaults[$argumentName] = $argumentValue;
    }

    /**
     * Return a text representation of the selected command and arguments
     *
     * @return string Information to display
     */
    public function getAdditionalInformation()
    {
        $label = $this->commandIdentifier;
        if (!empty($this->arguments)) {
            $arguments = [];
            foreach ($this->arguments as $argumentName => $argumentValue) {
                if ($argumentValue != $this->defaults[$argumentName]) {
                    array_push($arguments, $argumentName . '=' . $argumentValue);
                }
            }
            $label .= ' ' . implode(', ', $arguments);
        }
        return $label;
    }

    /**
     * @param \Exception $e
     */
    protected function logException(\Exception $e)
    {
        \TYPO3\CMS\Core\Utility\GeneralUtility::sysLog($e->getMessage(), $this->commandIdentifier, \TYPO3\CMS\Core\Utility\GeneralUtility::SYSLOG_SEVERITY_ERROR);
    }
}

redpajama_set_name: RedPajamaGithub


Achilles Recovery Day 13 - #whatwouldBillysay?
Thirteen days since my injury and 8 since the op. Due to the atrophy of my gastrocnemius and soleus muscles my cast has been working loose. So, I took a trip to the plaster room to get fitted out with a new model. While there I took the opportunity to have a look at the wound and check its progress.