text:
\section{#1}\label{sec:#1}}
\newcommand{\Subsection}[1]{\subsection{#1}\label{sec:#1}}
\newcommand{\Subsubsection}[1]{\subsubsection{#1}\label{sec:#1}}

\setcounter{secnumdepth}{0}
\setlength\titlebox{2.5in}
\title{Time your hedge with Deep Reinforcement Learning}
\author{
	Eric Benhamou \textsuperscript{\rm 1}, David Saltiel \textsuperscript{\rm 2,\rm 3}, Sandrine Ungari \textsuperscript{\rm 4}, Abhishek Mukhopadhyay \textsuperscript{\rm 5} \\
	\textsuperscript{\rm 1} MILES, LAMSADE, Dauphine university, France \texttt{eric.benhamou@lamsade.dauphine.fr}\\
	\textsuperscript{\rm 2} AI Square Connect, France \texttt{david.saltiel@aisquareconnect.com}\\
	\textsuperscript{\rm 3} LISIC, ULCO, France \\
	\textsuperscript{\rm 4} Societe Generale, Cross Asset Quantitative Research, UK,\\
	\textsuperscript{\rm 5} Societe Generale, Cross Asset Quantitative Research, France,\\
	\texttt{\{sandrine.ungari,abhishek.mukhopadhyay\}@sgcib.com}
}

\begin{document}

\maketitle
\thispagestyle{plain}
\pagestyle{plain}

\begin{abstract}
Can an asset manager plan the optimal timing for her/his hedging strategies given market conditions?
The standard approach based on Markowitz or other more or less sophisticated financial rules aims to find the best portfolio allocation thanks to forecasted expected returns and risk but fails to fully relate market conditions to hedging strategies decision. In contrast, Deep Reinforcement Learning (DRL) can tackle this challenge by creating a dynamic dependency between market information and hedging strategies allocation decisions. In this paper, we present a realistic and augmented DRL framework that: (i) uses additional contextual information to decide an action, (ii) has a one period lag between observations and actions to account for one day lag turnover of common asset managers to rebalance their hedge, (iii) is fully tested in terms of stability and robustness thanks to a  repetitive train test method called anchored walk forward training, similar in spirit to k fold cross validation for time series and (iv) allows managing leverage of our hedging strategy. Our experiment for an augmented asset manager interested in sizing and timing his hedges shows that our approach achieves superior returns and lower risk.
\end{abstract}

\Section{Introduction}
From an external point of view, the asset management (buy side) industry is a well-suited industry to apply machine learning as large amount of data are available thanks to the revolution of electronic trading and the methodical collection of data by asset managers or their acquisition from data providers. In addition, machine based decision can help reducing emotional bias and taking rational and systematic investment choices \cite{Kahneman_2011}. However, to date, the buy side industry is still largely relying on old and traditional methods to make investment decisions and in particular to choose portfolio allocation and hedging strategies. It is hardly using machine learning in investment decisions. \\
This is in sharp contrast with the ubiquitous usage of deep reinforcement learning (DRL) in other industries and in particular its use for solving challenging tasks like autonomous driving \cite{Wang2018DeepRL}, learning advanced locomotion and manipulation skills from raw sensory inputs \cite{Levine_2015,Levine_2016,Schulman_2015,Schulman_2017,Lillicrap_2015} or on a more conceptual side for reaching supra human level in popular games like Atari \cite{mnih-atari-2013}, Go \cite{Silver_2016,silver2017mastering}, StarCraft II \cite{Vinyals_2019}, etc ...\\

It therefore makes sense to investigate if DRL can help help in financial planning and in particular in creating augmented asset managers. To narrow down our problem, we are specifically interested in finding hedging strategies for a risky asset. To make things concrete and more illustrative, we represent this risky asset in our experiment with the MSCI World index that captures large and mid cap securities across 23 developed financial markets. The targeted hedging strategies are on purpose different in nature and spirit. They are appropriate under distinctive market conditions. Financial planning is therefore critical for deciding the appropriate timing when to add and remove these hedging strategies.

\Subsection{Related works}
At first, reinforcement learning was not used in portfolio allocation. Initial works focused on trying to make decisions using deep networks to forecast next period prices, \cite{Freitas_2009,Niaki2013,Heaton_2017}. Armed with the forecast, an augmented asset manager could solve its financial planning problem to decide the optimal portfolio allocations. However, this initial usage of machine learning contains multiple caveats. First, there is no guarantee that the forecast is reliable in the near future. On the contrary, it is a stylized fact that financial markets are non stationary and exhibit regime changes \cite{Salhi_2016,Dias_2015,Zheng_2019}, making the prediction exercise quite difficult and unreliable. Second, it does not target specifically the financial planning question of finding the optimal portfolio based on some reward metrics. Third, there is no consideration of online learning to adapt to changing environment as well as the incorporation of transaction costs.\\
A second stream of research around deep reinforcement learning has emerged to address these points \cite{Jiang_2016,Jiang_2017,Liang_2018,Yu_2019,Wang_2019,Liu_2020,Ye_2020,Li_2019,Xiong_2018,benhamou2020detecting}. The dynamic nature of reinforcement learning makes it an obvious candidate for changing environment  \cite{Jiang_2016,Jiang_2017,Liang_2018}. Transaction costs can be easily included in rules \cite{Liang_2018,Yu_2019,Wang_2019,Liu_2020,Ye_2020,Yu_2019}. However, these works, except \cite{Ye_2020} and \cite{benhamou2020detecting} rely only on time series of open high low close prices, which are known to be very noisy. Secondly, they all assume an immediate action after observing prices which is quite different from reality. Most asset managers need a one day turnaround to manage their new portfolio positions. Thirdly, except \cite{benhamou2020detecting}, they rely on a single reward function and do not measure the impact of the reward function. Last but not least, they only do one train and test period, never testing for model stability.

\Subsection{Contributions}
Our contributions are fourfold:
\begin{itemize}
\item \textbf{The addition of contextual information.} Using only past information is not sufficient to learn in a noisy and fast changing environment. The addition of contextual information improves results significantly. Technically, we create two sub-networks: one fed with direct observations (past prices and standard deviation) and another one with contextual information (level of risk aversion in financial markets, early warning indicators for future recession, corporate earnings etc...).

\item \textbf{One day lag between price observation and action.} We assume that prices are observed at time $t$ but action only occurs at time $t+1$, to be consistent with reality. This one day lag makes the RL problem more realistic but also more challenging.

\item \textbf{The walk-forward procedure.} Because of the non stationarity nature of time dependent data and especially financial data, it is crucial to test DRL models stability. We present a new methodology in DRL model evaluation referred to as walk forward analysis that iteratively trains and tests the model on extending data-set. This can be seen as the analogy of cross validation for time series. This allows validating that selected hyper parameters work well over time and that the resulting model is stable over time.

\item \textbf{Model leverage}. Not only do we do a multi inputs network, we also do a multi outputs network to compute at the same time the percentage in each hedging strategy and the overall leverage. This is a nice feature of this DRL model as it incorporates by design a leverage mechanism. To make sure the leverage is in line with the asset manager objective, we cap the leverage to the maximum authorized leverage, which is in our case 3. This byproduct of the method is another key difference with standard financial methods like Markwitz that do not care about leverage and only give a percentage for the hedging portfolio allocation.
\end{itemize}

\Section{Background and mathematical formulation}\label{sec:Background}
In standard reinforcement learning, models are based on Markov Decision Process (MDP) \cite{SuttonBarto_2018}.
A Markov decision process is defined as a tuple $\mathcal{M} = (\mathcal{X}, \mathcal{A}, p,r)$
where:
\begin{itemize}
\item $\mathcal{X}$ is the state space,
\item $\mathcal{A}$ is the action space,
\item $p(y|x, a)$ is the transition probability such that\\ $p(y|x, a) = \mathbb{P}(x_{t+1} = y|x_t = x, a_t = a)$,
\item $r(x, a, y)$ is the reward of transition (x, a, y).
\end{itemize}

MDP assumes that the we know all the states of the environment and have all the information to make the optimal decision in every state. The Markov property in addition implies that knowing the current state is sufficient.\\
From a practical standpoint, the general RL setting is modified by taking a pseudo state formed with a set of past observations $(o_{t-n}, o_{t-n-1}, \ldots, o_{t-1}, o_t)$. In practice to avoid large dimension and the curse of dimension, it is useful to reduce this set and take only a subset of these past observations with $j< n$ past observations, such that $0<i_1< \ldots < i_j$ and $i_k \in \mathbb{N}$ is an integer. The set $\delta_1 = (0,i_1, \ldots, i_j)$ is called the observation lags. In our experiment we typically use lag periods like (0, 1, 2, 3, 4, 20, 60) for daily data, where $(0,1,2,3,4)$ provides last week observation, $20$ is for the one-month ago observation (as there is approximately 20 business days in a month) and 60 the three-month ago observation.

\Subsection{Observations}
\Subsubsection{Regular observations}
There are two types of observations: regular and contextual information. Regular observations are data directly linked to the problem to solve. In the case of a trading framework, regular observations are past prices observed over a lag period $\delta = (0<i_1< \ldots < i_j)$. To renormalize data, we rather use past returns computed as  $r_t = \frac{p^k_t}{p^k_{t-1} } -1$ where $p^k_t$ is the price at time $t$ of the asset $k$. To give information about regime changes, our trading agent receives also empirical standard deviation computed over a sliding estimation window denoted by $d$ as follows $\sigma^k_t  = \sqrt{ \frac{1}{d} \sum_{u =t-d+1}^t \left( r_u - \mu \right)^2 }$, where the empirical mean $\mu$ is computed as $\mu = \frac{1}{d} \sum_{u =t-d+1}^t r_u$. Hence our regular observations is a three dimensional tensor $A_t = \left[ A^1_t, A^2_t\right]$
\begin{center}
with \,\, $A^1_t =  \left( \!
\begin{array}{c   }
r^1_{t-i_j} \,\,	... \,\, r^1_t \\
... \,\,... \,\, ...\\
r^m_{t-i_j} \,\,.... \,\, r^m_t
\end{array} \! \right)\! ,  \,\,
 A^2_t =  \left( \!
\begin{array}{c  }
\sigma^1_{t-i_j} 	\,\,	... \,\, \sigma^1_t\\
... \,\,... \,\, ...\\
\sigma^m_{t-i_j} \,\,.... \,\, \sigma^m_t
\end{array} \! \right)$
\end{center}
This setting with two layers (past returns and past volatilities) is quite different from the one presented in \cite{Jiang_2016,Jiang_2017,Liang_2018} that uses different layers representing closing, open high low prices. There are various remarks to be made. First, high low information does not make sense for portfolio strategies that are only evaluated daily, which is the case of all the funds. Secondly, open high low  prices tend to be highly correlated creating some noise in the inputs. Third, the concept of volatility is crucial to detect regime change and is surprisingly absent from these works as well as from other works like \cite{Yu_2019,Wang_2019,Liu_2020,Ye_2020,Li_2019,Xiong_2018}.

\Subsubsection{Context observation}\label{Context}
Contextual observations are additional information that provide intuition about current context. For our asset manager, they are other financial data not directly linked to its portfolio assumed to have some predictive power for portfolio assets. Contextual observations are stored in a 2D matrix denoted by $C_t$ with stacked past $p$ individual contextual observations. Among these observations, we have the maximum and minimum portfolio strategies return and the maximum portfolio strategies volatility. The latter information is like for regular observations motivated by the stylized fact that standard deviations are useful features to detect crisis. The contextual state writes as $
C^t =  \left( \!\!
\begin{array}{c  }
c^1_t 	\,\,	... \,\, c^1_{t-i_k}\\
... \,\,... \,\, ...\\
c^p_t 	\,\,.... \,\, c^p_{t-i_k}
\end{array} \!\!\! \right)$. The matrix nature of contextual states $C_t$ implies in particular that we will use 1D convolutions should we use convolutional layers. All in all, observations that are augmented observations, write as $O_t =[ A_t, C_t]$, with $A_t=[A^1_t, A^2_t]$ that will feed the two sub-networks of our global network.

\Subsection{Action}
In our deep reinforcement learning the augmented asset manager trading agent needs to decide at each period in which hedging strategy it invests. The augmented asset manager can invest in $l$ strategies that can be simple strategies or strategies that are also done by asset management agent. To cope with reality, the agent will only be able to act after one period. This is because asset managers have a one day turn around to change their position. We will see on experiments that this one day turnaround lag makes a big difference in results. As it has access to $l$ potential hedging strategies, the output is a $l$  dimension vector that provides how much it invest in each hedging strategy. For our deep network, this means that the last layer is a softmax layer to ensure that portfolio weights are between $0$ and $100\%$ and sum to $1$, denoted by $(p^1_t, ..., p^l_t)$. In addition, to include leverage, our deep network has a second output which is the overall leverage that is between 0 and a maximum leverage value (in our experiment 3), denoted by $lvg_t$. Hence the final allocation is given by $lvg_t \times (p^1_t, ..., p^l_t)$.

\Subsection{Reward}
There are multiple choices for our reward and it's a key point for the asset manager to decide the reward corresponding to his\ her risk profile.
\begin{itemize}
    \item A straightforward reward function is to compute the final net performance of the combination of our portfolio computed as the value of our portfolio at the last train date $t_T$ over the initial value of the portfolio $t_0$ minus one: $\frac{P_{t_T}}{P_{t_0}}-1$.
    \item Another natural reward function is to compute the Sharpe ratio. There are various ways to compute Sharpe ratio and we take explicitly the annualized Sharpe ratio.
    This annualized Sharpe ratio computed from daily data is defined as the ratio of the annualized return over the annualized volatility $\mu / \sigma$. The intuition of the Sharpe ratio is to account for risk when comparing returns with risk is represented by volatility.
    \item The last reward we are interested in is the Sortino ratio.
    This metric is a variation of the Sharpe ratio where the risk is computed by the downside standard deviation whose definition is to compute the standard deviation only on negative daily returns $(\tilde{r}_t)_{t=0..T}$ .
    Hence the downside standard deviation is computed by $\sqrt{250} \times  \text{StdDev}[ (\tilde{r}_t)_{t=0..T})]$.
\end{itemize}

\Subsection{Convolutional network}
The similarities with image recognition (where pixels are stored in 3 different matrices representing red, green and blue image) enable us using convolution networks for our deep neural network. The analogy goes even further as it is well known in image recognition that convolutional networks achieve strong performances thanks to their capacity to extract meaningful features and to have very limited parameters hence avoiding over-fitting. Indeed, convolution allows us to extract features; blindly weighting locally the variables over the tensor. There is however something to notice. We use a convolution layer with a convolution window or kernel with a single row and a resulting vertical stride of 1. This particularity enables us to avoid mixing data from different strategies. We only mix data of the same strategies but for different observation dates. Recall that in convolution network, the stride parameter controls how the filter convolves around our input. Likewise the size of the window also referred to as the kernel size controls how the filter applies to data. Thus, a kernel with a row of 1 and a stride with a row of 1 allows us to detect the vertical (temporal) relation for each strategy by shifting one unit at a time, without mixing any data from different strategies. This concept is illustrated in figure \ref{fig:conv1d}.
Because of this peculiarity, we can interpret our 2-D convolution as an iteration over a 1-D convolution network for each variable.
\begin{figure}[!htbp]
\centering
\includegraphics[width= \linewidth]{figures/conv1d.PNG}
\caption{2-D Convolution with stride of 1}\label{fig:conv1d}
\end{figure}

\Subsection{Multi inputs and outputs}
We display in figure \ref{fig:best_network} the architecture of our network. Because we feed our network with both data from the strategies to select but also contextual information, our network is a multiple inputs network.

\begin{figure}[!htbp]
\centering
\includegraphics[width= \linewidth]{figures/network.png}
\caption{network architecture obtained via tensorflow plotmodel function. Our network is very different from standard DRL networks that have single inputs and outputs. Contextual information introduces a second input while the leverage adds a second output}\label{fig:best_network}
\end{figure}

Additionally, as we want from these inputs to provide not only percentage in the different hedging strategies (with a softmax activation of a dense layer) but also the overall leverage (with a dense layer with one single ouput neurons), we also have a multi outputs network. Additional hyperparameters that are used in the network as L2 regularization with a coefficient of 1e-8.

\Subsection{Adversarial Policy Gradient}
To learn the parameters of our network depicted in \ref{fig:best_network}, we use a modified policy gradient algorithm called adversarial as we introduce noise in the data as suggested in \cite{Liang_2018}.. The idea of introducing noise in the data is to have some randomness in each training to make it more robust. This is somehow similar to drop out in deep networks where we randomly pertubate the network by randomly removing some neurons to make it more robust and less prone to overfitting.
A policy is a mapping from the observation space to the action space, $\pi:\mathcal{O}\rightarrow\mathcal{A}$.
To achieve this, a policy is specified by a deep network with a set of parameters $\vec \theta$. The action is a vector function of the observation given the parameters: $\vec a_t = \pi_{\vec \theta}(\bm o_t)$.
The performance metric of $\pi_{\vec \theta}$ for time interval $[0,t]$ is defined as the corresponding total reward function of the interval $
	J_{[0,t]}(\pi_{\vec \theta}) = R\left( \vec o_1,\pi_{\vec \theta}(o_1),\cdots,
		\vec o_{t},\pi_{\vec \theta}(o_{t}),\vec o_{t+1} \right)
	\label{eq:policy_value}$.
After random initialization, the parameters are continuously updated along the gradient direction with a learning rate $\lambda$:
$\vec\theta \longrightarrow \vec\theta + \lambda\nabla_{\vec\theta}J_{[0,t]}(\pi_{\vec \theta})$. The gradient ascent optimization is done with standard Adam (short for Adaptive Moment Estimation) optimizer to have the benefit of adaptive gradient descent with root mean square propagation \cite{kingma2014method}. The whole process is summarized in algorithm \ref{alg1}.

\begin{algorithm}[!htbp]
    \caption{Adversarial Policy Gradient}
    \label{alg1}
\begin{algorithmic}[1]
    \STATE Input: initial policy parameters $\theta$, empty replay buffer $\mathcal{D}$
\REPEAT
    \STATE reset replay buffer
    \WHILE{not terminal}
        \STATE Observe observation $o$ and select action $a = \pi_{\theta}(o)$ with probability $p$ and random action with probability $1-p$,
        \STATE Execute $a$ in the environment
        \STATE Observe next observation $o'$, reward $r$, and done signal $d$ to indicate whether $o'$ is terminal
        \STATE apply noise to next observation $o'$
        \STATE store $(o,a,o')$ in replay buffer $\mathcal{D}$
        \IF{Terminal}
            \FOR{however many updates in $\mathcal{D}$}
                \STATE compute final reward $R$
            \ENDFOR
            \STATE update network parameter with Adam gradient ascent
                $\vec\theta \longrightarrow \vec\theta + \lambda\nabla_{\vec\theta}J_{[0,t]}(\pi_{\vec \theta})$
        \ENDIF
    \ENDWHILE
\UNTIL{convergence}
\end{algorithmic}
\end{algorithm}

In our gradient ascent, we use a learning rate of 0.01, an adversarial Gaussian noise with a standard deviation of 0.002. We do up to 500 maximum iterations with an early stop condition if on the train set, there is no improvement over the last 50 iterations.

\Subsection{Walk forward analysis}
In machine learning, the standard approach is to do $k$-fold cross validation as shown in figure \ref{fig:cross_val}. This approach breaks the chronology of data and potentially uses past data in the test set. Rather, we can take sliding test set and take past data as training data as show in the two sub-figures on the right of figure \ref{fig:walk_forward}. To ensure some stability, we favor to add incrementally new data in the training set, at each new step. This method is sometimes referred to as anchored walk forward as we have anchored training data. The negative effect of using extending training data set is to adapt slowly to new information. To our experience, because we do not have so much data to train our DRL model, we use anchored walk forward to make sure we have enough training data. Last but not least, as the test set is always after the train set, walk forward analysis gives less steps compared to cross validation. In practice for our data set, we train our models from 2000 to end of 2006 (to have at least seven years of data) and use a repetitive test period of one year.

\def2{2}

\begin{figure}[!htbp]
\centering
\resizebox{ \linewidth} {!} {
\begin{tikzpicture}[]
\foreach \i [count=\j from 0] in {white,white,white,white,white,blue!20}
 \draw(2+\j,0) rectangle (2+\j+1,0.5)[fill=\i] ;
\foreach \i [count=\j from 0] in {white,white,white,white,blue!20,white}
 \draw(2+\j,0.6) rectangle (2+\j+1,1.1)[fill=\i] ;
\foreach \i [count=\j from 0] in {white,white,white,blue!20,white,white}
 \draw(2+\j,1.2) rectangle (2+\j+1,1.7)[fill=\i] ;
\foreach \i [count=\j from 0] in {white,white,blue!20,white,white,white}
 \draw(2+\j,1.8) rectangle (2+\j+1,2.3)[fill=\i] ;
\foreach \i [count=\j from 0] in {white,blue!20,white,white,white,white}
 \draw(2+\j,2.4) rectangle (2+\j+1,2.9)[fill=\i] ;
\foreach \i [count=\j from 0] in {blue!20,white,white,white,white,white}
 \draw(2+\j,3.0) rectangle (2+\j+1,3.5)[fill=\i] ;
\foreach \i [count=\j from 0] in { , , , , ,}
\draw(2 + 5.4-\j,\j*0.6) node[above]{Test};
\foreach \i [count=\j from 1] in { , , , , ,}
\draw(2-0.7,3.52-\j*0.6) node[above]{\# \j:};
\end{tikzpicture}
}
\caption{k-fold cross validation}
\label{fig:cross_val}
\end{figure}

\begin{figure}[!htbp]
\centering
\resizebox{\linewidth} {!} {
\begin{tikzpicture}[]
\foreach \i [count=\j from 0] in {white, white, white,white,white,blue!20}
 \draw(2+\j,0) rectangle (2+\j+1,0.5)[fill=\i] ;
\foreach \i [count=\j from 0] in {white, white,white,white,blue!20}
 \draw(2+\j,0.6) rectangle (2+\j+1,1.1)[fill=\i] ;
\foreach \i [count=\j from 0] in {white, white,white,blue!20}
 \draw(2+\j,1.2) rectangle (2+\j+1,1.7)[fill=\i] ;
\foreach \i [count=\j from 0] in {white, white,blue!20}
 \draw(2+\j,1.8) rectangle (2+\j+1,2.3)[fill=\i] ;
\foreach \i [count=\j from -1] in { , , , }
\draw(2+4.5- \j,\j*0.6+0.6) node[above]{Test};
\foreach \i [count=\j from 1] in { , , , }
\draw(2-0.7,2.42-\j*0.6) node[above]{\# \j:};
\end{tikzpicture}
}
\caption{anchored walk forward}
\label{fig:walk_forward}
\end{figure}

\Section{Experiments}
\Subsection{Goal of the experiment}
We are interested in planing a hedging strategy for a risky asset. The experiment is using daily data from 01/05/2000 to 19/06/2020. The risky asset is the MSCI world index. We choose this index because it is a good proxy for a wide range of asset manager portfolios. The hedging strategies are 4 SG-CIB proprietary systematic strategies further described below .

\Subsection{Data-set description}\label{Data-set}
Systematic strategies are similar to asset managers that invest in financial markets according to an adaptive, pre-defined trading rule. Here, we use 4 SG CIB proprietary 'hedging strategies', that tend to perform when stock markets are down:
\begin{itemize}
\item Directional hedges - react to small negative return in equities,
\item Gap risk hedges - perform well in sudden market crashes,
\item Proxy hedges - tend to perform in some market configurations, like for example when highly indebted stocks under-perform other stocks,
\item Duration hedges - invest in bond market, a classical diversifier to equity risk in finance.
\end{itemize}

The underlying financial instruments vary from put options, listed futures, single stocks, to government bonds. Some of those strategies are akin to an insurance contract and bear a negative cost over the long run. The challenge consists in balancing cost versus benefits.

In practice, asset managers have to decide how much of these hedging strategies are needed on top of an existing portfolio to achieve a better risk reward. The decision making process is often based on contextual information, such as the economic and geopolitical environment, the level of risk aversion among investors and other correlation regimes. The contextual information is modelled by a large range of features :

\begin{itemize}
    \item the level of risk aversion in financial markets, or market sentiment, measured as an indicator varying between 0 for maximum risk aversion and 1 for maximum risk appetite,
    \item the bond/equity historical correlation, a classical ex-post measure of the diversification benefits of a duration hedge, measured on a 1-month, 3-month and 1-year rolling window,
    \item The credit spreads of global corporate - investment grade, high yield, in Europe and in the US - known to be an early indicator of potential economic tensions,
    \item The equity implied volatility, a measure if the 'fear factor' in financial market,
    \item The spread between the yield of Italian government bonds and the German government bond, a measure of potential tensions in the European Union,
    \item The US Treasury slope, a classical early indicator for US recession,
    \item And some more financial variables, often used as a gauge for global trade and activity: the dollar, the level of rates in the US, the estimated earnings per shares (EPS).

\end{itemize}
A cross validation step selects the most relevant features. In the present case, the first three features are selected. The rebalancing of strategies in the portfolio comes with transaction costs, that can be quite high since hedges use options. Transactions costs are like frictions in physical systems. They are taken into account dynamically to penalise solutions with a high turnover rate.

\Subsection{Evaluation metrics}
Asset managers use a wide range of metrics to evaluate the success of their investment decision. For a thorough review of those metrics, see for example \cite{Cogneau_2009}. The metrics we are interested in for our hedging problem are listed below:
\begin{itemize}
\item annualized return defined as the average annualized compounded return,
\item annualized daily based Sharpe ratio defined as the ratio of the annualized return over the annualized daily based volatility $\mu / \sigma$,
\item Sortino ratio computed as the ratio of the annualized return overt the downside standard deviation,
\item maximum drawdown (max DD) computed as the maximum of all daily drawdowns. The daily drawdown is computed as the ratio of the difference between the running maximum of the portfolio value ($RM_T = \max_{t=0..T}(P_t)$ ) and the portfolio value over the running maximum of the portfolio value. Hence $DD_T = (RM_T - P_T) / RM_T$ and $MDD_T = \max_{t=0..T}(DD_t)$. It is the maximum loss in return that an investor will incur if she/he invested at the worst time (at peak).
\end{itemize}

\Subsection{Baseline}

\Subsubsection{Pure risky asset}
This first evaluation is to compare our portfolio composed only of the risky asset (in our case, the MSCI world index) with the one augmented by the trading agent and composed of the risky asset and the hedging overlay. If our agent is successful in identifying good hedging strategies, it should improve the overall portfolio and have a better performance than the risky asset.

\Subsubsection{Markowitz}
In Markowitz theory \cite{Markowitz_1952}, risk is represented by the variance of the portfolio. Hence the Markowitz portfolio consists in maximizing the expected return for a given level of risk, represented by a given variance. Using dual optimization, this is also equivalent to minimize variance for a given expected return, which is solved by standard quadratic programming optimization. Recall that we have $l$ possible strategies and we want to find the best allocation according to the Sharpe ratio. Let $w = (w_1, ..., w_l)$ be the allocation weights with $1 \geq w_i \geq 0$ for $i=0 ... l$, which is summarized by $1 \geq w \geq 0 $, with the additional constraints that these weights sum to 1: $\sum_{i=1}^l w_i = 1$. \\
Let $\mu = ( \mu_1, ..., \mu_l)^T$ be the expected returns for our $l$ strategies and $\Sigma$ the matrix of variance covariances of the $l$ strategies' returns. Let $r_{min}$ be the minimum expected return. The Markowitz optimization problem to solve that is done by standard quadratic programming is the following:

\begin{eqnarray*}
&\text{Minimize} & w^T \Sigma w     \\
&\text{subject to}& \mu^T w \geq r_{min}  ,  \sum_{i=1 \ldots l} w_i = 1, w \geq 0
\end{eqnarray*}

The Markowitz portfolio is a good benchmark very often used in portfolio theory as it allows investors to construct more efficient portfolios by controlling the variance of their strategies. One of the famous critic of this theory is that it controls the variance (and then the standard deviation) of the portfolio but it doesn't allow controlling a better risk indicator which is the downside standard deviation (representing the potential loss that may arise from risk compared to a minimum acceptable return).
Another limitation of this theory relies on the fact that it works under the assumption that investors are risk-averse. In other words, an investor prefers a portfolio with less risk for a given level of return and will only take on high-risk investments if he can expect a larger reward.

\Subsubsection{Follow the winner}
This is a simple strategy that consists in selecting the hedging strategy that was the best performer in the past year. If there is some persistence over time of the hedging strategies' performance, this simple methodology works well. It replicates standard investors behavior that tends to select strategies that performed well in the past.

\Subsubsection{Follow the loser}
As it name stands for, follow the loser is exactly the opposite of follow the winner. It assumes that there is some mean reversion in strategies' performance, meaning that strategies tend to perform equally well on long term and mean revert around their trend. Hence if a strategy did not perform well in the past, and if there is mean reversion, there is a lot of chance that this strategy will recover with its pairs.

\Subsection{Results and discussion}\label{sec:Results}
\begin{figure}[!htbp]
\centering
\includegraphics[width= 0.97 \linewidth]{figures/graphic_drl.png}
\includegraphics[width= 0.97 \linewidth]{figures/graphic_old_methods.png}
\includegraphics[width= 0.97 \linewidth]{figures/graph_best_methods.png}
\caption{performance of all models}\label{fig:all_models}
\end{figure}

\begin{figure}[!htbp]
\centering
\includegraphics[width= \linewidth]{figures/drl_weights.png}
\caption{DRL weights}\label{fig:drl_weights}
\end{figure}
\begin{figure}[!htbp]
\centering
\includegraphics[width= \linewidth]{figures/winner_weights.png}
\caption{Follow the winner weights}\label{fig:winner_weights}
\end{figure}
\begin{figure}[!htbp]
\centering
\includegraphics[width= \linewidth]{figures/markowitz_weights.png}
\caption{Markowitz weights}\label{fig:markowitz_weights}
\end{figure}

\begin{table}[!htbp]
  \centering
  \caption{Models comparison over 3 and 5 years}\label{tab:Model comparison}
\resizebox{.5\textwidth} {!} {
    \begin{tabular}{|l|rrrr|}
    \toprule
 &        & 3 Years &   &\\
    \toprule
        & \multicolumn{1}{l}{return } & \multicolumn{1}{l}{Sortino} & \multicolumn{1}{l}{Sharpe} & \multicolumn{1}{l|}{max DD}\\
    Risky asset & 10.27\% &       0.34  &      0.38  & -    0.34   \\
    DRL  & \textbf{22.45\%} & \textbf{1.18} & \textbf{1.17} & -0.27 \\
    Winner & 13.19\% & 0.66  & 0.72  & -0.35  \\
    Loser  & 9.30\% & 0.89  & 0.89  & \textbf{-0.15}    \\
    DRL no context & 8.11\% & 0.42  & 0.47  & -0.34   \\
    Markowitz & -0.31\% & -0.01  & -0.01  & -0.41   \\
    \toprule
     &        & 5 Years &   &\\
      \toprule
        & \multicolumn{1}{l}{return } & \multicolumn{1}{l}{Sortino} & \multicolumn{1}{l}{Sharpe} & \multicolumn{1}{l|}{max DD}\\
    Risky asset & 9.16\% &       0.54  &       0.57  & -     0.34    \\
    DRL  & \textbf{16.42\%} & \textbf{0.98} & \textbf{0.96} & -0.27 \\
    Winner &  10.84\% & 0.65  & 0.68  & -0.35  \\
    Loser  &  7.04\% & 0.78  & 0.76  & \textbf{-0.15}    \\
    DRL no context &  6.87\% & 0.44  & 0.47  & -0.34   \\
    Markowitz & -0.07\% & -0.00  & -0.00  & -0.41  \\
    \bottomrule
    \end{tabular}
}
\end{table}%

We compare the performance of the following 5 models: DRL model based on convolutional networks with contextual states (Sentiment indicator, 6 month correlation between equity and bonds and credit main index), same DRL model without contextual states, follow the winner, follow the loser and Markowitz portfolio. The resulting graphics are displayed in figure \ref{fig:all_models} with the risky asset position alone in blue and the other models in orange, green and red. To make figures readable, we first show the two DRL models, with the risky asset and clearly see the impact of contextual information as the DRL model (in orange) is well above the green curve (the same model without contextual information) and is also well above the risky asset position alone (the blue curve). We then plot more traditional models like Markowitz, follow the Winner (entitled for space reason Winner) and follow the Loser (entitled for the same reason Loser). We finally plot the two best performers: the DRL and the Follow the Winner model, emphasizing that the difference between DRL and Follow the Winner is mostly in years 2018 to 2020 that exhibit regime changes, with in particular the recent Covid crisis.

Out of these 5 models, only DRL and Follow the winner are able to provide significant net performance increase compared to the risky asset alone thanks to an efficient hedging strategy over the 2007 to 2020 period. The DRL model is in addition able to better adapt to the Covid crisis and to have better efficiency in net return but also Sharpe and Sortino ratios over 3 and 5 years as shown in table \ref{tab:Model comparison}. In addition, on the last graphic of figure \ref{fig:all_models}, we can remark that the DRL model has a tendency to move away from the blue curve (the risky asset) continuously and increasingly whereas the follow the winner model has moved away from the blue curve in 2015 and 2016 and tends to remain in parallel after this period, indicating that there is no continuous improvement of the model. The growing divergence of the DRL from the blue curve is a positive sign of its regular performance whci is illustrated in numbers in table \ref{tab:Model comparison}.

Moreover, when comparing the weights obtained by the different models (figures \ref{fig:drl_weights}, \ref{fig:winner_weights}, and \ref{fig:markowitz_weights}), we see that the bad performance of Markowitz can be a consequence of its diversification as it takes each year a non null position in the four hedging strategies and tends to change this allocation quite frequently. The rapid change of allocation is a sign of unstability of this method (which is a well known drawback of Markowitz).

In contrast, DRL and Follow the winner models tends to choose only one or two strategies, in a stock picking manner. DRL model tends to choose mostly duration hedge and is able to dynamically adapt its behavior over the last 3 years and to better manage the Covid crisis with a mix allocation between duration and proxy hedge.

In terms of the smallest maximum drawdown, the follow the loser model is able to significantly reduce maximum drawdown but at the price of a lower return, Sharpe and Sortino ratios. Removing contextual information deteriorates model performances significantly and is illustrated by the difference in term of return, Sharpe, Sortino ratio and maximum drawdown between the DRL and the DRL no context model. Last but not least, Markowitz model is not able to adapt to the new regime change of 2015 onwards despite its good performance from 2007 to 2015. It is the worst performer over the last 3 and 5 years because of this lack of adaptation.

For all models, we use the walk forward analysis as described earlier. Hence, we start training the models from 2000 to end of 2006 and use the best model on the test set in 2007. We then train the model from 2000 to end of 2007 and use the best model on the test set in 2008 and etc ... In total, we do 14 training (from 2007 to 2020). This process ensures that we detect models that are unstable overtime and is similar in spirit to delayed online training.  We also provide in table \ref{tab:model choice} different configurations (adversarial training, use of context, and use of day lag), which leads to a total of 16 models. The frist 8 models are the ones with a daylag sorted in order of decreasing performance. The best model is the one with a reward in net profit, adversarial training, use of context information with a total performance of 81.8 \%. We also provide the corresponding same models but with no day lag (model 9 to 16). These models are theoreticall and not considered as they do not cope with reality.

\begin{table}[!htpb]
  \centering
  \caption{Model comparison based on reward function, adversarial training (noise in data) and use of contextual state}
  \label{tab:model choice}%
    \resizebox{0.48 \textwidth}{!}{
    \begin{tabular}{|l l c c c c |}
    \toprule
    \# &    reward  & adversarial? & context? & day lag & performance  \\
    \midrule
    \textbf{1} & \textbf{Net\_Profit}     &     \textbf{Yes}    &     \textbf{Yes}    &     \textbf{Yes}    &     \textbf{81.8\%} \\
    2 & net profit     & No     & Yes    & Yes    & 75.2\% \\
    3 & Sortino     & No     & Yes    & Yes    & 26.5\% \\
    4 & Sortino     & Yes    & Yes    & Yes    & 26.3\% \\
    5 & Sortino     & Yes    & No     & Yes    & -16.7\% \\
    6 & net profit     & Yes    & No     & Yes    & -29.5\% \\
    7 & Sortino     & No     & No     & Yes    & -45.0\% \\
    8 & net profit     & No     & No     & Yes    & -47.7\% \\
    \midrule
    \midrule
    9 & net profit     & Yes    & Yes    & No     & 193.8\% \\
    10 & net profit     & No     & Yes    & No     & 152.3\% \\
    11 & Sortino     & No     & Yes    & No     & 45.3\% \\
    12 & Sortino     & Yes    & Yes    & No     & 29.3\% \\
    13 & Sortino     & Yes    & No     & No     & 16.9\% \\
    14 & net profit     & Yes    & No     & No     & 13.9\% \\
    15 & Sortino     & No     & No     & No     & 10.6\% \\
    16 & net profit     & No     & No     & No     & 8.6\% \\
    \bottomrule
    \end{tabular}
    }
\end{table}

\Subsection{Impact of context}
In table \ref{tab:model choice}, we provide a list of 16 models based on the following choices: the choice of the reward function (net profit or Sortino), the use of adversarial training with noise in data or not, the use of contextual states, and the use of day lag between observations and actions.

We see that the best DRL model with the day-lag turnover constraint is the one using convolutional networks, adversarial training, contextual states and net profit reward function. These 4 parameters are meaningful for our DRL model and change model performance substantially as illustrated by the table with a difference between model 1 (the best model) and model 8 (the worst model) of 129.5 \% (=81.8 \% - (-47.7 \%)).

To measure the impact of the contextual information for our best model, we can measure it simply by doing the difference between model 1 and model 6 (as there are the same model except the presence or absence of contextual information). We find a significant impact as it accounts for 111.4 \% (=81.8 \% - (-29.5 \%)). It is quite intuitive that adding a context should improve the model as we provide more meaningful information to the model.

\Subsection{Impact of one day lag}
Our model accounts for the fact that asset managers cannot immediately change their position at the close of the financial markets. It is easy to measure the impact of the one day lag as we simply need to take the difference of performance between model 9 and model 1. We find an impact of the one day lag of 112 \% (= 193.8 \% - 81.8\%). This is like for contextual information substantial. It is not surprising that a delayed action (with one period lag) after observation makes the learning process more challenging for the DRL agent as influence of variables tends to decrease with time. Surprisingly, this salient modeling characteristic is ignored in existing literature \cite{Jiang_2017,Liang_2018,Yu_2019,Wang_2019,Liu_2020,Ye_2020,Li_2019}.

\Subsection{Future work}
As nice as this work is, there is room for improvement as we have only tested a few possible hyper-parameters for our convolutional networks and could play with more layers, other design choice like combination of max pooling layers (like in image recognition) and ways to create more predictive contextual information.

\Section{Conclusion}
In this paper, we address the challenging task of financial planning in a noisy and self adapting environment with sequential, non-stationary and non-homogeneous observations. Our approach is based on deep reinforcement learning using contextual information thanks to a second sub-network. We also show that the additional constraint of a delayed action following observations has a substantial impact that should not be overlooked. We introduce the novel concept of walk forward analysis to test the robustness of the deep RL model. This is very important for regime changing environments that cannot be evaluated with a simple train validation test procedure, neither a $k$-fold cross validation as it ignores the strong chronological feature of observations.

For our trading agent, we take not only past performances of portfolio strategies over different rolling period, but also standard deviations to provide predictive variables for regime changes. Augmented states with contextual information make a big difference in the model and help the agent learning more efficiently in a noisy environment. On experiment, contextual based approach over-performs baseline methods like Markowitz or naive follow the winner and follow the loser. Last but not least, it is quite important to fine tune the numerous hyper-parameters of the contextual based DRL model, namely the various lags (lags period for the sub network fed by portfolio strategies past returns, lags period for common contextual features referred to as the common features in the paper), standard deviation period, learning rate, etc...

Despite the efficiency of contextual based DRL models, there is room for improvement. Other information like news could be incorporated to continue increasing model performance. For large stocks, like tech stocks, sentiment information based on social media activity could also be relevant.

\subsubsection{Acknowledgments.}
We would like to thank Beatrice Guez and Marc Pantic for meaningful remarks while working on this project. The views contained in this document are those of the authors and do not necessarily reflect the ones of SG CIB.

redpajama_set_name: RedPajamaArXiv


The Freeze spell is generally used in combination with the Hog Rider. However, with the presence of the Lumberjack to tank and rage the Balloon, the Freeze can be highly effective in this scenario. This spell is not commonly used in the higher arenas. But when it is utilized the right way, it can prove to be a solid offensive card. Here's the deck that works best with Freeze.
This deck was invented by dirt2222 in an effort to bring back Freeze into the game. And it worked wonders for the guy. It can take you to the Jungle Arena in no time. And if you use it right, you may even find yourself in the Legendary arena sooner than expected.
This deck is heavily dependent upon defense. Early in the game, you will have to defend as best you can with the help of Guards, Musketeer, Minions, and Ice Golem. In fact, all your cards except the Balloon can be used for defense. However, try to preserve the Balloon and Freeze for the double elixir stage.
Heavy beatdown decks may ask for a Freeze. Otherwise, cycle the cards with Arrows wherever necessary. Try to get positive Elixir trades. With the help of the Ice Golem, they won't be hard to come by. This versatile troop can take care of swarms and support troops without much hassle. Tanks can be slain with Minions, Guards, or Musketeer.
The Guards is another useful card in this deck. It can weather most spell storms rained by the opponent. The Log can only dismantle its shield. The Zap can barely inflict any damage upon it. Bring on the Fireball or even the Rocket, and the Guards will still be raging on the field.
Your offense will begin mid game. Counter pushes will play a big role here. After you have defended your mini tower well, deploy the Lumberjack ahead of your remaining troops and let them chip damage off the opponent's tower. The death-rage of the Lumberjack will enable the Musketeer and other support troops to take a couple more hits on their tower.
Your main offensive strategy will begin in the double elixir stage. Your dependable combo will be the Lumberjack-Freeze-Balloon. Deploy the Lumberjack near your mini tower. Follow it up with the Balloon. As soon as the Lumberjack is killed, covering the Balloon with Rage, drop the Freeze over the mini tower and other defensive troops. The mini tower will be destroyed in no time.
Cycle the deck with the Ice Golem for the tank and the Minions. Drop the Lumberjack again, followed by the Balloon to target their King Tower. With two Balloons on your side and a few support troops, it won't take long for you to get the three crown even.
The Lumberjack will be tough to replace in this deck. It plays a critical role during a full-fledged attack. However, you may try out the Mini Pekka in its place. The high damage provided by the troop might just help your Balloon to take down the mini tower.
Beatdown Decks: While you can easily take care of normal beatdowns with the help of the numerous high DPS troops in your repertoire, you will find it tough to deal with heavy beatdown decks. All air-decks like the Lavaloon will prove to be a formidable challenge to this deck. More often than not, you will find yourself in quite a fix while dealing with air-decks. However, they are not as frequent in Arena 8 and 9, thus enabling you to reach Hog Mountain without much trouble.


I mentioned in passing in a previous post how much I dislike the feeling of overeating. When I was younger, this wasn't a problem; I would eat to capacity and be done. Yes, I was inconveniently hungry again a couple hours later, but I stayed in good shape and didn't have to deal with that sluggish feeling one gets following a large meal.
As an adult, my taste buds have matured and thus has my appreciation for food. I am no longer such a picky eater and often find myself getting lost in a delicious meal, only to end up overdoing it. Having learned to cook, and discovering my passion for doing so, has only amplified this tendency.
As I grow older and my metabolism gets lazier, I find myself having to exercise and, at times, keep track of calories in order to avoid weight gain and the subsequent necessity to buy new clothes (which would be fun but is not currently in the budget).
In an effort to simplify (i.e. not count my calories all dang day), I've decided to start the 5/2 plan for weight loss/maintenance. For those who are unfamiliar, it's basically a plan where you eat normally 5 days a week, then mostly fast (guys get 600 cals/day, gals get 500) for 2.
Thus far I have made it through one day of fasting, which I felt was a major achievement. It was very difficult and unpleasant. I tried to think of other things besides for food but my mind always wandered back in that direction.
I chewed a lot of gum. I drank a lot of water/crystal light. I had to use the facilities frequently due to this liquid consumption.
I made the mistake of going grocery shopping.
I went on a short (tired) run through the neighborhood at dinner time, where smells of peoples' meals permeated my nose.
A Marco's pizza delivery driver drove past me three times (I have to a assume he was lost and not just trying to taunt me).
I split my calories between breakfast and dinner, (though I've heard having one big meal midday is preferable to some and may try this later) and actually felt somewhat satisfied by my salmon filet and broccoli dinner. They say it gets easier over time; let's hope so!
I also read about a method where you fast every other day, which is supposedly better for your system. I may try that at some point, but for now I think two days a week is plenty!
I'm afraid I might go to jail after killing someone!
Yes I did want to hurt someone the first day. Today is my second day and I do feel a little less hangry (as my friend Shahnaz would say).
Hang in there! You've got more willpower than you know!


text:
Q: Problem creating Bounty
Possible Duplicate:
Why can't I start a bounty?

I have been trying for the past 24 hours to create a bounty but the "Start Bounty" button is not responding. Any information on why please?

redpajama_set_name: RedPajamaStackExchange


text:
Prostate Cancer Foundation 2018
Sharing Clinical Information through Social Media - Todd Morgan & Dan Spratt
Dan Spratt, MD and Todd Morgan, MD discuss the role of social media in medical and scientific communication with Charles Ryan, MD. All three agree that social media, especially Twitter, helps to provide quicker access to breaking information at congresses when it is not possible to attend the live meeting. This communication outlet also typically provides more expert opinion and commentary that is typically not available in the publication. However, they all point out the concern that this information cannot be over interpreted as it is not peer reviewed information. Overall, they agree that when used correctly, social media provides timely access to breaking data.
Todd Matthew Morgan MD Associate Professor Specialties: Urology Area of Practice: Prostate Cancer, including robotic prostatectomy and active surveillance; Bladder Cancer, including bladder preservation approaches as well as open and robotic radical cystectomy; Kidney Cancer, including open and robotic approaches, and nephron-sparing (i.e. partial nephrectomy) when appropriate; and Penile Cancer, including robotic inguinal lymphadenectomy. Michigan Medicine Urology Oncology Clinic | Rogel Cancer Center
Daniel Spratt, MD Associate Professor Laurie Snow Research Professor of Radiation Oncology Associate Chair, Clinical Research Chair, Genitourinary Clinical Research, Rogel Cancer Center Director, Spine Oncology Program, Rogel Cancer Center
Charles J. Ryan, MD, B.J. Kennedy Chair in Clinical Medical Oncology at the University of Minnesota and Director of the Division of Hematology, Oncology and Transplantation. He previously held the position of Professor of Clinical Medicine and Urology and the Clinical Program Leader for Genitourinary Medical Oncology at the UCSF Helen Diller Family Comprehensive Cancer Center

redpajama_set_name: RedPajamaCommonCrawl


Online Personnel Management Systems Pty Ltd. – OPMS is a rostering and personnel management software.
"Eventful week at OPMS - meet the team"
"Get OPMS! And speak to the OPMS team about areas of weakness or concerns with your current processes/systems. Chances are there is already a solution"
"I never hear 'our system can't do that'…it's always 'Yes we can', 'Here's an alternative' or 'Let us work on it for you'"
Onsite Personnel Management System (OPMS) is a simple, flexible tool for managing staff and facilities globally. OPMS with its excellent rostering and training management capability is used widely in the resources sector. The software has evolved along with our clients' requirements, demonstrated by the broad range of modules.
A visual overview of employee placements and rosters with automated compliance checks.
Stores employee details, and work history allowing a variety of HR functions to be performed.
Generates emails and SMS to contact employees.
Control and display accommodation at sites.
Produces various letters on company letterhead templates.
Employee self service and operational interface.
Apply detailed criteria to determine suitable candidates.
Automatically identifies employees for transfers.
Set training requirements for each site and ensures compliance.
Manage vessel specifications and rostered vessel work activities.
We offer a variety of services that complement the OPMS software.
We work closely with our clients to develop and improve our software to meet business requirements.
We provide a high standard of phone, email and face to face support. We really get to know our users.
We provide training and a variety of instructional materials in basic and advanced use of the OPMS software.
We use our expert knowledge to assist clients in setting up the OPMS software to ensure AMSA and STCW compliance.
We combine industry expertise, technology and ongoing service support.
Save time when rostering and managing personnel by simplifying and automating manual processes.
2-4 hours of provided training is all it takes to be competent in the basic use of OPMS.
Scales well with increases in the number of facilities and employees. As you grow, so does your OPMS.
Swiftly navigates between facilities and easily creates data laden reports.
A visual representation of rostering and personnel management to increase visibility and ease of planning.
The vast number of reports able to be generated from OPMS provide many informative insights to better manage personnel.
OPMS can easily develop interfaces to work with your existing payroll, HR and travel booking software.
Log into the same live system where changes are applied to all other user sessions, providing certainty that all users are working on the same data.


1. Yes, the 2/12 version is correct; there is no new version.
2. Yes. That is correct.
I don't have any feedback yet, but I am simply putting a diagnosis and a GAF.. And in the report I am using both codes, to make sure attorneys who aren't yet up to speed understand..
On PR-2s and treatment billing, I have completed converted to ICD-10/DSM-5 without any problem. On Med-Legals, I do all 5 Axes in DSM-IV-TR (which includes the GAF) and add a section giving the DSM-5 DXs for purposes of completeness. I have noticed non-work comp IMEs indicating that they don't care if DSM-IV or DSM-5 is used; but, otherwise no mention of DSM-5 in Med-Legals.


text:
Q: How to Resolve deadlock on concurrent MERGE on two tables Recently I have been encountering deadlocks occasionally when I try to insert data concurrently on two tables.
Following is the table structure
Create Table TableA
(
  Id Bigint not null primary key ,
  FieldA1 nvarchar(50)
)
go

Create Table TableB
(
  Id Bigint not null primary key,
  TableAId Bigint not null constraint FK_TableA Foreign Key (TableAId) References TableB(Id),
  FieldB1 nvarchar(50)
)
go

CREATE type TableBParam as table
(
    Id Bigint,
    TableAId Bigint not null,
    FieldB1 nvarchar(50)
)

--Deadlock was observed on the save query
go
CREATE PROC SaveTableB
(
 @val [dbo].[TableBParam] READONLY
)
AS
BEGIN
SET NOCOUNT ON;

MERGE [dbo].[TableB] AS T
USING (SELECT * FROM @val) AS S
  ON ( T.Id = S.Id)
WHEN MATCHED THEN
    update set FieldB1 = S.FieldB1
WHEN NOT MATCHED THEN
    insert(TableAId, FieldB1) Values(S.TableAId, S.FieldB1);

END
go

On a single transaction I am trying to complete data insertion on tableA and Table B by calling respective stored procedure (SaveTableA and SaveTableB).
SaveTableA is same as SaveTableB.
.
Database has read committed snapshot isolation enabled.
wondering how deadlock occurs on insertion? From the above graph I infer that the SaveTableB from one thread is victim while other instance of same procedure on different thread was holding lock on primary key of TableA.
From this reference I understand that the lock here is on the Btree of the Index based on the reference https://technet.microsoft.com/en-us/library/ms189849(v=sql.105).aspx and the deadlock graph above. While the locked resource is on the clustered index BTree node itself,
From the above inference I am curious to understand following

*

*Does Insertion operation always holds lock at BTree level as seen the graph?

*Trying to understand why there is a lock on the clustered index of TableA when the statements involved in the deadlock are SaveTableB procedure from two different transactions?

*Since I am using the Merge statement here, is it the assumption that the btree lock on insert is hold on till the transaction ends or till the Merge statement ends?

Curious to understand that if it is possible to get deadlocks in this kind of scenario mentioned, trying to understand on how to approach it so that I will prepare myself to solve it. Thanks for helping me learn about this.

A: Concurrent MERGE statements will deadlock or produce PK violations by default, as the "scan" phase of the MERGE is performed without a restrictive lock.  You need to add a lock hint for this to work.  See Why is TSQL MERGE failing with a Primary Key Violation? Isn't it atomic? by David Browne for MERGE locking details.
Or just fix it like this:
MERGE [dbo].[TableB] with (serializable) AS T
USING (SELECT * FROM @val) AS S
  ON ( T.Id = S.Id)
WHEN MATCHED THEN
    update set FieldB1 = S.FieldB1
WHEN NOT MATCHED THEN
    insert(TableAId, FieldB1) Values(S.TableAId, S.FieldB1);

A: If running two querys that is in the opposite order can cause deadlock. Your problem is a classic deadlock example.

What will happen.

*

*Transaction 1 will lock TableA (SaveTableA)

*Transaction 2 will lock TableB (SaveTableB)

*Transaction 1 waits for Lock on TableB to be release to do the update/insert

*Transaction 2 waits for Lock on TableA to be released to do the update/insert

So to resolve your problem, dont run them at same time or update the logic to do something else.
If you in one of the procs add a Select statement to force a PAGE and UPDATE LOCK before doing the initial select/scan or use SERIALIZABLE. So in SaveTableB add something like this:
MERGE [dbo].[TableB] AS T  WITH (PAGLOCK,UPDLOCK)

Or
MERGE [dbo].[TableB] AS T  WITH (SERIALIZABLE)


Be a part of our fast growing team – Are you looking for employment in Essex or surrounding area?
Look no further as you've come to the right place- you could be just a few steps away from being a part of our fast growing online advertising team! Essex is a fresh website for us and there is no end to the amount of businesses here. We have a whole county of businesses to sign up, let's get out there so these businesses can start to receive the benefits that come with being listed on Things To Do In Essex.
We are currently looking for field sales and telephone sales executives who are driven, confident and enthusiastic. We're an energetic and hardworking team who love to work in a fun and rewarding environment. Experience in sales is not essential but will obviously help if you already have some experience in the field. The most effective method of selling we have found is face to face. When people can see what they are being sold and what the website is like it gets them more excited about advertising with us. Experience with the internet is pretty much a necessity, if you don't know much about the internet and how Google works then you may find it a bit more difficult to understand the concept of 'Things To Do In Essex'.


text:
Good Fences Make Bad Neighbors for Wild Animals
So these scientists are using high-tech methods to do something about it.
Michael Parks
(Matthew Brown/AP)
This story was originally published by High Country News and is reproduced here as part of the Climate Desk collaboration.
One smoke-tinged July morning on Horse Prairie—a plateau of big sagebrush and dusty washes overlooking Horse Prairie Creek in southwestern Montana—a man sat at the helm of a skid-steer loader. Attached to its front was a spool-like contraption called a Dakota wire winder and post puller. Four volunteers threw up their thumbs—Ready!—and the man flung a switch. The winder spun up, and a stretch of woven wire fence lying on the ground jerked into motion.
Soon, a hundred-plus years of tangled Western history had become a tidy bale.
Andrew Jakes joined the volunteers in a cheer. The group was the last in a two-week parade of helpers who had come out to Horse Prairie, and Jakes believed their hard work wrestling fences would be worth it.
he American West, research shows, has enough fencing to circle the equator 25 times.
Jakes is a biologist with the National Wildlife Federation and an expert on pronghorn antelope. In 2018, he and four colleagues published a paper calling for more research on how fences affect ecosystems. They also coined a term: "fence ecology." Today, the growing subdiscipline is not just revealing how fences can harm Western wildlife; it's also informing solutions.
Fence ecology research shows that the West is a wiry place, containing enough fencing to circle the equator 25 times. Sage grouse, peregrine falcons and other birds collide with fences, and ungulates must navigate an endless obstacle course. A 2021 paper found that pronghorn in Wyoming encountered fences an average of 249 times in a single year and changed their behavior around the barriers nearly 40% of the time. Fences often ensnare and kill large animals; woven wire with barbed wire on the top, like that on Horse Prairie, is particularly lethal. Fences also separate mothers from calves, exclude herds from prime habitat and exhaust and injure animals. "Probably a quarter to half of (pronghorn I've seen) have scarring at this point, from trying to get under," Jakes said.
A number of nonprofits, land trusts, ranchers, tribal nations and government agencies have already removed or modified thousands of miles of harmful fences. But because fences are difficult to study, these projects haven't had much science to guide them. There is no global fence map, as there is for roads; fences are hard to see on satellite imagery. And even as satellite-based mapping improves, researchers say it's difficult to distinguish between an impassable woven-wire fence and, for example, a dilapidated or smooth-wire fence that's more permeable to wildlife.
Recently, though, fence ecologists have begun to unravel some mysteries. Using models, they've estimated the locations of fences over large areas and painstakingly mapped them in a number of important habitats. And they're studying the paths of GPS-collared animals: If a mule deer makes a 90-degree turn in a section of roadless rangeland, for example, it could be because of a problem fence.
One researcher used modeling, mapping, and GPS-collar data to identify fencing that impedes wild grazers.
On Horse Prairie, one of Jakes' colleagues, Simon Buzzard, combined all three tools—modeling, mapping and GPS-collar data from a Montana Fish, Wildlife and Parks pronghorn study—to identify stretches of fencing that appeared to impede ungulate movement. "We had collars from 40 animals," Buzzard said. "Many of them stopped right here," at the 2.1-mile stretch that the volunteers took down in July.
The removal effort was the result of a cost-sharing agreement among the National Wildlife Federation and the landowners on either side of the fence: the Bureau of Land Management and two ranchers. The section that came out is the first of 10 miles of fence that Buzzard and Jakes hope to fix through a National Fish and Wildlife Foundation grant this year.
Still, the sheer scale of fencing in the West presents a daunting challenge, and many questions remain unanswered. For example, how do fences affect the long-term health of wildlife populations? And how do different species learn about and use wildlife-friendly fences? "Until we have a large amount modified and recorded," said Wenjing Xu, a Ph.D. candidate at UC Berkeley who has led or co-authored several recent fence ecology papers, "we don't know how well or fast (animals) will respond."
Back on Horse Prairie, piles of fencing were stacked on a flatbed trailer by noon. The following week, a contractor would put up a four-strand barrier with a smooth top wire 40 inches off the ground—low enough for "jumpers" like elk, deer and moose to go over—and a smooth bottom wire 16 inches above the earth, high enough for pronghorn, as well as calves of other ungulates, to slide under. (Jakes' research shows an 18-inch bottom wire is even better, but for now, he said, 16 inches is a major step forward.)
"Yesterday, we saw nine elk with one little calf," Buzzard said to Jakes as they drove across the newly reopened range. "If that old fence had been up, would that calf have been able to cross under?'
These Wildfire Thunderclouds Are Ravaging the West
A Severe Drought Is Threatening the Hoover Dam Reservoir—and Water Throughout the West
In New York City, Traffic Deaths Were Almost Equal to Shooting Deaths in 2022