text:
\section{Introduction}
The concept of \ac{SON} was introduced nearly a decade ago when various use cases of \ac{SON} were standardized by 3GPP \cite{TS32500}. However, the introduction of network slicing into 5G and beyond raises new challenges to \ac{SON}, while the slice-aware \ac{SON} has rarely been studied.
Slice-aware \ac{SON} requires solutions that satisfy the {\it slice-specific} service requirements, while the legacy \ac{SON} solutions optimize {\it cell-specific} or {\it cell-pair-specific} (dedicated with respect to a neighboring cell) parameters by observing cell-specific \acp{KPI}

In this paper, we study a particular use case of slice-aware \ac{SON}, namely, the slice-aware mobility robustness optimization (SAMRO). The legacy \ac{MRO} has the objective of minimizing the \ac{HO}-related \acp{RLF} and the number of unnecessary handovers. The decision variables are a set of cell-specific and cell-pair-specific \ac{HO} parameters. The complexity and dynamics of user mobility result in a \ac{HO} performance metric-related objective function that is in general non-convex, non-linear, and dependent on a high-dimensional \ac{HO} parameter space. Thus, practical solutions include heuristics \cite{nguyen2017mobility}, reinforcement learning \cite{mwanje2014distributed}, and Bayesian methods \cite{liao2013statistical}. However, these solutions, aiming to optimize the signal quality-based \ac{HO} performance metrics, cannot guarantee the per-slice service continuation in terms of throughput and latency. A recent work \cite{9405369} proposed to improve the quality of experience while optimizing the \ac{HO} performance. However, with the cell-specific parameters, the \ac{HO} optimization cannot be customized for every slice.

We propose a \ac{DRL}-based solution that provides a more seamless mobility service through the introduction of slice-specific \ac{HO} parameters and performance metrics. Although the newly introduced parameters increase the dimensions of the state and action spaces, by utilizing model-free actor-critic architectures such as \ac{TD3} \cite{fujimoto2018addressing}, we can work with large continuous state and action spaces with significantly reduced model complexity, compared to the value-based algorithms such as \ac{DQN} \cite{mnih2015human}.

Even with the help of the actor-critic architecture, we still face two challenges when developing a practical solution:
\begin{itemize}
\item {\it Large discrete action space}: In practical systems, \ac{HO} parameters are selected from finite discrete sets, but directly working with huge discrete action space leads to intractable complexity. How to project the continuous agent actions to the discrete operating actions?
\item {\it Effective training with knowledge transfer}: \ac{DRL} approaches usually require substantial exploration, while in practice online training with unconstrained exploration is expensive. On the other hand, a large amount of biased data -- usually collected in the safe operating space -- is available.
How to train more effectively by transferring knowledge from the biased offline data?
\end{itemize}
To overcome these challenges, we propose a two-step transfer learning-assisted \ac{SAMRO} scheme, consisting of regularized offline \ac{DRL} model training and online fine-tuning. The main contribution is threefold:
\begin{itemize}
\item {\it Mapping between continuous and discrete actions}: Motivated by \cite{dulac2015deep}, we use approximate nearest-neighbor methods to find the mapping between the continuous agent actions and the discrete operating actions.
\item {\it Regularized offline \ac{DRL} training}: We develop a regularized offline \ac{DRL} algorithm to exploit the offline data, while dealing with the bias by regularizing the objective with the density estimates of state-action pairs.
\item {\it Sample-efficient online fine-tuning}: We propose mixed experience replay during the online fine-tuning. The newly collected samples are mixed with a selected subset of offline samples for the batch training.
\end{itemize}
The rest of the paper is organized as follows. In Section \ref{sec:Model} we define the system model and the slice-specific \ac{HO} parameters and metrics. The \ac{SAMRO} problem is discussed in Section \ref{sec:Problem}. Section \ref{sec:Solution} introduces the proposed transfer learning and actor-critic-based solution. The numerical results are shown in Section \ref{sec:Numerical}. Section \ref{sec:Conclusion} summarizes the conclusions.

\section{System Model}\label{sec:Model}
We consider a set of cells in the neighboring area, denoted by $\set{N}$. Each cell can support numerous instances of different slices.
Let the set of slices be denoted by $\set{S}$. A service of slice $s\in\set{S}$ has a defined throughput requirement $\eta^{\ast}_s$ and latency requirement $d^{\ast}_s$.
Assume that there are a set of $B$ directional cell boundaries, denoted by $\set{B}:=\left\{(n_i,m_i): n_i, m_i\in\set{N}; i = 1, \ldots, B\right\}$. In practice, if the boundary $(n, m)\in\set{B}$, it means that cell $m$ is within the neighboring cell list of cell $n$. Note that the boundary is directional, i.e., $(n, m) \neq (m, n)$ for $m\neq n$. Assume that if $(n, m)\in\set{B}$, then $(m, n)\in\set{B}$. Therefore, there are $B/2$ combinations (regardless of the order) of the neighboring cell-pairs. We denote the set of cell-pairs as $\set{B}'$ with $|\set{B}'|=B/2$.

\subsection{\ac{HO} Process and Slice-Specific Parameters}\label{subsec:HO_Params}
In the legacy \ac{MRO}, for any user served by cell $n\in\set{N}$, a simplified \ac{HO} criterion\footnote{The complete version can be found in \cite[Section 5.5.4.4]{TS38331}, but many offsets are not needed for intra-frequency handover.} for a handover from cell $n$ to cell $m$ when $(n,m)\in\set{B}$ is that
$P_m(t) > P_n(t) + O_{n, m} $ holds for a time duration of $T_{n, m}$, where $P_m$ and $P_n$ (in dBm) indicates the received signal strength from cell $m$ and $n$ respectively, and $O_{n, m}$ (in dB) and $T_{n, m}$ (in ms) are the \ac{HOM} and \ac{TTT} from cell $n$ to cell $m$.

The \ac{HOM} and \ac{TTT} defined above are on the cell-pair basis. We propose to define them also on the slice basis, such that the \ac{HO} decision also considers whether the neighboring cell can provide the required service performance. The modified \ac{HO} criterion for a service in slice $s$ from cell $n$ to cell $m$ at time $t'$ (in ms) is given by
\begin{equation}
P_m(t)  > P_n(t) + O_{n,m,s},  \ \forall t \in [t'-T_{n,m,s}+1, t'].
\label{eqn:Slice_HO}
\end{equation}
Note that without loss of generality, \eqref{eqn:Slice_HO} can be generalized to mobility-specific or even user-specific \ac{HO} criteria. For example, we can group the users based on both mobility and slice classes, and define the user group-specific \ac{HO} parameters. Since mobility-specific handover has been well studied \cite{tesema2016evaluation}, in this paper we focus on the slice-specific aspect to ensure the service continuation.

\subsection{Slice-Specific \ac{HO} Performance Metrics}\label{subsec:HOMetrics}
The conventional \ac{HO} optimization aims to reduce the following four \ac{HO} events:
\begin{itemize}
\item {\it \ac{HOL}}: When a user is leaving the coverage area of its serving cell $n$ towards the target cell $m$, but the handover is triggered too late which causes \ac{RLF} before completing a handover.
\item {\it \ac{HOE}}: This occurs when the \ac{HO} decision is made too early. The neighboring cell cannot provide a sustainable signal quality to the user and a \ac{RLF} happens right after the handover.
\item {\it \ac{HOW}}: This happens when the user is handed over to a wrong cell. A \ac{RLF} is detected shortly after a successful handover to the target cell and then the user is connected to another neighboring cell.
\item {\it \ac{HOPP}}: When a user is handed over from cell $n$ to cell $m$, but after a short time period the handover from cell $m$ back to cell $n$ triggers.
\end{itemize}
A \ac{HO} attempt can either fail or succeed. The handover failures includes \ac{HOL}, \ac{HOE} and \ac{HOW} since they all cause \acp{RLF}, while the successful handovers includes \acp{HOPP}, although they cause unnecessary handovers. Let the counts of \ac{HO} attempts, successful \acp{HO}, \acp{HOL}, \acp{HOE}, \acp{HOW}, and \acp{HOPP} for slice $s$ and boundary $(n,m)$ be denoted by $N^{\operator{(HOA)}}_{n,m,s}, N^{\operator{(HOS)}}_{n,m,s}, N^{\operator{(HOL)}}_{n,m,s}, N^{\operator{(HOE)}}_{n,m,s}, N^{\operator{(HOW)}}_{n,m,s}, N^{\operator{(HOPP)}}_{n,m,s}$, respectively. We propose to include them in the \ac{KPI} reports for each slice $s\in \set{S}$ and each boundary $(n,m)\in\set{B}$.

We define the following two per-slice {\bf \ac{HO} metrics to minimize},  {\it \ac{HFR}} and {\it \ac{PPR}}:
\begin{align}
R^{\operator{(HF)}}_s & = \frac{\sum_{(n,m)\in\set{B}} N^{\operator{(HOL)}}_{n,m,s} + N^{\operator{(HOE)}}_{n,m,s} + N^{\operator{(HOW)}}_{n,m,s}}{\sum_{(n,m)\in\set{B}}N^{\operator{(HOA)}}_{n,m,s}} \label{eqn:HFR}\\
R^{\operator{(PP)}}_s & = \frac{\sum_{(n,m)\in\set{B}}N^{\operator{(HOPP)}}_{n,m,s}}{\sum_{(n,m)\in\set{B}}N^{\operator{(HOS)}}_{n,m,s}}. \label{eqn:SHOR}
\end{align}
Since $N^{\operator{(HOA)}}_{n,m,s} = N^{\operator{(HOS)}}_{n,m,s} + N^{\operator{(HOL)}}_{n,m,s} + N^{\operator{(HOE)}}_{n,m,s} + N^{\operator{(HOW)}}_{n,m,s}$ and $N^{\operator{(HOPP)}}_{n,m,s}\leq N^{\operator{(HOS)}}_{n,m,s}$, we have $R^{\operator{(HF)}}_s\in [0, 1]$ and $R^{\operator{(PP)}}_s\in [0, 1]$, $\forall s\in\set{S}$.

\subsection{Slice-Specific Service Quality Metrics}\label{subsec:ServiceMetrics}
As reflected in \eqref{eqn:Slice_HO}, the above-introduced \ac{HO} metrics are signal quality based. To observe the per-slice service performance, we define in below two {\bf service metrics to maximize}, {\it \ac{TSL}} $L^{\operator{(T)}}_{k,s}$ and {\it \ac{LSL}} $L^{\operator{(D)}}_{k,s}$ for each user $k$ associated to slice $s\in\set{S}$:
\begin{equation}
L^{\operator{(T)}}_{k,s}  = \min\left\{\frac{\eta_{k,s}}{\eta^{\ast}_s}, 1 \right\}\mbox{ and }
L^{\operator{(D)}}_{k,s}  = \min\left\{\frac{d^{\ast}_s}{d_{k,s}}, 1\right\} \label{eqn:ue_SLs},
\end{equation}
where $\eta_{s, k}$ and $d_{s, k}$ are the achieved throughput and latency of user $k$ associated to slice $s$, respectively. Note that $L^{\operator{(T)}}_{s,k}$ and $L^{\operator{(D)}}_{s,k}$ are upper bounded by $1$. Thus, as long as the service quality is satisfied, we have $L^{\operator{(T)}}_{s,k}= L^{\operator{(D)}}_{s,k} = 1$.

Let the set of users associated to cell $n$ and slice $s$ be denoted by $\set{K}_{n,s}$ with the cardinality $|\set{K}_{n,s}| = K_{n,s}$.
Let each cell $n$ report to the central agent the following three measures: 1) number of the users in each slice $K_{n,s}$, 2) the per-slice sum \ac{TSL} $L^{\operator{(T)}}_{n,s}:=\sum_{k\in\set{K}_{n,s}}L^{\operator{(T)}}_{k,s}$, and 3) the per-slice sum \ac{LSL} $L^{\operator{(D)}}_{n,s}:=\sum_{k\in\set{K}_{n,s}}L^{\operator{(D)}}_{k,s}$. The slice-specific average \ac{TSL} and \ac{LSL} for the considered area is given by
\begin{equation}
L^{\operator{(T)}}_s = \frac{\sum_{n\in\set{N}} L^{\operator{(T)}}_{n,s}}{\sum_{n\in\set{N}}  K_{n,s}} \mbox{ and } L^{\operator{(D)}}_s = \frac{\sum_{n\in\set{N}} L^{\operator{(D)}}_{n,s}}{\sum_{n\in\set{N}}  K_{n,s}}.
\label{eqn:TSL_LSL}
\end{equation}
With \eqref{eqn:ue_SLs}, we have $L^{\operator{(T)}}_s\in [0, 1]$ and $L^{\operator{(D)}}_s \in [0, 1]$, $\forall s\in\set{S}$.

\subsection{Markov Decision Process Model}\label{subsec:MDP}
We model the multi-cell system as a \ac{MDP}, defined by the tuple $\left(\set{S}, \set{A}, P(\cdot), r(\cdot), \gamma\right)$, where $P:\set{S}\times \set{A} \times \set{S} \to [0, 1]$ indicates the transition dynamics by a conditional distribution over the state space $\set{S}$ and the action space $\set{A}$, $r:\set{S}\times \set{A}\to{\field{R}}$ denotes the reward function, and $\gamma\in[0,1]$ is the discount factor. We define the action, state, and reward of the \ac{MDP} as follows.

\subsubsection{Action}
Assume \ac{HOM} $O_{n,m,s}\in\set{O}$ and $T_{n,m,s}\in\set{T}$, where $\set{O}$ and $\set{T}$ are in practice both finite discrete sets. The $2BS$-dimensional action $\ve{a}\in\set{A}$ is defined as
\begin{align}
\ve{a} := & [(O_{n,m,s}, T_{n,m,s}): O_{n,m,s}\in\set{O}, T_{n,m,s} \in\set{T}, \nonumber\\
&  (n, m)\in\set{B}, s\in\set{S}] \in \set{A}:=(\set{O}\times \set{T})^{BS}.
\label{eqn:action}
\end{align}

\subsubsection{State}
As shown in Table \ref{tab:state}, the network state $\ve{s} \in {\field{R}}^{(4N+2B)S}$ includes $4NS$ per-cell per-slice measurements and $2BS$ per-cell-pair per slice \ac{HO} event counts (with $B/2$ cell-pairs, we have $4*(B/2)*S=2BS$ per-cell-pair per-slice states). Note that we use the counts of \ac{HO} events directly instead of \ac{HFR} and \ac{PPR} defined in Section \ref{subsec:HOMetrics} because of two reasons: 1) sometimes the number of HO attempts is zero and using raw counts avoids zero denominators, and 2) raw counts may provide more information reflecting too-early and too-late \ac{HO} decisions.  Also, we reduce the dimension of state by adding the \ac{HO} event counts of the both directions of the cell boundaries. For the same reason we do not include the counts of \ac{HOE} and \ac{HOW} into the state, since we expect that the count of \acp{HOPP} indicates the too-early \ac{HO} decisions.

\subsubsection{Reward}
The reward function is a utility function based on the \ac{HO} metrics $R^{\operator{(HF)}}_s$ and $R^{\operator{(PP)}}_s$ defined in \eqref{eqn:HFR} and \eqref{eqn:SHOR} respectively, and the service metrics $L^{\operator{(T)}}_s$ and $L^{\operator{(D)}}_s$, for $s\in\set{S}$ defined in \eqref{eqn:TSL_LSL}.  For example, one option is given by:
\begin{equation}
r = \sum_{s\in\set{S}, X\in\{\operator{T}, \operator{D}\}} w_s^{(X)} L_s^{(X)} - \sum_{s\in\set{S}, X\in\{\operator{HF}, \operator{PP}\}} w_s^{(X)} R_s^{(X)}.
\label{eqn:reward}
\end{equation}
where $w_s^{(X)}$ for $s\in\set{S}$, $X\in\{\operator{T}, \operator{D}, \operator{HF}, \operator{PP}\}$ are the weight factors. Note that $L_s^{\operator{(D)}}$ is the latency satisfaction level that is {\bf inversely} proportional to latency -- a larger value indicates higher satisfaction by meeting the latency constraints \eqref{eqn:ue_SLs}. The designed reward function incentives to improve the service quality satisfaction level, as well as to minimize the handover failures and unnecessary handovers. Because all the metrics included in reward are within $[0, 1]$, we have the same scale for the metrics, and the reward is bounded.
 %
\begin{center}
\begin{table}[tbp]
 \caption{Measures included in the state}\label{tab:state}
\begin{tabular}{lllll}
\cline{1-2}
\multicolumn{1}{|l|}{Per-cell per-slice states} & \multicolumn{1}{l|}{Per-cell-pair per-slice states} &  &  &  \\ \cline{1-2}
\multicolumn{1}{|l|}
{\begin{tabular}[c]{@{}l@{}}
$\forall n\in\set{N}, s\in\set{S}:$ \\
$\cdot$ Load $l_{n,s}$\\
$\cdot$ Number of users $K_{n,s}$\\
$\cdot$ TSL $L_{n,s}^{\operator{(T)}}$\\
$\cdot$ LSL $L_{n,s}^{\operator{(D)}}$ \end{tabular}} &
\multicolumn{1}{l|}
{\begin{tabular}[c]{@{}l@{}}
$\forall (n,m)\in \set{B}', s\in \set{S}:$\\
$\cdot$ HO attempts $N^{\operator{(HOA)}}_{n,m,s} + N^{\operator{(HOA)}}_{m,n,s}$ \\
$\cdot$ Successful HOs $N^{\operator{(HOS)}}_{n,m,s} + N^{\operator{(HOS)}}_{m,n,s}$\\
$\cdot$ \acp{HOL} $N^{\operator{(HOL)}}_{n,m,s} + N^{\operator{(HOL)}}_{m,n,s}$\\
$\cdot$ \acp{HOPP} $N^{\operator{(HOPP)}}_{n,m,s} + N^{\operator{(HOPP)}}_{m,n,s}$\end{tabular}}
 &  &  &   \\ \cline{1-2}   &  &  &  & \\   &  &  & &
\end{tabular}
\end{table}
\end{center}
\section{Problem Statement}\label{sec:Problem}
As many classical reinforcement learning problems, we want to find a policy $\pi:\set{S}\to\set{A}$ which decides the \ac{HO} parameters $\ve{a}_t$ based on the network state $\ve{s}_t$, to maximize the expectation of the cumulative discounted reward of a finite horizon $H$:
\begin{equation}
\maximize_{\pi} \  {\field{E}}_{\pi}\left[\sum_{t=0}^H\gamma^t r(\ve{s}_t, \ve{a}_t)\right], \mbox{ s.t. } \ve{a}\in \set{A}.
\label{eqn:problem}
\end{equation}
However, we face the following two challenges. The first one is the complexity caused by the enormous discrete action space. In 3GPP, it is defined $\set{O} := \{-24, -23, \ldots, 23, 24\}$ (in dB) and $\set{T} := \{0, 40, 64, 80, 100, 128, 160, 256, 320, 480, 512, 640, 1024,\\ 1280, 2560, 5120\}$ (in ms) \cite{TS38331}, which means $|\set{O}| = 49$ and $|\set{T}| = 16$. Even for a neighboring area of $9$ cells, assuming that each cell only has $2$ neighboring cells, we have $36$ directional cell boundaries. With only $2$ slices per cell, the size of the action space yields $|\set{A}|=(49*16)^{72}> 1e208$. This is intractable with the value-approximation methods for discrete action space such as \ac{DQN}.

The second challenge is raised by the high cost of online interaction. Reinforcement learning, especially with large state and action spaces, usually requires sufficient exploration to converge. However, in real-world network systems, executing actions leading to unknown states may trigger risky network states. On the other hand, it is possible to collect an off-policy sample set $\left\{(\ve{s}_t, \ve{a}_t, \ve{s}_{t+1}, r_t): t\in{\field{N}}_0\right\}$ in the safe operating action space. Thus, our problem is to learn from a biased offline sample set with a minimal number of online interactions.

\section{Proposed Approach}\label{sec:Solution}
To deal with the large discrete action space, we propose a policy built upon the actor-critic algorithms with a projection policy that maps the continuous proto-action to its nearest neighbors in the discrete action space $\set{A}$. We  describe \ac{TD3} \cite{fujimoto2018addressing} as the applied actor-critic algorithm in Section \ref{subsec:Actor-critic}. Then, we propose the action projection policy in Section \ref{subsec:DiscAct}. To learn from the biased data, we propose in Section \ref{subsec:TFlearning} a transfer learning approach for \ac{DRL} that comprises two steps: 1) regularized offline \ac{DRL}, and 2) online \ac{DRL} model fine-tuning.

\subsection{Actor-Critic Algorithm}\label{subsec:Actor-critic}
The objective function in \eqref{eqn:problem} can be seen as a state-action value function $Q^{\pi}(\ve{s}, \ve{a})={\field{E}}^{\pi}\left[\sum_{t=0}^H\gamma^t r_t|\ve{s}_0=\ve{s}, \ve{a}_0=\ve{a}\right]$. The value function $Q^{\pi}$ can be expressed in a recursive manner using the Bellman equation \cite{bertsekas2012dynamic}:
\begin{equation}
Q^{\pi}(\ve{s}, \ve{a}) = r(\ve{s}, \ve{a}) + \gamma\sum_{\ve{s}'|\ve{s}, \ve{a}}Q^{\pi}(\ve{s}', \pi(\ve{s}')).
\label{eqn:Bellman}
\end{equation}
For value-based \ac{DRL} in discrete space, $Q^{\pi}(\ve{s}, \ve{a})$ is approximated by a neural network which takes both state and action as input, and output $|\set{A}|$ evaluations of $Q$ values corresponding to every possible action. Then, it chooses the action with the best estimation of the Q value to execute. Unfortunately, because its complexity grows linearly with $|\set{A}|$, it is intractable when the discrete action space is huge.

We consider next relaxing the action space to the continuous space, which enables us to apply the actor-critic method in the continuous action space. More details of action projection will be provided in Section \ref{subsec:DiscAct}.

The actor-critic method solves the optimization problem by using a critic $Q_{\theta}\left(\ve{s}_t, \ve{a}_t\right)$ to approximate the value function, i.e., $Q_{\theta}(\ve{s}_t, \ve{a}_t) \approx Q^\pi(\ve{s}_t, \ve{a}_t)$ and an actor $\pi_{\omega}(\ve{s}_t)$ to update the policy $\pi$ at every step in the direction suggested by critic. The critic and actor both modeled with neural networks are characterized by parameters $\theta$ and $\omega$, respectively. In this way, unlike \ac{DQN} having $|\set{A}|=(|\set{O}||\set{T}|)^{BS}$ outputs for our problem formulation (it estimates Q values corresponding to all possible actions), the actor has an output dimension of $2BS$ because it directly predicts the action $\ve{a}\in{\field{R}}^{2BS}$.

We follow a particular actor-critic algorithm, namely, the \ac{TD3} \cite{fujimoto2018addressing}, to train the agents, which has proven effectiveness when dealing with high dimensional and continuous state space. As an extension of \ac{DDPG} \cite{silver2014deterministic}, \ac{TD3} overcomes the \ac{DDPG}'s problem of overestimating Q values by introducing double critic structure for both current networks $Q_{\theta_1}, Q_{\theta_2}$ and target networks $Q_{\theta'_1}, Q_{\theta'_2}$. The minimum of the two Q values is used to represent the approximated Q value of the next state. Moreover, \ac{TD3} uses smoothing for target policy $\pi_{\omega'}$, i.e., adding noise to the target action to make it harder for the policy to exploit Q-function errors by smoothing out Q along changes in action. The target actions are computed by
\begin{equation}
    \label{eqn:target_action}
    \ve{a}'(\ve{s}_{t+1}) = clip\left(\pi_{\omega'}\left(\ve{s}_{t+1}\right) + clip(\epsilon, -c, c), \ve{a}_{\operator{L}}, \ve{a}_{\operator{H}}\right),
\end{equation}
where the added noise $\epsilon\sim {\field{N}}(0, \sigma)$ is clipped within $[-c, c]$ to keep the target close to the original action, and $\ve{a}_{\operator{L}}, \ve{a}_{\operator{H}}$ are the lower and upper bounds of the action, respectively.

The critics' parameters $\theta_i, i=1,2$ are updated with temporal difference (TD) learning, given by:
\begin{equation}
\label{eqn:update-critic}
\begin{aligned}
	L\left(\theta_i\right) & = {\field{E}}\left[\left(y_t - Q_{\theta_i}\left(\ve{s}_t, \ve{a}_t\right)\right)^2\right], i =1,2\\
\mbox{where }	y_t & = r_t + \gamma \min_{i=1,2} Q_{\theta'_i}\left(\ve{s}_{t+1}, \ve{a}'(\ve{s}_{t+1})\right).
\end{aligned}
\end{equation}

The actor is updated with the policy gradient on the expected accumulated regularized reward, give by
\begin{equation}

{\field{E}}\left[\nabla_{\ve{a}}Q_{\theta_1}(\ve{s},\ve{a})|_{\ve{s}=\ve{s}_t, \ve{a}=\pi_{\omega}(\ve{s}_t)}\nabla_{\omega}\pi_{\omega}\left(\ve{s}\right)|_{\ve{s}=\ve{s}_t}\right].\label{eqn:actorgradient}
\end{equation}

\subsection{Dealing with Large Discrete Action Space}\label{subsec:DiscAct}
In this part, we propose the method to relax the discrete action space to the continuous space, as well as to map the proto-action back to the discrete action by approximating its nearest neighbors in $\set{A}$.

Motivated by \cite{dulac2015deep}, given the proto-action $\tilde{\ve{a}}$ as the output of actor $\pi:\set{S}\to{\field{R}}^{2BS}$, it is very likely that $\tilde{\ve{a}}\notin\set{A}$. We need to design a policy to map $\tilde{\ve{a}}$ to an action in the desired discrete space $\hat{\ve{a}}\in\set{A}$.
The mapping policy is similar to the {\it Wolpertinger Policy} in \cite{dulac2015deep}, summarized in Algorithm \ref{alg:Wolpertinger}. We first find with policy $G_k$ (Algorithm \ref{alg:neighborselection}) the $k$ nearest neighbors of $\tilde{\ve{a}}$ in $\set{A}$, denotes by $\set{A}_k(\tilde{\ve{a}}):=\left\{\ve{a}^{(1)}, \ldots, \ve{a}^{(k)}\right\}$, then select the one with the best $Q$-estimates (estimated by the critic function) as the action applied to the environment. The mapped $\hat{\ve{a}}\in\set{A}$ is given by:
\begin{equation}
\hat{\ve{a}} = \argmax_{\ve{a}\in G_k\circ \pi_{\omega}(\ve{s})} Q_{\theta_1}\left(\ve{s}, \ve{a}\right).
\label{eqn:mapping_Q}
\end{equation}
\begin{algorithm}
\caption{Action Projection Policy}
\label{alg:Wolpertinger}
\begin{algorithmic}[1]
\State Get state $\ve{s}$
\State $\tilde{\ve{a}}=\pi_{\omega}(\ve{s})$  \Comment{Get proto-action from actor}
\State $\set{A}_k(\tilde{\ve{a}}) = G_k(\tilde{\ve{a}})$ \Comment{Get $k$ nearest neighbors of $\tilde{\ve{a}}$ }
\State $\hat{\ve{a}}=\argmax_{\ve{a}\in\set{A}_k(\tilde{\ve{a}})} Q_{\theta_1}\left(\ve{s}, \ve{a}\right)$
\State Apply $\hat{\ve{a}}$ to environment; receive $r, \ve{s}'$.
\end{algorithmic}
\end{algorithm}
As for the strategies of selecting $k$ nearest neighbors $G_k:{\field{R}}^{2BS}\to\set{A}^k$, in \cite{dulac2015deep} the authors use approximate nearest neighbor methods based on K-Means tree structure \cite{muja2014scalable} allowing for logarithmic-time search complexity relative to the size of action space $O\left(ZBS (\log|\set{A}|/\log k)\right)$, where $Z$ is the maximum number of points to examine (a hyperparameter of the algorithm). However, except for the search complexity, the tree construction complexity is also substantially high and requires extra large memory space. Considering that our action space is already well structured (containing gridded points in high-dimensional space), we propose a coarse, heuristic, but more efficient way to approximate the $k$ neighbors with the search complexity $O(kBS(|\set{O}|+|\set{T}|))$, described in Algorithm \ref{alg:neighborselection}. Let the $j$-th dimension of $\tilde{\ve{a}}$ be denoted by $\tilde{a}_j$. The intuition is to choose the nearest values in $\set{O}$ or $\set{T}$ along each dimension with the probability depending on the reciprocal of the distance from $\tilde{a}_j, \forall j$. The proposed heuristics can quickly generate $k$ neighbors of $\tilde{\ve{a}}$ in $\set{A}$.
\begin{algorithm}
\caption{Approximate $k$ neighbor selection}
\label{alg:neighborselection}
\begin{algorithmic}[1]
\State $\set{A}_k(\tilde{\ve{a}})\leftarrow\emptyset$
\For{the $i$-th neighbor to be generated, $i\in\{1, \ldots, k\}$}
	\For{$j\in\{1, \ldots, 2BS\}$}
	\State $\Omega \leftarrow \set{O}$ if the dimension is \ac{HOM}, o/w $\Omega \leftarrow \set{T}$
		\If{$\tilde{a}_j>=\max(\Omega)$ or $\tilde{a}_j<=\min(\Omega)$}
		\State $a^{(i)}_j \leftarrow \argmin_{x\in\Omega} |x-\tilde{a}_j|$
		\Else
		\State $a_j^{+} \leftarrow \argmin_{x\in\Omega, x\geq \tilde{a}_j} |x-\tilde{a}_j|$
		\State $\nu_j^{+} \leftarrow 1/(a_j^{+} - \tilde{a}_j)$
		\State $a_i^{-} \leftarrow \argmin_{x\in\Omega, x\leq \tilde{a}_j} |x-\tilde{a}_j|$
		\State $\nu_j^{-} \leftarrow  1/(\tilde{a}_j - a_j^{-})$
		\State $p_j^{+} \leftarrow \nu_j^{+}/(\nu_j^{+} + \nu_j^{-})$; $p_j^{-} \leftarrow 1-p_j^{+}$
		\State $\Pr\left\{a^{(k)}_j \leftarrow a_j^{+}\right\}=p_j^{+}$  and
		\State $\Pr\left\{a^{(k)}_j \leftarrow a_j^{-}\right\}=p_j^{-}$
		\EndIf
	\EndFor
	 \State $\ve{a}^{(i)} \leftarrow \left[a_1^{(i)}, \ldots, a_{2BS}^{(i)}\right]$
	\State $\set{A}_k(\tilde{\ve{a}})\leftarrow \set{A}_k(\tilde{\ve{a}})\cup \{\ve{a}^{(i)}\}$
\EndFor
\end{algorithmic}
\end{algorithm}
\subsection{Transfer Learning in Deep Reinforcement Learning}\label{subsec:TFlearning}
The proposed transfer learning approach comprises of two steps: 1) regularized offline actor-critic training, and 2) online fine-tuning with mixed replay buffer sampling.

\subsubsection{Regularized Offline Deep Reinforcement Learning}\label{subsec:ReOffDRL}
Off-policy batch reinforcement learning \cite{gao2020batch} performs the task of learning from a collected dataset $\set{D}:=\left\{\left(\ve{s}^{(i)}, \ve{a}^{(i)}, \ve{s}'^{(i)}, r^{(i)}\right): i = 1, \ldots, D\right\}$ without further interactions with the environment.
In the Telecom industries, however, the available data usually contains network configurations in a safe operating space. Such biased dataset causes the algorithms to fail because of a fundamental problem of {\it extrapolation error}, a phenomenon in which the model erroneously over-estimates the Q values of the unseen state-action pairs, and outputs the actions with over-estimated unrealistic rewards (see Fig. \ref{fig:extrapolation}).
To overcome the challenge, we propose to add a regularization term to the reward function, such that the probability of the state-action pair is also maximized. Note that this motivation is opposite to some of the online algorithms that encourage the exploration by maximizing the expected entropy of the policy. The rationale is that, for the offline optimization without interaction with the environment, exploring outside of training data distribution leads to unrealistic over-estimated values.

We replace the reward $r$ in \eqref{eqn:problem} and \eqref{eqn:Bellman} by
\begin{equation}
r'(\ve{s}_t, \ve{a}_t) = r(\ve{s}_t, \ve{a}_t) + \alpha\log p(\ve{s}_t, \ve{a}_t),
\label{eqn:mofdifiedRew}
\end{equation}
where $p(\ve{s}_t, \ve{a}_t)$ is the probability density function that $(\ve{s}_t, \ve{a}_t)$ appears in the training dataset, and $\alpha$ is a weight factor.
The remaining problem is how to compute the log-density estimation  $\log p(\ve{s}_t, \ve{a}_t)$. Density estimation for complex high-dimensional data is a challenging fundamental problem in statistical learning. Assuming that we observe a random vector $\ve{x}\in{\field{R}}^d$ and want to use a parametrized density model to approximate the log-density function $\log p_{\ve{x}}(\cdot)$. An efficient method is the energy-based models, i.e., to define an energy function $\set{E}_{\ve{x}}(\cdot)$ which is essentially an unnormalized log-density function for the given data, and define the score function $\phi(\ve{x}) = \nabla_{\ve{x}}\log p_{\ve{x}}(\ve{x}) = -\nabla_{\ve{x}}\set{E}_{\ve{x}}(\ve{x})$. The energy function can be then estimated using {\it score matching} \cite{hyvarinen2005estimation}. In this work, we apply a promising denoising score matching-based solution, \ac{DEEN}, which uses a neural network to model the energy $\set{E}_{\psi}\left(\ve{x}\right)$ characterized by parameters $\psi$.
Due to the limited space, we do not give full details but refer the interested readers to \cite{saremi2018deep}.  The regularized reward in \eqref{eqn:mofdifiedRew} is then given by
\begin{equation}
r'\left(\ve{s}_t, \ve{a}_t|\psi\right) = r(\ve{s}_t, \ve{a}_t) - \alpha \set{E}_{\psi}\left(\ve{s}_t, \ve{a}_t\right)
\label{eqn:rwd_Energy}
\end{equation}

It is also worth noting that the size of the offline dataset $\set{D}$ is limited, and  the collected operating actions are discrete. To fully exploit them to train the offline \ac{DRL} whose outputs of actor are continuous, we can use the inverse policy of the neighbor selection strategy in Algorithm \ref{alg:neighborselection} for the data augmentation of the training set. Similar to generating the discrete neighbors of a proto-action, we generate the $k$ neighboring continuous actions $\tilde{\set{A}}_k(\ve{a}):=\{\tilde{\ve{a}}^{(i)}: i = 1, \ldots, k\}$ near the discrete action in the dataset $\ve{a}\in\set{A}$.
In this way, from each sample $\left(\ve{s}, \ve{a}, \ve{s}', r\right)\in\set{D}$, we augment $k$ samples with actions in the continuous space $\{\left(\ve{s}, \tilde{\ve{a}}, \ve{s}', r\right): \forall \tilde{\ve{a}}\in \tilde{\set{A}}_k(\ve{a})\}$, and the training dataset is extend to $\overline{\set{D}}$ with size $kD$.

\subsubsection{The Online Fine-tuning of the Pretrained Models}\label{subsec:Finetuning}
We expect that the offline pretrained actor-critic agent provides us a good initialization of the online operation. However, to adapt to the real-time system that may have a distribution drift from the training data, we allow the model to fine-tune itself with the real-time interaction with the environment. To this end, we propose to build two replay buffers: $B^{\operator{(off)}}$ stores offline training data, while $B^{\operator{(on)}}$ stores the updated new samples. For each training update of the online agent, the mini-batch can be sampled from either $B^{\operator{(off)}}$ or $B^{\operator{(on)}}$. During the training, we gradually decrease the probability of sampling from $B^{\operator{(off)}}$, while increase the probability of sampling from $B^{\operator{(on)}}$. Meanwhile, along with the actor and critic networks, we also periodically update the energy estimation network $\set{E}_{\psi}\left(\ve{x}\right)$ based on the updated offline and online datasets.
The TD3-based SAMRO algorithm is then provided in Algorithm \ref{algo:onlineRL}.  Note that \ac{TD3} builds two critic networks and takes the minimum value between them to limit the over-estimation of the Q value.
\begin{algorithm}
\caption{TD3-based SAMRO with knowledge transfer}
\label{algo:onlineRL}
\begin{algorithmic}[1]
\Initialize{Critic networks $Q_{\theta_i}$, $i=1,2$ \\
Actor network $\pi_{\omega}$\\
Target critic networks $Q_{\theta'_i}$, $i=1,2$\\
Target actor network $\pi_{\omega'}$\\
Energy estimator network $\set{E}_{\psi}$\\
Replay buffers $B^{\operator{(off)}}\neq\emptyset$ and $B^{\operator{(on)}}\leftarrow \emptyset$\\
Buffer prioritization factor $\beta$
}
\For{$t=1,\ldots, T$}
\State Output action with exploration noise $\tilde{\ve{a}}\leftarrow \pi_{\omega}(\ve{s}) + \epsilon$
\State Select operating action $\hat{\ve{a}}$ using Algorithm \ref{alg:Wolpertinger}
\State Observe $r$ and new state $\ve{s}'$
\State Store transition tuple $(\ve{s}, \tilde{\ve{a}}, \ve{s}', r)$ in $B^{\operator{(on)}}$
\State Sample mini-batch of $M$ transitions from $B^{\operator{(on)}}$ with
\State probability $\beta$ while from $B^{\operator{(off)}}$ with probability $1-\beta$
\State $\tilde{\ve{a}}'\leftarrow \pi_{\omega'}(\ve{s}')$
\State Update $r'$ with \eqref{eqn:rwd_Energy}
\State $y\leftarrow r' + \gamma \min_{i=1,2}Q_{\theta_i'}\left(\ve{s}', \tilde{\ve{a}}'\right)$
\State Update critics' weights $\theta_i$, $i=1,2$ by minimizing
\State the sampled loss of \eqref{eqn:update-critic}
\If{$t\mod H^{\operator{(A)}}$}
\State Update actor weights $\omega$ using the sampled \State gradient of \eqref{eqn:actorgradient} and update the target networks'
\State weights $\theta'_i$, $i=1,2$ and $\omega'$
\EndIf
\If{$t\mod H^{\operator{(E)}}$}
\State Update energy estimator network weights $\psi$
\State Update $\beta$
\EndIf
\EndFor
\end{algorithmic}
\end{algorithm}
\section{Numerical Results}\label{sec:Numerical}
We test the SAMRO algorithm proposed in Section \ref{sec:Solution} with a realistic system-level \ac{SON} simulator, as an extension with network slicing of the emulator in \cite{network2009introducing}.

\subsubsection{Simulator Scenario}\label{subsec:simscenario}
 \begin{figure}[t]
    \centering
    \includegraphics[width=0.48\textwidth]{GeoLayout.pdf}
    \caption{The $9$-cell simulation environment. Red bar indicates the cell throughput at the snapshot. White areas are the buildings. Yellow circles define the mobility areas of the hotspots.}
    \label{fig:geo}
\end{figure}
We consider an environment with $3$ three-sector sites, i.e., $9$ cells, operating on the frequency band $2.4$ GHz, as shown in Fig. \ref{fig:geo}. The radio propagation model is Winner+ \cite{meinila2010d5} supported with ray tracing.  In Table \ref{tab:UE_group}, we define four groups of users associated to two slices: \lq\lq Slice 1\rq\rq \ supporting video traffic and  \lq\lq Slice 2\rq\rq \ supporting HTTP traffic. The users in groups $1$ and $2$ move uniformly randomly within the playground and those in hotspots move within the circled areas as shown in Fig. \ref{fig:geo}. Due to the limitation of the scheduler in simulator, we assume all slices have the same latency requirement of $1$ ms but different throughput requirements. we also generate a realistic traffic pattern to each of the user group as shown in Fig. \ref{fig:traffic}.
The agent interacts with the simulator with a time granularity of $900$ simulation slots, which reflects $15$-min of the real-life time. Namely, a one-day operation (in simulator) allows $96$ interactions with the simulated environment.
\begin{table}[tbp]
\caption{User Groups}
\label{tab:UE_group}
\begin{tabular}{|l|l|l|l|l|}
\hline
          & Group size & Traffic Type & Expected Rate & Speed  \\ \hline
Group $1$   & $25$         & Slice $1$ (Video)          & $5$ Mbit/s      & 6 km/h \\ \hline
Group $2$   & $25$         & Slice $2$ (HTTP)         & $3$ Mbit/s      & 3 km/h \\ \hline
Hotspot $1$ & $8$          & Slice $1$ (Video)        & $5$ Mbit/s      & 3 km/h \\ \hline
Hotspot $2$ & $8$          & Slice $1$ (Video)        & $5$ Mbit/s      & 3 km/h \\ \hline
\end{tabular}
\end{table}
\begin{figure}[t]
    \centering

    \includegraphics[width=0.48\textwidth]{traffic_hotspot.pdf}
    \caption{Number of connected users in Cell \lq\lq 3[2]\rq\rq \ with hotspot when \ac{HO} parameters are set to default.}
    \label{fig:traffic}
\end{figure}
\begin{figure}[t]
    \centering
    \includegraphics[width=0.48\textwidth]{extrapolation.pdf}
    \caption{Phenomenon of extrapolation error.}
    \label{fig:extrapolation}
\end{figure}

\subsubsection{Offline Regularized Deep Reinforcement Learning}\label{subsec:off-policy}
Following the state and action defined in Section \ref{subsec:MDP}, we define $N=9$, $S=2$, and $B=34$. Thus, the state $\ve{s}\in {\field{R}}^{208}$ and the proto-action $\tilde{\ve{a}}\in{\field{R}}^{136}$. We define $\set{O}:=\{-5,-4, \ldots, 5\}$ in dB and $\set{T}:=\{40, 64, 80, 100, 128, 160, 256, 320, 480, 512,$ $640, 1024, 1280, 2560, 5120\}$ in ms. Thus, we also define the upper and lower bounds for each dimension of the proto-action. We assume that in practice the default value of \ac{HOM} is $0$ dB and \ac{TTT} is $512$ ms.
We have collected $20,000$ offline biased samples as follows. Firstly, we generated independent normal distributed r.v.s for each dimension of $\tilde{\ve{a}}$ with the mean as the default value, and the variance of $3$ dB for \ac{HOM} and $300$ ms for \ac{TTT}, respectively. Then, we find the nearest value in $\set{O}$ or $\set{T}$ and send the discrete action back to the simulator. The reward weights $w_s^{\operator{(T)}}, w_s^{\operator{(D)}}, w_s^{\operator{(HF)}}, w_s^{\operator{(PP)}}$ are set as $1,1,1,0.3$, respectively. It is worth noting that sometimes scaling the reward values help improve the convergence. We scale the reward with a factor of $5$. With the defined weights, after the scaling the range of the reward is $[-6.5, 10]$. We choose small discount $\gamma=0.1$ because optimizing the network configuration for every $15$-min mainly influences the instantaneous reward.

As for the actor-critic networks, we use \ac{MLP} for the actor and critic networks, both with $3$ hidden layers, and numbers of neurons for each layer are $(128, 64, 32)$ and $(64,16,4)$, respectively. Learning rates of the actor and critic are $0.001$ and $0.002$, respectively. The batch size is $64$, and the optimizer is Adam. For the \ac{DEEN}, we also use $3$ layers with the sizes $(256, 64, 32)$. The batch size is $32$ and the noise scale is $0.1$. The update periods of actor and \ac{DEEN} are $H^{\operator{(A)}}=3$ and $H^{\operator{(E)}} = 100$ respectively.

Fig. \ref{fig:extrapolation} illustrates a typical extrapolation error when we train an offline \ac{DRL} without regularization in Fig. \ref{fig:extrapolation}. To verify this, we set the discount factor $\gamma=0$ such that the critic simply predicts the instantaneous reward and the policy should output the action with the best reward. However, after $30$ mini-batches, the agent starts over-estimating the reward for unseen actions (even exceed the upper-bound).

Fig. \ref{fig:off_training} shows the training loss functions of the actor and critics with the regularization. The regularized term help the negative actor loss (reflecting the Q estimate of the chosen next action) to be well bounded within the scaled reward region and prevent from outputting an action outside of the training data distribution.
\begin{figure}[t]
    \centering
    \includegraphics[width=0.48\textwidth]{Loss_1.pdf}
    \caption{Training loss of actor and critic networks.}
    \label{fig:off_training}
\end{figure}

\subsubsection{TD3-Based \ac{SAMRO} with Knowledge Transfer}\label{subsec:onlinetraining}
To show the advantage introduced by slice-specific \ac{HO} parameters and \acp{KPI}, we compare our \ac{SAMRO} approach with two baselines: 1) \ac{MRO} optimized with the same \ac{DRL} with transfer learning approach, but without slice-awareness, and 2) default setting of \ac{HOM} to be $0$ dB and \ac{TTT} to be $512$ ms.

Fig. \ref{fig:base_compare} shows that our approach helps both \ac{SAMRO} and \ac{MRO} converge faster and outperform the default setting with both higher start and higher asymptote, when assuming only two weeks of online training ($1344$ training interactions) and two days of testing ($192$ testing interactions).

Fig. \ref{fig:HO_service_compare} illustrates the empirical \acp{CDF} of {\it\ac{HO} metrics \ac{HO} failure ratio} and {\it ping-long \ac{HO} ratio} and the slice service metrics {\it throughput service level} and {\it latency service level}, respectively. We observe that \ac{SAMRO} further outperforms \ac{MRO} because it not only improves the \ac{HO} performance, but also reduces the violation of the throughput and latency constraints of each slice. For example, \ac{SAMRO} guarantees that more than $55\%$ and $70\%$ of the services satisfy the throughput requirements of Slice $1$ and $2$, respectively, while \ac{MRO} satisfies only less than $20\%$ and $40\%$, respectively. Moreover, \ac{SAMRO} achieves a good tradeoff between the too-late and too-early \ac{HO} metrics. It significantly reduces the too-early \ac{HO} events (e.g., ping-pong \acp{HO}), without increasing the \ac{HO} failures.

 %
 \begin{figure}[t]
    \centering
    \includegraphics[width=0.48\textwidth]{CompareReward.pdf}
    \caption{Online training against baselines.}
    \label{fig:base_compare}
\end{figure}

 \begin{figure*}[t]
    \centering
    \includegraphics[width=0.98\textwidth]{compareCDF.pdf}
    \caption{Comparison of the service and \ac{HO} quality.}
    \label{fig:HO_service_compare}
\end{figure*}

\section{Conclusions}\label{sec:Conclusion}
In this paper, we proposed a novel \ac{SAMRO} solution with the newly introduced slice-specific \ac{HO} parameters and \acp{KPI}, optimizing both service quality and handover performance for diverse slices. We also proposed a two-step transfer learning-empowered actor-critic scheme that can deal with large discrete spaces, and enable safe and effective online training with regularized objective function and mixed replay buffer sampling. The system-level simulation shows significant performance benefits versus the baseline algorithms without slice-awareness.
\input{acronyms}

\section*{Acknowledgement}
The authors would like to thank Senthil Kumaran K and Niraj Nanavaty for providing the practical insights.
\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


I'm pretty sure this is the "forgotten Lagoa", L. pyxidifera (Yellow Flannel Moth). I've only seen one photo of this species online as opposed to lots of the other two flannels (M. opercularis and L. crispata). I think it needs a guide page.
Moved your image to a new guide. We've been putting these under Megalopyge. I entered the synonym on the guide page, so we should be covered.


Dundela suffered a disappointing home defeat at the hands of Moyola Park in dramatic circumstances.
Despite once again being the better team they failed to capitalize on several occasions and with a little over ten minutes to go in the first half they got hit with a sucker punch when the visitors took the lead through Rafferty who was left unmarked to send Moyola in to the break with the narrow advantage.
Both teams came out in the second half and had chances blocked by both keepers with the away side looking the more likely to add to their lead however the score remained 0-1 until the final few minutes of the game when Dundela bounced back with an equalizer from a Sofley shot inside the box to make the score line 1-1.
However they were not happy with the point and as they attempted to get the winner they got caught short in defence and Mullan fired the winner for the visitors.
I am interested in Irish football.
I am collector badges & postcard Stadium.


text:
What's On In Perth
Things to do in Perth
Balhousie Castle
Huntingtower Castle
Scone Palace
Business Listing Options
Perth News & Events
"What's on in Perth" is the 'go to' website, packed with information for residents and visitors, to make your time in Perth an enjoyable. Great ideas on what to do and see, the best places to eat and stay, attractions, events, news and more…
Home British Council
Perth News October 5, 2020
Tayside and Fife schools encouraged to team up with international pupils through Government scheme
Schools in Tayside and Fife are being encouraged by the Foreign Secretary to sign up for a scheme which partners pupils with their peers in the developing world. Perth & Kinross – The Courier…
Perth News January 19, 2021
Tayside charities launch mental health survey, 12 months on from damning report
An extensive survey of psychiatric services in Tayside is under way, a year after an independent inquiry revealed a litany of shortcomings. Perth & Kinross – The Courier…
New £33.8m leisure hub outside Perth with 4-star hotel, museum and park and ride could create 170 jobs
One of Perthshire's biggest developers have unveiled proposals for a £33.8m leisure park on the edge of Perth, set to include a hotel, lodges and a museum. Perth & Kinross – The Courier…
Extra 30-bed ward readied at Dundee's Ninewells Hospital as Covid-19 takes its toll
An extra 30-bed ward has been prepared specifically for Covid-19 patients at Dundee's Ninewells Hospital as the health board strives to cope with increasing demand. Perth & Kinross – The Courier…
Tayside patients 'stood down' as hospitals edge closer to Covid-19 limit
Tayside health bosses have started to postpone procedures as staff respond to a large surge in Covid-19 cases. Perth & Kinross – The Courier…
Blairgowrie-born long-distance cyclist mounts campaign for Mary's Meals
British long-distance cyclist Mark Beaumont is leading a campaign to raise funds for Mary's Meals. Perth & Kinross – The Courier…
Copyright © 2020 *** What's On in Perth.net *** All Rights Reserved *** View our Privacy Policy Disclosure and Terms of Use and Disclaimer

redpajama_set_name: RedPajamaCommonCrawl


Our mission is to increase the understanding and appreciation of how gender and sexuality influence and frame our daily lives and social interactions.
To accomplish this, our office connects with the campus community to create safe and welcoming spaces for students and employees; advocate with and on their behalf for social justice and equality; and enhance critical inquiry into diversity and inclusion through programs and discussions.
The Queer Archives/LGBTQ Oral History Project is dedicated to capturing and preserving the experiences and reflections of Lafayette's LGBTQ alumni, faculty and staff in the Skillman Library College Archives.


WIN a Huawei Nova 2i Smartphone in February!
Entry is open to everyone in NZ, and runs from Feb 1st to Feb 21st inclusive. Full details on the KIWIreviews.nz website Competitions page.
Net Neutrality Was Repealed – Here's How Burger King Illustrated The Consequences.
Stand up for Whopper Neutrality - the same burger for everyone, served in order.
WIN a smartphone in February!
There's a brand new $500 smartphone up for grabs in February!
ePluribus.nz is proud to receive support from Well Above.


what do we do here, politely refuse help on the one hand OR end up doing the whole assignment on the other?
I think we should help anyone who requests it. If the poster explicitly indicates that the post is for a school assignment, perhaps we can initially offer some suggestions instead of solving the problem outright. But ultimately, I would put the responsibility for learning on the student, and not the volunteers on this forum.
Since there are many who utilize this forum to get their complete work/job assignments done, I am not sure why we would not help a student who has asked for help. If the poster (be it a student or someone employed somewhere) fails to learn from the posts and the solutions offered, that is their loss.
But with students specifically, I think we should be suggesting methodologies to solve problems and directing them to info or posts which may explain how a function/technique works, without specifically solving the problem for them.
In fact this approach should be done with all people, but often it is simpler and quicker to simply solve it and offer a brief explanation of how the solution was arrived at after the event.
I'm gonna share my personal thoughts about it and how they changed along the time.
I firstly took as a heresy those questions from students that come to this site for help with their homework and assignments. I tended to believe that it was unfair, so I avoided answering or sometimes posting a slightly ironic comment. I thought that the help (from here) should be delivered to people who needed aid in their work or life, not to cheating students.
At the same time I realized that sometime there were topics with questions that made me talk to myself wondering if that wasn't a clear and neat abuse of the help that could be got at this place. I remember many cases of people who asked almost for a full accounting or selling application for their own business without even helping or facilitating the minimum data. The most extreme that I remember was that of an airline owner who asked for a solution for an optimization of flight scheduling (1: would you -client- rely on such an airline with such a schedule? I don't.; 2) would he -owner- rely on a provided service of such an origin? I don't mean quality. He doesn't.).
and decided to analyze the problem from another viewpoint. Why should be I who decide to help or not a student if with Google he has full access to virtually any question or subject? Will my little grain of sand make the difference? No, absolutely not.
a) Those thankful, very thankful, up to saturation sometimes, mostly but not exclusively students, that cheating or not they try to accomplish a job: their homework or assignment.
b) Those ungrateful and sometimes unpolite, students or not and usually not students, that are clearly trying to take advantage of the collective knowledge not to solve a job or personal issue but to spare a few bucks.
And I changed my mind, I realized I was wrong at a first moment, so whenever I had the chance to choose between the firsts and the lasts, my option was, is and will be clear: the firsts, students or not.
At last these are forums, open to the community, neither guards nor policemen, just a few rules, being the common sense the major one. Forums at last. Nothing more.
I agree with above comments. In the past, I have ignored school assignments [Ditto SirJB7] but now that I have a young kid, I realize the importance of giving helping hand and showing him that the world is good place to be so that his neural networks develop with positivity. Maybe in future he'll be gracious enough to help others. The same probably applies to kids who come here in the want of help. In future, they might help others if they carry good memories of help given.
I just would like to point out that if someone says this is school assignment it actually shows the poster has chosen to be transparent. What if the poster chose to drop these two words from his post? Will we still spend some time guessing if it is real world assignment or not?
The other example that comes my mind is excel challenges section. I might be taken to task for saying that they sometimes serve a purpose more academic than practical but yet it is something that makes you study Excel hard and be better in your craft.
IMO, when a student comes here and asks for help (not just a direct solution, but guidance), I almost give them more applause than someone who doesn't. At least they have found a place to ask a question, and know how to do a search on the internet to find an answer. I smile to myself when I get coworkers coming up to me asking questions that I then answer by pulling up Google, typing in their question verbatim, and clicking on the first link for the answer.
I am going to come at this from a students perspective. This issue hits close to home as I did further study a couple of years ago. I am speaking specifically about questions typed into the forum word for word from an assignment or exam (I have seen this).
If when studying I was competing against other students that is fine but if I am competing directly against some of the gun slingers in here I am not OK with that. Someone else doing your assignments is not cool with me and defeats the purpose of trident trial and error. When you ask questions in a forum the answers are generally more tailored than what you may find on the net. Clearly a well crafted question will get through but if i suspect I am doing someone's assignment or take home exam that is when I pull up and say 'No way Cowboy'. I don't think there is any good which can come of it and I feel for the other students when I see experience players responding to word for word exam questions. I will happily point them at resources but not provide tailored assistance. Asking for general advice is different but it comes down to a judgement call and it is not black and white for mine.
Yes i have do help with Homework and Assignments.
First of all, you seem to be a spammer, and not only we don't like them but almost hate them as well.
Please act accordingly if wanna share this space with our community, otherwise... you yet know, don't you?
Here is a canned response that is used on another forum where I am active.
Nothing is more frustrating than to be working on homework and to get stuck. Without guidance, you might have no hope of finishing the assignment before the deadline. The Internet can be a big help in that respect, but without learning how to get the right answer, asking others to solve your homework questions for you simply becomes a downward spiral (and it's cheating).
We do not want to contribute to you cheating yourself out of your education, but we also acknowledge that seeking assistance to learn a concept is a legitimate request.
If you are genuinely interested in receiving help in the form of tutoring or coaching, then please rephrase the title of your original post to clearly indicate you are seeking coaching or tutoring help. Any forum members (who are willing to assist as a tutor) will modify their responses accordingly to facilitate your learning. Tutors don't tell you the answers, they help you figure it out for yourself; so don't expect answers, expect suggestions, or just plain hints. Also, be specific in describing the function/formula or technique you trying to learn, and tell us what you have attempted so far. Otherwise, expect your plea for homework answers to be ignored.


The Rasch Foundation has funded a wide variety of different projects. The following is only a partial list of charities that we have funded in the past. Please note that this list includes the recipients of both grant proposals and discretionary funding.
Major funding for Murals of Hope, a charity that aims to brighten the lives of sick and abused children by painting murals in hospitals and other health facilities and social agencies serving children.
Funding to develop a computerized patient care system that is remotely accessible by doctors using PDAs to improve the continuity of care provided to terminally ill patients of a Toronto area Palliative Care Centre.
Startup funding for Pups for Peace, a nonprofit group that buys, trains and deploys explosive-sniffing dogs throughout Israel to help prevent terrorist attacks.
Annual funding to hold a lectureship at Mount Sinai Hospital by a visiting expert to advance the education of resident doctors on the subject of Oncological Surgery.
Seed funding for Vartana, Canada's first nonprofit financial institution dedicated to meeting the needs of the voluntary sector.


text:
Sexual Conditions Home
WebMDRx Savings Card
Sexual Conditions Guide
Types & Causes
Related to Sexual Conditions
HPV/Genital Warts
Low T & Sex Drive
Sex Headaches
WebMD Connect to Care: Tests for STDs
Best Ways to Prevent Getting HIV
Delaying Sex After First Period Reduces Chance of STDs
By Jeff Levine
March 27, 2000 (Arlington, Va.) -- The longer an adolescent girl waits to have sex after she has her first period, the less likely she is to get a variety of sexually transmitted diseases (STDs). That conclusion comes from a new government-funded study presented here this week at the Annual Meeting of the Society for Adolescent Medicine.
Researchers from the Indiana University School of Medicine looked at more than 300 teenage women recruited from community health clinics. The adolescents were questioned about when they had their first period and how long they waited to have their initial sexual experience.
"The longer that interval is, the less risk of reporting an [sexually transmitted] infection," researcher Dennis Fortenberry, MD, a specialist in adolescent medicine at Indiana University, tells WebMD. The Indiana research was funded by the National Institute of Allergy and Infectious Diseases.
While a small percentage had sex before their first period, most experienced their first period around age 12 and then postponed sex until they were about 15 years old. The average interval between the initial period and sex was about 2 1/2 years. But about one-third of those in the study had sex around age 13. Fortenberry says that right after the first period, sensitivity to disease is quite high but seems to diminish over time.
The data show that for many STDs, such as chlamydia and herpes, there was a 200% reduction in risk for each year of delay between the first period and the first sexual experience. Gonorrhea infections also followed a similar pattern.
"I do have a concern that this would be taken into a fear-based message about sexual behavior, and I wouldn't like that," says Fortenberry. Instead, he says physicians and parents need to convince teenagers that it is important to wait as long as possible before becoming sexually active.
Counseling teens about sex by both physicians and parents is definitely needed. Another study released at the meeting looked at how much young people know about sex and STDs. Researchers at the University of Alabama School of Medicine in Birmingham (UAB) gave about 60 teenagers a questionnaire to gauge their sexual IQs, and they were appalled.
"The level of knowledge generally that these kids have with regard to STDs is pretty poor," lead researcher Jeanne Merchant, MPH, tells WebMD. For instance, most thought HIV was the most common STD in the U.S., ignoring chlamydia, the correct answer. Another misunderstanding -- getting an STD is likely to produce an obvious symptom.
The researchers emphasize that teens need to be taught to ask their questions early and not wait until they are already infected with a STD. "Without scaring them, they need to get the facts [about STDs]," says M. Kim Oh, MD, an adolescent medicine physician at UAB, who is on the research team.
What grade would the scientists give the teens for their knowledge of STDs? No higher than a "D." And that's their final answer.
Vital Information:
According to new research, the more time between a girl's first period and her first sexual experience, the less likely she is to contract an STD.
In a separate study, researchers found that adolescents don't know much about STDs, including which is the most common and whether or not they will have symptoms.
Physicians and parents should educate children on STD risks.
WebMD Health News
How Sexual Dysfunction Starts
13 Sex-Drive Killers
Condoms: What You Need to Know
Super Foods: Some Can Protect Your Sex Life
When to See a Sex Therapist
Get the Facts About Herpes
Safe Sex After 50: Preventing STDs, Using Condoms, and More
Women and Chlamydia
Symptoms of Chlamydia
Chlamydia -- The Basics
Exposing Chlamydia
The Scope of STDs
Treating Chlamydia
Chlamydia Topics
STDs 101
See pictures and get the facts.
Low Libido?
13 things that can kill your sex drive.
How to stop them from spreading.
Top Myths About HIV
Symptoms of STDs
Can You Get Herpes From a Toilet Seat?
What Guys Wish You Knew
Men's Biggest Bedroom Worries
The Truth About Orgasms
Common Sexual Problems in Women
Are You Having Safe Sex?
Does Arousal Change as We Age?
Your Birth Control Options
What Women Wish You Knew
Testosterone Too Low? How to Tell
Facts About the HPV Vaccine


Driving home from a tournament is when I do a lot of thinking about the event in my rearview mirror. It's a time for reflection and assessment — but never for beating a dead horse. In my business, it can be a challenge to stay positive, so one of my rules is that I can kick myself all the way home … but no further.
In evaluating my performance so far in 2017, I can tell you that Day 1 has been my Achilles heel. Instead of getting off to a good start and putting myself in position to win, I'm digging a hole that I have to get out of on Day 2. It makes for a challenging season.
In four of the five Elite tournaments so far this year, I've ended Day 1 in 66th place or worse. Fortunately, I've moved up on Day 2 each time and into the first cut at every event except one (I was 56th at Cherokee Lake). Then, I've had some solid Day 3s, where I've moved up a little bit more.
My season has been OK, but if I had some better first days it would be a lot better. If you're fishing the Elite Series, you just can't afford to give your competition any breaks. They're too good.
At Sam Rayburn Reservoir, I think I had way too much history in my head when the tournament started. Like everyone else, I was catching a lot of fish but it was tough to figure out what the very best pattern was — the one that gives you a chance to win.
I like fishing a tournament by the seat of my pants, figuring things out as I go along and making the adjustments necessary to do well. I try to get myself plugged into what's happening at the fishery and to go with the flow as the fishing changes with the conditions. That's served me well in the past, but this year I wonder if I'm not getting "dialed-in" enough during practice.
I talked about that a little in my last column. I said that I usually don't try to catch more than one or two bass in an area during practice and, as a result, I may not know a lot about what I really have with that area or pattern until the tournament starts.
My Day 1 issues this year have me wondering if maybe I need to stick more fish and get a lot better idea of the strength of an area and the size of the fish before competition starts. Maybe I'm trying to find too many different areas and patterns that could work rather than narrowing them down to just a few that should pay off based on a lot more information.
One thing I know for sure is that the competition is getting tougher and more talented every season. As each year goes by, the little adjustments that may seem minor become more and more important.
And I like that! I love that our sport is always changing, that you can't go out there and do the same thing over and over and have the same success.
I may or may not have figured out something about my Day 1 issues this year, but I guarantee I'll stay after it until the situation is fixed.