text:
\section{Introduction}

Modern datacenters consist of multiple servers jointly communicating at ultra high speeds. As the joint transmission rate of the servers surpasses the internal connection limitations (e.g., router processing rate) the communication may become congested. Congested networks suffer from reduced bandwidth utilization, the appearance of packet loss and increased application latency. Hence, avoiding and preventing congestion is an important task called congestion control.

Previous work on congestion control has mainly focused on rule-based methods. As these schemes are rule based, they are usually optimized for a single set of tasks (as we show in \cref{table: comparison}). Machine learning methods, as opposed to rule-based ones, are capable of learning and generalizing based on data and experience. Specifically, reinforcement learning (RL) automatically learns a control policy (transmission rate control in the context of congestion prevention) given an environment to interact with and a reward signal. Thus, an RL algorithm is (1) capable of solving the task without tedious manual tuning of control parameters and (2) successfully operate in a vast set of tasks, e.g., generalize, when provided with diverse training environments.

Most RL algorithms were designed under the assumption that the world can be adequately modeled as a Markov Decision Process (MDP). Unfortunately, this is seldom the case in realistic applications and specifically datacenter congestion control. As we show below, as opposed to the standard assumptions, this problem is partially-observable and consists of multiple-agents. These challenges prevent the standard RL methods from coping within such a complex environment, and we believe are one of the major reasons such methods are yet to be deployed in real world datacenters.

From an RL point of view, each agent controls the transmission of a single application. As there are multiple applications across multiple servers, this means multiple agents that, due to security reasons, are unaware of one another and unable to communicate -- hence multi-agent and partially observable. We observe that popular RL algorithms such as DQN \citep{mnih2015human}, REINFORCE \citep{williams1992simple} and PPO \citep{schulman2017proximal} fail on such tasks (\cref{tab: many-to-one comparison}).

To overcome these challenges, we present the {Analytic Deterministic Policy Gradient}{} ({ADPG}{}), a scheme that makes use of domain knowledge to estimate the gradient for a deterministic policy update. As this task lacks a ground truth reward function, we present a fitting reward function and show that this reward function, in a multi-agent partially-observable setting, leads to convergence to a global optimum.

\begin{table*}
\begin{center}
\begin{tabular}{c|cccc}
     & \textbf{Many to One} & \textbf{All to All} & \textbf{Long Short} & \textbf{Real World} \\
    \midrule
    Aurora / PPO / REINFORCE & \color{RubineRed}{\xmark} & \color{RubineRed}{\xmark} & \color{RubineRed}{\xmark} & \color{RubineRed}{\xmark} \\
    DCQCN & \color{ForestGreen}{\cmark} & \color{ForestGreen}{\cmark} & \color{RubineRed}{\xmark} & \color{ForestGreen}{\cmark} \\
    HPCC / SWIFT & \color{RubineRed}{\xmark} & \color{ForestGreen}{\cmark} & \color{ForestGreen}{\cmark} & \color{ForestGreen}{\cmark} \\
    \textbf{{ADPG}{}} (this paper) & \color{ForestGreen}{\cmark} & \color{ForestGreen}{\cmark} & \color{ForestGreen}{\cmark} & \color{ForestGreen}{\cmark} \\

    \bottomrule
\end{tabular}
\caption{Comparison of various approaches. A {\textit{v}} means the method has successfully controlled and prevented congestion in this task, whereas {\textit{x}} presents a failure. As can be seen, {ADPG}{} is the only learning-based method successful in tackling the task of congestion control, and the only method across all compared methods that succeeds in all tasks.}\label{table: comparison}
\end{center}
\end{table*}

To validate our claims, we develop an RL environment, based on a realistic networking simulator, and perform extensive experiments. The simulator, based on OMNeT++ \citep{varga2002omnet++}, emulates the behavior of state of the art hardware deployed in current datacenters: ConnectX-6Dx Network Interface Card (NIC). In addition, we further test the generalization and robustness of the agent by porting it to \textit{real hardware} and evaluating it there. Our experiments show that our method, {Analytic Deterministic Policy Gradient}{} ({ADPG}), learns a robust policy, in the sense that it is competitive in all the evaluated scenarios, both in simulation and when tested in the real world. Often, it outperforms the current state-of-the-art methods deployed in real datacenters.

\section{Related Work} \label{sec: related work}

\textbf{Multi-agent RL:} Recent work in games has shown the premise of multi-agent RL, where trained agents have competed with the top human players. These methods have been applied to tasks ranging from board games such as Go \citep{silver2017mastering} and up to high dimensional 3-dimensional multi-player games such as DOTA \citep{berner2019dota} and StarCraft \citep{vinyals2019grandmaster}. Despite these impressive achievements, we find that due to the combination of challenges in our task (partial observability, multi-objectiveness and multi-agent) the methods in these papers can't be applied in tasks such as datacenter congestion control.

\textbf{Hand-tuned congestion control:} Here, we focus on \textit{datacenter} congestion control. Previous work tackled this problem from various angles. \citet{alizadeh2010data} used a TCP-like protocol, increasing the rate until congestion is sensed, and then dramatically decreasing it. \citet{mittal2015timely,kumar2020swift} focused on round-trip latency measurements to react quickly to changes. \citet{zhu2015congestion} used a statistical signal provided by the switch (ECN), and \citet{li2019hpcc} added telemetry information, requiring specialized hardware, yet proves to benefit greatly in terms of reaction times.

\textbf{Optimization-based congestion control:} Although most previous work has focused on hand-tuned algorithmic behavior, two notable mentions have taken an optimization-based approach. \citet{dong2018pcc} presented the PCC-Vivace algorithm, which combines information from fixed time intervals such as bandwidth, latency inflation, and more. As it tackles the problem via online convex optimization, it is stateless; as such, it does not optimize for long-term behavior but rather focuses on the immediate reward (bandit setting).

PCC-Vivace was then extended in the Aurora system \citep{jay2019deep}. Aurora provides the monitor interval, defined in PCC-Vivace, as a state for a PPO \citep{schulman2017proximal} algorithm. While in this work we focus on a realistic multi-agent simulator, in \citet{jay2019deep} the focus was on a naive single-agent noisy connection. Thus, we observe that applying the PPO algorithm to our setting, as was done in \citet{jay2019deep}, results in stability and convergence issues (see \cref{table: comparison}). Hence, comparing to \citet{jay2019deep}, our novelty is twofold. (1) We consider a realistic setting that is used within real datacenters to evaluate congestion control algorithms prior to deployment, and (2) rising from the issues highlighted in this task, we present a novel on-policy deterministic policy gradient method that is capable of quickly converging to a satisfying solution.

\section{Networking Preliminaries}
\label{sec:networking}
In datacenters, traffic contains multiple concurrent data streams transmitting at high rates. The servers, also known as \emph{hosts}, are interconnected through a topology of \emph{switches}. A directional connection between two hosts that continuously transmits data is called a \emph{flow}. We assume, for simplicity, that the path of each flow is fixed.

Each host can hold multiple flows whose transmission rates are determined by a \emph{scheduler}. The scheduler iterates in a cyclic manner between the flows, also known as round-robin scheduling. Once scheduled, the flow transmits a burst of data. The burst's size generally depends on the requested transmission rate, the time it was last scheduled, and the maximal burst size limitation.

A flow's transmission is characterized by two primary values. \emph{Bandwidth}: the average amount of data transmitted, measured in Gbit per second; and \emph{latency}: the time it takes for a packet to reach its destination. \emph{Round-trip-time (RTT)} measures the latency of source$\rightarrow$destination$\rightarrow$source. While
the latency is often the metric of interest, most systems are only capable of measuring RTT.

\section{Congestion Control}\label{sec: cc}

\emph{Congestion} occurs when multiple flows cross paths, transmitting data through a single congestion point (switch or receiving server) at a rate faster than the congestion point can process. In this work, we assume that all connections have equal transmission rates, as typically occurs in most datacenters. Thus, a single flow can saturate an entire path by transmitting at the maximal rate.

Each congestion point in the network has an inbound buffer enabling it to cope with short periods where the inbound rate is higher than it can process. As this buffer begins to fill, the time (latency) it takes for each packet to reach its destination increases. When the buffer is full, any additional arriving packets are dropped.

\subsection{Objective}\label{cc: objective}

CC can be seen as a multi-agent problem. Assuming there are N flows, this results in N CC algorithms (agents) operating simultaneously. Assuming all agents have an infinite amount of traffic to transmit, their goal is to optimize the following metrics (where $\uparrow$/$\downarrow$ mean higher/lower is better, respectively):

\begin{enumerate}
    \item Switch bandwidth utilization ($\uparrow$) -- the \% from maximal transmission rate.

    \item Packet latency ($\downarrow$) -- the amount of time it takes for a packet to travel from the source to its destination.

    \item Packet-loss ($\downarrow$) -- the amount of data (\% of maximum transmission rate) dropped due to congestion.

    \item Fairness ($\uparrow$) -- a measure of similarity in the transmission rate between flows sharing a congested path. We consider $\frac{\min_\text{flows} BW}{\max_\text{flows} BW} \in [0, 1]$.
\end{enumerate}

These objectives may be contradictory. Minimizing latency often comes at the expense of maximizing throughput. Hence, multi-objective schemes present a Pareto-front \citep{liu2014multiobjective} for which optimality w.r.t. one objective may result in sub-optimality of another. However, while the metrics of interest are clear, the agent does not necessarily have access to signals representing them. For instance, fairness is a metric that involves all flows, yet the agent is unaware of how many other transmissions are active and what data they transmit. The agent only observes signals relevant to the flow it controls. As such, it is impossible for a flow to obtain an estimate of the current fairness in the system. Instead, we reach fairness by setting each flow's individual target adaptively, based on known relations between its current RTT and rate. More details on this are given in Sec.~\ref{sec: method}.

\section{Reinforcement Learning Preliminaries}\label{sec: rl}

We model the task of congestion control as a multi-agent partially-observable Markov decision process (POMDP) with multiple objectives and continuous actions, where all agents share the same policy. Each agent observes statistics relevant to itself and does not observe the entire global state.

A POMDP is defined as the tuple $(\mathcal{O}, \mathcal{S}, \mathcal{A},P,R)$ \citep{puterman1994markov,spaan2012partially}. An agent interacting with the environment at state $\state \in \mathcal{S}$ observes an observation $o(\state) \in \mathcal{O}$. After observing $o$, the agent selects a continuous action $\action \in \mathcal{A}$. In a POMDP, the observed state does not necessarily contain sufficient statistics for determining the optimal action. After performing an action, the environment transitions to a new state $\state'$ based on the transition kernel $P(\state' | \state, \action)$ and receives a reward $r(\state, \action) \in R$.

Let $\Pi$ be the set of stationary deterministic policies on $\mathcal{A}$, i.e., if $\pi\in\Pi$ then $\pi: \mathcal{O}\rightarrow \mathcal{A}$.  In this work, we focus on the \textit{average reward} performance metric, also known as the gain of the policy $\pi$, $\rho^\pi(\state) \equiv \lim_{T \rightarrow \infty} \frac{1}{T} {\mathbb{E}}^\pi[\sum_{t=0}^T r(\state_t,\action_t)\mid \state_0=\state]$, where ${\mathbb{E}}^\pi$ denotes the expectation w.r.t. the distribution induced by $\pi$. The goal is to find a policy $\pi^*,$ yielding the optimal gain $\rho^*$, i.e., for all $\state\in \mathcal{S}$, $\pi^*(o(\state)) \in \argmax_{\pi\in \Pi} \rho^\pi (\state)$ and the optimal gain is $\rho^{*}(\state) = \rho^{\pi^*}(\state)$.

\subsection{Reinforcement Learning For Congestion Control}
\label{sec: method}

The agent, a congestion control algorithm, controls the data transmission rate at the source. Specifically, the algorithm runs within the network-interface-card (NIC). At each decision point, the agent observes statistics correlated with the specific flow it controls. The agent then acts by determining a new transmission rate for that flow and observes the outcome of this action.
We define the four elements in $(\mathcal{O}, \mathcal{A},P,R)$ (\cref{sec: rl}).

\textbf{Observations.} The agent can only observe information relevant to the flow it controls. In this work, we consider the flow's transmission rate and the RTT measurement.

\textbf{Actions.} The optimal transmission rate depends on the number of agents simultaneously interacting in the network and on the network itself (bandwidth limitations and topology). As such, the optimal transmission rate will vary greatly across scenarios. To ensure the agent is agnostic to the specifics of the network and can easily generalize, we define the next transmission rate $\text{rate}_{t+1}$ as a multiplication of the previous rate with the action. I.e., $\text{rate}_{t+1} = \action_t \cdot \text{rate}_t$, where in our experiments $\action_t \in [0.8, 1.2]$.

\textbf{Transitions.} The transition $\state_t \rightarrow \state_{t}'$ depends on the dynamics of the environment and on the frequency at which the agent is polled to provide an action. Here, the agent acts (is asked to provide an updated transmission rate) once an RTT packet is received. This is similar to the definition of a monitor interval by \citet{dong2018pcc}, but while they considered fixed time intervals, we consider event-triggered (RTT) intervals.

\textbf{Reward.} As the task is a multi-agent partially observable problem, the reward must be designed such that there exists a single fixed-point equilibrium.

Based on \citet{appenzeller2004sizing}, a good approximation of the RTT inflation ($\text{RTT-inflation} = \frac{\text{RTT}}{\text{base-RTT}}$) in a bursty system, where all flows transmit at the ideal rate, behaves like $\sqrt{N}$, where $N$ is the number of flows. In this case, the combined transmission rate of all flows saturates the congestion point, the system is on the verge of congestion, and the major latency increase is due to the packets waiting in the congestion point's buffer. This latency is orders of magnitude higher than the empty-system routing latency. As such, we can assume that all flows sharing a congested path will observe a similar RTT inflation. We define
\begin{equation}
    r_t^i = -\left( \text{\bf{target}} - \frac{\text{RTT}^i_t}{\text{base-RTT}^i} \cdot \sqrt{\text{rate}^i_t} \right)^2 \,,
\end{equation}
where {\bf target} is a constant value shared by all flows, $\text{base-RTT}^i$ is defined as the RTT of flow $i$ in an empty system, and $\text{RTT}^i_t$ and $\text{rate}^i_t$ are respectively the RTT and transmission rate of flow $i$ at time $t$. $\frac{\text{RTT}^i_t}{\text{base-RTT}^i}$ is also called the RTT inflation of agent $i$ at time $t$. The ideal reward is obtained when $\textbf{target} = \frac{\text{RTT}^i_t}{\text{base-RTT}^i} \cdot \sqrt{\text{rate}^i_t}$. Hence, when the \textbf{target} is larger, the ideal operation point is obtained when $\frac{\text{RTT}^i_t}{\text{base-RTT}^i} \cdot \sqrt{\text{rate}^i_t}$ is larger. As increasing the transmission rate increases network utilization and thus the observed RTT, the two grow together. Such an operation point is less latency sensitive (RTT grows) but enjoys better utilization (higher rate).  As \cref{prop: reward is good} shows, maximizing this reward results in a fair solution.

\begin{proposition}\label{prop: reward is good}
    The fixed-point rate (solution) for all $N$ flows sharing a congested path is $\frac{\text{max rate}}{N}$.
\end{proposition}

Informally, the optimal reward for all agents is 0. An agent for which $\frac{\text{RTT}^i_t}{\text{base-RTT}^i} \cdot \sqrt{\text{rate}^i_t} > target$ needs to reduce the transmission rate, which in turn will also reduce the RTT. On the other hand, an agent below the target will act in the opposite direction. As all agents sharing the same congestion point observe approximately the same RTT, the fixed point solution is a fair solution. A formal proof is provided in the supplementary material, in addition to experiments showing how the target affects the behavior.

\section{Our Approach: Analytic  Deterministic Policy Gradient}\label{sec: impl}
In this section, we present the intricate combination of challenges arising in our setup. We explain why existing popular approaches are expected to fail, as we indeed observe and show later in experiments. We then introduce our algorithm that leverages the unique properties of the problem to overcome those challenges.

\textbf{The challenge.}
We address three challenges rising when learning in multi-agent partially-observable domains. The first is \textit{non-stationarity}. When multiple agents are trained in parallel while interacting in a shared   environment, it becomes non-stationary from the point of view of each agent, as other agents continually change. As a result, one is restricted to on-policy methods to ensure agents are trained on relevant data. Even had the environments changed slowly due to slow learning rate, off-policy methods were still not applicable. Such methods require access to the policies of other agents, which are out of reach in our case.

Second, \textit{partial-observability} hinders value-function estimation. Specifically, policy gradient methods that utilize value functions require direct access to states rather than observations to generate correct gradient estimations \citep{azizzadenesheli2018policy}[Theorem~3.2]. Instead, we shall directly use episodic reward trajectories as done in REINFORCE \citep{williams1992simple}.

The third challenge is \textit{instability} of stochastic policies. While REINFORCE avoids value-function estimation, it requires the policy to be stochastic. However, stochastic policies in our multi-agent partially-observable setup lead to highly unstable behavior. The multiple agents operate at the same time with the common goal of fairness. Thus, it is essential that they stabilize together to an equilibrium. We observe empirically that this is achievable only with deterministic policies. This also explains the failure of REINFORCE in our experiments later. Such instability  was also observed for PPO by \cite{touati2020stable}.

The combination of the above three challenges creates a unique set of limitations for a learning algorithm. Namely, it should be on-policy, should not depend on value-function estimation, and needs to support deterministic policies. We now propose an efficient approach that combines all these properties. It is achievable thanks to access to the derivative of the reward function.

\textbf{Our algorithm.}
To generate deterministic policies, as a first choice one might consider Deterministic Policy Gradient \citep[DPG]{silver2014deterministic}. However, it relies on value function estimation, as do its successors such as DDPG \citep{lillicrap2015continuous}.
Instead, we work around this by directly estimating the gradient via derivation of the reward function.

For $s\in {\mathcal S}$, the {Analytic Deterministic Policy Gradient}{} is defined as
\begin{align*}
    &\nabla_{\theta} \rho^{\pi_\theta} (s) = \nabla_{\theta}  \lim_{T \rightarrow \infty} \frac{1}{T} {\mathbb{E}} \left[ \sum_{t=0}^T r\big(o(\state_t), \pi_\theta(o(\state_t))\big) \right] \\
    &= \lim_{T \rightarrow \infty} \frac{1}{T}{\mathbb{E}} \left[ \sum_{t=0}^T \nabla_{\action} r(o(\state_t), \action)|_{\action=\action_t} \cdot \nabla_{\theta} \pi_\theta(o(\state_t)) \right] \, . \numberthis \label{eqn: on policy gradient}
\end{align*}
Similarly to DPG, it is on-policy. Moreover, despite the partial observability, the gradient estimation is unbiased since it relies on rollouts \citep{azizzadenesheli2018policy}[Eq.~(2)].

The gradient estimator used here is different from common estimators  \citep{sutton2000policy,silver2014deterministic} as it requires access to $\nabla_{\action} r(o(\state_t), \action)$.

\begin{claim}\label{claim: approximate gradient}
    The following is an analytical approximation of the deterministic gradient
    \begin{align*}
        \nabla_{\theta} \rho^{\pi_\theta} (\state) \approx \Bigg[  &\lim_{T \rightarrow \infty} \frac{1}{T} \sum_{t=0}^{T} \Big(\text{{\bf target}} \numberthis \label{eqn: approximate gradient}  \\
        &-  \text{rtt-inflation}_t  \cdot  \sqrt{\text{rate}_t} \Big) \Bigg] \nabla_\theta \pi_\theta (o(\state)) \, .
    \end{align*}
\end{claim}
in the supplementary material, we provide an extensive derivation of \cref{claim: approximate gradient}.

\begin{table*}[t]
    \centering
    \begin{tabular}{l|ccc|ccc|ccc|ccc}
        \multirow{2}{*}{\textbf{}} & \multicolumn{3}{c}{\textbf{128 to 1}} & \multicolumn{3}{c}{\textbf{1024 to 1}} & \multicolumn{3}{c}{\textbf{4096 to 1}} & \multicolumn{3}{c}{\textbf{8192 to 1}} \\
         & SU & FR & QL & SU & FR & QL & SU & FR & QL & SU & FR & QL \\

        \midrule
        Aurora  &
        \multicolumn{3}{c}{{packet loss}} & \multicolumn{3}{c}{{packet loss}} & \multicolumn{3}{c}{{packet loss}} & \multicolumn{3}{c}{{packet loss}} \\

        PPO & 1 & 26 & 3 & \multicolumn{3}{c}{{packet loss}} & \multicolumn{3}{c}{{packet loss}} & \multicolumn{3}{c}{{packet loss}} \\

        REINFORCE & 51 & 100 & 3 & \textbf{74} & \textbf{70} & \textbf{7} & 53 & 45 & 22 & \multicolumn{3}{c}{{packet loss}} \\

        DCQCN & \textbf{100} & \textbf{56} & \textbf{11} & \textbf{100} & \textbf{50} & \textbf{13} & \textbf{95} & \textbf{65} & \textbf{12} & \textbf{95} & \textbf{64} & \textbf{12} \\

        HPCC & \textbf{83} & \textbf{96} & \textbf{5} & 59 & 48 & 27 & \multicolumn{3}{c}{{packet loss}} & \multicolumn{3}{c}{{packet loss}} \\

        SWIFT & 97 & 94 & 26 & \textbf{89} & \textbf{96} & \textbf{27} & \textbf{88} & \textbf{85} & \textbf{77} & \multicolumn{3}{c}{{packet loss}}
        \\
        {ADPG}{} (ours) & \textbf{92} & \textbf{95} & \textbf{8} & \textbf{90} & \textbf{70} & \textbf{15} & 91 & 44 & 26 & 92 & 29 & 42 \\
        \bottomrule
    \end{tabular}
    \caption{\textbf{Many-to-one} test results. Numerical comparison of {ADPG}{} (our method) with various baselines. We color the tests that failed (extensive periods of packet loss) in red. As the task has multiple objectives, Bold face denotes results that are not Pareto-dominated (\cref{def: domination}).}
    \label{tab: many-to-one comparison}
\end{table*}

\section{Experiments} \label{sec: experiments}
We evaluate our approach in two domains. First, in a  simulated domain, using a hardware emulator  built on top of OMNeT++ \citep{varga2002omnet++}. Then, the trained agent is also deployed on real hardware and tested in a real environment.

\textbf{Emulated settings:} We focus on 3 major benchmarks:\\
\textbf{(1) Many-to-one: $\mathbf{N \rightarrow 1}$}. $N$ senders transmit data through a single switch to a single receiver. We evaluate the agents on $2^i \rightarrow 1$, for $i \in \{7, 10, 12, 13\}$. The exact configuration is presented in the supplementary material.\\
\textbf{(2) All-to-all:} Multiple servers transmitting data to all other servers. Given $N$ servers, there are $N$ congestion points (switches). Data sent towards server $i$ routes through switch port $i$. This synchronized traffic causes high system load.\\
\textbf{(3) Long-short:}
This scenario evaluates how each agent reacts to changes. A single flow (the `long' flow) transmits an infinite amount of data, while several short flows randomly interrupt it with a short data transmission. The goal is to test how fast the long flow reacts and reduces its transmission rate and how fast the short flows increase their transmission rate. Once the short flows finish transmitting, the long flow should quickly recover to the full line rate. We follow the process from interruption until full recovery. An example of ideal behaviors is presented in the supplementary material.

\textbf{Evaluation setup:} To show our method \emph{generalizes to unseen scenarios}, motivating the use in the real world, we split the scenarios to train and test sets. We \textbf{train} the agents only in the many-to-one domain, on the following scenarios simultaneously: $2 \rightarrow 1, 4 \rightarrow 1,$ and $8 \rightarrow 1$. \textbf{Evaluation} (test) is performed on many-to-one, all-to-all, and long-short scenarios. We provide an extensive overview of the training process, including the technical challenges of the asynchronous CC task, in the supplementary material.

\textbf{Compared Baselines.}
We compare with three RL benchmarks \textbf{(1)~REINFORCE} \citep{williams1992simple} \textbf{(2)~PPO} \citep{schulman2017proximal}, \textbf{(3)~Aurora} \citep{jay2019deep} and three rule-based approaches \textbf{(4)~DCQCN} \citep{zhu2015congestion}, \textbf{(5) HPCC} \citep{li2019hpcc} and \textbf{(6) SWIFT} \citep{kumar2020swift}. For RL-based methods, we used ``official" implementations and for rule-based methods we used in-house implementations that are currently deployed in datacenters. Additional details are provided in the supplementary material.

\textbf{Evaluation metrics:} We report three metrics: \textbf{(1) SU:} Switch Utilization, measured in \%, higher is better, \textbf{(2) FR:} Fairness, defined as $\frac{min rate \cdot 100}{max rate}$, higher is better, \textbf{(3) QL:} Queue Latency, measured in $\mu$ seconds, lower is better.

In our experiments, bold face denotes results that are not dominated (in the Pareto sense).
\begin{definition}[Domination]\label{def: domination}
    Result A dominates B if for all metrics $m \in M$ $A(m) \approx B(m)$ and there exists at least one metric $m$ such that $A(m) \gg B(m)$.
    Based on feedback from datacenter managers, we determined that $||A(m) - B(m)|| \leq 5$ to be similar $\forall m$.
\end{definition}

\subsection{Experiments With Simulated Data}\label{sec: sim experiments}
Simulating a networking environment is hard to do correctly. The behavior may be sensitive to hardware limitations, and should take into account packet loss that may occur due to other agents interacting with the network.

To test our approach, we used a hardware emulator built on top of OMNeT++ \citep{varga2002omnet++}. OMNet++ is a sophisticated simulator that is used by the largest data centers to benchmark congestion control algorithms prior to deployment. It accurately emulates hardware devices, including limitations of parallel compute by on-board CPU. While no simulation is perfect, this simulator minimizes the difference between the simulated environment and the real world.

\begin{table}[t]
    \centering
    \label{tab: all-to-all comparison}
    \begin{tabular}{l|ccc|ccc}

        \multirow{2}{*}{\textbf{}} & \multicolumn{3}{c}{\textbf{4 hosts}} & \multicolumn{3}{c}{\textbf{8 hosts}} \\
         & SU & FR & QL & SU & FR & QL \\
        \midrule

        DCQCN & \textbf{89} & \textbf{93} & \textbf{4} & 87 & 66 & 5 \\

        HPCC & 71 & 18 & 3 & \textbf{69} & \textbf{60} & \textbf{3} \\

        SWIFT & \textbf{93} & \textbf{100} & \textbf{10} & \textbf{92} & \textbf{100} & \textbf{11} \\
        {{ADPG}{}} & \textbf{94} & \textbf{77} & \textbf{6} & \textbf{94} & \textbf{97} & \textbf{8} \\
        \bottomrule
    \end{tabular}
    \caption{\textbf{All-to-all} test results. In these tests, none of the algorithms exhibited packet loss.}
\end{table}

\textbf{Many-to-one:} \cref{tab: many-to-one comparison} gives the results for the many-to-one scenario. Competing methods are competitive when the number of flows and interactions are small. However, with more flows, all methods aside from {ADPG}{} and DCQCN have extensive periods of packet loss and high latency. Across all tasks, previous RL methods (REINFORCE, PPO and Aurora) fail, either converging to extremely safe behavior or constantly transmitting at full-line-rate entirely ignoring latency and packet loss. This validates our arguments that as these methods rely on strong assumptions, they are unfit for many real-world tasks.

\begin{figure*}[t]
    \centering
    \textbf{2 to 1}\\
    \begin{subfigure}[b]{0.192\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/RL_long_short_2_to_1}
        \caption{{ADPG}}
        \label{subfig : rl long short 2}
    \end{subfigure}
    \begin{subfigure}[b]{0.161\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/DCQCN_long_short_2_to_1}
        \caption{DCQCN}
        \label{subfig : dc2qcn long short 2}
    \end{subfigure}
    \begin{subfigure}[b]{0.161\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/HPCC_long_short_2_to_1}
        \caption{HPCC}
        \label{subfig : hpcc long short 2}
    \end{subfigure}
    \begin{subfigure}[b]{0.161\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/SWIFT_long_short_2_to_1}
        \caption{SWIFT}
        \label{subfig : swift long short 2}
    \end{subfigure}\\

    \textbf{8 to 1}\\
    \begin{subfigure}[b]{0.196\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/RL_long_short_8_to_1}
        \caption{{ADPG}}
        \label{subfig : rl long short 8}
    \end{subfigure}
    \begin{subfigure}[b]{0.161\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/DCQCN_long_short_8_to_1}
        \caption{DCQCN}
        \label{subfig : dc2qcn long short 8}
    \end{subfigure}
    \begin{subfigure}[b]{0.161\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/HPCC_long_short_8_to_1}
        \caption{HPCC}
        \label{subfig : hpcc long short 8}
    \end{subfigure}
    \begin{subfigure}[b]{0.161\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/SWIFT_long_short_8_to_1}
        \caption{SWIFT}
        \label{subfig : swift long short 8}
    \end{subfigure}
    \caption{ \textbf{Long Short} test results. The goal is to recover fast, but also avoid packet loss. Higher buffer utilization means higher latency. A fully utilized buffer (100\% utilization of the 5MB allocated) leads to packet loss. In these tests, none of the algorithms encountered packet loss. The top row (\cref{subfig : rl long short 2,subfig : dc2qcn long short 2,subfig : hpcc long short 2,subfig : swift long short 2}) presents the results of a long-short test with 2 flows, and the bottom row (\cref{subfig : rl long short 8,subfig : dc2qcn long short 8,subfig : hpcc long short 8,subfig : swift long short 8}) presents a test with 8 flows. We plot the bandwidth utilization of the long flow and the buffer utilization in the switch. Recovery time is measured as the time it takes the long flow to return to maximal utilization. As can be seen, in both scenarios, DCQCN does not recover within a reasonable time. In addition, in HPCC, the long flow does not reach 100\% utilization, even when there are no additional flows.}
    \label{fig: long short}
\end{figure*}

\textbf{All-to-all:} Each method provides a different trade-off balancing between utilization, fairness and latency -- see \cref{tab: all-to-all comparison}. Similarly to the many-to-one scenario, ADPG is capable of generalizing and exhibits competitive behavior.

\textbf{Long-short:} \cref{fig: long short} depicts performance for the long-short scenario, numerical results are in the supplementary material.
Here, algorithms are quantified based on how quickly they react to changes. {ADPG}{} was not trained with this scenario, but performs well.
Here, HPCC was fastest to react. This is expected since it was specifically designed to handle  long-short scenarios \citep{li2019hpcc}. Unlike  HPCC, {ADPG}{} achieves 100\% utilization before and after the interruption and recovers faster than both SWIFT and DCQCN (which fails in this scenario).

\textbf{Summary:}  We find  that {ADPG}{} learns a robust policy. Although in certain tasks the baselines marginally outperformed it, {ADPG}{} always obtained competitive performance. In addition, {ADPG}{} is the only learning-based method capable of successfully solving the task and thus controlling the network congestion. As {ADPG}{} was trained only in low-scale many-to-one scenarios, it highlights the ability of our method to generalize and learn a robust behavior (successful behavior within higher scale and diverse test scenarios), that we believe will perform well in a real datacenter.

\subsection{Experiments In The Real-World}\label{sec: real world}

Going beyond the simulated environment, we deployed trained agents in real hardware devices.
As existing hardware lacks the accelerators required for fast inference, the reaction time (forward pass) was on the order of $~400 \mu sec$. This was slower than used in our simulated experiments, which were designed to simulate future hardware with $\bigO(1 \mu sec)$ reaction time. Importantly, the baselines we compare with do not rely on DNN inference, thus react within $\bigO(1 \mu sec)$.

\begin{table}[t]
    \centering
    \label{tab: real many-to-one comparison}
    \begin{tabular}{l|c|c|c}
         & \multicolumn{1}{c}{{2 to 1}} & \multicolumn{1}{c}{{1024 to 1}} & \multicolumn{1}{c}{{2000 to 1}} \\
        \midrule

        DCQCN & \textbf{89}  $\pm$ 0.1 & \textbf{86}  $\pm$ 1.3 & \textbf{78}  $\pm$ 0.6 \\

        HPCC & \textbf{84} $\pm$ 0.2 & 68 $\pm$ 0.1 & 72 $\pm$ 0.1 \\
        SWIFT & \textbf{89} $\pm$ 0.0 & \textbf{82} $\pm$ 0.2 & \textbf{83} $\pm$ 0.5 \\
        {ADPG}{} & \textbf{86}  $\pm$ 1.2 & \textbf{82}  $\pm$ 3.9 & 72  $\pm$ 3.2 \\
        \bottomrule
    \end{tabular}
    \caption{\textbf{Real-world, Many-to-one} test results, comparing switch utilization \%.}
\end{table}

We deployed our trained agent from \cref{sec: sim experiments} after performing quantization and porting code to native C. Results are in \cref{tab: real many-to-one comparison}. Even though the ADPG agent was trained under the assumption of a much faster reaction time, it performs on-par with other methods,
outperforms HPCC in the many-to-one test and is competitive to DCQCN in the low-scale setting. We expect dramatic improvement with faster hardware (future generation).

\section{Summary}

AI centric algorithms have the ability to leverage direct interaction with the datacenter in order to optimize performance. We presented the fundamental challenges this task presents when using reinforcement learning tools. These challenges prevent popular algorithms from successfully solving the task. We proposed an on-policy deterministic policy gradient algorithm and presented a way to use domain knowledge to obtain an analytical gradient estimation.

Experiments in a simulated environment, a realistic OMNeT++ network simulator commonly used to benchmark CC algorithms for deployment in real datacenters, show that our method {ADPG}{} successfully addresses this task. More generally, they demonstrate the efficacy and generalization capabilities of an RL approach which is in contrast to the hand-crafted algorithms that currently dominate the field of CC. While some baselines achieve outstanding performance in specific evaluation scenarios, they catastrophically fail in others. In contrast, {ADPG}{} learned a robust policy that performed well across all scenarios we tested, and often obtained the best results. We also show that {ADPG}{} generalizes to unseen domains and is capable of operating at a large-scale without incurring packet loss, a setting where most competing methods fail.

Finally, we deployed our trained agent in real hardware. Current hardware suffers from high latency, an issue that we expect to solve with the next generation NICs. Regardless, our method was capable of successfully obtaining competitive behavior, thus increasing our confidence in its applicability when combined with future low-latency hardware.

While there are yet many challenges on the path of applying RL in the real world, we believe these results to be a promising start. While the baseline algorithms have gone through years of meticulous hand tuned optimization, {ADPG}{} was capable of quickly obtaining competitive behavior both in simulation and in the real world.

\section{Introduction}

Most RL algorithms were designed under the assumption that the world can be adequately modeled as a Markov decision process.
However, in applications this is seldom the case. In this work we study a real world application of RL to data-center congestion control. In this application, there is partial observability and a complex multi-objective. Moreover, there are multiple decision makers that have to  make concurrent decisions that can affect each other. We show that even though that the problem at hand doesn't fit the standard model, we can nevertheless devise an RL framework that outperforms state-of-the-art tailored heuristics. Moreover, we show experimentally that the RL framework generalizes well.

We will provide a full description of the Congestion Control (CC) problem in Section \ref{sec: cc}, but from a bird's eye view, it is enough to think of it as a multi-agent, multi-objective, partially observed problem where each decision maker
receives a goal (target). The target makes it easy to tune the behavior to fit the requirements (i.e., how latency sensitive the system is).
We model the task of datacenter congestion control as a reinforcement learning problem. We observe that standard algorithms \citep{mnih2015human,lillicrap2015continuous,schulman2017proximal} are incapable of solving this task. We introduce a novel on-policy deterministic-policy-gradient scheme. This method enjoys both the stability of deterministic algorithms and the ability to tackle partially observable problems (as per on-policy algorithms).

To validate our claims, we develop a RL GYM \citep{brockman2016openai} environment, based on a realistic simulator and perform extensive experiments. The simulator is based on OMNeT++ \citep{varga2002omnet++} and emulates the behavior of ConnextX-6Dx network interface cards (state of the art hardware deployed in current datacenters)\footnote{To encourage additional research in this field, the code will be open-sourced after the review phase is over.}. Our experiments show that our method, PCC-RL, learns a robust policy, in the sense that it is competitive in all the evaluated scenarios; often outperforming the current state-of-the-art methods.

Our contributions are as follows. (i) We formulate the problem of datacenter congestion control as a partially-observable multi-agent multi-objective reinforcement learning task. (ii) We present the challenges of this realistic formulation and propose a novel on-policy deterministic-policy-gradient method to solve it. (iii) We provide an RL training and evaluation suite for training and testing RL agents within a realistic simulator. Finally,  (iv) we ensure our agent satisfies compute and memory constraints such that it can be easily deployed in future datacenter network devices.

\section{Congestion Control}\label{sec: cc}

Congestion is a state observed in datacenters when an interface receives data faster than it can process. Each interface in the network has an input buffer, \cref{fig: example topology}, that enables it to cope with short-term inbound bursts. These buffers operate on a first-in-first-out (FIFO) basis. When these buffers are full, newly arriving packets are dropped. Congestion control is an algorithmic task of preventing this state.

A FIFO buffer means that a packet arriving to the end of the line. will need to wait additional time until it is processed. This time (latency) increases as the buffer fills up. A unique property in datacenter congestion control is that many applications are latency sensitive. For instance, financial applications require near-zero latency, thus the task of congestion control is not only to avoid packet loss, but also minimize latency and maximize throughput.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figures/computer_network_diagram}
    \caption{Example of a simple many-to-one topology. Multiple senders transmit data through a single switch, the congestion point, towards a single receiver. As the source nodes have the potential to transmit at a combine rate faster than the switch can process, its input buffer might fill up and the switch may become congested.}
    \label{fig: example topology}
\end{figure}

Previous methods either focused on hand-designed algorithms \citep[DCTCP]{alizadeh2010data}, \citep[DCQCN]{zhu2015congestion}, \citep[Timely]{mittal2015timely}, \citep[HPCC]{li2019hpcc}, \citep[SWIFT]{kumar2020swift}, going through meticulous trial-and-error loops in an attempt to reach optimal behavior. As these methods are manually tuned, they often don't generalize well to new domains (topologies, data types, etc...), thus requiring on-site experts to adapt the parameters to the customer's (datacenter) requirements.

We focus on the setting of congestion control (CC) within a datacenter. In this case, the data traffic is dense, the possible transmission rates are high, and achieving low latency is of utmost importance. The network contains multiple servers that are interconnected through a topology of switches (also known as routers). A directional connection, a continuous transmission of data, between two servers is called a flow. We assume, for simplicity, that the path of each flow is fixed.

Each server can hold multiple flows and a scheduler, operating in a round-robin manner, determines the next flow to transmit data. Once scheduled, the flow will transmit a burst of data. The size of the burst depends on the requested transmission rate, when it was last scheduled and the maximal burst size limitation.

\begin{figure}[t]
    \centering
    \includegraphics[trim=80 130 300 20,clip,width=\linewidth]{figures/packet_flow}
    \caption{Schema of the packet flow. The sender transmits a bursts of data packets and at the end transmits an {\color{mygreen} RTT packet}.}
    \label{fig: packet flow diagram}
\end{figure}

Congestion occurs when multiple flows cross paths, transmitting data through a single congestion point (switch or receiving server) at a rate faster than the congestion point can process. In this work, we assume that all connections have equal transmission rates, which typically occurs in most datacenters, thus a single flow can saturate an entire path by transmitting at the maximal rate.

Each congestion point in the network has an inbound buffer, enabling it to cope with short periods of time where the inbound rate is higher than what it can process. As this buffer begins to fill, the time (latency) it takes for each packet to reach its destination increases. When the buffer is full, any additional arriving packets are dropped.

There are various methods to measure or estimate the congestion within the network. The ECN \citep{ramakrishnan2001rfc3168} protocol considers marking packets with an increasing probability as the buffer fills up. DCTCP \citep{alizadeh2010data} and DCQCN \citep{zhu2015congestion} utilize the marked packets to recognize when congestion starts to occur and react accordingly. In addition, algorithms such as HPCC \citep{li2019hpcc} utilize advanced switches that support network telemetry functionality. Network telemetry functions enable the switch to return information to the source, such as the exact buffer utilization and the switch port utilization.

However, while the ECN and telemetry signals provide important information about the system, they require specialized hardware (switches). An ideal solution is one that can be easily deployed within existing networks. Such solutions are based on the round-trip-time (RTT), as used by Timely \citep{mittal2015timely} and SWIFT \citep{kumar2020swift}. They measure congestion based on how much the RTT has changed over time, compared to the RTT within an empty system.

In order to avoid congestion, each server employs a CC algorithm, controlling the rate of transmission of the source. The goal of the CC algorithm is to control the transmission rate of the flows such that congestion is avoided. Specifically, a functional CC algorithm should not encounter packet loss and should reduce latency while maximizing bandwidth in a fair manner (see \cref{cc: objective}).

\cref{fig: 4 to 1 packet flow} illustrates the behavior of our algorithm and what is expected in CC. When multiple flows share a single congestion point, they should quickly converge and transmit at an equal portion of the line rate. While the transmission rates may jitter, we are interested in the average rates and latency.

\subsection{Setup}

While many methods consider CC algorithms that operate within the host (server), ConnectX devices run the CC algorithm within the network-interface-card (NIC) itself. This enables faster reaction times, additional capabilities and reduces the computational overhead required from the host.

An example of a specialized hardware capability is the RTT measurement packets. While there are various ways of measuring the RTT, they are often inaccurate. The RTT measurement packet is a specialized hardware packet transmitted once every time a data burst ends. The behavior is presented in \cref{fig: packet flow diagram}. Upon reaching the destination, the RTT packet is quickly routed back towards to source, without going through the host. This enables a precise measurement of the RTT within the network itself.

We compare to 3 baseline algorithms -- DC2QCN, HPCC and SWIFT. These methods lack an official implementation, we compare to an unofficial implementation that is currently deployed and used daily in real datacenters.

\subsection{Objective}\label{cc: objective}
This task can be seen as a distributed system where there are N agents (assuming there are N flows then there are N CC agents operating in parallel). Assuming all agents have an infinite amount of traffic to transmit, their goal is to reach a joint fair-optimal transmission rate. We measure the following objectives:

\begin{defn}[joint fair-optimal transmission rate]\label{def: fair optimal transmission}
    ~
    \begin{enumerate}
        \item Switch bandwidth utilization -- the \% from maximal transmission rate.

        \item Packet latency -- the amount of time it takes for a packet to travel from the source to it's destination.

        \item Packet-loss -- the amount of data (\% of maximum transmission rate) dropped due to congestion.

        \item Fairness -- a measure of similarity in the transmission rate between flows sharing a congested path. We consider $\frac{\min_\text{flows} BW}{\max_\text{flows} BW} \in [0, 1]$.
    \end{enumerate}
\end{defn}

The goal of the CC agent is to maximize the \emph{bandwidth utilization} and \emph{fairness}, and minimize the \emph{latency} and \emph{packet-loss}; a multi-objective problem. As such, this formulation may result in a Pareto-front \citep{liu2014multiobjective} where being optimal w.r.t. to one objective may result in sub-optimal behavior on another. However, while the metrics of interest are clear, the agent does not necessarily have access to signals representing them. For instance, fairness is a metric over all flows, yet the agent observes signals relevant only to the flow it controls. Thus it is impossible for a flow to obtain an estimate of the current fairness in the system.

This problem exhibits additional complexities. As the agent only observes information relevant to the flow it controls, this task is partially observable. The observation might lack sufficient statistics required to determine the optimal policy. Moreover, the network does not react instantaneously to a change in the transmission rate. It takes time for the decisions to propagate, estimated at $\bigO(RTT)$.

\begin{figure}[t]
    \centering
    ~~~~~~~~~\textbf{Four PCC-RL agents converging to a fair solution}
    \includegraphics[width=\linewidth]{figures/4_to_1_timeseries}
    \caption{PCC-RL agents controlling 4 flows, each running on a different host. Only a single flow is active at the beginning and end of the simulation. The congestion point is limited to $100 [GByte/s] = 12.5 [Gbit/s]$. We can see that the flows quickly converge to a region of the fair transmission rate ($\frac{12.5}{4}$).}
    \label{fig: 4 to 1 packet flow}
\end{figure}

\section{Reinforcement learning}\label{sec: rl}

We model the task of congestion control as a multi-agent partially-observable multi-objective markov-decision-process, where all agents share the same policy. Each agent observes statistics relevant to itself, and does not observe the entire global state (e.g., how many other flows are also interacting with the system). We present this model below, building up from the foundations of the basic MDP model.

\subsection{Markov Decision Process}\label{sec:mdp}
We consider an infinite-horizon Markov Decision Process (MDP). An MDP is defined as the tuple $(\mathcal{S}, \mathcal{A},P,R)$ \cite{puterman1994markov}. ${\mathcal S}$ is the state space, e.g., the signals observed by the CC algorithm. ${\mathcal A}$ is the action space, e.g., the maximal rate the flow is allowed to transmit. $P \equiv P(\state'|\state,\action)$ is the transition kernel, determining the random process at which the system transitions from state $\state$ after an action $\action$ was performed. In the CC problem, the system transitions to the next state based on the actions performed by all agents. Finally, $R \equiv r(\state,\action)$ is a reward function.

As the behavior should be always optimal with respect to every state, we consider the average reward metric defined as follows. We denote $\Pi$ as the set of stationary deterministic policies on $\mathcal{A}$, i.e., if $\pi\in\Pi$ then $\pi: \mathcal{S}\rightarrow \mathcal{A}$.  Let $\rho^\pi \in \mathbb{R}^{|\mathcal{S}|}$ be the gain of a policy $\pi,$ defined in state $\state$ as $\rho^\pi(\state) \equiv \lim_{T \rightarrow \infty} \frac{1}{T} {\mathbb{E}}^\pi[\sum_{t=0}^T r(\state_t,\action_t)\mid \state_0=\state]$, where ${\mathbb{E}}^\pi$ denotes the expectation w.r.t. the distribution induced by $\pi$ and conditioned on the event $\{\state_0=\state\}.$

The goal is to find a policy $\pi^*,$ yielding the optimal gain $\rho^*$, i.e., for all $\state\in \mathcal{S}$, $\pi^*(\state) \in \argmax_{\pi\in \Pi} \rho^\pi (\state)$
and the optimal gain is $\rho^{*}(\state) = \rho^{\pi^*}(\state)$. A well known result \cite{puterman1994markov}[Theorem 6.2.10], is that there always exists an optimal policy which is stationary and deterministic.

\subsection{Partially Observable MDPs}
Although the standard MDP framework builds the foundation for most RL work, here we must consider an extension that refers to partially observable information \citep{spaan2012partially}. As is in our congestion control problem, while the state $\state$ contains the entire information required, the agent is presented with an observation $\obs$, i.e., the agent only observes signals relevant to the flow it controls yet does not know important information such as the amount of flows sharing its path. Thus, the observation does not necessarily contain the sufficient statistics for determining the optimal action.

\subsection{Partially Observable Multiplayer Games with Identical Policies}
A final extension we must consider is called multiplayer games. While the POMDP framework considers a single agent interacting with the environment, in this task we consider multiple agents interacting in parallel. In our setting, we are interested in identical congestion control algorithms, thus, denoting the action performed by agent $i$ at state $\state$ as $\action^i$, we can define the joint value by:

\begin{align*}
    & G^\pi(\obs(\state)) \equiv \numberthis \\
    & \lim_{T \rightarrow \infty} \! \frac{1}{T} {\mathbb{E}}^\pi \! \left[ \sum_{t=0}^T r(\obs(\state_t),\action_t^0, \ldots, \action_t^N) \! \mid \! \state_0=\state \right] \! .
\end{align*}

\section{Related Work} \label{sec: related work}

\textbf{Hand-tuned methods:} While there has been a vast amount of work on congestion control, we focus on datacenter congestion control methods. Previous work tackled this problem from various angles. \citet{alizadeh2010data} used a TCP-like method, which slightly increases the transmission rate so long as there is no congestion and dramatically decreases it once congestion begins to accumulate. \citet{mittal2015timely,kumar2020swift} focus on the packet's round-trip measurement (latency) in order to react quickly to changes. \citet{zhu2015congestion} utilize congestion notification packets, a statistical signal provided by the switch, and \citet{li2019hpcc} add telemetry information that requires specialized hardware yet proves to benefit greatly in terms of reaction times.

\textbf{Optimization-based methods:} Although most previous work has focused on hand-tuned algorithmic behavior, two notable works have taken an optimization-based approach. \citet{dong2018pcc} presented the PCC-Vivace algorithm, an online-convex-optimization method. PCC-Vivace combines information over fixed timeframes (monitor intervals) and extracts certain statistics such as bandwidth, latency inflation and more. As it tackles the problem via online-convex-optimization, it is state-less and adapts the behavior in an attempt to minimize a given objective. The state-less behavior means that it does not optimize for long-term behavior but rather focuses on the immediate reward (bandit setting). This work was then extended in the Aurora system \citep{jay2019deep}. Aurora provides the monitor interval, defined in PCC-Vivace, as a state for a PPO \citep{schulman2017proximal} algorithm. Although these methods work in a naive single-agent setting, we observed in inability to converge to satisfying behavior in the realistic multi-player setting.

\textbf{Multi-objective RL:} This task can also be cast as a multi-objective RL problem \citep{mannor2004geometric} and solved by combining the various metrics into a scalar reward function \citep{jay2019deep} (e.g., $r = a \cdot \text{bandwidth} - b \cdot \text{latency} - c \cdot \text{packet loss}$, where $a,b,c$ are constant coefficients $> 0$). However, in practice, the exact coefficients are selected through a time consuming and a computationally intensive process of \textit{hyper-parameter} tuning. Previous work \citep{leike2017ai,mania2018simple,tessler2018reward} has shown that these coefficients do not generalize, a coefficient which leads to a satisfying behavior on one domain may lead to catastrophic failure on the other. We propose an alternative approach, in which we present a reward function that is domain agnostic. Our reward has a single fixed point solution, which is optimal for any domain.

\section{Method}

\subsection{Congestion control using reinforcement learning}

The POMDP framework from \cref{sec: rl}, requires the definition of the 4 elements -- $(\mathcal{O}, \mathcal{A},P,R)$. The agent, a congestion control algorithm, resides within the NIC and controls the rate of the flows within that NIC. At each decision point, the agent observes statistics correlated to the specific flow it controls. The agent then acts by determining a new transmission rate and observes the outcome of this action.

\textbf{Observations.} As the agent can only observe information relevant to the flow it controls, we consider: the flow's transmission \emph{rate}, \emph{RTT} measurement and number of \emph{CNP} and \emph{NACK} packets received. The CNP and NACK packets represent events happening in the network. A CNP packet is transmitted to the source once an ECN-marked packet reaches the destination. A NACK packet signals to the source that packets have dropped (e.g., due to congestion) and should be re-transmitted.

\textbf{Actions.} The optimal transmission rate depends on the number of agents simultaneously interacting in the network and on the network itself (bandwidth limitations and topology). As such, the optimal transmission rate will vary greatly across scenarios. To prevent the agent from overfitting to a specific transmission rate, we define the action as a multiplication of the previous rate. I.e., $\text{rate}_{t+1} = \action_t \cdot \text{rate}_t$.

\textbf{Transitions.} The transition $\state_t \rightarrow \state_{t'}$ depends on the dynamics of the environment and on the frequency at which the agent is required to provide an action. Here, the agent acts once an RTT packet is received. This scheme is similar to the definition of a monitor interval, but while \citet{dong2018pcc} considered a time interval, we consider event (RTT) triggered intervals.

\textbf{Reward.} As the task is a multi-player partially observable problem, the reward must be designed such that there exists a single fixed point equilibrium -- the joint fair-optimal transmission rate, Definition \ref{def: fair optimal transmission}. We define the reward as $r_t = -(\text{target} - \frac{\text{RTT}^i_t}{\text{base RTT}^i} \cdot \sqrt{\text{rate}^i_t})^2$, where target is a constant value shared by all flows, $\text{base RTT}^i$ is defined as the RTT of flow $i$ in an empty system and $\text{RTT}^i_t$ and $\text{rate}^i_t$ are respectively the RTT transmission rate of flow $i$. $\frac{\text{RTT}^i_t}{\text{base RTT}^i}$ is also called the rtt inflation of agent $i$ at time $t$.

Based on \citet{kumar2020swift}, a good approximation of the RTT inflation in a bursty system, where all flows transmit at the ideal rate, behaves like $\sqrt{N}$. Where $N$ is the number of flows. As the system, at the optimal point, is on the verge of congestion, the major latency increase is due to the packets waiting in the congestion point. As such, we can assume that all flows sharing a congested path will observe a similar $\text{rtt inflation}_t \approx \frac{\text{RTT}^i_t}{\text{base RTT}^i}$. As \cref{prop: reward is good} shows, maximizing this reward results in an optimal-fair solution.

\begin{proposition}\label{prop: reward is good}
    The fixed-point solution for all $N$ flows sharing a congested path is a transmission rate of $\frac{1}{N}$.
\end{proposition}

The proof relies on the assumption that all flows sharing a congested path observe the same rtt inflation. Although the network topology affects the packet path and thus the latency, this latency is minimal when compared to the queue latency of a congested switch.

\begin{proof}
    A fixed point solution can be obtained in two scenarios, all flows sharing a congested path observe (1) $\text{rtt inflation} \cdot \sqrt{rate} < \text{target}$ or (2) $\text{rtt inflation} \cdot \sqrt{rate} = \text{target}$. $\text{rtt inflation} \cdot \sqrt{rate} > \text{target}$ is not a stable solution, as the flow can reduce the transmission rate to approach the target.

    \begin{enumerate}
        \item As the value is below the target, minimizing the distance to the target means maximizing the transmission rate. A stable solution below the target is obtained when the flows are transmitting at full line rate and yet there is no congestion. As we assumed all the ports can transmit at the same rate, this scenario may occur only when there is a single flow transmitting.

        \item For any $i, j$ sharing a congested path, we assume that $\text{rtt inflation}_i = \text{rtt inflation}_j$. Thus, as all flows observe $\text{rtt inflation} \cdot \sqrt{rate} = \text{target}$, we conclude that $\sqrt{rate}_i = \sqrt{rate}_j = \frac{1}{N}, \forall i,j$.
    \end{enumerate}
\end{proof}

Below we discuss how this is implemented in practice.

\subsection{Implementation}

Thriving for simplicity, we initially attempted to solving this task using standard RL techniques such as DQN \citep{mnih2015human}, DDPG \citep{lillicrap2015continuous} and PPO \citep{schulman2017proximal}. Due to the challenges this task exhibits, namely partial observability and multi-agent optimization, these methods did not converge. Due to the partial observability, on-policy methods are the best fit, and as the goal is to converge to a stable multi-agent equilibria, deterministic policies are easier to manage.

To overcome these limitations, we develop an on-policy deterministic policy-gradient \citep[DPG]{silver2014deterministic} training procedure. In DPG, the goal is to estimate $\nabla_{\theta} G^{\pi_\theta}$, the gradient of value of the current policy, with respect to the policy's parameters $\theta$. By taking a gradient step within this direction, the policy is improving and thus under standard assumptions will converge to the optimal policy.

Learning a critic requires off-policy methods, which we observed not to work well. Hence, we focus on estimating $\nabla_{\theta} G^{\pi_\theta}$ from a sampled trajectory rollout.

\begin{align*}
    \nabla_{\theta} G^{\pi_\theta} = &\nabla_{\theta} \lim_{T \rightarrow \infty} \frac{1}{T} {\mathbb{E}} \left[ \sum_{t=0}^T r(\state_t, \pi_\theta(\state_t)) \right] \numberthis \label{eqn: gradient}\\
    = &\lim_{T \rightarrow \infty} \frac{1}{T} \sum_{t=0}^T \nabla_{\action} r(\state_t, \action)|_{\action=\action_t} \cdot \nabla_{\theta} \pi_\theta(\state_t) \\
    = &- \lim_{T \rightarrow \infty} \frac{1}{T} \\
    & \cdot \sum_{t=0}^T \nabla_{\action} \left(\text{target} - \text{rtt inflation}^i \cdot \sqrt{\text{rate}^i_t} \right)^2 \mid_{\action = \action_t} \\
    &\cdot \nabla_{\theta} \pi_\theta(\state_t)
\end{align*}

where both the rate and the RTT depend on the history and the action $\action$. Using the chain rule we can estimate the gradient of the reward $\nabla_{\action} r(\state_t, \action)$:

\begin{align*}
    \nabla_{\action} r(\state_t, \action) = \left( \text{target} - \text{rtt inflation}_t \cdot \sqrt{\text{rate}_t} \right) \cdot \numberthis\\
    \nabla_{\action} \left( \text{rtt inflation}_t \cdot \sqrt{\text{rate}_t} \right)
\end{align*}

\begin{table*}
    \centering
    \begin{tabular}{c||c|c|c|c||c|c|c|c||c|c|c|c||c|c|c|c}
        \multirow{2}{*}{\textbf{Alg.}} & \multicolumn{4}{c}{\textbf{128 to 1}} & \multicolumn{4}{c}{\textbf{1024 to 1}} & \multicolumn{4}{c}{\textbf{4096 to 1}} & \multicolumn{4}{c}{\textbf{8192 to 1}} \\
         & SU & FR & QL & DR & SU & FR & QL & DR & SU & FR & QL & DR & SU & FR & QL & DR \\
        \hline\hline
        \textbf{PCC-RL} & \textbf{92} & \textbf{95} & \textbf{8} & \textbf{0} & 90 & 70 & 15 & 0 & \textbf{91} & \textbf{44} & \textbf{26} & \textbf{0} & \textbf{92} & \textbf{29} & \textbf{42} & \textbf{0} \\
        \hline
        \textbf{DC2QCN} & 96 & 84 & 8 & 0 & \textbf{88} & \textbf{82} & \textbf{17} & \textbf{0} & \cellcolor{red!40}85 & \cellcolor{red!40}67 & \cellcolor{red!40}110 & \cellcolor{red!40}0.2 & \cellcolor{red!40}100 & \cellcolor{red!40}72 & \cellcolor{red!40}157 & \cellcolor{red!40}1.3 \\
        \hline
        \textbf{HPCC} & 83 & 96 & 5 & 0 & 59 & 48 & 27 & 0 & \cellcolor{red!40}73 & \cellcolor{red!40}13 & \cellcolor{red!40}79 & \cellcolor{red!40}0.2 & \cellcolor{red!40}86 & \cellcolor{red!40}8 & \cellcolor{red!40}125 & \cellcolor{red!40}0.9 \\
        \hline
        \textbf{SWIFT} & 98 & 99 & 40 & 0 & \textbf{91} & \textbf{98} & \textbf{66} & \textbf{0} & \cellcolor{red!40}90 & \cellcolor{red!40}56 & \cellcolor{red!40}120 & \cellcolor{red!40}0.1 & \cellcolor{red!40}92 & \cellcolor{red!40}50 & \cellcolor{red!40}123 & \cellcolor{red!40}0.2
    \end{tabular}
    \caption{\textbf{Many-to-one:} Numerical comparison of PCC-RL (our method) with DC2QCN (unpublished followup to \citet{zhu2015congestion}), HPCC \citep{li2019hpcc} and SWIFT \citep{kumar2020swift}. The columns represent \textbf{SU} Switch Utilization [\%], higher is better, \textbf{FR} Fairness defined as $\frac{min rate \cdot 100}{max rate}$, higher is better, \textbf{QL} Queue Latency [$\mu$ sec], lower is better, and \textbf{DR} Drop rate [Gbit/s], lower is better. As the goal of a congestion control algorithm is to prevent congestion, we color the tests that failed (extensive periods of packet loss) in red. We mark the best performing methods on each tests in bold. For 1024, we mark both DC2QCN and SWIFT as it is up to the end user to determine which outcome is preferred.}
    \label{tab: many-to-one comparison}
\end{table*}

where target is a constant. Notice that both $\text{rtt inflation}_t$ and $\sqrt{\text{rate}_t}$ are monotonically increasing in $\action$\footnote{The action is a scalar determining by how much to change the transmission rate. A faster transmission rate leads to higher RTT inflation and transmission rate.}. Thus, the signs of $\text{rtt inflation}_t$ and $\sqrt{\text{rate}_t}$ are identical and $\nabla_{\action}(\text{rtt inflation}_t \cdot \sqrt{\text{rate}_t})$ is always non-negative.

However, estimating the exact value $\nabla_{\action}(\text{rtt inflation}_t \cdot \sqrt{\text{rate}_t})$ is non-trivial. As the sign is always non-negative, we replace it with a positive constant. While the direction of the gradient remains the same, the size may differ.

\begin{align*}
    &\nabla_{\theta} G^{\pi_\theta} (\state) \approx \numberthis \label{eqn: approximate gradient} \\
    & \left[ \lim_{T \rightarrow \infty} \frac{1}{T} \sum_{t=0}^{T} \left(\text{target} \! - \! \text{rtt inflation}_t \! \cdot \! \sqrt{\text{rate}_t} \right) \right] \nabla_\theta \pi_\theta (\state)
\end{align*}

In layman's terms -- if $\text{rtt inflation}_t * \sqrt{\text{rate}_t}$ is above the target, the gradient will push the action towards decreasing the rate. On the other hand, if it is below the target it will push it towards increasing the transmission rate. As all flows observe approximately the same $\text{rtt inflation}_\tau$, the objective drives them towards the fixed point solution. As shown in \cref{prop: reward is good}, when all flows transmit at the same rate $\frac{1}{N}$ and the system is slightly congested as proposed by \citet{kumar2020swift}.

Finally, the true estimation of the gradient is obtained for $T \rightarrow \infty$. Our final approximation for this gradient is by averaging over a finite $T$. As the domain is characterized by fast mixing, we can use a finite, yet sufficiently long, trajectory gradient estimation.

\section{The Challenges of Real World Deployment}\label{sec: real world}

In addition to enabling RL training in this task, our goal is to provide a method that can run in real-time on-device (on a NIC). This requirement presents two limitations. (1) The problem is asyncronous, while the CC algorithm is determining the transmission rate limitation, the flows continue to transmit data. As such, the network must be efficient and fast such that a forward pass can be performed within $\bigO(RTT) = \bigO(\mu sec)$. (2) Each NIC can control thousands of flows. As we require real-time reaction, the algorithm must utilize fast, yet small, memory. Hence, the amount of memory stored per each flow must be minimal.

The proposed architecture is composed of 2 fully connected layers $(in \rightarrow 32 \rightarrow 16)$ followed by an LSTM $(16 \rightarrow 16)$ and a final fully connected layer $(16 \rightarrow 1)$. Such an architecture, combined with ReLU activation functions, enables fast inference using common deep learning accelerators (DLA).

\section{Experiments} \label{sec: experiments}
In prove our method can generalize to unseen scenarios, motivating the use in the real world, we split the scenarios to train and test sets. Training the agents is performed only within the many-to-one domain, see below, on in 3 scenarios: $2 \rightarrow 1, 4 \rightarrow 1$ and $8 \rightarrow 1$. Evaluation is performed on a set of many-to-one, all-to-all, and long-short scenarios.

\textbf{Many-to-one:} this scenario ($N \rightarrow 1$) emulates multiple senders $N$ transmitting data through a single switch $\rightarrow$ towards a single receiver $1$ (\cref{fig: example topology}). We evaluate the agents on $2^i \rightarrow 1$, for $i \in {4, 5, \ldots, 13}$.

\begin{figure}[t]
    \centering
    \begin{subfigure}[b]{0.49\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/128_to_1}
        \caption{128 to 1}
        \label{subfig : 128 to 1}
    \end{subfigure}
    \begin{subfigure}[b]{0.49\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/1024_to_1}
        \caption{1024 to 1}
        \label{subfig : 1024 to 1}
    \end{subfigure}
    \begin{subfigure}[b]{0.49\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/4096_to_1}
        \caption{4096 to 1}
        \label{subfig : 4096 to 1}
    \end{subfigure}
    \begin{subfigure}[b]{0.49\linewidth}
        \centering
        \includegraphics[width=\linewidth]{figures/8192_to_1}
        \caption{8192 to 1}
        \label{subfig : 8192 to 1}
    \end{subfigure}\\~\\
    \includegraphics[width=0.8\linewidth]{figures/legend}
    \caption{Radar plots comparing the various algorithms. Each color represents a single algorithm. The axis are normalized to $[0,1]$, where higher (closer to the edge) is better. While all methods are competitive at low scale (\cref{subfig : 128 to 1}), PCC-RL outperforms at large scale (thousands of flows in parallel).}
    \label{fig: many to 1}
\end{figure}

\textbf{All-to-all:} the all-to-all scenarios emulate multiple servers each transmitting data to all other servers. In this case, given there are $N$ servers, there will also be $N$ switches. All data sent towards server $i$ will route through switch $i$. This setting tests the ability of the various algorithms to cope under increased noise.

\textbf{Long-short:} in addition to testing the ability of the algorithms to converge to a stable point solution, the long-short scenario evaluates the ability of each agent to react to changes -- an elephant/mouse test. A single flow (elephant) transmits an infinite amount of data, while a set of short-flows (mice) randomly interrupt it with a short data transmission. The goal is to test how fast the `elephant' reacts and reduces it's transmission rate to enable the `mice' to transmit their data. Once the short flows are done, the goal of the long flow is to recover quickly to full line rate. We measure the time this entire process takes, from interruption until full recovery.

\subsection{Baseline comparison}

The results for the many-to-one tests are presented graphically in \cref{fig: many to 1} and numerically in \cref{tab: many-to-one comparison}. We observe that while most methods are competitive at small scale (smaller amount of flows, thus less interactions), at large-scale deployments, all methods aside from PCC-RL encounter extensive periods of packet loss. PCC-RL is the only method capable of maintaining high switch utilization, while keeping the latency low and a fairness at a reasonable level.

In addition, we evaluate the various methods on an all-to-all setting. Here, there are $N$ hosts and $2 \cdot N$ (a total of $2N^2$) flows running on each. At each host, flow $i$ constantly sends data towards host $i \;\mathrm{mod}\; N$ through switch $i \;\mathrm{mod}\; N$. The results are presented in \cref{tab: all-to-all comparison}. Although SWIFT performed well at low scale many-to-one tests, when transitioning to the all-to-all setting it is incapable of retaining high switch utilization. Similarly to the many-to-one setting, PCC-RL performs competitively and outperforms at the higher scale setting.

\begin{table}[t]
    \centering
    \begin{tabular}{c||c|c|c||c|c|c}
        \multirow{2}{*}{\textbf{Alg.}} & \multicolumn{3}{c}{\textbf{4 hosts}} & \multicolumn{3}{c}{\textbf{8 hosts}} \\
         & SU & FR & QL & SU & FR & QL \\
        \hline\hline
        \textbf{PCC-RL} & \textbf{94} & \textbf{77} & \textbf{6} & \textbf{94} & \textbf{97} & \textbf{8} \\
        \hline
        \textbf{DC2QCN} & \textbf{90} & \textbf{91} & \textbf{5} & 91 & 89 & 6 \\
        \hline
        \textbf{HPCC} & 71 & 18 & 3 & 69 & 60 & 3 \\
        \hline
        \textbf{SWIFT} & 76 & 100 & 11 & 76 & 98 & 13
    \end{tabular}
    \caption{\textbf{All-to-all:} In these tests, none of the algorithms exhibited packet loss. For 4 hosts, we mark both PCC-RL and DC2QCN as it is up to the end user to determine which outcome is preferred.}
    \label{tab: all-to-all comparison}
\end{table}

Finally, we compared the methods on a long-short setting where the algorithms are tested on their ability to quickly react to changes, presented in \cref{tab: long-short comparison}. Although PCC-RL did not encounter this scenario during training, it was able to generalize and outperform the various baselines on the 2 and 128 flows scenarios -- 1 and 127 interrupting flows (mice) respectively and 1 long flow (elephant). Although at higher scale PCC-RL was outperformed by HPCC and SWIFT, we believe this can be improved in the future by adding a `recovery-time' objective during training.

\begin{table}[t]
    \centering
    \begin{tabular}{c||c|c|c|c}
        \backslashbox{\textbf{Alg.}}{\textbf{Flows}} & \textbf{2} & \textbf{128} & \textbf{1024} & \textbf{2048} \\
        \hline\hline
        \textbf{PCC-RL} & \textbf{6e-7} & \textbf{8e-4} & 3e-2 & 3e-2 \\
        \hline
        \textbf{DC2QCN} & 3e-2 & 5e-2 & \cellcolor{red!40}- & \cellcolor{red!40}- \\
        \hline
        \textbf{HPCC} & 3e-5 & 1e-2 & 2e-2 & 3e-2 \\
        \hline
        \textbf{SWIFT} & 1e-3 & 1e-2 & \textbf{1e-2} & \textbf{2e-2}
    \end{tabular}
    \caption{\textbf{Long-short:} The results represent the time it takes (in $\mu$sec) from the moment the \emph{short} flows interrupt and start sending, until they finish and the \emph{long} flow recovers to full line rate transmission. DC2QCN does not recover fast enough and has thus failed the high scale recovery tests.}
    \label{tab: long-short comparison}
\end{table}

\textbf{Summary:} We observe that PCC-RL is capable of learning a robust policy. Although in certain tasks, e.g., long-short 1024 to 1, the baselines may outperform it, PCC-RL always obtained competitive performance. In addition, when considering large scale scenarios, where thousands of flows interact in parallel, we observed that PCC-RL is the only method capable of avoiding packet loss and thus control the network congestion. As PCC-RL was trained only on a low-scale scenario, it highlights the ability of our method to learn robust behavior that we believe will work in a real datacenter.

\subsection{Selecting the operation point}

As we have seen in \cref{tab: many-to-one comparison,tab: all-to-all comparison,tab: long-short comparison}, the variously algorithms exhibit different behaviors. For instance, as seen in \cref{tab: many-to-one comparison} under the \emph{1024 to 1} evaluation, both DC2QCN and SWIFT obtain good results. It is not clear whether one is absolutely better than the other. The decision depends on the use case. Certain datacenters are very sensitive to latency and would thus prefer DC2QCN, in others this might be less of an issue they they would prefer behavior such as SWIFT that provides higher fairness and bandwidth outputs.

PCC-RL is controlled by a single parameter, the \emph{target}. By changing the target, the behavior deployed within the datacenter is controlled. When the required behavior is low latency, the target value can be taken closer to 0, whereas in a system is less latency sensitive, the user can try higher values which should promote improved fairness and utilization.

\begin{table}
    \centering
    \begin{tabular}{c||c|c|c|c}
        \multirow{2}{*}{\textbf{Target}} & \multicolumn{4}{c}{\textbf{8192 to 1}} \\
         & SU & FR & QL & DR \\
        \hline\hline
        \textbf{Strict} & \textbf{51} & \textbf{99} & \textbf{12} & \textbf{0} \\
        \hline
        \textbf{Standard} & \textbf{92} & \textbf{29} & \textbf{42} & \textbf{0} \\
        \hline
        \textbf{Loose} & \textbf{92} & \textbf{35} & \textbf{55} & \textbf{0}
    \end{tabular}
    \caption{\textbf{Many-to-one:} Numerical comparison of PCC-RL with various selected operation points, where Strict, Standard and Loose refer to targets of 1, 2, and 20 respectively. The Standard results are those presented in the previous section.}
    \label{tab: operation point many-to-one comparison}
\end{table}

\begin{table}[t]
    \centering
    \begin{tabular}{c||c|c|c||c|c|c}
        \multirow{2}{*}{\textbf{Target}} & \multicolumn{3}{c}{\textbf{4 hosts}} & \multicolumn{3}{c}{\textbf{8 hosts}} \\
         & SU & FR & QL & SU & FR & QL \\
        \hline\hline
        \textbf{Strict} & 60 & 30 & 4 & 74 & 52 & 13 \\
        \hline
        \textbf{Standard} & \textbf{94} & \textbf{77} & \textbf{6} & \textbf{94} & \textbf{97} & \textbf{8} \\
        \hline
        \textbf{Loose} & 73 & 18 & 8 & 71 & 21 & 9
    \end{tabular}
    \caption{\textbf{All-to-all:} Numerical comparison of PCC-RL with various selected operation points, where Strict, Standard and Loose refer to targets of 1, 2, and 20 respectively. The Standard results are those presented in the previous section.}
    \label{tab: operation point all-to-all comparison}
\end{table}

\begin{table}[t]
    \centering
    \begin{tabular}{c||c|c|c|c}
        \backslashbox{\textbf{Target}}{\textbf{Flows}} & \textbf{2} & \textbf{128} & \textbf{1024} & \textbf{2048} \\
        \hline\hline
        \textbf{Strict} & 5e-2 & 5e-2 & \cellcolor{red!40}- & \cellcolor{red!40}- \\
        \hline
        \textbf{Standard} & 5e-3 & 5e-3 & \cellcolor{red!40}- & \cellcolor{red!40}- \\
        \hline
        \textbf{Loose} & \textbf{6e-7} & \textbf{8e-4} & \textbf{3e-2} & \textbf{3e-2}
    \end{tabular}
    \caption{\textbf{Long-short:} Numerical comparison of PCC-RL with various selected operation points, where Strict, Standard and Loose refer to targets of 1, 2, and 20 respectively. The Loose results are those presented in the previous section.}
    \label{tab: operation point long-short comparison}
\end{table}

The results in \cref{tab: operation point many-to-one comparison,tab: operation point all-to-all comparison,tab: operation point long-short comparison} present an interesting image on how PCC-RL behaves when the operation point is changed. We compare 3 operation points, `strict', `standard' and `loose', corresponding to targets of 1, 2 and 20 respectively.

As expected, a stricter target (closer to 0) results in lower latency whereas looser results in the opposite. As the agent attempts to maintain a stricter latency profile, it is required to keep the buffer utilization within the switch as low as possible. This results in a dramatic decrease in switch utilization (\cref{tab: operation point long-short comparison}). On the other hand, a looser profile suffers from higher latency but is capable of reacting faster to the appearance of new flows (\cref{tab: operation point long-short comparison}), which can be explained by the new flows joining faster due to the larger ``allowance", and attains better fairness across flows (\cref{tab: operation point many-to-one comparison}).
\section{Summary}

As opposed to hand-crafted algorithms, an RL agent optimizes it's behavior while interacting with the environment. Thus, it is capable of learning the delicate intricacies and maximizes the performance.

Our experiments focused on a realistic network simulator, based on OMNeT++, used to benchmark congestion control algorithms for deployment within real datacenters. We showed that PCC-RL learns a robust behavior, generalizes to unseen domains, and is the only algorithm capable of operating at large scale while avoiding packet loss.

While developing PCC-RL, our focus was an algorithm that can easily be deployed in real datacenters, and potentially even trained on-device. We took into consideration memory and compute requirements and ensured our method not only obtains competitive behavior, but also meets the requirements for running in real-time in a NIC. Also, as datacenters often have different priorities, changing the behavior simply requires selecting a different target. We hope that in the future, PCC-RL will be utilized to improve performance across real datacenters.

\section{Future work}

As this work is the first successful application of RL to a realistic congestion control setting, it can be extended in a multitude of directions. First and foremost, while in \cref{sec: real world} we presented our efforts to making our method deployable on real systems, deploying requires specialized hardware which is not currently available. It is of great interest to see how well this method can generalize and succeed when deployed in real datacenters. Furthermore, deployment success can be greatly improved by successfully quantizing \citep{hubara2017quantized} the neural network.

In addition, our focus was on ``plug-and-play" functionality where we aim to deploy our method in existing networks with minimal changes. As such, we only considered the RTT measurement and the flow transmittion rate, signals that are available to the NIC itself. Future work can take a different angle and look at what novel network features would facilitate faster and better learning.

Finally, the network consists of more than just network-interface-cards. Can congestion be improved further by applying learning paradigms to the switches? This is an important question that has received less focus from the research community.

redpajama_set_name: RedPajamaArXiv


text:
The Library of Congress Has Digitized 155 Persian Texts Dating Back to the 13th Century
Offerings include a book of poetry featuring the epic Shahnameh and a biography of Shah Jahan, the emperor who built the Taj Mahal
Meilan Solly
Associate Editor, History
These illustrations come from a miniature book of classical Persian poetry. Library of Congress, African and Middle East Division, Near East Section Persian Manuscript Collection
A sumptuously illustrated edition of classic Persian poetry by such luminaries as Saadi Shirazi and Jami, a 16th-century gold leaf map found in a volume chronicling the "wonders of creation," and a 17th-century prayer book filled with colorful floral patterns are among the 155 Persian language texts now available via the Library of Congress' online catalogue.
Spanning nearly 1,000 years and encompassing subjects as varied as literature, philosophy, religion, science and history, the newly digitized trove draws on documents from Persian-speaking countries including Iran, Afghanistan and Tajikistan, as well as places like India, Central Asia, the Caucasus and regions formerly controlled by the Ottoman Empire. The texts' wide-ranging origins, in the words of reference specialist Hirad Dinavari, speak to the collection's "diversity and cosmopolitan nature."
"We nowadays are programmed to think Persia equates with Iran, but when you look at this it is a multiregional collection," Dinavari expands in an interview with Atlas Obscura's Jonathan Carey. "It's not homogenous, many contributed to it. Some were Indian, some were Turkic, Central Asian. Various people of various ethnic groups contributed to this tradition."
According to a press release, the digital catalogue includes a copy of the Shahnameh, an epic exploration of pre-Islamic Persia consisting of 62 stories divided into 990 chapters of 50,000 rhyming couplets, and a biography of Shah Jahan, the 17th-century Mughal emperor best known for building the Taj Mahal. Materials written in multiple languages, including Arabic and Turkish, are also present.
The manuscripts were digitized in honor of Persian New Year, or Nowruz Library of Congress, African and Middle East Division, Near East Section Persian Manuscript Collection
A depiction of animal handlers in James Skinner's history of the Indian caste system Library of Congress, African and Middle East Division, Near East Section Persian Manuscript Collection
Another highlight is the History of the Origin and Distinguishing Marks of the Different Castes of India, an 1825 text penned by James Skinner, a Scottish-Indian lieutenant-colonel who served in the British military. The "voluminous treatise," according to the Perso-Indica database, focuses on the Indian caste system, as influenced by profession and religious order, and is based on Sanskrit sources later translated into Persian.
Dinavari tells Carey that the book is a prime example of the "cultural fusion" fostered by Persian's widespread use. (As a 2014 Library of Congress exhibition titled A Thousand Years of the Persian Book pointed out, Persian was once a lingua franca, or common cultural language, across a diverse swath of Asia and the Middle East.) Although the majority of Skinner's work details the tribes, traditions and professions of Hindu India, it is still a Persian text—albeit one peppered with terms more commonly heard in the regional vernacular of India. The volume is even more unusual for its emphasis on everyday locals' lives over the exploits of those at the top of society.
Much of the LOC's rare Persian language collection stems from the efforts of Kirkor Minassian, an antiquities dealer and collector who specialized in Islamic and Near-Eastern artifacts and procured texts for the library during the 1930s. Since that time, the LOC has acquired a small number of additional manuscripts at auction, as well as through donations.
According to the library's website, researchers from the Near East Section, spurred the popularity of a 2014 exhibition of more than 40 rare manuscripts and lithographic books, began digitizing the LOC's Persian texts in 2015.
The collection's digital debut was timed to coincide with Persian New Year, or Nowruz, which takes place during the vernal equinox and marks the dawning of a new chapter in life.
Manuscripts representing the majority of texts are currently available online. Assorted lithographs, early imprint books and Islamic book bindings are set to follow over the next several months.
Meilan Solly | | READ MORE
Meilan Solly is Smithsonian magazine's associate digital editor, history.
Asian History Books Libraries Library of Congress Literature Middle East

redpajama_set_name: RedPajamaCommonCrawl


Boost your vitamin intake and mood with this collection of vibrant and delicious winter blends using kale, broccoli, zucchini and more.
Accidentally getting jalapeño or chili juice in the eye is no laughing matter. Here are a few remedies to help ease the pain.
Have you tried cooking with this delicious white crumbly cheese that forms the heart and soul of the Greek kitchen? See how versatile is, from salads to bakes.
A lot of work goes into the $100 24k gold donut from Brooklyn's Manila Social Club. Check out how this decadent treat is made.


text:
Q: Proof that the inverse of an analytic function is analytic which uses only real analysis. I would like to prove the following result:
Let $f:R\to R$ be such that $f(x)=\sum\limits_{k=0}^\infty a_k(x-c)^k$ for all $x$ in some open set $O\subset R$.
Suppose that $f'(c)\ne 0$.
Then there is a function $g:T\to R$ such that $g(f(x))=x$ and $g(x)=\sum\limits_{k=0}^\infty b_k(x-f(c))^k$, where $T$ is an open set of $R$ containing $f(c)$.
I know how to prove that $f$ is locally invertible but not how to prove that its inverse can be written as a power series on some open set.
There are many proofs for the inverse function theorem which use complex analysis but I would like to write a proof which uses only real analysis. I have been writing a book of proofs of many of the important theorems in calculus and I don't want to introduce complex analysis for the proof of one theorem.
Thanks, Andrew Murdza.

A: This proof works over any Cauchy-complete field $\mathbb F$ with a non-trivial absolute value. The basic examples are $\mathbb R$ (real numbers), $\mathbb C$ (complex numbers), $\mathbb Q_p$ ($p$-adic numbers), and $\mathbb F_p((X))$ (formal Laurent series over a finite field). In fact $\mathbb F$ necessarily contains one of these four as a sub-field, but we don't need to know that.

After shifting/translating (which is invertible and preserves analyticity), we can assume that $c=f(c)=0$. Likewise, after scaling, we can assume that $f'(c)=1$, but let's not do that yet.
$$f(x)=\sum_{n\geq1}a_nx^n,\quad|x|\leq R$$
$$\sum_{n\geq1}|a_n|R^n<\infty$$
$$g(y)=\sum_{n\geq1}b_ny^n,\quad|y|\leq S$$
$$\sum_{n\geq1}|b_n|S^n<\infty$$
We're given $a_n$ and $R>0$, and we want to find $b_n$ such that $g(f(x))=x$ for all $x$ in some neighbourhood of $0$ and $f(g(y))=y$ for all $y$ in some neighbourhood of $0$.
Expanding the formal power series $g(f(X))$ (as shown at the bottom of this answer) and equating to the identity series $e(X)=X$, we can see that the required coefficients $b_n$ exist and are unique. Each one is a polynomial in $a_k$ for $k\leq n$, with integer coefficients, divided by some power of $a_1\neq0$. Thus $f$ has a unique formal left-inverse: $g\circ f=e$. By similar reasoning, $f$ has a unique formal right-inverse: $f\circ h=e$. Since composition of formal power series is associative, we have $g=g\circ e=g\circ(f\circ h)=(g\circ f)\circ h=e\circ h=h$, so the two inverses are in fact the same.
We only need to show that $g(y)$ converges for some $y\neq0$. That is, we need to find $S>0$ such that $\sum_n|b_n|S^n<\infty$.

Wolfram MathWorld gives this formula for the coefficients:
$$b_n=\sum_{k_2+2k_3+3k_4+\cdots=n-1}\frac{(2k_2+3k_3+4k_4+\cdots)!}{n!\;k_2!\;k_3!\;k_4!\;\cdots}\left(\frac{1}{a_1}\right)^n\left(-\frac{a_2}{a_1}\right)^{k_2}\left(-\frac{a_3}{a_1}\right)^{k_3}\left(-\frac{a_4}{a_1}\right)^{k_4}\cdots$$
I've reformulated it somewhat. I haven't verified it, but we can look for a proof of this formula elsewhere.
That factorial stuff looks like a fraction, which might be undefined if $\mathbb F$ has positive characteristic. But we've already seen that the coefficients are integers, so that's not a problem. We can evaluate the fraction in $\mathbb Q$, getting a result in $\mathbb N$, and then map that to $\mathbb F$ in the natural way.
Now let's scale $f$ so that $a_1=1$. (This simplifies the formula, but may actually make it harder to interpret and to verify, which is why I haven't done it yet.)
Applying the triangle inequality:
$$|b_n|\leq\sum_{k_2+2k_3+3k_4+\cdots=n-1}\frac{(2k_2+3k_3+4k_4+\cdots)!}{n!\;k_2!\;k_3!\;k_4!\;\cdots}|a_2|^{k_2}|a_3|^{k_3}|a_4|^{k_4}\cdots$$
$$\sum_{n\geq1}|b_n||y|^n\leq\sum_{n\geq1}\;\sum_{k_2+2k_3+3k_4+\cdots=n-1}\frac{(2k_2+3k_3+4k_4+\cdots)!}{n!\;k_2!\;k_3!\;k_4!\;\cdots}|y|^n|a_2|^{k_2}|a_3|^{k_3}|a_4|^{k_4}\cdots$$
Rearranging the series, and eliminating the variable $n$:
$$=\sum_{k_2,k_3,k_4,\cdots}\frac{(2k_2+3k_3+4k_4+\cdots)!}{(1+k_2+2k_3+3k_4+\cdots)!\;k_2!\;k_3!\;k_4!\;\cdots}|y|^{1+k_2+2k_3+3k_4+\cdots}|a_2|^{k_2}|a_3|^{k_3}|a_4|^{k_4}\cdots$$
Introducing a new variable $m$:
$$=\sum_m\sum_{k_2+k_3+k_4+\cdots=m}\frac{(2k_2+3k_3+4k_4+\cdots)!}{(1-m+2k_2+3k_3+4k_4+\cdots)!\;k_2!\;k_3!\;k_4!\;\cdots}|y|^{1-m+2k_2+3k_3+4k_4+\cdots}|a_2|^{k_2}|a_3|^{k_3}|a_4|^{k_4}\cdots$$
Distributing factors of $|y|$:
$$=\sum_m|y|^{1-m}\sum_{k_2+k_3+k_4+\cdots=m}\frac{(2k_2+3k_3+4k_4+\cdots)!}{(1-m+2k_2+3k_3+4k_4+\cdots)!\;k_2!\;k_3!\;k_4!\;\cdots}|a_2y^2|^{k_2}|a_3y^3|^{k_3}|a_4y^4|^{k_4}\cdots$$
Breaking off the $m=0$ term, where the inner sum reduces to a single term with $k_2=k_3=k_4=\cdots=0$:
$$=|y|+\sum_{m\geq1}|y|^{1-m}\sum_{k_2+k_3+k_4+\cdots=m}\frac{(2k_2+3k_3+4k_4+\cdots)!}{(1-m+2k_2+3k_3+4k_4+\cdots)!\;k_2!\;k_3!\;k_4!\;\cdots}|a_2y^2|^{k_2}|a_3y^3|^{k_3}|a_4y^4|^{k_4}\cdots$$
Rewriting the factorial stuff with a binomial coefficient, and using $\binom nk\leq\sum_k\binom nk=2^n$:
$$=|y|+\sum_{m\geq1}|y|^{1-m}\sum_{k_2+k_3+k_4+\cdots=m}\binom{2k_2+3k_3+4k_4+\cdots}{m-1}\frac{(m-1)!}{k_2!\;k_3!\;k_4!\;\cdots}|a_2y^2|^{k_2}|a_3y^3|^{k_3}|a_4y^4|^{k_4}\cdots$$
$$\leq|y|+\sum_{m\geq1}|y|^{1-m}\sum_{k_2+k_3+k_4+\cdots=m}2^{2k_2+3k_3+4k_4+\cdots}\frac{(m-1)!}{k_2!\;k_3!\;k_4!\;\cdots}|a_2y^2|^{k_2}|a_3y^3|^{k_3}|a_4y^4|^{k_4}\cdots$$
Distributing factors of $2$:
$$=|y|+\sum_{m\geq1}|y|^{1-m}\sum_{k_2+k_3+k_4+\cdots=m}\frac{(m-1)!}{k_2!\;k_3!\;k_4!\;\cdots}\big(2^2|a_2y^2|\big)^{k_2}\big(2^3|a_3y^3|\big)^{k_3}\big(2^4|a_4y^4|\big)^{k_4}\cdots$$
$$=|y|+\sum_{m\geq1}\frac{|y|^{1-m}}{m}\sum_{k_2+k_3+k_4+\cdots=m}\frac{m!}{k_2!\;k_3!\;k_4!\;\cdots}\big(2^2|a_2y^2|\big)^{k_2}\big(2^3|a_3y^3|\big)^{k_3}\big(2^4|a_4y^4|\big)^{k_4}\cdots$$
Applying the multinomial formula:
$$=|y|+\sum_{m\geq1}\frac{|y|^{1-m}}{m}\Big(2^2|a_2y^2|+2^3|a_3y^3|+2^4|a_4y^4|+\cdots\Big)^m$$
$$=|y|+\sum_{m\geq1}\frac{2^2|y|^{1+m}}{m}\Big(2^0|a_2y^0|+2^1|a_3y^1|+2^2|a_4y^2|+\cdots\Big)^m$$
The parenthesized series will converge if $|y|\leq R/2$, since we assumed $\sum_n|a_n|R^n$ converges:
$$=|y|+\sum_{m\geq1}\frac{2^2|y|^{1+m}}{m}\left(\sum_{n\geq2}|a_n|(2|y|)^{n-2}\right)^m$$
$$\leq|y|+\sum_{m\geq1}\frac{2^2|y|^{1+m}}{m}\left(\sum_{n\geq2}|a_n|R^{n-2}\right)^m$$
$$=|y|+4|y|\sum_{m\geq1}\frac{1}{m}\left(|y|\sum_{n\geq2}|a_n|R^{n-2}\right)^m$$
The outer series will converge if $|y|<1/\sum_{n\geq2}|a_n|R^{n-2}$, since the Taylor series of the real logarithm is $-\ln(1-u)=\sum_{m\geq1}u^m/m$ for $-1<u<1$:
$$=|y|-4|y|\ln\left(1-|y|\sum_{n\geq2}|a_n|R^{n-2}\right)$$
$$<\infty$$
Thus, we've shown that $g(y)$ converges absolutely, with a radius at least
$$S=\min\left\{\frac{R}{2},\;\frac{0.99R^2}{\sum_{n\geq2}|a_n|R^n}\right\}.$$

redpajama_set_name: RedPajamaStackExchange


May be useful for the material if it doesn't fit the window. Don't have exact measurements as can't find tape measure!
Two large pallets available to collect from Pollokshields - daytime only. Please email or text.


Don't give away your power.
Many times, I've witnessed women giving their power away to men. You start out strong, confident and having a full life, and as the relationship progresses, you wake up one day realizing you haven't spoken to your friends in months and your whole life revolves around supporting the man in your life.
You're guilty of having unhealthy boundaries in a relationship, and it can start affecting your life completely.
Boundaries are very useful because they act like a "fence" protecting your property. When you have healthy boundaries set up that you stick to, it gives you more freedom to make better choices in your romantic life.
Another advantage to understanding how to set healthy boundaries is that they help you easily recognize others who take personal responsibility and respect themselves, making it easier to zero in and attract (or stay with) the right mate.
Many women I coach often arrive at my doorstep with a history of unhappiness and dissatisfaction in their past relationships. We end up finding out that they've compromised too much at some point to avoid being alone.
After a lifetime of dating and having relationships, I have experienced all of these at one time or another. In one relationship I had many years ago, I fell head over heels in love and suddenly found myself in category 3. He was gorgeous, charming and romantic... and had divorced 4 times, was emotionally unavailable and still had feelings for his last wife.
It was so frustrating because we had a very real, deep connection but he kept (emotionally) going in and out of the relationship. He'd be all in, and then suddenly all out.
What I needed and desired was his full attention and commitment, but I kept settling by forgiving his behavior in the hopes he would eventually change. After a year of this, I finally rose up and honored myself by ending the relationship.
To get you started on the road to honoring yourself, the first thing to do is throw out the list you have of your perfect partner.
Athletic, perfect body, powerful leader, wealthy, ambitious, fun, witty, smart... I have yet to meet anyone who got divorced or broke up because their partner wasn't witty enough!.
Instead, I want you to get clear and make a list with your preferences and the character of your ideal partner. An example of a character trait would be how they treat themselves, you and those around them. Do they treat waiters like servants or do they treat everyone with respect and a sense of equality?
Your preferences are your common interests, goals, and values. When you list those out, think about what you could live with in another person and also what you could not live without.
Just like the relationship I was in years ago, I got clear that I couldn't live without a man being emotionally ready for a relationship. Look at your past or present relationships for clues.
This article was originally published at AttractYourKing. Reprinted with permission from the author.