This adorable 2 bedroom 1 bath home is centrally located in Lecompte near shopping and restaurants. This home has large bed rooms with ample space. It qualifies for Rural development and is perfect for the first time home buyer.
Listing provided courtesy of Richard C. Hayes of Key Realty Llc.


At Piccolo Marketing, we understand that our friends in small business have dreams and goals just as big as large corporations. However, we're left to dreaming big and achieving our goals with smaller budgets and less manpower. Fortunately, social media, when leveraged properly, can be a low-cost tool that delivers results in less time than a dedicated department.
Consider the extent to which social media, digital programming, and basic technology impacts your everyday personal life. You can use those same principles in your marketing strategy and turn your social media into a productive member of your team.
As a small business owner, you likely started your company with a goal or passion at the center of your business model. Let your light shine! Strive to show customers your passion for your business, service, or industry. You can show your spirit by giving back and setting yourself up as a trusted source for the latest tips or industry news.
In terms of actual posting tips, don't get lost behind the business name. Be funny. Be serious. Be sad. Be YOU. When it comes down to it, people are drawn to people, not names or products.
While the delivery of the product or service matters, experience is just as important to the customer. Would you rather work with the friendly small business owner down the street who you pass and greet walking your dog each morning or the faceless name on the other side of the phone you reached after guessing your way through telephone prompts? I'm guessing the former.
The customer experience is more than just the face to face time you spend with a customer. It's the feeling they get when they enter your store or their reaction to a post you shared on Facebook. Ensure your social media lines up with the experience the customer will receive once you get the opportunity to make a sale.
I repeat. Be. Mobile. Friendly.
It's no longer acceptable that your digital presence is NOT mobile friendly. How likely are you to search for something on your phone versus a tablet or desktop computer? It's important your online presence is transferrable, presentable, and easy to navigate across a variety of platforms.
Need help to organize, implement, and/or maintain your social media strategy? Give us a call at Piccolo Marketing at 615-348-7768 to get started today!
For more small business tips on how to use social media to boost sales, click here.


text:
Q: C# Web API : using web service for multiple databases I created a web service project to provide data to a C# Winforms project. The service transfers data from datacontext created with Linq to SQL Server with controllers.
In SQL Server, the database is rebuilt for each customer and each customer connects to its own database. For this reason, the connection string changes for each client.
The situation that gets complicated for about 30 databases is as follows:

*

*Should I set up a web service for each client?

*Does a single service serve more than one connection string?

*If you suggest a single service path, will too many requests cause performance issues?

*If you suggest a single service path, how to redirect to the database name in the service?

What are your suggestions?

redpajama_set_name: RedPajamaStackExchange


Smoke Alarms Australia recently contributed to Zoom Real Estate Burwood's 2018 Holiday Food Drive. This was a great initiative run by Zoom Real Estate at the end of last year that raised an incredible 244.5kgs worth of Food items, which they donated to Food Bank NSW & ACT to help provide over 489 meals to those less fortunate over the holidays.
Congratulations and thanks to Zoom Real Estate for letting us get involved in such a great cause!


text:
package org.hscieripple.patient.appointments.model;

import java.io.Serializable;
import java.util.Date;
import java.util.List;

public class HSCIEAppointmentDetails implements Serializable {

    private String source;
    private String sourceId;
    private String author;
    private Date dateCreated;
    private String serviceTeam;
    private Date dateOfAppointment;
    private Date timeOfAppointment;
    private String location;
    private String status;

    public String getSource() {
        return source;
    }

    public void setSource(String source) {
        this.source = source;
    }

    public String getSourceId() {
        return sourceId;
    }

    public void setSourceId(String sourceId) {
        this.sourceId = sourceId;
    }

    public String getAuthor() {
        return author;
    }

    public void setAuthor(String author) {
        this.author = author;
    }

    public Date getDateCreated() {
        return dateCreated;
    }

    public void setDateCreated(Date dateCreated) {
        this.dateCreated = dateCreated;
    }

    public String getServiceTeam() {
        return serviceTeam;
    }

    public void setServiceTeam(String serviceTeam) {
        this.serviceTeam = serviceTeam;
    }

    public Date getDateOfAppointment() {
        return dateOfAppointment;
    }

    public void setDateOfAppointment(Date dateOfAppointment) {
        this.dateOfAppointment = dateOfAppointment;
    }

    public Date getTimeOfAppointment() {
        return timeOfAppointment;
    }

    public void setTimeOfAppointment(Date timeOfAppointment) {
        this.timeOfAppointment = timeOfAppointment;
    }

    public String getLocation() {
        return location;
    }

    public void setLocation(String location) {
        this.location = location;
    }

    public String getStatus() {
        return status;
    }

    public void setStatus(String status) {
        this.status = status;
    }
}

redpajama_set_name: RedPajamaGithub


ACCA in its last installment of a report on late payment, known as "Ending Late Payment", clearly stated that change is required from officials of SME financing on the way they manage and control late payments and trade credits.
As per ACCA, policy makers of government and its officials have a strong responsibility to make a balanced financial infrastructure under which they are able to minimize the concept of late payment and boost trade credits around the globe.
ACCA stated that this requires to be improved as early as possible because late payment has been seriously damaging the overall business. The damage is higher in small business as compared to large corporates. According to an estimate this issue is double in small businesses.
ACCA's policy advisor for global SME, RosanaMirkovic said that delayed payment is a part of business life and is a type of a credit. In a perfect world, late payment presents a controllable risk to organizations.
He continued to argue that there is a need to re think the habits of late payment. Policymakers by improving effectiveness and efficiency of the courts can encourage the people to avoid late payments and provide an easy access to trade credit.
The report highlights how buyers and suppliers can improve this situation, by interacting with each other more closely and by improving client / customer relationships. From buyer's perspective, they can protect themselves by fulfilling the clauses of contract and from supplier's point of view a pattern of due diligence should be followed.
1. There should be clear and unbiased information in agreement between buyers and suppliers.
2. All the invoices should be monitored until the payment is cleared.
3. The cost and results of providing credit to buyers should be kept in mind by suppliers.
4. Both the parties need to build a relationship of understanding, integrity and honesty.
6. Businesses need to improve their internal structures related to disputed matters that could result in late payment.


text:
Home » News » Announcements » Welcome to the new Hampton Roads Transportation Accountability Commission website!
Welcome to the new Hampton Roads Transportation Accountability Commission website!
Added on September 24, 2014
by HRTAC Staff
This website was designed to provide information about transportation projects in Hampton Roads to the public.
On March 8, 2014, the General Assembly adopted HB1253/SB513, which provides for the creation of the Hampton Roads Transportation Accountability Commission (HRTAC) in order to manage the Hampton Roads Transportation Fund (HRTF) revenues for the Hampton Roads region. Signed into law on April 3, 2014 by Governor Terry McAuliffe, the HRTAC became effective on July 1, 2014. Learn more about HRTAC >
Comprised of locally elected officials, the commission has the organizational structure and leadership in place to determine how new regional money - approximately $200 million annually - will be invested in transportation projects. See who the HRTAC Members are >
This website will be continuously updated with new information. Please browse the following areas that may interest you:
See what's on the Meetings & Events Calendar >
Stay up to date with HRTAC News >
Learn how to Get Involved >
Read about HRTAC in the News >

redpajama_set_name: RedPajamaCommonCrawl


The Expert Advisor works on the basis of determining the signal patterns of the ZigZag indicator of various tamframes (multi-patterns).
The combination of the descending patterns of the three timeframes is a signal for opening a long position due to the delay rules. Simply put, when the market has already won back the fall, the indicator patterns only indicated this fall.
Recommended trading tools: 15m: GBPUSD, EURGBP, EURUSD, EURJPY, GBPJPY, USDCAD.
Current settings (default) are universal for recommended trading instruments.
To work correctly, the advisor requires a low spread and the use of a remote server (VPS).
Comment to orders - Comment on opened orders.


To dream of a carcass signifies the complete ending of an era in your life. There is no coming back or retreating from the fact that it is now over. Acceptance is the key to making the most of this ending. Dreaming of a carcass that moves or speaks implies the inability to accept that it is time to move on.


text:
\section{Introduction}

Routing in wireless ad-hoc networks is a complex problem involving sequential
decisions in each hop in order to build up a route that optimizes certain
network utilities. Due to the lack of centralized control, routing in wireless
ad-hoc networks needs to be performed in a distributed manner. Starting from
the source node, the selection of next node in each hop requires considerations
of multiple often conflicting factors. Most existing routing algorithms assume
that the network has a fixed set of links and aim at network-layer objectives
such as congestion control, minimizing transmission delay, or maximizing
successful transmission probability. These existing approaches cannot be readily
applied to the wireless ad-hoc network setting, because they do not account for
physical-layer considerations such as the flexibility for each node to connect
to different neighboring nodes in a shared wireless medium, the mutual interference
between the transmission links, and most importantly the abilities for each link
to utilize different spectrum slots. Recognizing this gap, this paper advocates a
joint routing and spectrum access protocol in the \emph{physical layer}, taking
into account the channel strengths and the interference levels in different
spectrum slots. As these factors are largely determined by the geographical
locations of the mobile nodes (assuming a path-loss model for wireless propagation),
this motivates us to consider a new approach to routing based on
the locations of each node and its neighbours and the physical-layer
characteristics of the surrounding wireless environment.

As routing amounts to successive selections of nodes and spectrum slots in each
hop, the number of possibilities in the overall optimization problem is
combinatorial in nature, therefore finding the globally optimal route by
exhaustive search is computationally
prohibitive in large networks. Decentralized discrete optimization algorithms
previously proposed for routing \cite{gafni, DSDV, murthy, spine, shiguang,
tse} are mostly heuristic in nature.  From a different perspective, routing
involves sequential decision making, in which each decision is a function of
the current state of the network. Thus, routing can be modeled as a Markov
decision process \cite{markov} and naturally fits into the realm of
reinforcement learning; see \cite{DRL_survey} for a comprehensive survey of the
applications of reinforcement learning to communications and networking.  In
this direction, many previous works \cite{qrouting, choi, chang, forster,
santhi, alharbi, varun, leonardo, rahul} have employed the classical
\emph{Q-learning} \cite{qlearning} algorithm to train agents to find the
optimal route. In most of these works, a distinct agent is
associated with each transmission node. Each agent maintains a list of
reachable neighbors and a $Q$-value table of ``fitness scores" for these
neighbors as the potential next hop (e.g. time or number of hops to reach the
destination). After training, the route is collectively determined by all the
agents, each selecting the neighbor with the highest fitness scores as the next
hop.  Moreover, \cite{drlnetworking} explores the use of deep reinforcement
learning \cite{DQN} as a complement to a proposed heuristic routing approach.
Finally, \cite{nurmi} focuses on stochastic routing, where each node is
characterized by a forwarding probability depending on its selfishness and
remaining energy.  Reinforcement learning is used on each node to come up with
the forwarding probability.

Collectively, the approaches in the above existing works in using reinforcement
learning on routing for ad-hoc wireless networks are restricted by all or at
least some of the following modeling and design choices:
\begin{enumerate}[label=\textbf{C.\arabic*}]
    \item The network model assumes a fixed set of connections where two nodes are either reachable, or not. \label{enum:2}
    \item A distinct agent is associated with every node in the network and is trained specifically for this node.\label{enum:1}
    \item The same network must be used for both training and testing.  \label{enum:3}
\end{enumerate}
Each of these modeling and design choices imposes limitations and has drawbacks.
\ref{enum:2} abstracts away pivotal physical-layer characteristics of the network,
and cannot account for objectives such as signal-to-noise-and-interference (SINR)
based quality-of-service (QoS). \ref{enum:1} requires the number of agents to
match to the network size, therefore limiting scalability and generalization capability.
\ref{enum:3} further limits agents' abilities to adapt to network changes.
Attempting to address the limitation of \ref{enum:3}, \cite{chang} let agents
\emph{explore} all new neighbors opportunistically, which is essentially
retraining.  But as the layout of an ad-hoc network can frequently change, any
solution designed under \ref{enum:3} is inherently insufficient. Further, to
address the limitation of \ref{enum:1}, \cite{lspirouting} proposes the use of
a specific $Q$-function estimator \cite{lspi} for the agent, allowing the same
agent to be applied to different nodes. But, none of these works perform
routing based on physical-layer attributes.

In this work, we recognize the critical importance of making routing decisions
based on the physical-layer characteristics of the network, and moreover the
importance of optimizing spectrum access jointly with routing decisions. This is
because the allocation of which frequency bands to transmit for a particular
link significantly affects the interference levels of all nearby links,
including the previous and subsequent links in the same flow.  While spectrum
access has been the focus in many previous work in wireless network optimization,
including conventional optimization algorithms \cite{dual,yiping,songgao},
supervised learning \cite{peiliang}, and reinforcement learning \cite{qingzhao,mosleh},
all these works assume fixed wireless connections, thus do not make routing
decision jointly with spectrum optimization. While there exists
conventional optimization based works that focus on both routing and spectrum
access \cite{chunsheng,yongk,talay}, these works do not take advantage of
learning based method. We mention here that \cite{sudeep} explores
learning automata based optimization over wireless sensor networks that
includes both routing and spectrum access, but its focus is exclusively on
energy conservation.

The main contribution of this paper is a novel and scalable deep reinforcement
learning approach to simultaneous routing and spectrum access for ad-hoc
wireless networks based on physical-layer inputs, constraints, and objectives.
We focus on routing as the main task, and solve the spectrum access problem
along the routing process by training an reinforcement agent to determine the
suitabilities of different frequency bands for each link. For routing, as a main novelty, we
associate one agent to each \emph{data flow} from the source node to the destination.
The agent executes sequential node-selection tasks along the route.
The fact that the agent takes physical-layer information as input is crucial --- this allows
the same agent to be used for all nodes along the route, thus providing scalability.
Furthermore, we show
that the same model parameters for the agent can also be shared across
distinct data flows, or even generalized across different ad-hoc networks.
Finally, the agent can also
be designed to adapt to network characteristics such as the density of
neighbors.

The rest of the paper is organized as follows. Section~\ref{sec:problem}
establishes the system model and formulates the optimization problem.
Section~\ref{sec:method} proposes a deep reinforcement learning based approach
for routing and spectrum access in wireless ad-hoc networks. The performance
of the proposed method is analyzed in Section~\ref{sec:results}. As extensions,
Section~\ref{sec:delay} incorporates delay into the objective;
Section~\ref{sec:powercontrol} performs power control to further improve the data rate.
Finally, conclusions are drawn in Section~\ref{sec:conclusion}.

\section{Routing in Ad-hoc Wireless Network}\label{sec:problem}

\subsection{Network Model}

Consider a wireless ad-hoc network supporting $F$ data flows, each with a fixed
source and a fixed destination node, and routed through multiple hops over
a subset of $N$ intermediary relay nodes in the network. We assume that there
are $B$ available frequency bands each of bandwidth $W$ Hz,
and each hop uses one of the $B$ frequency bands for signal transmission and reception. We use $\mathcal{F}$ to
denote the set of data flows; $\mathcal{S}$ and $\mathcal{T}$ to denote the set
of sources and destinations, respectively; $\mathcal{N}$ to denote the set
of relay nodes; and $\mathcal{B}$ to denote the set of available frequency
bands, with $|\mathcal{F}|=F$, $|\mathcal{N}|=N$, and $|\mathcal{B}|=B$.

To utilize reinforcement learning, the environment around each node
needs to be stationary. To this end, we characterize the ad-hoc
network through a node density distribution function. Specifically,
we assume the density of neighboring nodes is uniform locally across a sub-region.

\subsection{Routing and Spectrum Access}

The task of routing is to select an ordered list of relay nodes
for each flow $f\in\mathcal{F}$. The task of spectrum access is to select a
frequency band in $\mathcal{B}$ for each hop. Mathematically, we
represent the route for flow $f$ as an ordered list denoted by
$\mathbf{n}^{(f)}$:
\begin{align}\label{equ:nodelist}
	\mathbf{n}^{(f)}=(n^{(f)}_0, n^{(f)}_1, n^{(f)}_2\dots n^{(f)}_h, n^{(f)}_{h+1})
\end{align}
where $ \mathbf{n}^{(f)}[0] = n^{(f)}_0 = s_f\in\mathcal{S}$ represents the source
node for $f$,
$ \mathbf{n}^{(f)}[h+1] = n^{(f)}_{h+1}=t_f\in\mathcal{T}$ represents the destination
node for $f$, and
$ \mathbf{n}^{(f)}[i] = n^{(f)}_i \in\mathcal{N}, i=1,\cdots,h$ are the relay nodes.
We represent the spectrum
access for flow $f$ as an ordered list denoted by $\mathbf{b}^{(f)}$, containing
the frequency bands of each hop
\begin{align}\label{equ:bandlist}
    \mathbf{b}^{(f)}=(b^{(f)}_0, b^{(f)}_1, b^{(f)}_2\dots b^{(f)}_h)
\end{align}
where $\mathbf{b}^{(f)}[j] = b^{(f)}_j \in\mathcal{B}, j=0,\cdots,h$.

Given $\mathbf{n}^{(f)}$ and $\mathbf{b}^{(f)}$, we have a complete description of how the data flow $f$ is connected for data transmission, as following:
\begin{align}\label{equ:dataflow}
    n^{(f)}_0\xrightarrow{b^{(f)}_0}n^{(f)}_1\xrightarrow{b^{(f)}_1}n^{(f)}_2\xrightarrow{b^{(f)}_2}\dots\to n^{(f)}_h\xrightarrow{b^{(f)}_h}n^{(f)}_{h+1}
\end{align}
In establishing $f$, there are $h$ routing decisions for selecting the nodes
for the $h$ hops,
as well as $h+1$ spectrum access decisions for selecting the $h+1$ transmission bands.
Collectively, these routing and spectrum access decisions overall the flows are denoted as:
\begin{subequations}\label{equ:solset}
\begin{align}
	\mathbf{N}_{\mathcal F} =  \{\mathbf{n}^{(f)}, \ \forall f\in\mathcal{F}\}, \\
	\mathbf{B}_{\mathcal F} =  \{\mathbf{b}^{(f)}, \ \forall f\in\mathcal{F}\}.
\end{align}
\end{subequations}

We restrict each hop of any data flow to use only one frequency band, which is already implicitly reflected in (\ref{equ:dataflow}).
We do not allow a node to transmit and receive in the same frequency band, but
we do allow for the possibility that an intermediary node can serve as a relay for
multiple data flows, as long as all of its incoming and outgoing
hops occupy distinct frequency bands.
Mathematically, this means for a flow $f$ with $h$ relays
\begin{align}
	b^{(f)}_i \neq b^{(f)}_{i+1}, \quad \forall i = 0, 1, \cdots, h-1;
\end{align}
and if $n_{i+1}^{(f_1)} = n_{j+1}^{(f_2)}$ for some $f_1, f_2 \in \mathcal{F}$, then
\begin{align}
	b^{(f_1)}_i \neq b^{(f_2)}_{j+1}, {\rm \ \ \ and\ \ \ \ } b^{(f_2)}_j \neq b^{(f_1)}_{i+1}.
\end{align}
Finally, to eliminate loops in routing (which is never helpful), we explicitly
enforce that each data flow can visit an intermediary node at most once
\begin{align}
    n^{(f)}_i\neq n^{(f)}_j, \quad \forall i\neq j.
\end{align}

\subsection{Performance Metrics}\label{sec:physicalobjectives}

At the physical layer, the maximum transmission rate between a pair of nodes
over a frequency band is characterized by the capacity of the underlying
wireless channel, which is a function of the signal-to-noise-and-interference
(SINR) at the receiver.
In details, consider a link with transmitting node $i\in \mathcal{S} \cup \mathcal{N}$
and receiving node $j \in \mathcal{T} \cup \mathcal{N}$, over
the frequency band $b \in \mathcal{B}$ with bandwidth $W$, with $p_i$ as the
transmit power of node $i$ and
$\sigma^2$ as the background noise power in each frequency band, and
$h_{ij,b}\in\mathcal{C}$ as the channel strength from node $i$ to node $j$
in the band $b$, the maximum transmission rate of this link is related to
the SINR as follows:
\begin{subequations}\label{equ:rate}
\begin{align}
\text{SINR}_{(i,j,b)} =& \frac{|h_{ij,b}|^2p_ix_{i,b}}{\sum_{\substack{k\neq i,j \\ k\in \mathcal{S} \cup \mathcal{N}}}|h_{kj,b}|^2 p_kx_{k,b} + \sigma^2} \label{equ:SINR} \\
R_{(i,j,b)} =& W\log\left(1+\text{SINR}_{(i,j,b)}\right) \label{equ:instantRate}
\end{align}
\end{subequations}
The binary variable $x_{i,b}\:(i\in\mathcal{S}\cup\mathcal{N}, b\in\mathcal{B})$ indicates whether the node $i$ is transmitting on the band $b$ or idle, and is determined by the routing and spectrum access solution (i.e. if node $i$ is used by any route to transmit over the band $b$ or not). The mapping from $\mathbf{n}^{(f)}$ and $\mathbf{b}^{(f)}$ to $x$ is:
\begin{align}
    x_{i,b}=
    \begin{cases}
	    1 \quad\text{if}\quad\exists f,k: \mathbf{n}^{(f)}[k]=i {\ \rm and\ } \mathbf{b}^{(f)}[k]=b \\
    0 \quad\text{otherwise.}
    \end{cases}
\end{align}

We assume full-buffered transmission in which the data is continuously
transmitted along the route. In this case, the overall data rate for each
flow is determined by its \emph{bottleneck} link capacity (i.e. the
minimum data rate among all its links). Specifically, consider the
flow $f$ with its routing and spectrum access specified as in (\ref{equ:dataflow}),
the overall data rate of the flow $f$ is
\begin{align}\label{equ:bottleneckrate}
	R^{(f)} = \min_{i=0,1,2,\dots,h}R_{(n^{(f)}_i, n^{(f)}_{i+1}, b^{(f)}_i)}.
\end{align}
The concept of the bottleneck link is visually illustrated in Fig.~\ref{fig:bottleneck}.
We denote the set of the overall data rates of all flows to be
\begin{align}\label{equ:rateset}
	\mathbf{R}_{\mathcal F} = \{R^{(f)}, \ \forall f\in\mathcal{F}\}.
\end{align}

\begin{figure}
    \centering
    \ifOneColumn
        \includegraphics[width=0.55\textwidth]{Figures/Figure_Routing_Bottleneck.pdf}
    \else
        \includegraphics[width=0.4\textwidth]{Figures/Figure_Routing_Bottleneck.pdf}
    \fi
    \caption{Bottleneck links (marked with thin arrows) due to strong interference (marked with dotted lines) from nearby links}
    \label{fig:bottleneck}
\end{figure}

\subsection{Optimization Problem Formulation}

The objective for optimal routing and spectrum access is to determine the
set of solutions as in (\ref{equ:dataflow}) for each data flow in order to
optimize some global objectives across all the flows in the network. The
global objective is called network utility, denoted as $U(\cdot)$, which
is a function of the data rates of all the flows (e.g. the sum data rates
or the minimum data rate).
Mathematically, the optimization problem is formulated as follows:
\begin{subequations}\label{equ:problem}
\begin{align}
	\underset{{\mathbf N}_\mathcal{F},{\mathbf B}_\mathcal{F}} {\text{maximize}}\quad &  U({\mathbf R}_\mathcal{F}) \\
    \text{subject to}\quad& (\ref{equ:nodelist}),(\ref{equ:bandlist}),(\ref{equ:solset})-(\ref{equ:rateset}).
\end{align}
\end{subequations}
Note that the data rates of different links both within each flow and across the flows have strong inter-dependencies
due to the mutual interference.  Specifically, if two links share the same frequency
band, then the transmitted signal of one link appears as interference for the other
link. Thus, the spectrum allocation problem is tightly coupled with the routing problem.
Consequently, there is strong trade-off among the data rates of all flows in the network.

Optimizing just a single flow is already challenging: it is a discrete
optimization problem and is non-convex due to the interference among the links
within the route as in (\ref{equ:rate}). Optimizing multiple flows is even more
so due to the interactions between them. The approach taken in this paper is to
let a single reinforcement learning agent optimize one flow at a time by moving
along the frontier nodes of the flow. Then, we optimize the next flow, and do
so successively, in effect letting flows compete with each other. As shown later in
the paper, this would already lead to highly effective solutions to the problem
(\ref{equ:problem}) for common network utilities such as the sum rate or the minimum rate.

\section{Flow based Deep Reinforcement Learning for Routing and Spectrum Access}
\label{sec:method}

The ad-hoc network routing and spectrum access problem can be modeled as a
Markov decision process, but because the distributed agents can only observe
local information, the problem of distributed routing and spectrum access
across multiple data flows belongs to the class of \emph{multi-player partially
observable Markov decision process} (MP-POMDP).  In this section, we propose
a novel reinforcement learning approach to tackle this challenging problem.

\subsection{Single Agent for Multiple Nodes in a Flow}\label{sec:assoc}

A key innovation of this paper as compared to prior work is to associate a
single reinforcement learning agent to all the nodes in the flow.
The idea is that as a route is being established on a hop-by-hop basis,
the agent moves along with the frontier node of the partially
established route to decide the best next hop as well as the best frequency
band to reach the next node from the frontier node. Compared to most prior
approaches that train a different agent for each node in the network to perform
either routing or spectrum access, this \emph{agent-to-flow} association has
the key advantage that it significantly reduces the number of parameters that
need to be trained, because a single agent now works for all nodes in the flow.
This improves scalability, and further also improves generalization ability
because the same agent can be used across multiple data flows and even
across distinct ad-hoc networks.
Fig.~\ref{fig:agent} provides a visualization of the proposed
agent-to-flow association for establishing the next hop in at a frontier
node of a flow.

When optimizing multiple flows, we adopt a \emph{sequential} ordering for
establishing routes and spectrum allocation: i.e., we let one flow establish
its entire route and frequency band selections before the next flow, and do so
over multiple rounds to allow the interference pattern to be fully established.

\begin{figure}
    \centering
    \ifOneColumn
        \includegraphics[width=0.55\textwidth]{Figures/Figure_Routing_Agent.pdf}
    \else
        \includegraphics[width=0.4\textwidth]{Figures/Figure_Routing_Agent.pdf}
    \fi
    \caption{A reinforcement learning agent at the frontier node of the flow determines the best next-hop and the best frequency band based on state values S1 to S4 for each of $c$ neighbors in each of $B$ frequency bands.}
    \label{fig:agent}
\end{figure}

\subsection{Single Agent for Multiple Frequency Bands}\label{sec:spectrum_access}

We aim to develop a reinforcement learning agent to simultaneously perform
routing and spectrum access for all the hops in each flow. One
natural design for the agent is to aggregate features from all frequency bands,
and to decide the optimal \emph{next node and frequency band} combination.
This, however, significantly increases the resulting state and action spaces
for the agent.
Furthermore, this design does not allow the agent to be generalized to
scenarios with different number of frequency bands.

To reduce the state space and action space and to achieve higher model parameter
efficiency, we propose an approach of designing an agent for a single frequency
band, then replicating it across $B$ bands.
The agent evaluates the suitability of the next node for all frequency bands, one band at a time,
then selects the best node and frequency band combination.
This process is illustrated in Fig.~\ref{fig:routemultibands}. In essence, the
agent achieves equivalent state space and action space over all the frequency
bands, with parameter cost of just one band.

\begin{figure}
    \centering
    \ifOneColumn
        \includegraphics[width=0.7\textwidth]{Figures/Figure_Routing_Multiple_Bands.pdf}
    \else
        \includegraphics[width=0.48\textwidth]{Figures/Figure_Routing_Multiple_Bands.pdf}
    \fi
    \caption{The same reinforcement learning agent is reused across all the frequency bands. The optimal next-hop node and frequency combination is determined from the outputs of the agent.}
    \label{fig:routemultibands}
\end{figure}

\subsection{Action Space and State Space}\label{sec:stateactionspace}

We define the action space and the state space for the
reinforcement learning agent for one frequency band. The same
agent is reused across all $B$ frequency bands. Within each frequency band, the
agent's action corresponds to selecting a next-hop candidate node to transmit.
As this paper adopts a physical-layer network model in which the connections
are not predetermined, we consider a fixed number of $c$ available neighbors with
strongest channels to the frontier node as candidates. In addition, to ensure sufficient exploration
capability, we also add one \emph{reprobe} action, which means that if the
agent decides none of the $c$ strongest neighbors is a suitable next hop (in
the current frequency band), it will proceed to probe the next $c$ strongest
neighbors, until a suitable next hop is found. Thus, over a single frequency band,
the agent's action space $\mathcal{A}$ consists of $c+1$ actions: one
action for each of the $c$ strongest neighbors, and one action for reprobing.
Note that if the agent chooses to reprobe, the agent stays at the current frontier node in the current time.
At the next time step, the agent will explore the next $c$ closest neighbors and collect the state information corresponding to these new set of neighbors.

The agent's state space $\mathcal{S}$ over one frequency band should summarize
crucial factors about each of the $c$ strongest neighbors in that frequency
band as the agent moves along the frontier node. We adopt a pathloss based network
model in which the SINR is a function of the geographical locations of the
neighbors relative to the frontier node and to the destination, as well as the
interference each neighbor is subjected to. To this end, the agent gathers the
following information from each neighbor to form its state:
\begin{enumerate}[label=\textbf{S.\arabic*}]
\item Distance between the neighbor and the frontier node.\label{state:1}
\item Distance between the neighbor and the destination.  \label{state:2}
\item Angle difference between the direction from the frontier node to the neighbor and the direction from the frontier node to the destination. \label{state:3}
\item Total interference the neighbor is exposed to in the current frequency band.\label{state:4}
\end{enumerate}
Here, \ref{state:2} and \ref{state:3} indicate the amount of progress toward the
destination node that can be made if the neighbor is chosen as the next hop, while
\ref{state:1} and \ref{state:4} together allow an estimation of the SINR to the
neighbor in the current frequency band. In total, the state over one frequency band is a $4c$-component vector. Fig.~\ref{fig:agent} shows the four state features for each of the $c$ closest neighbors.

Before we complete the picture by defining the agent's reward, we next provide a brief overview of reinforcement learning, and illustrate why the standard \emph{Q-Learning} needs to be modified in order to tackle the routing problem.

\subsection{Conventional Q-Learning}
\label{sec:qlearning}

Q-learning \cite{qlearning} is an off-policy, \emph{value-based} reinforcement
learning algorithm for \emph{model-free} control. Q-learning is suited for
routing and spectrum access for wireless ad-hoc networks, because:
\begin{itemize}
\item Being a value-based algorithm, Q-learning estimates a value for each
state-action pair. In the routing problem, each action corresponds to a
combination of next hop node and frequency band selection. These actions can be
quantitatively evaluated and compared based on the overall network objective
(such as the bottleneck rate of the flow). The value-based Q-learning algorithm
then allows these quantities to be mapped and learned depending on the
state-action values.
\item Being a model-free algorithm, Q-learning allows us to bypass explicit
modeling and directly determine the optimal policy. This is useful, because in our setting, the state of the network is only partially observed by the agent, which makes the state transitions of the network difficult to model.
\end{itemize}
The Q-learning algorithm is based on the following definition of the
\emph{action value} denoted as $Q$:
\begin{align}
     Q_\pi(s_t, a_t) =\:&\mathbb{E}_{\pi}[r_t+\gamma r_{t+1}+\gamma^2 r_{t+2}+\dots \ | \ s_t,a_t] \label{equ:futurerewardmontecarlo} \\
     =\:& \mathbb{E}_{\pi}[r_t + \gamma Q_\pi(s_{t+1},a_{t+1}) \ | \ s_t,a_t] \label{equ:futurerewardbootstrap}
\end{align}
where $r_t$ is the reward at time $t$ and $\gamma$ is a discount factor.
Here, $Q_\pi(s_t, a_t)$ is the expected \emph{future cumulative reward} after the agent executes the action $a_t$ from the state $s_t$, with the expectation taken under certain policy $\pi$.
Given an estimate of $Q_\pi(s_t, a_t)$, one can improve the policy by acting greedily with respect to $Q_\pi(s_t,a_t)$, i.e.,
\begin{align}\label{equ:greedyimprov}
    \pi'(s_t) = \text{argmax}_a Q_\pi(s_t, a).
\end{align}
We can then re-estimate the values of $Q_\pi'(s_t, a_t)$ through either
(\ref{equ:futurerewardmontecarlo}) or (\ref{equ:futurerewardbootstrap}),
and continue to improve the policy using (\ref{equ:greedyimprov}).
When the process converges, we reach an optimal policy $\pi^*$
(and its corresponding value functions $Q_{\pi^*}(s_t, a_t)$).
Such a process is referred to as the \emph{policy iteration}, and
the optimal policy takes on the following self-reference relation:
\begin{align}
    \pi^*(s_t) = \text{argmax}_a Q_{\pi^*}(s_t, a).
\end{align}

When estimating $Q$ (which is commonly referred to as policy evaluation), there are two common approaches, each with its own benefits and shortcomings:
\begin{enumerate}
    \item Monte-Carlo Estimation, corresponding to (\ref{equ:futurerewardmontecarlo}), which provides a unbiased estimation, but has high variance;
    \item Temporal Difference Estimation, corresponding to (\ref{equ:futurerewardbootstrap}), which has low variance, but is biased due to its self-referential nature.
\end{enumerate}

The standard Q-learning algorithm performs the policy evaluation with temporal difference estimation, and updates the $Q$ estimation based on the Bellman optimality equation \cite{bellman} as follows:
\ifOneColumn
    \begin{align}
        Q^{\rm new}_{\pi}(s_t, a_t) = Q^{\rm old}_{\pi}(s_t, a_t) +
    	\alpha(r_t+\gamma \max_{a} Q^{\rm old}_{\pi}(s_{t+1}, a)-Q^{\rm old}_{\pi}(s_t, a_t)), \label{equ:updateactionvalue}
    \end{align}
\else
    \begin{multline}
        Q^{\rm new}_{\pi}(s_t, a_t) = Q^{\rm old}_{\pi}(s_t, a_t) + \\
    	\alpha(r_t+\gamma \max_{a} Q^{\rm old}_{\pi}(s_{t+1}, a)-Q^{\rm old}_{\pi}(s_t, a_t)), \label{equ:updateactionvalue}
    \end{multline}
\fi
where $\alpha$ is the step size.
To accelerate the algorithm, Q-learning only performs (\ref{equ:updateactionvalue}) once before the policy improvement in each round of the policy iteration.
(One step of the policy evaluation in a policy iteration process is also known as the \emph{value iteration} process.)

There are, however, two major issues when applying the standard Q-learning
algorithm to the routing and spectrum access problem for the wireless ad-hoc
network. First, the conventional Q-learning assumes a reward definition $r_t$ as
the \emph{immediate} reward that the agent receives after executing action $a_t$
in state $s_t$. These immediate rewards form the cumulative reward $Q(s_t, a_t)$,
which is the objective that the agent aims to optimize. In the routing problem, the objective
(\ref{equ:problem}) is determined by the bottleneck rates in the flows. Decomposing
these bottleneck rates into a set of immediate rewards is, however, not straightforward.

Secondly, while the reward definition can be modified to account for the optimization
of bottleneck rate, the standard training process for Q-learning, which is based on
temporal difference estimation, is known to suffer from high biases. In our experience for
training the reinforcement learning agent for routing with the bottleneck rate as
the reward, the learned agent's state-action
value estimations are noticeably off with large biases throughout the
state space, rendering the agent incapable of producing high quality solutions
for routing or spectrum access.

\subsection{Novel Reward and Action Value Estimation}\label{sec:reward}

To apply Q-learning to the routing problem, we need first to incorporate the bottleneck rate into the reward
definition. The most intuitive way to do so is to set the reward to be all zero
along the route until the last hop, which takes a reward value of (\ref{equ:bottleneckrate}),
i.e., the minimum rate across all the hops in the route. However, this definition
has the drawback that such a reward and the resulting Q-values for one hop could
well be determined by an earlier hop, making them independent of the agent's
choice of actions. Consequently, this makes it impossible for the agent to interpret
the rewards during training.

To simultaneously address the problem of decomposing the bottleneck rate to
immediate rewards as well as the problem of biased estimation for temporal
difference state-action value estimation, this paper proposes a novel structure
for rewarding the agent. Instead of identifying an instantaneous reward $r_t$
then computing $Q$ as the cumulative sum of $r_t$, we directly assign a future
cumulative reward for each state-action pair. Such a new definition of future
cumulative reward shares the same semantic meaning as the $Q$ function in
conventional Q-learning, but is not a result of cumulative instantaneous rewards.
The main advantage of this approach is that it makes the optimization of network
objectives (such as the bottleneck rate in our case) that cannot be easily
decomposed as sum of instantaneous rewards possible.

Specifically, at any given node, we choose to use the \textbf{bottleneck link
SINR from that node \textit{onwards} in the route} as its future cumulative
reward.  Note there is a one-to-one relationship between rate and SINR.
More precisely, consider the data flow $f\in\mathcal{F}$ established in
(\ref{equ:dataflow}). During training, at frontier node $n^{(f)}_t, (0\leq
t\leq h)$, the agent observes the state $s_t$ and takes the action $a_t$
(leading to node $n^{(f)}_{t+1}$ using frequency band $b^{(f)}_t$).
After the route is fully established, the state-action pairs along the route
are then assigned a future cumulative rewards, which we denote as $\widetilde{Q}$,
defined as follows:
\begin{align}\label{equ:qhat}
    \widetilde{Q}(s_t, a_t) = \min_{i=t\dots h}\text{SINR}_{(n^{(f)}_i, n^{(f)}_{i+1}, b^{(f)}_i)}.
\end{align}
We emphasize that $\widetilde{Q}$ plays the same role as $Q$ value in
Q-learning in that it incorporates future information (and no past information)
and is used as the metric for determining the optimal actions.

In the works on classical Q-learning \cite{qlearning} and deep Q-learning
\cite{DQN,dueling,doubledqn}, the target for $Q(s_t,a_t)$ is computed based on
the temporal difference estimation with bootstrapping, which can cause high
biases as already mentioned. Instead, the $\widetilde{Q}$ defined in
(\ref{equ:qhat}) can be computed directly based on Monte-Carlo estimation,
which is unbiased. This turns out to be a second crucial advantage.

We do note that Q-Learning is not originally designed to work with Monte-Carlo
estimation. The above unconventional algorithm design, when used in conjunction with
\emph{experience replay} for deep reinforcement learning,
does give rise to one slight technical difficulty: each time the $\widetilde{Q}$ is
updated, it corresponds to the policy evaluation of a slightly older version
of policy (following which the samples are collected to run Monte-Carlo
sampling). In the next section, we provide a variation to the experience
replay technique to address this difficulty.

\subsection{Deep Q-Learning Based Routing and Spectrum Access}
\label{sec:dqnmodel}

Since the states are continuous variables, the agent needs to be able to
generalize over unseen portions of the state space $\mathcal{S}$. To this end,
we utilize deep reinforcement learning, specifically deep Q-learning
\cite{DQN}, in which a neural network, namely \emph{DQN}, is trained to predict
$Q$ values (in our case, the $\widetilde{Q}$ values) given the state-action
inputs.

We format each state input for a single frequency band as a vector of length
$4c$ (four features for each of the $c$ strongest neighbors). These input
vectors are processed with fully connected layers with non-linearities.  We
adopt the state-of-the-art \emph{dueling-DQN} network architecture
\cite{dueling} for our agent, as illustrated in Fig.~\ref{fig:DDQN}.  The
dueling DQN consists of two separate estimators: one for estimating the state
value and the other for estimating the action advantage. Such a
separation has been shown to be beneficial for the agent to better learn the distinctions between actions.

\begin{figure}
    \centering
    \ifOneColumn
        \includegraphics[width=0.55\textwidth]{Figures/Figure_Dueling_DQN_Agent.pdf}
    \else
        \includegraphics[width=0.48\textwidth]{Figures/Figure_Dueling_DQN_Agent.pdf}
    \fi
    \caption{Dueling-DQN Structure}
    \label{fig:DDQN}
\end{figure}

The DQN is trained following the experience-replay technique as in \cite{DQN},
with uniform sampling from the replay buffer. The agent follows the
\emph{$\epsilon$-greedy} \cite{epsilongreedy} policy to gather experiences for
the replay buffer. More specifically, with probability $1-\epsilon$, the agent
acts greedily according to the agent's current estimation of $\widetilde{Q}$:
the action with the highest $\widetilde{Q}$ value across the $B$ frequency
bands is selected (as in Fig.~\ref{fig:routemultibands}).  With
probability $\epsilon$, the agent selects a random node as the next hop and a
random frequency band to reach that node.  If the selected node is not within
the $c$ strongest neighbors to the agent, we store a reprobing transaction.
Together, this allows us to produce a series of agent transactions
for routing from the source to the destination.
Once a route is fully established, we compute the rewards as defined by the
onward bottleneck SINR for each node in the route, and store the corresponding
(state, action, reward) tuple in the replay buffer, which is used for subsequent DQN training.
Note that since the agent is trained to work in one frequency band at a time,
the state is a $4c$-vector and the action is a value in $\{1, \dots, c+1\}$.
In case of greedy exploitation, we store the state vector for the frequency
band in which the highest $\widetilde{Q}$ is achieved; in case of random
exploration, we store the state vector for the randomly chosen frequency band.
The same is true for the reprobe transactions.
The entire process of DQN training with experience replay is illustrated in Fig.~\ref{fig:replay}.

\begin{figure}
    \centering
    \ifOneColumn
        \includegraphics[width=0.75\textwidth]{Figures/Figure_Routing_Experiences.pdf}
    \else
        \includegraphics[width=0.5\textwidth]{Figures/Figure_Routing_Experiences.pdf}
    \fi
    \caption{Training of Dueling-DQN with experience replay}
    \label{fig:replay}
\end{figure}

We also propose a slight variation at the end of the training process with
experience replay to address the technical difficulty of using Monte-Carlo
estimation for Q-learning. Normally, when training using the $\epsilon$-greedy policy,
the agent always has a small (but non-zero) $\epsilon$ value, even at the end
of the training process. To ensure that the policy evaluation uses samples
from the most up-to-date policy, we reset $\epsilon=0$ at the end of training,
and let the agent further collect and train for an extended period
of time, (which we refer to as the \emph{extended training}). With $\epsilon=0$,
the agent's policy reduces to the most up-to-date greedy policy.
Correspondingly, Monte-Carlo estimation based on samples generated in this manner
provides a policy evaluation for the most up-to-date policy. This would then allow
the policy iteration process to produce the final optimal policy for the agent.

To showcase the generalization ability of our method, we only train the agent
on one specific data flow during experience gathering.
Specifically, we first use a simple \emph{closest-to-destination
among the strongest neighbors} heuristic to produce routes of all but one data flow. We then
assign the agent to the last flow to gather
experiences and conduct training. As shown later in Section~\ref{sec:results},
such lightweight training strategy is already sufficient to produce an agent that
generalizes well across all data flows as well as across different ad-hoc networks.
It does not need to be re-trained when the network topology changes, unlike in methods such as in \cite{chang}.

\subsection{Agent Policy Enhancements}\label{sec:enhance}
From domain knowledge, we propose two enhancements to the agent's policy:
\begin{itemize}
    \item If the agent chooses a neighbor that does not have the strongest channel to the frontier node, then all neighbors with stronger channels to the frontier node are excluded for future consideration for the next hop.
    \item If the destination node appears within the agent's exploration scope (i.e. one of the $c$ strongest neighbors), then the agent would not choose any neighbor which has a weaker channel to the frontier node than the destination node.
\end{itemize}
Both enhancements effectively prevent agents from taking non-essential back-and-forth hops.

\subsection{Multi-Agent Distributed Optimization}\label{sec:rewardimp}

The agent learns and acts according to $\widetilde{Q}$ in (\ref{equ:qhat}) in order
to optimize the data rate of its own flow. If instead we optimize with respect to a global reward as function of multiple flows, we can encounter the
\emph{credit mis-assignment} problem \cite{localreward}. We observe, however, that in a wireless
ad-hoc network, where the major performance limiting factor is the mutual
interference, route optimization in a flow tends to lead to interference
avoidance which benefits other data flows as well. Therefore in our problem,
designing the agent to act selfishly in multiple rounds actually allows every
agent to adjust according to each other's routes, thus achieving \emph{implicit
cooperation}.

\subsection{Fairness Among Data Flows}\label{sec:fairness}
In many applications, fairness among the data flows is an important design objective.
We empirically observe that if routing and spectrum access are established
sequentially across the data flows in multiple rounds, the data flows established
relatively later tend to have better bottleneck rates. The reason is that
the later flows can adjust according to the interference levels and utilize
spectrum more efficiently. Therefore, to promote fairness among the data flows,
we propose the following: At the end of each round, we compute the bottleneck
rates for all data flows, then optimize routing and spectrum for the data flows in
the next round in the decreasing order of the bottleneck rates. Flows with weak bottleneck rates
in the previous round obtain the advantage of being able adjust their routes and
frequency bands later in the next round, and as a result achieve better data rates.

\section{Simulation Results}\label{sec:results}

\subsection{Ad-hoc Network Setup}\label{sec:origin_setting}
We consider wireless ad-hoc networks in a 500$\times$500$m^2$ region with $F=3$ data flows. The data flows are to be routed with $B=8$ available frequency bands, with each band having 5MHz bandwidth. The node density profile is specified over nine equally divided sub-regions, with $(6, 8, 7, 6, 5, 10, 8, 9, 6)$ nodes located randomly within each sub-region. We consider the short-range outdoor model ITU-1411 with a distance-dependent path-loss to model all wireless channels, over all frequency bands at 2.4GHz carrier frequency. All antennas have a height of 1.5m and 2.5dBi antenna gain. We assume a transmit power level of 30dBm for all nodes; the background noise level is at -130dBm/Hz.

We randomly generate 290,000 ad-hoc network layouts to train our agent. Among these, 20,000 layouts are used for random exploration based routing to generate the agent's initial experience, 250,000 layouts are used for the $\epsilon$-greedy policy based routing to train the agent, and the remaining 20,000 layouts are used for extended training with $\epsilon$ set to 0.
For testing, we randomly generate 500 new ad-hoc network layouts according to the same distribution.

\subsection{Agent and Reward Specification}\label{sec:trainingspecs}

We use $c=10$ as the number of the strongest neighbors the agent explores each time. Correspondingly, the inputs to the dueling-DQN are 40-component vectors. The state inputs are processed by sequential fully-connected layers, organized into a state-value prediction branch and an action-advantage prediction branch, as in \cite{dueling}. The specifications for the neural network are summarized in Table~\ref{tab:neuralnetspecs}.

\begin{table}[t]
\caption{Design Parameters for the DDQN Agent}
\centering
\begin{tabular}{|l|l|c|}
\hline
\textbf{Parameters} & \multicolumn{2}{c|} {\textbf{Number of Neurons}}  \\
\hline
\multirow{2}*{\shortstack[l]{Initial Main-Branch\\ fully-connected layers}}
& 1st & 150 \\ \cline{2-3}
& 2nd & 150 \\
\hline
\multirow{2}*{\shortstack[l]{State-Value \\ fully-connected layers}}
& 1st & 100 \\ \cline{2-3}
& 2nd & 1 (1 state value) \\
\hline
\multirow{2}*{\shortstack[l]{Action-Advantage \\ fully-connected layers}}
& 1st & 100 \\ \cline{2-3}
& 2nd & 11 (11 actions) \\
\hline
\end{tabular}
\label{tab:neuralnetspecs}
\end{table}

This paper uses the bottleneck SINR onwards in the route as the
future cumulative reward $\widetilde{Q}$ assigned to each state-action pair.
For practical training, we express SINR in dB scale in order to have a more
suitable range of values. Further, we add a constant bias to ensure that the
reward is almost always positive. This is so that this reward definition can be
extended to the case in which a discount factor related to delay can later be added.
Specifically, in actual training, we pre-process the SINR in the definition
of the $\widetilde{Q}$ function as follows:
\begin{align}\label{equ:normalizesinr}
    \widetilde{Q}(s_t, a_t) = \min_{i=t\dots h}
	10\log_{10}\left(\text{SINR}_{(n^{(f)}_i, n^{(f)}_{i+1}, b^{(f)}_i)}\right) + \mathrm{bias}.
\end{align}
This pre-processing step makes the reward much easier to learn.

\subsection{Sum-Rate and Min-Rate Performance}\label{sec:sumrate}

We present the sum-rate and min-rate testing results for deep reinforcement learning agent (i.e., the dueling-DQN, or DDQN) as compared to a comprehensive list of benchmarks. These benchmarks are greedy in nature, since to our knowledge there currently do not exist efficient globally optimal algorithms for ad-hoc network routing based on physical-layer attributes. The benchmarks are as following:
\begin{itemize}
    \item \textbf{Strongest neighbor: } Select the neighbor with the strongest wireless channel from the frontier node.
    \item \textbf{Best direction among strongest neighbors:} Select the neighbor with the best next hop direction (i.e. the smallest angle offset between the agent-to-neighbor direction and the agent-to-terminal direction).
    \item \textbf{Closest to destination among strongest neighbors:} Select the neighbor closest to the destination node. If all neighbors are further away from the destination compared with the agent, the agent will reprobe.
    \item \textbf{Least interfered among strongest neighbors: } Select the neighbor exposed to the lowest amount of interference.
    \item \textbf{Highest data rate among strongest neighbors: } Select the neighbor with the highest link capacity.
    \item \textbf{Destination direct: } Directly route from the source to the destination node.
\end{itemize}
These benchmarks incorporate a comprehensive set of important criteria, including but not limited to all the state features \ref{state:1}-\ref{state:4}.

We follow the multi-round sequential routing for all the methods.
In simulations, two rounds appear to be sufficient already. For the DDQN agent,
spectrum access is solved as in Section~\ref{sec:spectrum_access}. For all the
benchmarks, we select the best frequency bands based on the state feature \ref{state:4}.
The sum-rate and min-rate results are summarized in
Table~\ref{tab:sumrate}. The CDF curves of sum-rate results are presented in
Fig.~\ref{fig:sumrate}. As shown, our proposed method achieves significantly
better performance than all benchmarks in both sum-rate and min-rate
performances. We emphasize that this performance is achieved by re-using a
single agent across all data flows, across all frequency bands, and over all
testing network layouts.

\begin{table}[t]
	\caption{Average Sum-Rate and Min-Rate Performances (Mbps)}
\centering
\ifOneColumn
    \begin{tabular}{|c|c|c|}
    \hline
    \textbf{Methods} & \textbf{Sum Rate} & \textbf{Min Rate} \\[2pt]
    \hline
    DDQN agent & 4.90 & 0.619 \\
    \hline
    Closest-to-destination among strongest neighbors & 1.76 & 0.238 \\
    \hline
    Best-direction among strongest neighbors & 2.92 & 0.352 \\
    \hline
    Largest-data-rate among strongest neighbors & 0.44 & 0.023 \\
    \hline
    Strongest neighbor & 0.43 & 0.007 \\
    \hline
    Lowest-interference among strongest neighbors & 0.55 & 0.040 \\
    \hline
    Destination direct & 0.02 & 0.003 \\
    \hline
\else
    \begin{tabular}{|p{0.235\textwidth}|c|c|}
    \hline
    \textbf{Methods} & \textbf{Sum Rate} & \textbf{Min Rate} \\[2pt]
    \hline
    DDQN agent & 4.90 & 0.619 \\
    \hline
    \shortstack[l]{\strut Closest-to-destination among \\ strongest neighbors} & 1.76 & 0.238 \\
    \hline
    \shortstack[l]{\strut Best-direction among \\ strongest neighbors} & 2.92 & 0.352 \\
    \hline
    \shortstack[l]{\strut Largest-data-rate \\ among strongest neighbors} & 0.44 & 0.023 \\
    \hline
    Strongest neighbor & 0.43 & 0.007 \\
    \hline
    \shortstack[l]{\strut Lowest-interference among \\ strongest neighbors} & 0.55 & 0.040 \\
    \hline
    Destination direct & 0.02 & 0.003 \\
    \hline
\fi
\end{tabular}
\label{tab:sumrate}
\end{table}

\begin{figure}
    \centering
    \ifOneColumn
        \includegraphics[width=0.7\textwidth]{Figures/sumrate_CDF}
    \else
        \includegraphics[width=0.49\textwidth]{Figures/sumrate_CDF}
    \fi
	\caption{Cumulative distribution function of sum rate over 3 flows in 500 networks generated with original settings.}
    \label{fig:sumrate}
\end{figure}

To better understand how our method excels, we provide a visualization of routes formed by the DDQN agent together with selected benchmarks over a random ad-hoc network layout in Fig.~\ref{fig:routes}. As shown, our agent intelligently spreads out all data flows as well as switching frequency bands for transmission to mitigate mutual interference, while still maintaining strong and properly directed links to the destination to form the routes.

\begin{figure}
    \centering
    \includegraphics[width=0.48\textwidth]{Figures/routes.eps}
    \caption{Routes achieved on three data flows (Data flows are differentiated by colors, while frequency bands are differentiated by line styles).}
    \label{fig:routes}
\end{figure}

To further illustrate this point, in Table~\ref{tab:spectrumefficiency}, we
show the spectral efficiency in bps/Hz achieved when the number of available
frequency bands varies from 2 to 8. It is clear that the DDQN agent is able to
achieve higher spectrum efficiencies when larger number of frequency bands are available.
These results show that the DDQN agent is able to intelligently utilize all
frequency resources and mitigate interference by spreading transmission powers
across different frequency bands.

\begin{table}[t]
    \centering
    \caption{Spectrum Efficiency Analysis for DDQN Agent}
    \begin{tabular}{|c|c|c|}
        \hline
        $B$ & Sum Rate (bps/Hz) & Min Rate (bps/Hz) \\
        \hline
        2 & 0.033 & 0.003 \\
        \hline
        4 & 0.078 & 0.008 \\
        \hline
        8 & 0.123 & 0.015 \\
        \hline
    \end{tabular}
    \label{tab:spectrumefficiency}
\end{table}

\subsection{Generalization Performance}

To test the generalizability of the proposed method, we directly take the agent
trained under the original setting and reuse it for a much larger ad-hoc network of $F=10$ data flows in a 5000$\times$5000$m^2$ region, with $B=32$ available frequency bands. We place larger number of $(19, 16, 21, 18, 14, 24, 17, 20, 19)$ nodes over nine evenly divided sub-regions. The sum-rate results are shown in Fig.~\ref{fig:sumrate_large}, which shows that our approach still significantly outperforms the benchmarks.

We do note that if we increase the number of relays nodes more aggressively, the performance of the DDQN agent drops below the strongest benchmarks tested. The potential reason is due to the fact that the DDQN agent learns to prioritize shorter links. Using shorter links is beneficial for obtaining higher direct-channel strengths, yet also leads to more active nodes and thus more interference within the entire wireless network. As the DDQN agent is trained under the network settings as in Section~\ref{sec:origin_setting}, it can achieve the right balance for optimal performances. But, when the network settings are drastically changed, the trade-off that the agent learns by utilizing relatively shorter links might not have the right balance.

Nevertheless,
the results presented in this subsection still illustrate that the proposed method has the capability to adapt to networks that are much larger in size, in the number of mobile nodes, and in the number of frequency bands, than the networks used for training.

\begin{figure}
    \centering
    \ifOneColumn
        \includegraphics[width=0.7\textwidth]{Figures/sumrate_CDF_large}
    \else
        \includegraphics[width=0.49\textwidth]{Figures/sumrate_CDF_large}
    \fi
	\caption{Cumulative distribution function of sum rate over 10 flows in 500 large-scale networks.}
    \label{fig:sumrate_large}
\end{figure}

\subsection{Optimal Relaying Distance}

The heart of the routing and spectrum access problem is to
achieve the right balance between using nearby nodes as relays (which have
strong channels) versus directly transmitting to farther-away nodes (which results in
fewer number of transmissions in the network hence lower overall interference).
The DDQN agent learns this balance in a data driven fashion.

For a network with a fixed node density, it is conceivable that a heuristics
can be designed based on identifying an optimal relay distance, so that it can
approach the performance of DDQN.  Indeed, Table~\ref{tab:linklength} shows the
performance of the ``closest-to-destination'' benchmark, but the set of
neighbors to explore is varied from 2 to 25. We see that the sum-rate and
min-rate performances are the best when 4 neighbors are explored, which
correspond to a medium relaying distance.  Although this benchmark still does
not outperform DDQN, its performance comes close.  The point is, however, that
that this optimal number of neighbors to explore is difficult to determine in
advance, while the proposed DDQN agent is able to find it through training.

\begin{table}[t]
    \centering
    \caption{Closest-to-Destination Benchmark with Different Number of Neighbors Explored}
    \begin{tabular}{|c|c|c|}
    \ifOneColumn
        \hline
        \# of Neighbors Explored & Sum Rate (Mbps) & Min Rate (Mbps) \\
        \hline
        2 & 4.02 & 0.430\\
        \hline
        4 & 4.56 & 0.517\\
        \hline
        6 & 3.79 & 0.455\\
        \hline
        8 & 2.67 & 0.311\\
        \hline
        10 & 1.76 & 0.238 \\
        \hline
        15 & 0.89 & 0.119 \\
        \hline
        25 & 0.33 & 0.050\\
        \hline
    \else
        \hline
        \shortstack[c]{\strut \# of Neighbors \\ Explored} & \shortstack[c]{Sum Rate \\ (Mbps)} & \shortstack[c]{Min Rate \\ (Mbps)} \\
        \hline
        2 & 4.02 & 0.430\\
        \hline
        4 & 4.56 & 0.517\\
        \hline
        6 & 3.79 & 0.455\\
        \hline
        8 & 2.67 & 0.311\\
        \hline
        10 & 1.76 & 0.238 \\
        \hline
        15 & 0.89 & 0.119 \\
        \hline
        25 & 0.33 & 0.050\\
        \hline
    \fi
    \end{tabular}
    \label{tab:linklength}
\end{table}

In the same vein, it is also conceivable that an optimal spectrum access
heuristic can be designed. For example, we found empirically that if the agent
always chooses the
frequency band with the least amount of interference, it would generally work
very well. In fact, the DDQN agent chooses the frequency band with the least
interference about 70\% of the time. Nevertheless, the proposed Q-learning
based approach is still justified, because there is no easy way to come up with
these heuristics, especially when the choice of the frequency band needs to be
coupled with the choice of the next hop. Further, without DDQN, it is not
straightforward to evaluate the optimality of these heuristics.

\subsection{Fairness Among Flows}

In the proposed DDQN, we encourage fairness between data flows by ordering
the optimization of the flows in the reverse order of the data rates
achieved in the previous round. To validate the achieved fairness using this approach,
we provide numerical simulations of bottleneck rates achieved by each flow
over multiple testing layouts drawn from the same setting. The results are
shown in Fig.~\ref{fig:fairness}. The distributions of the bottleneck rates are
very close for all the data flows, indicating that fairness is achieved over
the long run.

\begin{figure}
    \centering
    \ifOneColumn
        \includegraphics[width=0.7\textwidth]{Figures/rate_distribution}
    \else
        \includegraphics[width=0.47\textwidth]{Figures/rate_distribution}
    \fi
	\caption{Bottleneck rates achieved by 3 data flows in 500 ad-hoc networks.}
    \label{fig:fairness}
\end{figure}

\section{Incorporating Delay in the Objective}\label{sec:delay}

Up to this point, we have exclusively focused on optimizing network utilities
based on the bottleneck rate of each flow. On the other hand, \emph{delay} is also an
important performance metric, which has been the focus of many prior routing work.
As an extension of the proposed approach, in this section, we show how delay
performance can be incorporated into the network objective, and propose training
techniques for the reinforcement learning agent to achieve a tradeoff between the
bottleneck rate and time delay---two quantities which often conflict with each
other.

Similar to most prior routing works, we quantify the transmission delay of
a given flow as the \emph{number of hops} from the source to the
destination. To incorporate the number of hops into the reward function
of the DDQN agent, we propose to augment the reward function $\widetilde{Q}$ as
\ifOneColumn
    \begin{align}\label{equ:qhatdecay}
    	\widetilde{Q}(s_t, a_t) = \left\{\min_{i=t\dots h} 10\log_{10}\left(\text{SINR}_{(n^{(f)}_i, n^{(f)}_{i+1}, b^{(f)}_i)}\right) + \mathrm{bias} \right\} \cdot \lambda^{h-t}
    \end{align}
\else
    \begin{multline}\label{equ:qhatdecay}
    	\widetilde{Q}(s_t, a_t) = \\ \left\{\min_{i=t\dots h} 10\log_{10}\left(\text{SINR}_{(n^{(f)}_i, n^{(f)}_{i+1}, b^{(f)}_i)}\right) + \mathrm{bias} \right\} \cdot \lambda^{h-t}
    \end{multline}
\fi
where $h-t$ is the number of hops from node $n_{t+1}$ to the destination
(i.e., remaining number of hops after executing action $a_t$); and $\lambda \in (0,1)$
is a penalizing factor. With $\lambda$ closer to
zero, the agent would prioritize more on minimizing time delays.
Note that the purpose of the constant bias term is to ensure that the reward value is always positive so that it can be properly penalized by the penalty factor.

To explore such rate-delay
tradeoff, we train the DDQN agent with varying values of $\lambda$, each time from
scratch. Once the agent model is trained, we evaluate its performance on the testing ad-hoc
networks. The observed performance tradeoff between bottleneck rate and time
delay is shown in Table~\ref{tab:delaytradeoff}.

\begin{table}[t]
	\caption{Tradeoff between bottleneck rate and time delay}
    \centering
    \ifOneColumn
        \begin{tabular}{|c|c|c|c|c|}
        \hline
        ${\lambda}$ & Sum Rate (Mbps) & Min Rate (Mbps) & \# of hops per data flow & \# of reprobes per data flow\\
        \hline
        ${1}$ & 4.90 & 0.619 & 13.2 & 0.01 \\
        \hline
        ${0.8}$ & 2.07 & 0.343 & 5.3 & 0.20 \\
        \hline
        ${0.6}$ & 0.31 & 0.050 & 2.9 & 5.00 \\
        \hline
        \end{tabular}
    \else
        \begin{tabular}{|c|p{0.072\textwidth}|p{0.07\textwidth}|p{0.099\textwidth}|p{0.1\textwidth}|}
        \hline
        ${\lambda}$ & \shortstack[c]{Sum Rate \\ (Mbps)} & \shortstack[c]{Min Rate \\ (Mbps)} & \shortstack[l]{\strut \# of hops \\ per data flow} & \shortstack[l]{\strut \# of reprobes \\ per data flow}\\
        \hline
        ${1}$ & 4.90 & 0.619 & 13.2 & 0.01 \\
        \hline
        ${0.8}$ & 2.07 & 0.343 & 5.3 & 0.20 \\
        \hline
        ${0.6}$ & 0.31 & 0.050 & 2.9 & 5.00 \\
        \hline
        \end{tabular}
    \fi
\label{tab:delaytradeoff}
\end{table}

As shown in the results, the tradeoff is effectively achieved as $\lambda$ decreases from 1 to 0.
Specifically, with a low value of $\lambda$, the agent automatically learns to utilize the reprobe action
more aggressively to achieve fewer number of links per data flow. These results
suggest that with a properly modified reward definition and action space, the DDQN
agent can achieve a desired tradeoff between objectives that conflict with each other.
We note here that empirically when $\lambda < 0.8$, the DDQN agent reaches the destination on average faster than the ``Closest-to-destination among strongest neighbors'' benchmark.

\section{Benefit of Power Control}\label{sec:powercontrol}

This paper has so far assumed that each node always transmits at its fixed
power level. However, after determining the optimal routes and the set of
frequency bands, the bottleneck rates can be further improved if we allow
the transmission powers of the nodes to vary.

Power control among interfering links is by itself already a highly challenging
optimization problem. The problem becomes even more complex if power control
is performed in combination with routing and optimal spectrum access.
As this paper mainly focuses on routing, we will leave optimal power control
for future study. Instead, in this section, we simply apply
power control after the routes and frequency bands are fixed.

We propose the following straightforward and computationally efficient
heuristic for power control within each flow.  The heuristic is based on the
fact that only the bottleneck links affect the overall data rate of the flows.
Therefore, for each of the remaining links, we can decrease its transmission power
as long as its transmission rate does not drop below the bottleneck rate.
Decreasing these \emph{excessive} transmission power levels leads to less
interference, which helps improve the bottleneck rates.

In details, we fix the transmission power for the bottleneck link, then reduce
the transmission powers for all other links to approximately align their
link rates to the bottleneck.
Specifically, consider flow $f\in\mathcal{F}$ with the routing and
spectrum access solution as in (\ref{equ:dataflow}), with the
transmission power levels as $\{p^{(f)}_{0}, p^{(f)}_{1} \dots p^{(f)}_{h}\}$. We then adjust
the transmission power for every node $n^{(f)}_{i}, 0\leq i\leq h$ as
\begin{align}\label{equ:powercontrol}
	p^{(f)}_{i,{\rm adjusted}} = \frac{\text{SINR}_\mathrm{bottleneck}^{(f)}}{\text{SINR}_{(n^{(f)}_{i},n^{(f)}_{i+1},b^{(f)}_{i})}}p^{(f)}_{i}
\end{align}
where the SINR values are computed as in (\ref{equ:SINR}), and $\text{SINR}_{\mathrm{bottleneck}}^{(f)}$ denotes the bottleneck SINR for $f$.

After adjusting the powers for one flow, the bottleneck SINRs for all other flows would increase due to the reduced interference.
We recompute $\text{SINR}_{\mathrm{bottleneck}}^{(f)}$ for all the flows, and
go through all the flows successively in one round, from the data flow with highest bottleneck SINR value to the lowest.

While this method is heuristic, it is sufficient to illustrate the benefit of
power control. In Table~\ref{tab:powercontrol}, we provide a comparison of
achieved rates before and after power control, with the routing and spectrum access
solutions already determined by the DDQN agent. As the results suggest, with even a
simple power control heuristic within each data flow, the bottleneck
rates can already be improved. These results show the efficacy of this
simple power control method, as well as the potential for more sophisticated
rate optimization, even after the routes are already fixed.
\begin{table}[t]
	\caption{Power Control within Each Data Flow}
    \centering
    \begin{tabular}{|c|c|c|}
    \ifOneColumn
        \hline
        Methods & Sum Rate (Mbps) & Min Rate (Mbps) \\
        \hline
        No Power Control & 4.88 & 0.608 \\
        \hline
        With Power Control & 5.29 & 0.693  \\
        \hline
    \else
        \hline
        Methods & \shortstack[c]{\strut Sum Rate \\ (Mbps)} & \shortstack[c]{\strut Min Rate \\ (Mbps)} \\
        \hline
        No Power Control & 4.88 & 0.608 \\
        \hline
        With Power Control & 5.29 & 0.693  \\
        \hline
    \fi
    \end{tabular}
\label{tab:powercontrol}
\end{table}

\section{Conclusion}\label{sec:conclusion}

This paper proposes a physical-layer based reinforcement learning approach to
the wireless ad-hoc network routing and spectrum access problem. By training a
universal agent along the different nodes in a flow and across different
frequency bands, and by using the physical environment as the input, together with
a novel definition of reward function that accounts for the bottleneck rate
in the route, we arrive at a highly scalable algorithm for simultaneous routing
and spectrum access which can be adapted to the varying network layout
characteristics and generalized to large-scale networks. The numerical results
provide insight into the optimal relaying distance in an ad-hoc network
setting.  We further propose methods to provide fairness, as well as extensions
for incorporating delay into the optimization objective, and heuristic power
control methods that can further improve the overall performance of the network.

The proposed method achieves scalability by recognizing that as long as the
physical environment around each node is relatively stationary, a single agent
can be trained to perform resource allocation tasks efficiently across the
entire network. Thus, for ad-hoc networks, just as shown in early work
\cite{cui_spatial} that the optimal scheduling strategy can be learned based on
the physical environment using a universal deep learning agent, this paper
shows that a universal reinforcement learning agent can learn the optimal
routing and spectrum allocation strategy based on physical inputs.  Together,
these results show the feasibility of using machine learning to solve
communication and networking problems that are otherwise difficult to tackle.

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


text:
Cinnamon Deceptions (Angel in the Kitchen)
On Tuesday we discussed the "11 herbs and spices" in Kentucky Fried Chicken. Remember the difference between herbs and spices?
Simon (right): "Sigh! I'm tired of performing this song about herbs. We need to try something really different." Garfunkel (left): "Hm, let's do one about spices."
Herbs come from the green, leafy parts of certain plants. Basil, oregano, and mint are herbs; and the 1970s folk-rock duo Simon and Garfunkel included a few others in one of their somber songs: "…Parsley, sage, rosemary and thyme." (Of course, when we were kids we thought they were crooning about breakfast sausage and the sands of TIME.)
Spices are derived from parts of the plant other than the green leafy bits. Cloves (flower buds), ginger (roots) and pepper (seed pods). And then there's … CINNAMON. Oh yes, how can we forget cinnamon. To mention that we love it would be an understatement. In French toast, how much cinnamon is enough? Answer: More!
Are there any songs that mention cinnamon? None we know. But the Bible signifies this aromatic and flavorful spice in several passages. It was included in God's recipe for holy anointing oil, when "…The Lord said to Moses, 'Take the finest spices: of liquid myrrh … of sweet-smelling cinnamon … and of aromatic cane … and of olive oil…; and you shall make of these a sacred anointing oil….'" (Exodus 30:22-25 RSV)
Solomon wrote "I have perfumed my bed with myrrh, aloes and cinnamon." (Proverbs 7:17 NIV) Other references to cinnamon are found in Solomon's Song of Songs 4:14; and in Revelation 18:13. We like to think that the Creator of the Universe loved cinnamon as much as we do.
But what's so special about this particular spice? Well, in addition to tasting and smelling absolutely divine, cinnamon is loaded with fiber and antioxidants, and it contains a unique compound called cinnamaldehyde. These components make cinnamon conducive to good health. Indeed, the spice is known to produce 7 health benefits. (SEVEN!! — Hey, that's God's favorite number!) Cinnamon: (1) can lower bad cholesterol or LDL; (2) help treat Type 2 diabetes; (3) has antifungal, antibacterial, and antiviral properties; (4) can reduce the symptoms of Alzheimer's and Parkinson's; (5) has anti-carcinogenic properties; (6) has anti-inflammatory properties; (7) and can help manage polycystic ovarian syndrome, or PCOS.
And did we mention it's delicious in French toast? No wonder cinnamon was once valued almost as highly as gold, and was considered a gift fit for a king … or a god. Cool, right? But times change. So does "stuff"! Not all is as it seems. And not all cinnamon is really cinnamon!
The English word cinnamon can be traced back to the Hebrew qinnamon, a spice imported to Egypt from India. This is known scientifically as Cinnamomum verum. You may recognize the Latin adjective verum, from which we derive such words as TRUTH and VERIFY (as in "Trust … but verify!")
"True Cinnamon" is produced from the inner bark of an evergreen tree that's native to Sri Lanka. TRUE cinnamon was rare and expensive during Biblical Times. It was delicious, aromatic, and had many health benefits. It's what God requested be mixed into His Holy anointing oil.
But there's another spice currently marketed under the name "cinnamon." It's less expensive and more readily available. It's produced from the bark of trees originating in China and now cultivated throughout Asia. Spice manufacturers call it Cassia, to distinguish it from the real deal. Cassia looks like cinnamon, tastes like cinnamon — but contains none of the cinnamaldehyde or antioxidants that promote true health.
Eat up! Heh-heh!
To the contrary, Cassia contains massive concentrations of a blood-thinning compound called coumarin. Consuming large amounts of this ersatz cinnamon (as a health supplement, for instance; or maniacally tossed it into French toast batter) can lead to kidney and liver damage.
Who knew that something traditionally prized and sought after, something beneficial and even Biblical, could be replaced with something TOXIC? But again, the same can be said of many things, including the evening "news" and a lot of stuff that gets posted on the internet. So we can no longer accept things at face value.
Nor can we always go by the "label." This also applies to people — who often aren't what they seem or proclaim to be. How many of us have learned this truth the hard way? How many of us have trusted someone only to find out too late that he or she had a hidden (and self-centered) agenda? Even the Bible warns us against "wolves in sheep's clothing." (Matthew 7:15)
Toast or TOXIC!
Let's be clear on this point: Not everyone who calls themselves a "Christian" is truly a follower of Christ. Again, labels are convenient and ultimately can be deceiving. Even Satan sports a misleading label. Many of us (thanks to pop culture) imagine him as this red dude with horns and a pitch fork, but the Bible states, "…Satan disguises himself as an angel of light." (2 Corinthians 11:14 NLT) Why? Because he is "a liar — indeed, the inventor of the lie!" (John 8:44 Complete Jewish Bible)
In regards to all things, ask the Lord to help you distinguish the truth from the lie. Ask Him to grant you the ability to discern truth from error. And always do a little homework. Ask yourself, "Does this or that line up with the Bible (the infallible Word of God)?"
"Is it nourishing and beneficial? Or is it something spiritually toxic in disguise. Above all, is it TRUE?"
Don't swallow life's "cinnamon deceptions." Jesus said, "Take heed that no one deceives you." (John 24:4 NKJV) The Lord also said, "Not all who sound religious are really godly people." (Matthew 7:21 TLB) "The way to identify a tree or a person is by the kind of fruit produced." (Matthew 7:20 TLB)
Previous Previous post: Good Gravy or Grave Glop? (Angel in the Kitchen)
Next Next post: Hump Day Hope


We are making our learning resources freely available as Open Educational Resources (OER) so they are more readily available for organisations to use for their learning development needs.
OER are any type of learning materials that are in the public domain or introduced with an open license, like Creative Commons, which means anyone can legally and freely copy, use, adapt and re-share them.
Our OER are MS Word versions of our learning apps and websites that also include guidance on how to reuse the content.
Anyone can use our OER and they will be of particular interest to those delivering development or supervision to social services staff, including trainers, lecturers, managers and supervisors.
Find our OER on the SSSC Learning Zone. If you have any questions or want to find out more email sssclearningtech@sssc.uk.com.


The Muse 3 has been designed with impressive flying characteristics that will surprise the eager intermediate pilot fresh out of training. The Muse 3 is in its own class called relaxed sport because of its fun handling and performance usually found in more advanced paragliders. You will love how easy it does everything well in a calm yet sporty feel. Critics say it's so easy to fly it practically fly's itself!
Forget about thinking you have to buy a new glider every year. The newly designed Muse 3 embodies designs from MacPara's other intermediate performance paragliders. The bonus is that it offers incredible safety and stability that is really fun to fly. This allows pilots to fly more often without the worry of feeling like their flight may be jeopardy. Safety and performance matched together makes this a popular choice for pilots who take a causal approach to flying on weekends. The Muse 3 is certified in the EN A glider category.
The Muse 3 is simple to control on the ground and in the air. You will feel like a pro with its exceptional take off characteristics to get you flying right after inflation with no hassles. It will stay above your head without the tendency to collapse or wander off course. The safety in the Muse 3 is best admired by pilots that are looking for a paraglider resistant to collapse. With this glider you can keep your eyes on the scenery instead of being worried about what's coming next. Flying the Muse 3 you will notice immediately how well it is able to perform everything you need to enjoy the flight without feeling like your missing out. MacPara hit the target by designing a fun performance glider with high stability characteristics.
Redesigned to Make You Look Good!
Based on the success of the Muse 2, the Muse 3 continues making history because of its popularity and beloved launching characteristic that MacPara paragliders are known for. It has become one of the bestselling gliders because of how happy pilots feel flying it. Many gliders competing in the entry level glider class are boring once you learn the basics of launching and landing. Not the Muse 3. You would be hard pressed to think you could out fly this glider. Some even say it's a perfect glider for learning acrobatics because of its performance and built in safety recovery characteristics. Your abilities to play like a bird will exceed your expectations. You will be glad to know that the safety of the Muse 3 is also found using full speed bar.
The new Muse 3 has given more attention to the airfoil resulting is a noticeably better glide ratio and in flight stability. The smooth shape of the airfoil nose is maintained by long lasting Mylar reinforcement combined with new age plastic rods. These high tech rods give the front of the glider stability in the air after it effortlessly scoops up the air for a perfect launch.
The Muse 3 is yet another quality made glider by MacPara that will continue to perform for the years to come. The construction using diagonal ribs and a system of load-bearing tapes gives the canopy stability and helps keep a clean profile. The Muse 3 has 45 cells and a nice elliptically shaped canopy with an aspect ratio of 5.00. The glider is light. Just 5.1 kg for the medium size which results in very easy launches with or without a paramotor.
Intensive testing resulted in a well balanced glider with simple take-off characteristics and easy behavior during extreme maneuvers. The canopy is very resistant to collapse keeping your flight on track. In the event of a collapse the opening is smooth and fast. In an induced collapse the glider gives a rotation of only 30 degrees. The canopy of the Muse 3 is well damped in all axes but offers lively sporty handling to keep you smiling!
The light brake pressure allows pleasant turns during long thermal flights and increases progressively towards the end to offer a good safety margin. It has a large brake range offering good pilot input to know how the glider is flying above you. The Muse 3 is 4 liner paraglider with a 5 point riser system and split "A" risers for easy big-ears. The risers have 2 suspension points giving the option of free flying or paramotoring. This is a nice option for pilots that don't want to buy two paragliders. More and more pilots are realizing the Muse 3 does it all well! The transition from free flight to powered flight is very simple: hook up the carabineers in the second position and off you go with shortened riser length for powered flight! There is no need to extend the brake lines.
Regardless if you are using the Muse 3 for paragliding or paramotoring it has been certified to fly perfectly in thermals and cruise like a soaring eagle. The Muse 3 is designed for pilots that want the most out of their flight without the headache and stress of waiting for "things" to go wrong in turbulence or powerful thermals. Undoubtedly, the Muse 3 lets you fly with ease.
The Muse 3 stood out among the EN-A gliders in the Testival issue of July 2011. It was the representative which came closest to being an intermediate glider.
In a solo test we now take a thorough look at look at how well equipped this Czech glider really is.
In ancient Greece, the Muse was believed to be a source of inspiration, which should spur creative achievement. This is a beautiful image, which aptly seems to be especially true for the brand new EN-A category glider from Mac Para Technology. From the beginning the engineer, director and designer Peter Recek is not unknown, quite the opposite. He is a veteran of the flying scene, former member of the Czech national team and privately a nice fellow. Despite his success Peter speaks modestly about his designs even though his gliders are now and were, time and again at the top of the development. Most recently, he landed a groundbreaking development leap with the Magus XC2.
Interesting is the view at the reorientation of the manufacturer on the performance.
Despite the lack of mention in the specification Peter reported a significant increase in performance even with the new Muse. He speaks of a glide ratio of 8.3, one rarely achieved in this category. But the true qualities of the Muse 3 lie beyond the numbers.
To comply with the inflexible requirements of the class for trim, material selection and security is not easy. The keyword is Safety: EN / LTF A class certification is not obtained every day, so the balance must fit. Not an easy target.
As mentioned in our Testival issue, Mac Para offers in the safest A – category, two representatives. The Muse should move to the upper limit of the class to include the company requirements for powered flight. From this point of view climbing performance is important, to keep fuel consumption low. This orientation is also evident from the technical specifications, with 45 cells and an aspect ratio of 5.0, the design is clearly oriented to the typical intermediate rather than to the rest of the A-gliders. Like other manufacturers, the MAC PARA R&D team developed a completely new airfoil to fulfil the new requirements. In addition to the joy of climbing, the glider should increase the dynamic and handling. The basic shape of the Muse 3 is an ellipse, but this is hardly recognizable due to the glider's design. On the leading edge are the omnipresent nylon rods and classic Mylar.
The line concept is itself not very spectacular: The usual four rows, each with 3 main lines illustrate the requirement of powered flying. For paramotor use this is a better guarantee of the continuity of the trimming. Sheathed gallery lines speak the same language. This requirement is also obvious from the risers which have a separate hook-in point for powered flying. A glider for everything: not a bad thing! The workmanship of canopy, lines and risers are all very conscientious and there are no grounds for criticism.
The sensitive pilot will notice immediately already at take off that the Muse 3 is more consistent with intermediate gliders. Front launching the climb is characterized by high directional stability, but the canopy must be braked at the top otherwise it will have a tendency to overfly the pilot. What possibly can look slightly tricky on the practice slope proves to be no problem after getting used to it, mainly because the canopy responds to the commands relatively quickly. The sorting of lines is average for the number of lines and makes inflation very nice.
The result is a ready "starter" which is not foolproof, but in the hands of a talented beginner or advanced pilot it is a great joy.
The reverse launch is similar. Especially pleasant is the ease with which the canopy can be brought over pilot's head when sitting back. The responsiveness of the Muse 3 during the launch can be easily controlled by the brakes. This excludes the needs for frantic reaction from the pilot to suppress the canopy with the brakes which results only in indifference in many nursery slope canopies. Clearly the Muse 3 asks for and provides more than conventional devices in its class.
Whilst checking the reactions on the ground during launch the pilot begins to understand that thoughts about the handling are coming true. Nevertheless the ease with which the Muse launches is fantastic.The wing flies from the first second completely intuitively and unaided. Responses to brake input and/or weight shift are answered promptly and accurately. The wing quickly and willingly accepts all inputs. The brake pressure depending on the wing loading is medium to low, giving comfortable and fatigue-free flying. Only the stall point is not entirely easy to find, through its sheer inaccessibility makes this hardly relevant. Provoking the stall from normal reactions is not an option.
Highly encouraging is the "thermal capture", "thermal handling" and the "climbing joy" of this new A-glider. While many other in the class counter entry into thermals by unpleasant sitting back, this Czech remains neutral. Usually stable above the pilot, steady height gain is only noticeable from the sound of the vario. Powerful and nevertheless well damped, the wing can still be turned wonderfully gently, which is by far not a must. Actually this is what separates the wheat from the chaff!
Turbulent air under the Muse is hardly noticeable, in accordance with its classification but more than most of its A-class colleagues. In comparison the feedback is immediate and direct via the brake pressure and is good. The Muse 3 neither dives, sits-back, or levers out of the thermal, to achieve this it must have no yawing tendency.
Pitch and roll damping are more within the range of a basic intermediate and it flies exactly like a glider of this class. Provoked deformations reward the newcomer with very pleasant behaviour. Collapses of the leading edge progress laterally, so that the rear portion of the canopy usually remains inflated. Even accelerated, it is very difficult to collapse larger areas of the canopy. Even with a completely passive pilot, the span only starts to deviate from course in slow motion; however the folded portion continuously opens starting from the centre progressively to the outer wing.
Such a collapse or any cravat of the wing is hardly possible. Even on full bar during the testing stage no turns greater than 45 ° were flown. Indeed: an A-class glider!
No-one would ascribe freestyle suitability to an A-glider. However if the controls are operated bravely, then the canopy builds an unhurried pleasant dynamic.
Hard to believe, but high wingovers are easily flyable, enabled by the gliders high manoeuvrability and responsiveness.
Suitability for SAT, Helicopter and Full-Stall are hardly relevant for its main target group. Nevertheless: the wing is like no other for learning these manoeuvres the dynamic is there, although buried deep under a huge safety cushion.
The final delight is the speed system. It operates with relative ease and the glider reaches a respectable maximum speed of 49 km/h. It is fascinating how docile the Muse 3 is on full bar. Even big thermal transfers could not induce a deflation, even performance-reducing deformation of the leading edge did not occur during our testing.
Thanks to the split A-risers the ears quickly folded without too much resistance. The effectiveness was moderate to high. For opening you must give the Muse some time. If necessary a supportive, single braking impulse helps.
For a clean B-stall the pilot must grasp the lines and pull vigorously. The forces are considerable, but not insurmountable. Also during the stall they remain high. The manoeuvre appears to be quite stable, a tendency to warp could not be found. The sink rate reached only about 6 m/s. After exiting, the glider starts to fly immediately, without over-shooting.
Even after extensive flying and despite the gentleness of the extreme flight manoeuvres one question arises at the end: How can this glider be an A-category glider? The Muse3 looks like an intermediate, flies like an Intermediate, acts like an intermediate, shows adequate dynamics and all this at much greater safety margins. We searched in vain for a catch. In the light of appealing performance, the good and especially stable top speed, the amazing freestyle suitability or suitability for powered flying, the question remains, what more could you wish for. You can't ascribe full beginner capability to the Muse, but with a little practice during training it should be possible to upgrade to the Muse. Why not fly a fun glider from the very beginning? The potential target group for the Muse 3 is very wide. Even after many years of flying experience the Muse 3 never will act like a tank and never act as barrier to the fun of flying. Quite the contrary: The additional safety cushion allows no fear flying; great handling leaves nothing to be desired even by top pilots.
Amazing at the end: In addition to the excellent free-flying properties, paramotoring with the Muse is not only possible but also allowed and encouraged. One thing is certain: It will be difficult to send back this glider!
I have made a plurality of flights with the Muse 3, in our home areas Ternberg, Micheldorf, Bischling and the Dachstein / all in Austria.
***** Robust, proven materials. The only downside is the cheap-looking plastic pulleys.
**** Very good workmanship with impeccable sewing. Suturing of the loops but without further tension distribution.
**** Reliable climb, high directional stability, braking impulse at the zenith necessary.
**** Good control with brakes or D-risers, high directional stability, braking impulse at the zenith necessary.
***** Unrivalled high in comparison to class.
Steering ***** Good response to inputs, medium control travel, excellent damping.
***** Very high stability even accelerated, rotation under 45 °, leisurely opening from the centre of the wing.
***** Complies with all the virtues: efficiency, ease of movement, stability.
**** Simple introduction, moderate folding depth, leisurely opening.
**** High entry and follow-up forces, moderate efficiency, stable stall phase, barely any tendency to over-shoot during recovery.
**** Leisurely development of sink rates after some persuasion, moderate efficiency.Active recovery recommended but not required.
Technical specialities Thin plastic rod reinforcements in the leading edge, PPG suspension points incorporated into the risers.
Suitability Talented newcomers, but also experienced mountaineers, freestyle aspirants, powered flight pilots, fun flyer, infrequent flyer, frequent flyer.
The Muse 3 is manufactured using the paragliding industries well established lightweight material with excellent durability; Skytex 40, from Porcher Sport (NCV). Because for any glider, the part that always wears out first is the leading edge, we use Skytex 45 with coating E85A for this. For the less stressed areas of the sail we use Skytex 40 with E38A coating. For the supporting and diagonal ribs we use Skytex 40 with "hard finish" coating E29A. This gives the advantage of a sail that is both lightweight and very durable.
CLICK HERE you can printed special design on paper, or print to pdf.


text:
/*!
 *  Font Awesome 4.7.0 by @davegandy - http://fontawesome.io - @fontawesome
 *  License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License)
 */
/* FONT PATH
 * -------------------------- */
@font-face {
  font-family: 'FontAwesome';
  src: url('https://cdn.joaobordalo.com/fonts/fontawesome/fontawesome-webfont.eot?v=4.7.0');
  src: url('https://cdn.joaobordalo.com/fonts/fontawesome/fontawesome-webfont.eot?#iefix&v=4.7.0') format('embedded-opentype'),
       url('https://cdn.joaobordalo.com/fonts/fontawesome/fontawesome-webfont.woff2?v=4.7.0') format('woff2'),
       url('https://cdn.joaobordalo.com/fonts/fontawesome-webfont.woff?v=4.7.0') format('woff'),
       url('https://cdn.joaobordalo.com/fonts/fontawesome-webfont.ttf?v=4.7.0') format('truetype'),
       url('https://cdn.joaobordalo.com/fonts/fontawesome-webfont.svg?v=4.7.0#fontawesomeregular') format('svg');
  font-weight: normal;
  font-style: normal;
  font-display: swap;
}
.fa {
  display: inline-block;
  font: normal normal normal 14px/1 FontAwesome;
  font-size: inherit;
  text-rendering: auto;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}

.fa-volume-up:before { content: "\f028"; }
.fa-pause:before { content: "\f04c"; }
.fa-play:before { content: "\f04b"; }
.fa-stop:before { content: "\f04d"; }
.fa-search:before { content: "\f002"; }
.fa-close:before { content: "\f00d"; }
.fa-rss:before { content: "\f09e"; }
.fa-adjust:before{ content:"\f042"; }
.fa-twitter:before { content: "\f099"; }
.fa-github:before { content: "\f09b"; }
.fa-certificate:before { content: "\f0a3"; }