This entry was posted in Uncategorized on July 13, 2018 by deefduke.
It took us 63 hours to drive straight through to Soldotna, 2,846 miles, from our house in Jefferson. The car ran great, no flats on the boat trailer, the trailer pulled great, and the passengers got along great. We got pretty tired sleeping in the car, but kept rotating drivers and managed. The biggest problem was the last 200 miles was very bad road because of thermal frost heaving, which made it very hard to sleep when you bounced out out of your seat every few minutes. But we are now at "Tim's Place" now waiting for the fish to come in. I am working on the boat getting it ready to take out in the ocean. I am hoping to take it out tomorrow and catch a bunch of halibut. We drove to Ninilchik where they back your boat in the surf with a big skidder and checked it out. It looks like it will be an adventure for sure. We went into local "Sportsman's" store in Soldotna to buy gear to fish Halibut with. I decided to ask one of the salespeople how to fish for Halibut in Cook Inlet. I walked in and started looking at the gear for Halibut fishing and a young man walked up behind me and asked me if I was going Halibut fishing. I responded "yes", and I am from Oregon and don't have a clue what I am doing. He said, " wow, I am moving to Oregon to go to Western Oregon University, and I have been a Halibut guide for the last 4 years. He proceeded to show me exactly what I needed to buy and what to do. He said he really wanted to go bass fishing when he got to Oregon so I told him I would take him when he got to Oregon and to come to Jefferson to Jefferson Baptist Church and ask for me. He then told me what church he went to and that he was a Christian. It was a great divine appointment.
This entry was posted in Uncategorized on July 11, 2018 by deefduke.
We have been driving about 5 1/2 hours, and it is just about midnight. Only 45 more hours to go or thereabouts. So far everything is going good, car running good, boat towing good, and passengers being good. I drove the first 3 hours and have been sleeping very sound for the last 2 hours and intend to go back to sleep as soon as I finish this blog. The most beautiful drive in the world is the Alaskan Highway, and we will get into some of the really pretty scenery about the time it gets light. I intend to memorize a bunch of Bible verses and watch you tube video's on how to catch halibut in Alaska with the time I will have while riding. Well, I am going back to sleep!
This entry was posted in Uncategorized on July 7, 2018 by deefduke.
I worked all day, daylight to dark on my boat. I sanded some of the spots that had epoxy runs on them, did some bondo work on low spots, sanded some more, and painted the whole boat again. Just wanted it to look pretty. We are leaving for Alaska tomorrow evening at 6 pm. We are leaving then to be able to get through Seattle without the traffic. We are going to drive straight through without stopping to sleep, just trading off and sleeping in the car. It is going to take approximately 50 hours to reach Soldotna. I am planning on bringing back 500 lbs of sockeye salmon and halibut fillets. That will be nice!
In the gospel of John in chapter 21 Jesus asks John, " do you love Me more than these?" "These" that Jesus is referring to is the fishing boats and nets that the disciples had just been in when Jesus came to fix them breakfast. I need to replay that conversation regularly, "Do I love Jesus more than fishing?" The answer will be demonstrated by my life.
This entry was posted in Uncategorized on July 6, 2018 by deefduke.
Proverbs 3:13-17 How blessed is the man who finds wisdom, and the man who gains understanding. For her profit is better than the profit of silver and her gain better than fine gold. She is more precious than jewels.
Wisdom is the main thing, the most important thing, nothing else is as important as wisdom is. Wisdom is foundational to good marriages, wise parents raise good kids, wise people manage their finances well, wise people get along with others, wise people keep God first in their lives.
There are not very many truly wise people around, because the price is to high for most. To receive wisdom there must be a very strong desire for wisdom, a strong hungering and thirsting for more and more wisdom. A casual interest for wisdom won't work. If you aren't consumed with getting wisdom you won't.
To receive wisdom there must be a strong and consistent seeking of wisdom. Proverbs says to seek it like gold, silver, and treasure. Seeking is pro-active, that is we actively do something that is a proven way to acquire more wisdom. I like to say that seeking wisdom is like fishing. You want to catch fish so you, but it isn't automatic that you will. A few people catch a lot of fish, and a lot of people catch a few fish. We are leaving for Alaska Friday morning to fish for Halibut and sockeye salmon.
The greatest source of wisdom is experience that is gained while in pursuit of wisdom.
This entry was posted in Uncategorized on July 5, 2018 by deefduke.
Before I can leave for Alaska to go fishing this Friday I have a long list of things I need to get done. There are some things on the list that are under the category of "it would be nice to get that done before I leave", and then there are those things that are super important and really ought to be done before I go, and then there are those things that have to get done or I can't go. Obviously the sub-group on my Alaska to do list "things that must get done or I can't go" have the highest priority, and I will focus on getting those checked off before moving to the others. Today it was interesting to me to note how much I was drawn to do the things that weren't on the "have to or I am stuck home" list. Why would I spend time on things that don't have to get done and ignore things that absolutely must get done. I do it all the time, why? It really is stupid, and nonsensical, but I do it anyway. I don't know the answer for sure, but my guess is that the things that must get done, as I look at the list of them, aren't very fun, or exciting, they all would easily be described by my Dad as chores. An aspect of the character trait of self-control is choosing to do what is most important, and must get done. I really want to go fishing in Alaska so I have a strong motivation that greatly helps my self control. In all of life a key principle of self control is reward. I am working at attaching a big reward to the most important activities in my life even if I have to manufacture the rewards so that I maintain focus on the things that really matter.
This entry was posted in Uncategorized on July 3, 2018 by deefduke.


text:
\section{Introduction}
\label{sec:introduction}

Interconnected electrical power grids are a critical infrastructure of modern society. The operation of these grids is a demanding control task, requiring uninterrupted monitoring by skilled experts and frequent interventions to safely and reliably transport electricity from the producers (generators) to all connected consumers (loads)~\citep{Kelly2020gridrl}. Load flow calculation is a well-established method for the simulation of grid states and it is routinely used by grid operators to ensure that all security constraints are met~\citep{stott1974loadflow}. The existence of fast and reliable simulators pave the way for powerful tools based on Reinforcement Learning (RL)~\citep{marot2019l2rpn}, supporting operators with the increasingly difficult challenges in the ongoing effort to decarbonize the energy sector and ensure security of supply~\citep{marott2022towards,marot2022perspectives}.

The shift towards renewable electricity generation is advancing rapidly. Globally, 80\% of the newly installed generation capacity is already based on renewable sources, where solar and wind account for the largest share\footnote{Data provided by the International Renewable Energy Agency, \href{https://pxweb.irena.org/pxweb/en/IRENASTAT}{IRENASTAT}, retrieved Sept 2022.} (see Figure~\ref{fig:renewables_and_congestion}, left). Compared to non-renewable technologies such as thermal generation from fossil fuel, solar and wind energy is highly volatile, confronting grid operators with large power gradients and increased uncertainties. This makes more frequent interventions necessary in order to prevent individual grid elements from exceeding operational limits (grid congestion)~\citep{han2015congcasestudy}.
Figure~\ref{fig:renewables_and_congestion}, right shows an exemplary congestion scenario.

\begin{figure}[t]
  \centerline{\includegraphics[width=.43\textwidth]{figures/generation_stats.pdf}\includegraphics[width=.57\textwidth]{figures/redispatching.png}}
  \caption{Left: Total global generator capacity expansion per year (orange: non-renewables, blue: renewables, green: share of new renewable generation)
Right: Simplified congestion scenario with surplus power from wind farms and transmission to large remote consumers and a neighbouring country. Load flow analysis reveals a congested transmission line. Curtailing wind power and compensating for the spilled renewable energy by ramping up a thermal generator is the prevalent remedial action applied in practice.}
  \label{fig:renewables_and_congestion}
\end{figure}

\textbf{Power Grid Basics.}
Laying out the electrophysical foundations and covering all common grid elements
is beyond the scope of this paper.
Fortunately, this complexity is largely encapsulated in the simulation tools of the grid operators. A more detailed presentation is provided in~\citep{Kelly2020gridrl}.

\textit{Grid model}. In essence, a power grid connects generators (producers) to loads (consumers) and is modeled as an undirected multigraph,
with edges representing transmission lines (carrying energy over long distances) and nodes representing substations (switchable connection points)\footnote{Throughout the paper we adopt the view of a transmission grid operator, but note that the same concepts apply to the distribution grid as well.}.
Relevant grid elements are presented in more detail in Section~\ref{sec:env_disign}.

\textit{Congestion management}. Given an expected load and generation scenario,
grid security analysis is conducted as part of the operational planning process~\citep{shahidehpour2005gridsecurity}. If the security analysis reveals violations of operational constraints,
grid operators are required to intervene to keep the grid stable and reliable.

\textit{Costly and non-costly measures}. Operators can relieve congested lines by altering the power entering and leaving the grid, which involves high costs from modifying previously agreed transmissions and procuring energy from different generators~\citep{VANDENBERGH2015redispatching}. From an economic perspective, it is highly desirable to jointly optimize these costly measures with the non-costly measure of dynamically reconfiguring the grid by making use of existing switchable equipment~\citep{hedman2011toporeview}.

\textbf{Related Work.}
Optimal line switching was first proposed by~\citet{fisher2008topo} as a mixed linear integer program, based on a linear approximation of the load flow calculation (DC formulation). \citet{hedman2009lineswitching} added security analysis and a heuristic strategy to obtain approximate results for the IEEE 118 bus system.
Note that line switching does not make use of the flow rerouting ability of the substations. Subsequent work~\citep{heidarifar2016topo, xiao2018topo, li2019topo, zhou2021topo} includes more complete modeling of substations at the circuit breaker level, but continues to rely on linear approximations of the grid dynamics in order to apply commercial solvers.
Recent editions of the Learning to run a Power Network Challenge~\citep{marot2021retrospective,marot2021l2rpntrust}
reveal the first promising results of RL-based topology optimizers.
\citet{zhou2021es} propose to combine search-based planning with the predictions of a policy network
learned via large scale Evolutionary Strategy (ES) training.
Their approach is able to discover long-term strategies,
but struggles with the inherent limitation of being sample inefficient.
\citet{roychowdhury2022toporl} train a value network with Dueling DQN (DDQN)
for line switching and topology control at a substation level.
The Authors then evaluate their approach on the IEEE 14 bus system.
Using the same testbed~\citet{subramanian2021simplerl} approached the problem
by utilizing a cross-entropy optimizer for topological switching.
By directly predicting the desired target topology and subsequently planning an action sequence towards this target topology
~\citet{Yoon2021afterstate} propose an off-policy actor-critic approach
adopting the hierarchical policy together with the afterstate representation.
Related to our approach,~\citet{taha2022mcts} propose to learn a line load dynamics model, which is then
utilized in a tree search for planning viable topological remedial actions.
However, this tree search is not yet guided by a learned policy network,
making it inefficient and unlikely to find optimal solutions for upcoming congestion situations
within the~exponential topology exploration space.

\textbf{Our Contributions}.
We pick up this line of research and propose an AlphaZero-based topology optimization agent~\citep{silver2017alphazero}
to address power grid congestion management.
Our choice is motivated by recent successes of AlphaZero
for similar large-scale combinatorial optimization problems.
We benchmark our topology agent on a 118 substation power grid
and demonstrate how to successfully combine it with traditional congestion management measures.
Finally, we discuss the~practical relevance of such an agent
using the examples of day-ahead planning and real-time remedial action recommendation,
and outline promising further research directions.

\section{Environment Design and Interaction Workflow}
\label{sec:env_disign}
This section describes the power grid environment, its observation and action spaces, reward as well as custom modules.

\subsection{Power Grid Environment Overview}
\label{subsec:power_grid}

We utilize the Grid2Op framework~\citet{grid2op} provided by RTE France, Europe's largest grid operator. A power grid consists of generators, loads, storages, substations, and power lines. We provide a brief overview below; please refer to the \href{https://grid2op.readthedocs.io/en/latest/}{official documentation} for the full specification.

\begin{figure}[h]
  \centerline{\includegraphics[width=0.99\textwidth]{figures/topology_action_example.pdf}}
  \caption{Example topology action. Node splitting is performed by switching the overflowing line to the second bus, changing the topology of the network and resolving the overflow. Note that in the Grid2Op framework~\citet{grid2op}, each substation features two buses. In the real world, a~substation might feature even more than two buses.
  (Figure adopted from~\citep{marot2021retrospective}.)}
  \label{fig:topology_action_example}
\end{figure}

\textbf{Generators, Loads and Storages.} \emph{Generators} produce power. Renewable generators (sun, wind, hydro) follow simulated weather patterns; thermic generators (coal, gas, nuclear) follow predefined output curves. As a congestion management measure, the output of generators can be influenced---thermics can be redispatched (up or down) and renewables curtailed (down only). \emph{Loads} consume power, following a predefined pattern unknown to the agent. \emph{Storages} store power across time steps, and can be gradually charged or discharged.

\textbf{Transmission: Substations and Lines.} \emph{Substations} connect elements (generators, loads, lines), and consist of two buses. All connected elements need to be assigned to a bus, configuring the topology of the grid. For an example of how bus switching affects the topology of a network, refer to Figure~\ref{fig:topology_action_example}. \emph{Power lines} connect substations. Each line has a thermal limit (i.e., capacity), which it can exceed only for a small number of time steps---then an automated disconnect occurs. Severe overflows (>~200~\%) cause an immediate disconnect, and often lead to cascading line failures.

\textbf{Objective.} The goal is to keep the grid in operation, avoiding congestions and line disconnects
that can lead to isolation of loads or generators---i.e., a diverging power flow (black out).
The secondary objective is to minimize operational costs incurred by power loss and congestion management.

\subsection{Observation Space}
\label{subsec:observation_space}

\textbf{Overview.} We utilize the \texttt{CompleteObservation}\footnote{Please refer to the documentation of \texttt{CompleteObservation} for a complete description (including, e.g., maintenance and cool down times): \url{https://grid2op.readthedocs.io/en/latest/observation.html}}, which we further process
into feature representations.
The observation space includes (most notably, but not limited to):
\textit{generator and loads state}, including power production/consumption and its forecasts;
\textit{power line state}, including the power flow at origin and extremity, and the~current load as a fraction of capacity (denoted as~$\rho$, one of the most important metrics);
\textit{topology configuration}, with the power lines status and bus configurations for each element; and, finally,
\textit{redispatch, curtailment and storage states}.

\textbf{Simulation.} A limited simulation functionality for planning purposes is available to the~agent. Compared to the complete power grid environment (which is not available to the agent), it features only forecasts of future load, generator, and weather patterns. Hence, while the agent can test the~general impact of an~action, the accuracy varies.

\subsection{Action Space and Combinatorial Complexity}
\label{subsec:action_space}

An action can modify topology, redispatching, curtailment and storage elements\footnote{For a more information, please review the action documentation: \url{https://grid2op.readthedocs.io/en/latest/action.html} and operational cost overview: \url{https://codalab.lisn.upsaclay.fr/competitions/5410\#learn_the_details-evaluation}}.

\textbf{Non-Costly Measures.} \textit{Topology actions} (the only non-costly measure) modify the bus configuration of one~grid element (either all connections on one line or one substation). As each substation features two buses, there are overall $2^{k}$ possible topological configurations, where $k$ is the number of connections in the grid (i.e., all elements on all substations). While not all configurations are valid,
this still yields an exponential exploration space, presenting a challenge for both RL-based and traditional approaches.

\textbf{Costly Measures.} \emph{Redispatching actions} (incurring redispatching costs) modify the target output of the redispatchable generators (up or down), and need to be within the per-generator bounds. \emph{Curtailment actions} (incurring curtailment costs) limit the production of renewable generators. \emph{Storage actions} (incurring loading, preserving, and discharging costs) direct the behavior of storages.

Note that in a real grid, a few more congestion remediation measures are available, such as counter trading, demand-side management or load shedding. However, the actions described above (and simulated in the Grid2Op environment) represent the main remediation options at hand.

\textbf{Action Space.} Our topology optimization agent utilizes the unitary action space provided by Grid2Op. This space represents all individual available topology actions (i.e., manipulating all combinations of bus connections, per grid element) as integers, creating a discrete action space. To reduce the action space size, we then perform action masking and action reduction (see Section~\ref{sec:experiments}).

\subsection{Environment Dynamics, Interaction Workflow and Reward}
\label{subsec:env_dynamics}

This section describes custom modules engaging in observation processing and environment control.

\textbf{Grid State Observer and Safe-State Skipping.} In most time steps, the grid is running safely, and launching our congestion remediation agent is neither necessary nor desirable (due to additional operational costs). Hence, we integrate a \emph{Grid State Observer} that determines whether the current state is considered safe by evaluating the current maximum line load\footnote{The \emph{Grid State Observer} is a customizable module and it can consider also, e.g., an n-1 contingency analysis.}.
In safe states, the congestion remediation agent is skipped, and either a recovery (see below) or a no-operation action is performed.

\textbf{Topology Recovery.} In the default topology, all connections are switched to the same bus, i.e., no node splitting takes place (see Figure~\ref{fig:topology_action_example} for a node splitting explanation). We find that this ,,fully connected`` setting is rather resilient, and a good starting point for congestion management. However, due to the reduced action set (see Section~\ref{sec:experiments}), the agent does not always have all the actions available required to recover this topology. Hence, in safe states, we perform automated topology recovery.

\textbf{Rewards.} In case of a congestion, the topology optimization agent is triggered. We utilize a shaped reward based on the cumulative sum of all overflowing line loads, which the agent aims to minimize.

\section{Topology Optimization with AlphaZero and Heuristic Value Functions}
\label{sec:alpha_zero}
In this section, we outline our customized AlphaZero algorithm employed with a heuristic value function. AlphaZero~\citep{silver2017alphazero}
constitutes an RL algorithm using a general-purpose Monte Carlo Tree Search (MCTS). Each search consists of a fixed number of simulations, starting at the root node, and traversing the tree until a new leaf node is added to the tree. Which action is selected at each branching in the tree is guided by the probabilistic upper confidence tree (PUCT) bound~\citep{rosin2011multi} taking into account the visitation count of the current and potential child node, the action probability sampled from the policy network and the value assigned to the resulting state when performing the respective action. Once the desired number of simulations has been reached, the action to perform is sampled from all expanded children of the root node w.r.t. the probability distribution obtained by taking the softmax of their visitation counts. In the greedy setting, the action with the highest visitation count is selected.
The original implementation utilizes this MCTS formulation and self-play~\citep{silver2017human} to collect state-action pairs (or trajectories).
These state-action pairs are then used to continuously update the policy network with a behavioural cloning algorithm. The three games studied have a reward of $0$ in all but the last time-step, where the reward is $1$ if won, $-1$ if lost and $0$ if the game draws. This 'terminal value' is then paired with all states of the corresponding trajectory and used for value function learning.

MuZero~\citep{schrittwieser2020mastering} is a model-based extension of AlphaZero,
which still relies on MCTS but generalizes to single-player domains and infinite horizon problems with intermediate rewards. While the policy targets stay the same as in AlphaZero,
the value targets need to be adapted for the intermediate rewards. Specifically the return $z_t$ is computed for each state in the trajectory with $z_t = r_{t} + \gamma r_{t+1}  + \gamma^2 r_{t+2}  + \dots + \gamma^n \nu_{t+n} $ where $\nu_t$ is the prediction of the return by the value network for time step $t$. Note, that the prediction $\nu$ can be neglected if the given environment has no infinite horizon but a max reachable time step $t_{max}$ and $n=t_{max}$.

While managing a power grid is not a two-player game like Chess or Go,
it still bears some inherent similarities
when it comes to the long-term effect a single action can have on the development of the grid (``game'')
as well as the exponential nature of the action space (see Section~\ref{subsec:action_space}).
Since AlphaZero has already proven superior performance for problems
where long-term planning in large combinatorial action spaces is required,
we see it as a natural fit for the topology optimization problem in large power grids.
Furthermore, power grid management is a domain
where fast and accurate simulations have been optimized for decades
and are already widely used in practice, e.g., for day-ahead or intra-day operation planning.
The nature of the optimization problem
as well as the fact that power grid load flow simulators are already established in practice
set the stage for our modified version of the AlphaZero algorithm in the single-player infinite horizon power grid setting also utilizing modifications of MuZero.

\begin{figure}
  \centerline{\includegraphics[width=1\textwidth]{figures/mcts_tree_example.png}}
  \caption{Example of a grid topology MCTS tree. Nodes represent states  (root is the current state); edges are simulated actions. Yellow nodes are critical states (maximum line load > 98\%), red nodes are black-out states and blue nodes are critical states that internally skipped two or more steps because they were deemed safe by the \emph{Grid State Observer} (see Section~\ref{subsec:env_dynamics}). Edges correspond to topology actions with the action ID (in brackets) and the corresponding upper confidence bound.}
  \label{fig:mcts_tree}
\end{figure}

\subsection{Early Stopping and Action Selection}
Considering that AlphaZero is a tabula rasa learning algorithm, an extensive amount of data needs to be collected during the training process, which requires large-scale parallelization and as a consequence a lot of compute resources. For Chess, Shogi and Go~\citet{silver2017alphazero} report using 5,000 first-generation TPUs~\citet{jouppi2017datacenter} to generate self-play games and 64 second-generation TPUs to train the neural networks.
To substantially reduce the amount of resources required to train such a model to only 64 CPU cores to generate trajectories and 1 GPU to train the neural networks, we devised an MCTS early stopping mechanism that terminates the simulation search
once a good (enough) solution has been found.
For our experiments we use two stopping criteria: the number of distinct recovery nodes found in the tree, and the existence of a node that reaches the end of a training episode (i.e., 2016 steps in our case).
Recovery nodes (blue nodes in Figure~\ref{fig:mcts_tree})
represent states that were able to skip at least $t_{skipped}$ environment steps
in a stable grid configuration (i.e., deemed safe by the \emph{Grid State Observer})
before entering a situation where intervention by the agent is required again (i.e., unsafe state).
Once the number of distinct recovery states in the tree exceeds the threshold $t_{stopping}$ the simulation phase is (early-) stopped and the algorithm proceeds to the action selection phase. As an example, if $t_{skipped}=10$ and $t_{stopping}=6$ then the MCTS search with the current tree depicted in Figure~\ref{fig:mcts_tree} would be stopped at this point.
Our experiments in Section~\ref{sec:experiments} show that the effective number of MCTS simulation steps
until the early stopping criterion is hit
steadily decreases throughout training as the prior of the policy network improves.

Since in many cases MCTS is aborted early due to early stopping,
the action selection based on the visitation count (used in the original algorithm) has to be adjusted.
With the main objective in mind -- keeping the grid up and running
in a stable state for as long as possible (see Section~\ref{subsec:power_grid}) -- we define the action strategy as: choose the action that corresponds to the child node leading to the maximum number of reachable steps in the tree.
In our example in Figure~\ref{fig:mcts_tree}, this corresponds to picking the action with id $1646$.
As~\citet{Browne2012mcts} already suggested in their survey paper
we are also able to improve the efficiency of MCTS by
incorporating domain knowledge into the search process.

\subsection{Heuristic Value Function}
Another modification we make to the AlphaZero algorithm is
the use of an engineered heuristic value function instead of a learned neural network.
This design choice enables us,
similar to early stopping and action selection,
to incorporate domain knowledge into the algorithm
resulting in a speed up (reduced training time) and more stable learning behaviour.
In our experiments learning the value function was inferior
compared to the heuristic value with respect to
the required number of training samples,
final performance of the agent as well as training stability\footnote{
This might only hold in our low resource training regime (1~GPU, 64 CPU cores) compared to the original AlphaZero training setup and might flip when scaling up compute.}.
Additionally, in contrast to Chess, Go or Shogi
it is easier to design a non-sparse reward for the power grid environment.
In general, the value for a node is the accumulated value of the tree branch below
divided by the visitation count of the node.
Whenever a new leaf node is added (expanded) to the MCTS tree
its initial value $\nu$ is predicted using either the value network
or in our case a heuristic value function.
Subsequently, the value gets back-propagated up the tree to the root node using the discounting factor $\gamma$. Averaging the value of a node in such a way further stabilizes the search especially if the reward is intermediate as is the case with the power grid environment.
The heuristic value function for state $S_t$ employed in our experiments
is based on the assumption that the current reward will remain similar for subsequent states.
Although not entirely correct, this assumption is still reasonable
as power grids are not expected to exhibit substantial state changes within a few time steps
except for unforeseeable events, which are also not predictable by a learned value function.
In particular, the heuristic value is defined as the asymptotic value
$\nu_t = \sum_{j=t}^{t+h} \gamma^j r_j $
with some large enough horizon $h$ (depending on $\gamma$)
where $r_j$ is the reward received at time step $j$.

\section{Joint Topology Optimization and Redispatching}
\label{sec:join_topo_redis}

This section describes the redispatching controller
used as a baseline agent in~our experiments,
and how it is combined with our learned topology agent.

\textbf{Redispatching Controller.}
Conceptually, joint optimization is applicable to any redispatcher,
including the mature systems deployed in grid operation environments.
For our experiments, we opted for a runtime-efficient approximation approach,
based on Cross-Entropy (CE) optimization~\citep{boer2005ce}\footnote{
Competitive with the Grid2op's official convex optimization baseline.}.
As a cost function, we simulate the sampled action (see Section~\ref{subsec:observation_space}) and calculate the sum of all overflowing line loads (or take the maximum line load if no overflows remain).

\textbf{Redispatching vs. Topology Optimization.}
During congestion, we attempt to select the~most suitable remediation measure at~hand.
In cases when a \textit{topology-only action} (from our learned topology agent)
or a \textit{redispatching-only action} (from the redispatching controller) resolves the congestion (in simulation),
it is selected for execution.
If both resolve the congestion (all loads $<98\%$)
the topology action is preferred to save redispatching costs.
In cases when none of these two resolves the congestion, we employ superimposed redispatching.

\textbf{Superimposed Redispatching.} We expect that not all topology actions are equally well-suited to be combined with redispatching.
Hence, we work with the top-5 topology optimization candidates, re-running the redispatching optimization for each one, taking the proposed topology action into account. This enables the redispatching optimizer to tailor the redispatching action to the proposed topology optimization, and provides a joint topology and redispatch action. We simulate the candidates and pick the action that results in the lowest simulated remaining congestion.

\iffalse
\textbf{Combining Redispatching with Topology Optimization.}
During congestion, our approach attempts to select the most suitable remediation measure for the~situation at~hand, combining the outputs of~our topology agent and redispatching optimizer.

In cases when a \textbf{plain topology} or a \textbf{plain redispatching} action resolves the congestion (i.e., the agent's simulation suggests that all lines will return below their thermal limits), the plain action that reduces the remaining congestion further is chosen.
In cases when none of these two resolves the congestion, we employ \textbf{superimposed redispatching}.

We expect that not all topology actions are equally well suited to be combined with redispatching.
Hence, we work with the top-5 topology optimization candidates, re-running the redispatching optimization for each one, taking the proposed topology action into account. This enables the redispatching optimizer to tailor the redispatching action to the proposed topology optimization, and provides a joint topology and redispatch action. We simulate the candidates and pick the action that results in the lowest simulated remaining congestion.
\fi

\section{Experimental Evaluation}
\label{sec:experiments}
This section describes agent training,
experimentally evaluates the congestion management potential of the proposed topology optimizer
and tests its interoperability with traditional measures.

\textbf{Experimental Environment.}
Experiments are carried out on the WCCI 2022 power grid
consisting of 118 substations, 186 power lines, 62 generators, 91 loads and 7 storages (see Appendix).
Each episode is a replay of generated loads and generation patterns of 2016 time steps ($\Delta_t=5min$)
resulting in a total time span of seven days.
The environment is designed to also test the ability of agents to establish n-1 grid stability
by randomly disconnecting one out of a predefined set of power lines
at randomly sampled time steps through the episode replays
(also referred to as \emph{opponent attacks}).

\textbf{Unitary Action Reduction.}
The WCCI 2022 grid allows for a total of 72958 topological actions (see~Section~\ref{subsec:action_space}).
Given the size of this action space, it is infeasible to
directly train a policy with AlphaZero and a reasonable amount of compute budget.
Both the output layer of the policy network predicting the prior
as well as the branching of MCTS nodes would exhibit a dimensionality of 72958.
To mitigate this issues we reduce this original action set
by running a brute force search for the top 2000 most frequent actions~\citep{marot2021retrospective}.
This reduced set is the basis for all topology agents in the experiments below.
Researching more elegant solutions for dealing with the large action space
is a promising open research direction,
which we discuss further in Section~\ref{sec:practical_relevance}.

\textbf{Training.} For training our topology policy network with AlphaZero, we have 1662 generated training replays (scenarios) available.
For each training rollout, the random generator of the opponent is seeded differently,
which allows us to increase the effective number of training episodes available.
Figure~\ref{fig:training_curves} shows the evolution of some evaluation measures of the topology agent during training.
\begin{figure}[h]
  \centerline{\includegraphics[width=.98\textwidth]{figures/training_curves.pdf}}
  \caption{Evolution of the topology agent during training
  (mean values per epoch).

  }
  \label{fig:training_curves}
\end{figure}

We can see that the average ratio of steps survived already improves to over 65\% after the first training epoch,
and continues to increase throughout the entire training process to 75\%.
Interestingly, along with the performance improvement,
we observe a decrease in the number of substations required for relieving congestions,
indicating that the agent discovers more efficient strategies to utilize the available grid topology.
This is also in line with the right-most plot showing a continuous increase of the average MCTS search depth,
allowing the agent to plan and test topological action sequences several time steps into the future.
Note that this increase in the temporal planning horizon takes place even though
the number of simulations required until the early-stopping criteria are met
continues to decrease.
This behavior is a result of the improved accuracy of the learned policy network
because it provides the prior of the PUCT action selection strategy of MCTS.

\textbf{Evaluation.} For evaluation we consider the 52 held out scenarios provided along with the WCCI 2022 grid
and ten different random seeds resulting in 520 distinct evaluation episodes.

As baselines (BL) we consider
\textbf{NoOp (BL)}, an agent taking no grid operations at all,
\textbf{R (BL)}, relying solely on traditional congestion management measures such as redispatching and curtailment
as well as \textbf{T (brute-force, BL)}, a brute force topology search,
trying all available actions by testing them with the forecasting simulation
on the current grid state and selecting the one with minimum remaining congestion
(analogous to the action space reduction described above).

Given the policy network produced by AlphaZero training,
we formulate three different agents  relying solely on topology optimization.
\textbf{T (arg-max)} directly selects the top prediction of the policy network for execution.
\textbf{T (top-25)} simulates the top 25 predictions of the policy network on the current grid state and selects the one with minimum remaining congestion
analogously to the brute force baseline.
\textbf{T (MCTS)} runs a full topology optimization tree search as during AlphaZero training,
utilizing the learned policy network as a prior for the tree policy
along with environment oracle information (i.e., it is able to look into the future).
Hence, it can be seen as a theoretical performance upper bound for topology-based congestion management on the evaluation grid.

Finally, we define a combined agent \textbf{T (top-5) + R},
testing the interoperability of the trained topology agent with joint redispatching.
This is relevant for two main reasons:
(1) from a grid operation perspective, it is the most relevant setting
with respect to deploying a topology optimization agent in practice, and
(2) from a RL perspective, it is of interest because redispatching was never applied during training,
and the experiments reveal the robustness of the topology agent with respect to the changed observations (distribution shift).
Further details for this agent are provided in Section~\ref{sec:join_topo_redis}.

\begin{table}[h]
  \caption{Comparison of policies on the 52 WCCI 2022 evaluation episodes and ten different random seeds.
  All measures except \emph{Steps Survived} are averaged by the total steps survived
  to allow for a fair comparison independent of the agent's performance.}
  \label{tab:sample-table}
  \centering
  \begin{tabular}{lcccc}
    \toprule
    Agent      & Steps Survived  & Step Time[ms] & Redispatch[MW] & Curtailment[MW] \\
    \midrule
    NoOp (BL)               & 19.2\% & 8.9 & 0.0 & 0.0 \\
    R (BL)                  & 74.5\% & 31.0 & 504.2 & 484.4 \\
    \midrule
    T (brute-force, BL)     & 61.1\% & 153.3 & 0.0 & 0.0 \\
    T (arg-max)             & 50.4\% & 13.4 & 0.0 & 0.0 \\
    T (top-25)              & 65.3\% & 34.2 & 0.0 & 0.0 \\

    T (MCTS, oracle)        & 76.9\% & 1714.2 & 0.0 & 0.0 \\
    \midrule

    T (top-5) + R           & 82.1\% & 53.3 & 202.8 & 193.4 \\
    \bottomrule
  \end{tabular}
\end{table}
Table~\ref{tab:sample-table} summarizes the performance of the respective agents
and compares them based on the overall steps survived,
the average time required to take one environment step (measured on a single CPU core),
as well as the energy consumed by traditional congestion management measures.
The NoOp baseline maintains the grid for about 19\% of overall steps
and does not introduce any congestion management costs.
R (BL), the agent applying traditional congestion management measures,
maintains the grid for 74.5\% of the steps,
but at the average redispatching and curtailment costs of 988MW per step.
By contrast, the topology-based agents do not introduce any congestion management costs.
While T (brute-force, BL) maintains the grid for 61\% of the steps
the plain policy network T (arg-max) only achieves 50\%.
T (top-25) survives 65\% of the steps and hence outperforms the brute force agent.
This is an interesting results
as it shows that the already reduced action space dimensionality of 2000 still benefits from simulation testing.
More importantly it also shows that it is superior to the greedy brute force search
revealing long-term dependencies between subsequent topological actions.
This observation becomes even more evident
when taking the performance of the oracle MCTS agent into account,
which achieves almost 77\% of steps survived.

T (top-5) + R, the agent combining topology optimization with traditional measures,
achieves 82\% of overall maintained steps,
outperforming all remaining agents
while at the same time reducing the average required redispatching power to 396.2 MW;
only 40\% of the original baseline costs.
This is a promising finding as it confirms that
topology optimization is indeed complementary to traditional congestion management measures
but more importantly bears a great potential for
stabilizing the grid and reducing cost while at the same time reducing emissions.
A closer look at the selected actions additionally
shows that the top topology action candidate
is not always the one most compatible with superimposed redispatching (only in 65\% of the cases).
This suggests that incorporating redispatching already in the training process of the topology agent
is another promising research direction.

\section{Conclusions and Practical Relevance for the Real Power Grid}
\label{sec:practical_relevance}
We propose an AlphaZero-based power grid topology agent
and show how to combine it with traditional congestion management measures.
This joint agent improves the grid resilience
while at the same time reducing costly measures to only 40\% of the original costs.
In the last section of this paper, we
emphasize the practical relevance of the proposed agents with the example of two real world use cases,
and discuss open research problems.

\textbf{Use Case 1: Day-Ahead Planning.}
A common practice in real-world grid operation is day-ahead planning,
where a congestion management plan is created for the~entire day ahead,
based on the market outcomes, available demand and renewable generation forecasts,
and a load flow simulation of the real grid.
Both the availability of forecasts and of the load flow simulator
make the~day-ahead planning routine well suited for RL-based power grid operation planning.

\textbf{Use Case 2: AI-Assisted Control.}
 AI-assisted control addresses the exploration problem by utilizing an RL-based agent for generating remediation action candidates. These are then evaluated by a human (or any conventional process) in order to maintain the strict safety and interpretability requirements that power grids, as critical infrastructure, place on all implemented measures.
 We present a web application that demonstrates an elementary version of AI-assisted control with multi-candidate, multi-step congestion remediation recommendations on a simulated power grid\footnote{Please see the Appendix for more information, or visit our demo at:
 \url{http://grid-demo.enlite.ai/}
 }.

\textbf{Open Problem 1: Redispatching Integration.}
Our experimental results reveal that the efficient and effective combination of redispatching with topology optimization is a promising open research direction.
In practice, costly congestion management measures are usually favored over topology corrections.
At the same time, our approach shows that joint actions have a strong potential. However, the current setup does not yet allow for more coherent and co-operative joint actions as it is, e.g., infeasible to train our topology optimization agent together with existing redispatching controllers, due to computational limitations.
Exploring ways for computationally more efficient redispatching would open the door
to combining the two paradigms already during training.
This would allow the topology agent to adjust its control strategies with respect to the redispatcher\footnote{
Our initial attempts to approximate redispatching with behavioural cloning have not been successful.}.

\textbf{Open Problem 2: Action Representation.}
To help mitigate the exploration problem, agents have so far been either developed and tested on moderately sized power grids,
still exhibiting a~tractable exploration space, or,
as in our case, utilize only a drastically reduced set of available topology actions.
We think that devising improved, potentially domain-knowledge informed action space formulations would allow the agent to better leverage the full optimization potential of the underlying power grid.

\textbf{Conclusions.} Given the promising results, we see RL-based optimization
as a promising avenue for unlocking the potential of topology-based congestion management
and the efficient utilization of renewable energy generation, contributing to the societal goal of decarbonizing the energy sector.

\medskip

\begin{ack}
We would like to thank RTE France for developing and providing the Grid2Op environment
enabling the line of RL(Agent)-based research on power grids.
We would also like to thank Antoine Marot (RTE France) and Jan Viebahn (TenneT) for useful discussions
and Benjamin Donnot (RTE France) for his fast \textit{lightsim2grid} loadflow simulation backend.
\end{ack}

\small
\bibliographystyle{authordate1}

\section{Introduction}
\label{sec:introduction}

Interconnected electrical power grids are a critical infrastructure of modern society. The operation of these grids is a demanding control task, requiring uninterrupted monitoring by skilled experts and frequent interventions to safely and reliably transport electricity from the producers (generators) to all connected consumers (loads)~\citep{Kelly2020gridrl}. Load flow calculation is a well-established method for the simulation of grid states and it is routinely used by grid operators to ensure that all security constraints are met~\citep{stott1974loadflow}. The existence of fast and reliable simulators pave the way for powerful tools based on Reinforcement Learning (RL)~\citep{marot2019l2rpn}, supporting operators with the increasingly difficult challenges in the ongoing effort to decarbonize the energy sector and ensure security of supply~\citep{marott2022towards,marot2022perspectives}.

The shift towards renewable electricity generation is advancing rapidly. Globally, 80\% of the newly installed generation capacity is already based on renewable sources, where solar and wind account for the largest share\footnote{Data provided by the International Renewable Energy Agency, \href{https://pxweb.irena.org/pxweb/en/IRENASTAT}{IRENASTAT}, retrieved Sept 2022.} (see Figure~\ref{fig:renewables_and_congestion}, left). Compared to non-renewable technologies such as thermal generation from fossil fuel, solar and wind energy is highly volatile, confronting grid operators with large power gradients and increased uncertainties. This makes more frequent interventions necessary in order to prevent individual grid elements from exceeding operational limits (grid congestion)~\citep{han2015congcasestudy}.
Figure~\ref{fig:renewables_and_congestion}, right shows an exemplary congestion scenario.

\begin{figure}[t]
  \centerline{\includegraphics[width=.43\textwidth]{figures/generation_stats.pdf}\includegraphics[width=.57\textwidth]{figures/redispatching.png}}
  \caption{Left: Total global generator capacity expansion per year (orange: non-renewables, blue: renewables, green: share of new renewable generation)
Right: Simplified congestion scenario with surplus power from wind farms and transmission to large remote consumers and a neighbouring country. Load flow analysis reveals a congested transmission line. Curtailing wind power and compensating for the spilled renewable energy by ramping up a thermal generator is the prevalent remedial action applied in practice.}
  \label{fig:renewables_and_congestion}
\end{figure}

\textbf{Power Grid Basics.}
Laying out the electrophysical foundations and covering all common grid elements
is beyond the scope of this paper.
Fortunately, this complexity is largely encapsulated in the simulation tools of the grid operators. A more detailed presentation is provided in~\citep{Kelly2020gridrl}.

\textit{Grid model}. In essence, a power grid connects generators (producers) to loads (consumers) and is modeled as an undirected multigraph,
with edges representing transmission lines (carrying energy over long distances) and nodes representing substations (switchable connection points)\footnote{Throughout the paper we adopt the view of a transmission grid operator, but note that the same concepts apply to the distribution grid as well.}.
Relevant grid elements are presented in more detail in Section~\ref{sec:env_disign}.

\textit{Congestion management}. Given an expected load and generation scenario,
grid security analysis is conducted as part of the operational planning process~\citep{shahidehpour2005gridsecurity}. If the security analysis reveals violations of operational constraints,
grid operators are required to intervene to keep the grid stable and reliable.

\textit{Costly and non-costly measures}. Operators can relieve congested lines by altering the power entering and leaving the grid, which involves high costs from modifying previously agreed transmissions and procuring energy from different generators~\citep{VANDENBERGH2015redispatching}. From an economic perspective, it is highly desirable to jointly optimize these costly measures with the non-costly measure of dynamically reconfiguring the grid by making use of existing switchable equipment~\citep{hedman2011toporeview}.

\textbf{Related Work.}
Optimal line switching was first proposed by~\citet{fisher2008topo} as a mixed linear integer program, based on a linear approximation of the load flow calculation (DC formulation). \citet{hedman2009lineswitching} added security analysis and a heuristic strategy to obtain approximate results for the IEEE 118 bus system.
Note that line switching does not make use of the flow rerouting ability of the substations. Subsequent work~\citep{heidarifar2016topo, xiao2018topo, li2019topo, zhou2021topo} includes more complete modeling of substations at the circuit breaker level, but continues to rely on linear approximations of the grid dynamics in order to apply commercial solvers.
Recent editions of the Learning to run a Power Network Challenge~\citep{marot2021retrospective,marot2021l2rpntrust}
reveal the first promising results of RL-based topology optimizers.
\citet{zhou2021es} propose to combine search-based planning with the predictions of a policy network
learned via large scale Evolutionary Strategy (ES) training.
Their approach is able to discover long-term strategies,
but struggles with the inherent limitation of being sample inefficient.
\citet{roychowdhury2022toporl} train a value network with Dueling DQN (DDQN)
for line switching and topology control at a substation level.
The Authors then evaluate their approach on the IEEE 14 bus system.
Using the same testbed~\citet{subramanian2021simplerl} approached the problem
by utilizing a cross-entropy optimizer for topological switching.
By directly predicting the desired target topology and subsequently planning an action sequence towards this target topology
~\citet{Yoon2021afterstate} propose an off-policy actor-critic approach
adopting the hierarchical policy together with the afterstate representation.
Related to our approach,~\citet{taha2022mcts} propose to learn a line load dynamics model, which is then
utilized in a tree search for planning viable topological remedial actions.
However, this tree search is not yet guided by a learned policy network,
making it inefficient and unlikely to find optimal solutions for upcoming congestion situations
within the~exponential topology exploration space.

\textbf{Our Contributions}.
We pick up this line of research and propose an AlphaZero-based topology optimization agent~\citep{silver2017alphazero}
to address power grid congestion management.
Our choice is motivated by recent successes of AlphaZero
for similar large-scale combinatorial optimization problems.
We benchmark our topology agent on a 118 substation power grid
and demonstrate how to successfully combine it with traditional congestion management measures.
Finally, we discuss the~practical relevance of such an agent
using the examples of day-ahead planning and real-time remedial action recommendation,
and outline promising further research directions.

\section{Environment Design and Interaction Workflow}
\label{sec:env_disign}
This section describes the power grid environment, its observation and action spaces, reward as well as custom modules.

\subsection{Power Grid Environment Overview}
\label{subsec:power_grid}

We utilize the Grid2Op framework~\citet{grid2op} provided by RTE France, Europe's largest grid operator. A power grid consists of generators, loads, storages, substations, and power lines. We provide a brief overview below; please refer to the \href{https://grid2op.readthedocs.io/en/latest/}{official documentation} for the full specification.

\begin{figure}[h]
  \centerline{\includegraphics[width=0.99\textwidth]{figures/topology_action_example.pdf}}
  \caption{Example topology action. Node splitting is performed by switching the overflowing line to the second bus, changing the topology of the network and resolving the overflow. Note that in the Grid2Op framework~\citet{grid2op}, each substation features two buses. In the real world, a~substation might feature even more than two buses.
  (Figure adopted from~\citep{marot2021retrospective}.)}
  \label{fig:topology_action_example}
\end{figure}

\textbf{Generators, Loads and Storages.} \emph{Generators} produce power. Renewable generators (sun, wind, hydro) follow simulated weather patterns; thermic generators (coal, gas, nuclear) follow predefined output curves. As a congestion management measure, the output of generators can be influenced---thermics can be redispatched (up or down) and renewables curtailed (down only). \emph{Loads} consume power, following a predefined pattern unknown to the agent. \emph{Storages} store power across time steps, and can be gradually charged or discharged.

\textbf{Transmission: Substations and Lines.} \emph{Substations} connect elements (generators, loads, lines), and consist of two buses. All connected elements need to be assigned to a bus, configuring the topology of the grid. For an example of how bus switching affects the topology of a network, refer to Figure~\ref{fig:topology_action_example}. \emph{Power lines} connect substations. Each line has a thermal limit (i.e., capacity), which it can exceed only for a small number of time steps---then an automated disconnect occurs. Severe overflows (>~200~\%) cause an immediate disconnect, and often lead to cascading line failures.

\textbf{Objective.} The goal is to keep the grid in operation, avoiding congestions and line disconnects
that can lead to isolation of loads or generators---i.e., a diverging power flow (black out).
The secondary objective is to minimize operational costs incurred by power loss and congestion management.

\subsection{Observation Space}
\label{subsec:observation_space}

\textbf{Overview.} We utilize the \texttt{CompleteObservation}\footnote{Please refer to the documentation of \texttt{CompleteObservation} for a complete description (including, e.g., maintenance and cool down times): \url{https://grid2op.readthedocs.io/en/latest/observation.html}}, which we further process
into feature representations.
The observation space includes (most notably, but not limited to):
\textit{generator and loads state}, including power production/consumption and its forecasts;
\textit{power line state}, including the power flow at origin and extremity, and the~current load as a fraction of capacity (denoted as~$\rho$, one of the most important metrics);
\textit{topology configuration}, with the power lines status and bus configurations for each element; and, finally,
\textit{redispatch, curtailment and storage states}.

\textbf{Simulation.} A limited simulation functionality for planning purposes is available to the~agent. Compared to the complete power grid environment (which is not available to the agent), it features only forecasts of future load, generator, and weather patterns. Hence, while the agent can test the~general impact of an~action, the accuracy varies.

\subsection{Action Space and Combinatorial Complexity}
\label{subsec:action_space}

An action can modify topology, redispatching, curtailment and storage elements\footnote{For a more information, please review the action documentation: \url{https://grid2op.readthedocs.io/en/latest/action.html} and operational cost overview: \url{https://codalab.lisn.upsaclay.fr/competitions/5410\#learn_the_details-evaluation}}.

\textbf{Non-Costly Measures.} \textit{Topology actions} (the only non-costly measure) modify the bus configuration of one~grid element (either all connections on one line or one substation). As each substation features two buses, there are overall $2^{k}$ possible topological configurations, where $k$ is the number of connections in the grid (i.e., all elements on all substations). While not all configurations are valid,
this still yields an exponential exploration space, presenting a challenge for both RL-based and traditional approaches.

\textbf{Costly Measures.} \emph{Redispatching actions} (incurring redispatching costs) modify the target output of the redispatchable generators (up or down), and need to be within the per-generator bounds. \emph{Curtailment actions} (incurring curtailment costs) limit the production of renewable generators. \emph{Storage actions} (incurring loading, preserving, and discharging costs) direct the behavior of storages.

Note that in a real grid, a few more congestion remediation measures are available, such as counter trading, demand-side management or load shedding. However, the actions described above (and simulated in the Grid2Op environment) represent the main remediation options at hand.

\textbf{Action Space.} Our topology optimization agent utilizes the unitary action space provided by Grid2Op. This space represents all individual available topology actions (i.e., manipulating all combinations of bus connections, per grid element) as integers, creating a discrete action space. To reduce the action space size, we then perform action masking and action reduction (see Section~\ref{sec:experiments}).

\subsection{Environment Dynamics, Interaction Workflow and Reward}
\label{subsec:env_dynamics}

This section describes custom modules engaging in observation processing and environment control.

\textbf{Grid State Observer and Safe-State Skipping.} In most time steps, the grid is running safely, and launching our congestion remediation agent is neither necessary nor desirable (due to additional operational costs). Hence, we integrate a \emph{Grid State Observer} that determines whether the current state is considered safe by evaluating the current maximum line load\footnote{The \emph{Grid State Observer} is a customizable module and it can consider also, e.g., an n-1 contingency analysis.}.
In safe states, the congestion remediation agent is skipped, and either a recovery (see below) or a no-operation action is performed.

\textbf{Topology Recovery.} In the default topology, all connections are switched to the same bus, i.e., no node splitting takes place (see Figure~\ref{fig:topology_action_example} for a node splitting explanation). We find that this ,,fully connected`` setting is rather resilient, and a good starting point for congestion management. However, due to the reduced action set (see Section~\ref{sec:experiments}), the agent does not always have all the actions available required to recover this topology. Hence, in safe states, we perform automated topology recovery.

\textbf{Rewards.} In case of a congestion, the topology optimization agent is triggered. We utilize a shaped reward based on the cumulative sum of all overflowing line loads, which the agent aims to minimize.

\section{Topology Optimization with AlphaZero and Heuristic Value Functions}
\label{sec:alpha_zero}
In this section, we outline our customized AlphaZero algorithm employed with a heuristic value function. AlphaZero~\citep{silver2017alphazero}
constitutes an RL algorithm using a general-purpose Monte Carlo Tree Search (MCTS). Each search consists of a fixed number of simulations, starting at the root node, and traversing the tree until a new leaf node is added to the tree. Which action is selected at each branching in the tree is guided by the probabilistic upper confidence tree (PUCT) bound~\citep{rosin2011multi} taking into account the visitation count of the current and potential child node, the action probability sampled from the policy network and the value assigned to the resulting state when performing the respective action. Once the desired number of simulations has been reached, the action to perform is sampled from all expanded children of the root node w.r.t. the probability distribution obtained by taking the softmax of their visitation counts. In the greedy setting, the action with the highest visitation count is selected.
The original implementation utilizes this MCTS formulation and self-play~\citep{silver2017human} to collect state-action pairs (or trajectories).
These state-action pairs are then used to continuously update the policy network with a behavioural cloning algorithm. The three games studied have a reward of $0$ in all but the last time-step, where the reward is $1$ if won, $-1$ if lost and $0$ if the game draws. This 'terminal value' is then paired with all states of the corresponding trajectory and used for value function learning.

MuZero~\citep{schrittwieser2020mastering} is a model-based extension of AlphaZero,
which still relies on MCTS but generalizes to single-player domains and infinite horizon problems with intermediate rewards. While the policy targets stay the same as in AlphaZero,
the value targets need to be adapted for the intermediate rewards. Specifically the return $z_t$ is computed for each state in the trajectory with $z_t = r_{t} + \gamma r_{t+1}  + \gamma^2 r_{t+2}  + \dots + \gamma^n \nu_{t+n} $ where $\nu_t$ is the prediction of the return by the value network for time step $t$. Note, that the prediction $\nu$ can be neglected if the given environment has no infinite horizon but a max reachable time step $t_{max}$ and $n=t_{max}$.

While managing a power grid is not a two-player game like Chess or Go,
it still bears some inherent similarities
when it comes to the long-term effect a single action can have on the development of the grid (``game'')
as well as the exponential nature of the action space (see Section~\ref{subsec:action_space}).
Since AlphaZero has already proven superior performance for problems
where long-term planning in large combinatorial action spaces is required,
we see it as a natural fit for the topology optimization problem in large power grids.
Furthermore, power grid management is a domain
where fast and accurate simulations have been optimized for decades
and are already widely used in practice, e.g., for day-ahead or intra-day operation planning.
The nature of the optimization problem
as well as the fact that power grid load flow simulators are already established in practice
set the stage for our modified version of the AlphaZero algorithm in the single-player infinite horizon power grid setting also utilizing modifications of MuZero.

\begin{figure}
  \centerline{\includegraphics[width=1\textwidth]{figures/mcts_tree_example.png}}
  \caption{Example of a grid topology MCTS tree. Nodes represent states  (root is the current state); edges are simulated actions. Yellow nodes are critical states (maximum line load > 98\%), red nodes are black-out states and blue nodes are critical states that internally skipped two or more steps because they were deemed safe by the \emph{Grid State Observer} (see Section~\ref{subsec:env_dynamics}). Edges correspond to topology actions with the action ID (in brackets) and the corresponding upper confidence bound.}
  \label{fig:mcts_tree}
\end{figure}

\subsection{Early Stopping and Action Selection}
Considering that AlphaZero is a tabula rasa learning algorithm, an extensive amount of data needs to be collected during the training process, which requires large-scale parallelization and as a consequence a lot of compute resources. For Chess, Shogi and Go~\citet{silver2017alphazero} report using 5,000 first-generation TPUs~\citet{jouppi2017datacenter} to generate self-play games and 64 second-generation TPUs to train the neural networks.
To substantially reduce the amount of resources required to train such a model to only 64 CPU cores to generate trajectories and 1 GPU to train the neural networks, we devised an MCTS early stopping mechanism that terminates the simulation search
once a good (enough) solution has been found.
For our experiments we use two stopping criteria: the number of distinct recovery nodes found in the tree, and the existence of a node that reaches the end of a training episode (i.e., 2016 steps in our case).
Recovery nodes (blue nodes in Figure~\ref{fig:mcts_tree})
represent states that were able to skip at least $t_{skipped}$ environment steps
in a stable grid configuration (i.e., deemed safe by the \emph{Grid State Observer})
before entering a situation where intervention by the agent is required again (i.e., unsafe state).
Once the number of distinct recovery states in the tree exceeds the threshold $t_{stopping}$ the simulation phase is (early-) stopped and the algorithm proceeds to the action selection phase. As an example, if $t_{skipped}=10$ and $t_{stopping}=6$ then the MCTS search with the current tree depicted in Figure~\ref{fig:mcts_tree} would be stopped at this point.
Our experiments in Section~\ref{sec:experiments} show that the effective number of MCTS simulation steps
until the early stopping criterion is hit
steadily decreases throughout training as the prior of the policy network improves.

Since in many cases MCTS is aborted early due to early stopping,
the action selection based on the visitation count (used in the original algorithm) has to be adjusted.
With the main objective in mind -- keeping the grid up and running
in a stable state for as long as possible (see Section~\ref{subsec:power_grid}) -- we define the action strategy as: choose the action that corresponds to the child node leading to the maximum number of reachable steps in the tree.
In our example in Figure~\ref{fig:mcts_tree}, this corresponds to picking the action with id $1646$.
As~\citet{Browne2012mcts} already suggested in their survey paper
we are also able to improve the efficiency of MCTS by
incorporating domain knowledge into the search process.

\subsection{Heuristic Value Function}
Another modification we make to the AlphaZero algorithm is
the use of an engineered heuristic value function instead of a learned neural network.
This design choice enables us,
similar to early stopping and action selection,
to incorporate domain knowledge into the algorithm
resulting in a speed up (reduced training time) and more stable learning behaviour.
In our experiments learning the value function was inferior
compared to the heuristic value with respect to
the required number of training samples,
final performance of the agent as well as training stability\footnote{
This might only hold in our low resource training regime (1~GPU, 64 CPU cores) compared to the original AlphaZero training setup and might flip when scaling up compute.}.
Additionally, in contrast to Chess, Go or Shogi
it is easier to design a non-sparse reward for the power grid environment.
In general, the value for a node is the accumulated value of the tree branch below
divided by the visitation count of the node.
Whenever a new leaf node is added (expanded) to the MCTS tree
its initial value $\nu$ is predicted using either the value network
or in our case a heuristic value function.
Subsequently, the value gets back-propagated up the tree to the root node using the discounting factor $\gamma$. Averaging the value of a node in such a way further stabilizes the search especially if the reward is intermediate as is the case with the power grid environment.
The heuristic value function for state $S_t$ employed in our experiments
is based on the assumption that the current reward will remain similar for subsequent states.
Although not entirely correct, this assumption is still reasonable
as power grids are not expected to exhibit substantial state changes within a few time steps
except for unforeseeable events, which are also not predictable by a learned value function.
In particular, the heuristic value is defined as the asymptotic value
$\nu_t = \sum_{j=t}^{t+h} \gamma^j r_j $
with some large enough horizon $h$ (depending on $\gamma$)
where $r_j$ is the reward received at time step $j$.

\section{Joint Topology Optimization and Redispatching}
\label{sec:join_topo_redis}

This section describes the redispatching controller
used as a baseline agent in~our experiments,
and how it is combined with our learned topology agent.

\textbf{Redispatching Controller.}
Conceptually, joint optimization is applicable to any redispatcher,
including the mature systems deployed in grid operation environments.
For our experiments, we opted for a runtime-efficient approximation approach,
based on Cross-Entropy (CE) optimization~\citep{boer2005ce}\footnote{
Competitive with the Grid2op's official convex optimization baseline.}.
As a cost function, we simulate the sampled action (see Section~\ref{subsec:observation_space}) and calculate the sum of all overflowing line loads (or take the maximum line load if no overflows remain).

\textbf{Redispatching vs. Topology Optimization.}
During congestion, we attempt to select the~most suitable remediation measure at~hand.
In cases when a \textit{topology-only action} (from our learned topology agent)
or a \textit{redispatching-only action} (from the redispatching controller) resolves the congestion (in simulation),
it is selected for execution.
If both resolve the congestion (all loads $<98\%$)
the topology action is preferred to save redispatching costs.
In cases when none of these two resolves the congestion, we employ superimposed redispatching.

\textbf{Superimposed Redispatching.} We expect that not all topology actions are equally well-suited to be combined with redispatching.
Hence, we work with the top-5 topology optimization candidates, re-running the redispatching optimization for each one, taking the proposed topology action into account. This enables the redispatching optimizer to tailor the redispatching action to the proposed topology optimization, and provides a joint topology and redispatch action. We simulate the candidates and pick the action that results in the lowest simulated remaining congestion.

\iffalse
\textbf{Combining Redispatching with Topology Optimization.}
During congestion, our approach attempts to select the most suitable remediation measure for the~situation at~hand, combining the outputs of~our topology agent and redispatching optimizer.

In cases when a \textbf{plain topology} or a \textbf{plain redispatching} action resolves the congestion (i.e., the agent's simulation suggests that all lines will return below their thermal limits), the plain action that reduces the remaining congestion further is chosen.
In cases when none of these two resolves the congestion, we employ \textbf{superimposed redispatching}.

We expect that not all topology actions are equally well suited to be combined with redispatching.
Hence, we work with the top-5 topology optimization candidates, re-running the redispatching optimization for each one, taking the proposed topology action into account. This enables the redispatching optimizer to tailor the redispatching action to the proposed topology optimization, and provides a joint topology and redispatch action. We simulate the candidates and pick the action that results in the lowest simulated remaining congestion.
\fi

\section{Experimental Evaluation}
\label{sec:experiments}
This section describes agent training,
experimentally evaluates the congestion management potential of the proposed topology optimizer
and tests its interoperability with traditional measures.

\textbf{Experimental Environment.}
Experiments are carried out on the WCCI 2022 power grid
consisting of 118 substations, 186 power lines, 62 generators, 91 loads and 7 storages (see Appendix).
Each episode is a replay of generated loads and generation patterns of 2016 time steps ($\Delta_t=5min$)
resulting in a total time span of seven days.
The environment is designed to also test the ability of agents to establish n-1 grid stability
by randomly disconnecting one out of a predefined set of power lines
at randomly sampled time steps through the episode replays
(also referred to as \emph{opponent attacks}).

\textbf{Unitary Action Reduction.}
The WCCI 2022 grid allows for a total of 72958 topological actions (see~Section~\ref{subsec:action_space}).
Given the size of this action space, it is infeasible to
directly train a policy with AlphaZero and a reasonable amount of compute budget.
Both the output layer of the policy network predicting the prior
as well as the branching of MCTS nodes would exhibit a dimensionality of 72958.
To mitigate this issues we reduce this original action set
by running a brute force search for the top 2000 most frequent actions~\citep{marot2021retrospective}.
This reduced set is the basis for all topology agents in the experiments below.
Researching more elegant solutions for dealing with the large action space
is a promising open research direction,
which we discuss further in Section~\ref{sec:practical_relevance}.

\textbf{Training.} For training our topology policy network with AlphaZero, we have 1662 generated training replays (scenarios) available.
For each training rollout, the random generator of the opponent is seeded differently,
which allows us to increase the effective number of training episodes available.
Figure~\ref{fig:training_curves} shows the evolution of some evaluation measures of the topology agent during training.
\begin{figure}[h]
  \centerline{\includegraphics[width=.98\textwidth]{figures/training_curves.pdf}}
  \caption{Evolution of the topology agent during training
  (mean values per epoch).

  }
  \label{fig:training_curves}
\end{figure}

We can see that the average ratio of steps survived already improves to over 65\% after the first training epoch,
and continues to increase throughout the entire training process to 75\%.
Interestingly, along with the performance improvement,
we observe a decrease in the number of substations required for relieving congestions,
indicating that the agent discovers more efficient strategies to utilize the available grid topology.
This is also in line with the right-most plot showing a continuous increase of the average MCTS search depth,
allowing the agent to plan and test topological action sequences several time steps into the future.
Note that this increase in the temporal planning horizon takes place even though
the number of simulations required until the early-stopping criteria are met
continues to decrease.
This behavior is a result of the improved accuracy of the learned policy network
because it provides the prior of the PUCT action selection strategy of MCTS.

\textbf{Evaluation.} For evaluation we consider the 52 held out scenarios provided along with the WCCI 2022 grid
and ten different random seeds resulting in 520 distinct evaluation episodes.

As baselines (BL) we consider
\textbf{NoOp (BL)}, an agent taking no grid operations at all,
\textbf{R (BL)}, relying solely on traditional congestion management measures such as redispatching and curtailment
as well as \textbf{T (brute-force, BL)}, a brute force topology search,
trying all available actions by testing them with the forecasting simulation
on the current grid state and selecting the one with minimum remaining congestion
(analogous to the action space reduction described above).

Given the policy network produced by AlphaZero training,
we formulate three different agents  relying solely on topology optimization.
\textbf{T (arg-max)} directly selects the top prediction of the policy network for execution.
\textbf{T (top-25)} simulates the top 25 predictions of the policy network on the current grid state and selects the one with minimum remaining congestion
analogously to the brute force baseline.
\textbf{T (MCTS)} runs a full topology optimization tree search as during AlphaZero training,
utilizing the learned policy network as a prior for the tree policy
along with environment oracle information (i.e., it is able to look into the future).
Hence, it can be seen as a theoretical performance upper bound for topology-based congestion management on the evaluation grid.

Finally, we define a combined agent \textbf{T (top-5) + R},
testing the interoperability of the trained topology agent with joint redispatching.
This is relevant for two main reasons:
(1) from a grid operation perspective, it is the most relevant setting
with respect to deploying a topology optimization agent in practice, and
(2) from a RL perspective, it is of interest because redispatching was never applied during training,
and the experiments reveal the robustness of the topology agent with respect to the changed observations (distribution shift).
Further details for this agent are provided in Section~\ref{sec:join_topo_redis}.

\begin{table}[h]
  \caption{Comparison of policies on the 52 WCCI 2022 evaluation episodes and ten different random seeds.
  All measures except \emph{Steps Survived} are averaged by the total steps survived
  to allow for a fair comparison independent of the agent's performance.}
  \label{tab:sample-table}
  \centering
  \begin{tabular}{lcccc}
    \toprule
    Agent      & Steps Survived  & Step Time[ms] & Redispatch[MW] & Curtailment[MW] \\
    \midrule
    NoOp (BL)               & 19.2\% & 8.9 & 0.0 & 0.0 \\
    R (BL)                  & 74.5\% & 31.0 & 504.2 & 484.4 \\
    \midrule
    T (brute-force, BL)     & 61.1\% & 153.3 & 0.0 & 0.0 \\
    T (arg-max)             & 50.4\% & 13.4 & 0.0 & 0.0 \\
    T (top-25)              & 65.3\% & 34.2 & 0.0 & 0.0 \\

    T (MCTS, oracle)        & 76.9\% & 1714.2 & 0.0 & 0.0 \\
    \midrule

    T (top-5) + R           & 82.1\% & 53.3 & 202.8 & 193.4 \\
    \bottomrule
  \end{tabular}
\end{table}
Table~\ref{tab:sample-table} summarizes the performance of the respective agents
and compares them based on the overall steps survived,
the average time required to take one environment step (measured on a single CPU core),
as well as the energy consumed by traditional congestion management measures.
The NoOp baseline maintains the grid for about 19\% of overall steps
and does not introduce any congestion management costs.
R (BL), the agent applying traditional congestion management measures,
maintains the grid for 74.5\% of the steps,
but at the average redispatching and curtailment costs of 988MW per step.
By contrast, the topology-based agents do not introduce any congestion management costs.
While T (brute-force, BL) maintains the grid for 61\% of the steps
the plain policy network T (arg-max) only achieves 50\%.
T (top-25) survives 65\% of the steps and hence outperforms the brute force agent.
This is an interesting results
as it shows that the already reduced action space dimensionality of 2000 still benefits from simulation testing.
More importantly it also shows that it is superior to the greedy brute force search
revealing long-term dependencies between subsequent topological actions.
This observation becomes even more evident
when taking the performance of the oracle MCTS agent into account,
which achieves almost 77\% of steps survived.

T (top-5) + R, the agent combining topology optimization with traditional measures,
achieves 82\% of overall maintained steps,
outperforming all remaining agents
while at the same time reducing the average required redispatching power to 396.2 MW;
only 40\% of the original baseline costs.
This is a promising finding as it confirms that
topology optimization is indeed complementary to traditional congestion management measures
but more importantly bears a great potential for
stabilizing the grid and reducing cost while at the same time reducing emissions.
A closer look at the selected actions additionally
shows that the top topology action candidate
is not always the one most compatible with superimposed redispatching (only in 65\% of the cases).
This suggests that incorporating redispatching already in the training process of the topology agent
is another promising research direction.

\section{Conclusions and Practical Relevance for the Real Power Grid}
\label{sec:practical_relevance}
We propose an AlphaZero-based power grid topology agent
and show how to combine it with traditional congestion management measures.
This joint agent improves the grid resilience
while at the same time reducing costly measures to only 40\% of the original costs.
In the last section of this paper, we
emphasize the practical relevance of the proposed agents with the example of two real world use cases,
and discuss open research problems.

\textbf{Use Case 1: Day-Ahead Planning.}
A common practice in real-world grid operation is day-ahead planning,
where a congestion management plan is created for the~entire day ahead,
based on the market outcomes, available demand and renewable generation forecasts,
and a load flow simulation of the real grid.
Both the availability of forecasts and of the load flow simulator
make the~day-ahead planning routine well suited for RL-based power grid operation planning.

\textbf{Use Case 2: AI-Assisted Control.}
 AI-assisted control addresses the exploration problem by utilizing an RL-based agent for generating remediation action candidates. These are then evaluated by a human (or any conventional process) in order to maintain the strict safety and interpretability requirements that power grids, as critical infrastructure, place on all implemented measures.
 We present a web application that demonstrates an elementary version of AI-assisted control with multi-candidate, multi-step congestion remediation recommendations on a simulated power grid\footnote{Please see the Appendix for more information, or visit our demo at:
 \url{http://grid-demo.enlite.ai/}
 }.

\textbf{Open Problem 1: Redispatching Integration.}
Our experimental results reveal that the efficient and effective combination of redispatching with topology optimization is a promising open research direction.
In practice, costly congestion management measures are usually favored over topology corrections.
At the same time, our approach shows that joint actions have a strong potential. However, the current setup does not yet allow for more coherent and co-operative joint actions as it is, e.g., infeasible to train our topology optimization agent together with existing redispatching controllers, due to computational limitations.
Exploring ways for computationally more efficient redispatching would open the door
to combining the two paradigms already during training.
This would allow the topology agent to adjust its control strategies with respect to the redispatcher\footnote{
Our initial attempts to approximate redispatching with behavioural cloning have not been successful.}.

\textbf{Open Problem 2: Action Representation.}
To help mitigate the exploration problem, agents have so far been either developed and tested on moderately sized power grids,
still exhibiting a~tractable exploration space, or,
as in our case, utilize only a drastically reduced set of available topology actions.
We think that devising improved, potentially domain-knowledge informed action space formulations would allow the agent to better leverage the full optimization potential of the underlying power grid.

\textbf{Conclusions.} Given the promising results, we see RL-based optimization
as a promising avenue for unlocking the potential of topology-based congestion management
and the efficient utilization of renewable energy generation, contributing to the societal goal of decarbonizing the energy sector.

\medskip

\begin{ack}
We would like to thank RTE France for developing and providing the Grid2Op environment
enabling the line of RL(Agent)-based research on power grids.
We would also like to thank Antoine Marot (RTE France) and Jan Viebahn (TenneT) for useful discussions
and Benjamin Donnot (RTE France) for his fast \textit{lightsim2grid} loadflow simulation backend.
\end{ack}

\small
\bibliographystyle{authordate1}

redpajama_set_name: RedPajamaArXiv


text:
Northern Rock customers to suffer say National Homebuyers
There are concerns that Northern Rock mortgage borrowers could face financial difficulties as a result of the bank's nationalisation.
[UKPRwire, Wed Feb 27 2008] Alistair Darling announced that the bank would become publicly owned yesterday (February 17th).
The move has sparked huge debate about the reputation of British banking but mortgage advisor Ray Boulger has highlighted that many borrowers may face bigger problems in the form of financial difficulties.
"At the moment most people who have a mortgage with Northern Rock which is coming to the end of the deal have been moving away for the last few months, just because Northern Rock's deals are uncompetitive," Mr Boulger told Channel Four's News at Noon.
"That is what most people will continue to do. If they can't move away for any reason, perhaps because their credit status is not very good, or because the value of their property has not risen and they are still borrowing over 100 per cent, they may have no choice but to stay with Northern Rock and either go onto their standard variable rate or perhaps take another uncompetitive deal," he added.
Fast property sale expert Julian King adds, "People looking for a quick property sale are encouraged to seek an alternative to the traditional open market approach.
"Selling a property through an estate agent now incurs not only a commission payment but also the cost of a Hips pack. By selling to fast property buyers like National Homebuyers or Sure Offer, the homeowner can avoid the costs and the likely delays and uncertainties of selling on the open market."
Mr King is a director of National Homebuyers, the UK's largest quick sale property firm. He is also a founder member of APBA [Approved Property Buyers Association] which is calling for regulation in the quick sale and Sell and Rent Back industries.
Company: National Homebuyers
Contact Name: Jayne Cornford
Contact Email: jayne.cornford@nationalhomebuyers.co.uk
Contact Phone: 08709798118

redpajama_set_name: RedPajamaCommonCrawl


I need a bar code scanner for automotive VIN scanning to use at an auction.
Barcode › Questions & Answers › I need a bar code scanner for automotive VIN scanning to use at an auction.
Guest I need a bar code scanner for automotive VIN scanning to use at an auction.
Since VIN codes are often located in the dash or door of the vehicle I would suggest an imager based scanner like the Honeywell Xenon 1900. This scanner is a camera based imager which allows reading through a windshield with much better performance than a typical laser scanner. If you need something more mobile there is the Xenon 1902 which is a cordless version of the scanner.


For more information, please send an email to cars@gardenstatemodela.org.
The GSMAR logo is the property of Garden State Model A Region of the AACA.
No part may be copied or reprinted without the written consent of the owner.
Please send comments about this website to webmaster@gardenstatemodela.org.


Description: Clove is derived from a slender evergreen tree that grows up to 12m. At the start of the rainy season, long buds appear. These change colour and are beaten from the tree and dried. These are the cloves sold commercially.
Common uses: Clove bud oil is an effective agent for minor pains and aches, especially dental pain, and is helpful when battling colds and flu.
Blends well with: spice oils, citronella, grapefruit, lemon, orange, peppermint, rosemary, rose.
Aromatic scent: smells spicy and rich just like cloves.
History: The word clove comes from the Latin word 'clavus', meaning nail, since the shaft and head of the clove bud resembles a nail. Cloves and nutmeg were among the most precious of items in Europe in the 16th and 17th centuries when they were worth more than their weight in gold.


Panasonic Lumix DMC-GH4 is a new exceptional Mirrorless Micro Four Thirds Digital Camera which able to capture professional photo quality with exceptional moire suppression includes recording UHD 4K 3840×2160 30p/24p and cinematic DCI 4K 4096×2160 video at 24p. This camera also has 3.0-inch 1,036k-Dot OLED Monitor, 2,359K-Dot OLED Live View Finder, High-Speed 49-Point Autofocus, and Support for 59.94p, 23.98p, 50p, & 24p including 4:2:2 8-Bit or 10-Bit HDMI Output. While the body is made of Magnesium Alloy and Weather-Sealed. We'll update this review soon.
Listed below are some of the rivals of the Panasonic Lumix DMC-GH4.
This entry was posted on Monday, February 10th, 2014 at 2:11 am and is filed under Compact System Cameras, Digital Camera, Panasonic.	You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.


text:
Partnerships & Capabilities
The RELLIS Campus
Discover Rellis
Texas A&M University System Opens RELLIS Academic Complex
BRYAN, Texas —Earlier today, Texas A&M University System officials celebrated the grand opening of the first Texas A&M System academic building on the 2,000-acre RELLIS Campus in Bryan, Texas.
"The RELLIS Academic Alliance Complex is a state-of-the-art facility that will help us lead the way in educating the next generation of Texans," said Texas A&M University System Chancellor John Sharp. "We're bringing the expertise from all across the A&M System directly to our students in one innovation location and providing a totally unique educational experience."
The 68,000 square-foot building features 10 classrooms, group learning and collaboration spaces, modern allied health teaching laboratories, upper-division science laboratories and more than 40 faculty offices. The $42 million three-story building is part of the planned RELLIS Academic Complex, which will accommodate 2,500 upper-division students once completed.
Currently, nine Texas A&M System regional universities offer 17 bachelor's degree programs at RELLIS in subjects ranging from criminal justice to marketing and accounting. Upper-division students at RELLIS will begin using the new building this fall.
Earlier this month, the Texas A&M System Board of Regents approved the construction of phase two of the RELLIS Academic Alliance Complex. Expected to be completed in November 2020, this building will provide the needed space for engineering labs and classrooms needed for expected upcoming engineering programs at RELLIS.
In fall 2019, the Blinn College District opened a 83,000-square-foot facility on the RELLIS Campus. The classrooms and laboratories provided space for all students during the first academic year. Upper-division students at RELLIS will now seamlessly transfer across the street to the new complex for their bachelor's degree program courses.
For more information about the RELLIS Academic Alliance visit https://rellis.tamus.edu/academicalliance/.
About the RELLIS Campus:
Founded in 2016 by The Texas A&M University System, the RELLIS campus in Bryan, Texas, fosters cutting-edge research, technology development, workforce training and two- and four-year college degrees by tapping the System's state agencies and multiple universities, along with academic, corporate and government partners outside the System.
Contact: Laylan Copelin
Vice Chancellor of Marketing and Communications
(512) 289-2782 cell
lcopelin@tamus.edu
The Right Formula »
« Nancy Shankle Jordan Named RELLIS Academic Alliance Provost
RELLIS Academic Alliance Hosts Graduation Ceremony
AT&T and The Texas A&M University System's RELLIS Campus to Launch 5G Technology to Promote 5G Innovation, Exploration, Education and Training
RELLIS Academic Alliance Hosts First Graduation Ceremony
TTI, Neology, Inc., Extend Research Partnership to Support Mobility Innovation
RELLIS offers Application Waiver at Transfer Fair
RELLIS is currently seeking corporate, governmental and education partners for applied research opportunities.
3100 TX-47, Bryan, TX 77807
Contact Us: Phone/Email/Mailing
Copyright ©2019 The Texas A&M University System
News Old
Procedures, Rules and SAPs
Risk, Fraud, Misconduct Hotline


If you love sports and enjoy games on your TV whenever you can you'll be able to add a fun dimension to your love for sports by betting on the outcome.betting confidence This could help keep you interested in sports as well as allow you to earn money if you manage to win most of your bets. However, since gambling on sports is not legal in the USA, you will have to shift to the net to effortlessly place your bets. Here's how you can bet at sports betting sites without facing any glitches.
The hardware that you'll require to go to online betting sites is simply a computer or a laptop with an connection to the internet. Choose a corner in your home where one can play without getting disturbed. Additionally, you will require your charge card and may need to provide the specifics of your card to the website where you desire to play in. That is all of the hardware required for anyone to start betting. You should now look for the top betting sites which will accept bets from US citizens. Just type in a search for the top 100 sports betting sites on any search and scrutinize the outcome.
You should never blindly open a free account with any bookmaker on any website before cross-verifying the credentials. There are several black sheep within the online market specializing in relieving people of their money by canceling bets, or just not paying out the winning amounts. Ensure that you have references from someone who has played in that site.
You can find mind-numbing sports to select from like boxing, golf, football, baseball, basketball, tennis among numerous others. These could possibly be played at college levels or even at pro levels. Horse racing and car races too are popular among many betting enthusiasts. You can bet on teams, their players, the whole scores and also the range by which any team could win or lose, among a number of other criteria, hence turning betting to a virtually intriguing sport itself.
When you do reach the home page of the betting website, you'll be invited to start a merchant account with them. You might also be offered starting or joining-up bonuses that will help you to get started. You will have to submit your charge card details plus the entire process of registration at any site will probably be over in a matter of minutes.
You may have to transfer-in a starting deposit too. Many sites offer additional ways to send out your winnings like by bank transfers or by cheques. Make use of the option most suitable for you personally. You'll have to open accounts with many other betting sites so as to augment your chances of winning on your bet. Once the registration is complete as well as your money is transferred in, you'll be able to start betting on your favorite sport. Know the conditions and terms of each site before betting so that there isn't any trouble at a later date.
Hence, although seeking out the truly good sites could be a little difficult, once you do find a good site then it's extremely simple to open an account and begin betting. There are lots of excellent sports betting sites that will allow one to merge your fascination with sports with a chance to win money and when you do locate them then its simply smooth sailing.


All allergens are in CAPITALS. May also contain NUTS. Contains NO WHEAT, GLUTEN, GELATINE, EGGS and alcohol. Some vegetarians may not consume (cochineal) E120 which is in the transfer as a red colouring. Not suitable for vegans.


Maps of Africa - Barry Lawrence Ruderman Antique Maps Inc.
La table & description universelle de toute l'Afrique, uoire estendue outre les limites de Ptol.
Africae nova descriptio . . .
Africae Antiquae et quarundam Europae Asiaque Adiacentium Regionum . . .
A New and most Exact map of Africa Described by N.I. Vischer and don into English and corrected to I. Bleau and Others with the Habits of the people and ye manner of the cheife sitties ye like never before . . .
Africae, Described. The manners of their Habits and buildings newly done into English by I.S. . .
L'Afrique Divisee suivant l'estendue des ses principales parties . . .


If you're in the Ottawa area this Thursday at 7:00 PM, why not join some of the gang from the Ludwig von Mises Institute of Canada. I'll be hosting the meet-up and giving a light, non-technical, and very casual speech on the recession and "quantitative easing."
You may find more details here. See you there.


A Health Savings Account (HSA) is a tax-exempt, custodial account used solely for paying qualified medical and dental expenses. If you are covered by a HSA qualified, high-deductible health plan, consider opening your HSA today. Start reaping the tax savings* this acccount is designed to provide.
There is no minimum opening deposit to open an HSA and no minimum balance to maintain. Earn interest on every dollar and receive a monthly statement. Pay your qualified medical expenses with your free Instant Issue debit card. Utilize all our electronic banking solutions to make paying and managing your medical and dental bills a breeze!


SKU: pntmawsb-1. Categories: New Items, Pork Pies & Meat Products, Bacon & Bangers. Tags: Apple Wood Smoked Bacon, Tommy Moloney.
Tommy Moloney Apple Wood Smoked Bacon. This newest addition to Tommy's bacon line comes as an Apple Wood Smoked Bacon, these lean cuts are 65% less fat than American bacon, cured using our traditional recipe.
This Traditional Apple Wood Smoked Bacon comes from a famous Limerick about a butcher… All the seasoning comes from Ireland to give you the exact same taste as if you were back Home. Also Tommy Maloney has won over ten national awards in Ireland.