extfieldsofstudy:
Computer Science
Mathematics

text:
IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control

Scaling adaptive traffic-signal control involves dealing with combinatorial state and action spaces. Multi-agent reinforcement learning attempts to address this challenge by distributing control to specialized agents. However, specialization hinders generalization and transferability, and the computational graphs underlying neural-networks architectures -- dominating in the multi-agent setting -- do not offer the flexibility to handle an arbitrary number of entities which changes both between road networks, and over time as vehicles traverse the network. We introduce Inductive Graph Reinforcement Learning (IG-RL) based on graph-convolutional networks which adapts to the structure of any road network, to learn detailed representations of traffic-controllers and their surroundings. Our decentralized approach enables learning of a transferable-adaptive-traffic-signal-control policy. After being trained on an arbitrary set of road networks, our model can generalize to new road networks, traffic distributions, and traffic regimes, with no additional training and a constant number of parameters, enabling greater scalability compared to prior methods. Furthermore, our approach can exploit the granularity of available data by capturing the (dynamic) demand at both the lane and the vehicle levels. The proposed method is tested on both road networks and traffic settings never experienced during training. We compare IG-RL to multi-agent reinforcement learning and domain-specific baselines. In both synthetic road networks and in a larger experiment involving the control of the 3,971 traffic signals of Manhattan, we show that different instantiations of IG-RL outperform baselines.

I. INTRODUCTION
T HE steady growth of the world's population, combined with a lack of space in urban areas, leads to intractable road congestion, the social and environmental costs of which are well documented [1]. The adaptive control of traffic signal systems based on real-time traffic dynamics could play a key role in alleviating congestion. The framing of adaptive traffic signal control (ATSC) as a Markov decision process (MDP) and the use of reinforcement learning (RL) to solve it via © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. experiencing the traffic system is a promising way to move beyond heuristic assumptions [2], [3], [4], [5].

A. Contribution
We introduce inductive graph reinforcement learning (IG-RL), which combines the inductive capabilities of graphconvolutional networks (GCNs) [16] with a new decentralized RL (DEC-RL) framework. In our independent Q-Learning (IQL) formulation, multi-agent RL is replaced by a shared policy, learned and applied in a decentralized fashion.
We define the topology of the computational graph of the GCN based on the current dynamic state of the road network (e.g., including the position of moving entities such as cars).
The GCN learns and exploits representations of a neighborhood of arbitrary order for every entity of the road network, in the form of node embeddings. TSCs' node embeddings are used to evaluate action-values (i.e., Q-values). The entire model is differentiable and we train it using backpropagation of the temporal difference error (TD error), following a standard Deep Q-Learning setting [17]. By having the computational graph adapt to the road network's state, we are able to: • Train a ubiquitous policy which can adapt to new road networks, including topologies and traffic never experienced during training. • Exploit the vehicular data at its finest granularity by representing every vehicle as a node and its corresponding vectorized representation (i.e., embedding). Current neural-network architectures used in RL-ATSC do not enable dealing with a changing number of inputs and state dimensionality. For this reason, in a traffic scenario where various types of entities such as cars and pedestrians enter, move inside of, and leave the network, these methods do not enable full granularity exploitation. They typically resort, upstream of learning, to aggregations at the lane-level (e.g., the number of vehicles approaching the intersection) [2], [18], [19] or queues lengths [2], [18], [19]. Alternatively, they fix the number of entities to be represented in detail. For instance, [2] represents the cumulative delay of the first vehicle. To test our claims, we define a new evaluation setting in which RL-ATSC methods are tested on road networks which are never experienced during training. In particular, we design two experiments involving non-stationary traffic distributions and different road networks.
The first experiment, on synthetic road networks, evaluates IG-RL's performance against both learned and domain-specific baselines. We compare a specialist instance of IG-RL trained on the road network used during evaluation to a generalist instance that is trained on a set of road networks not including the target road network. These instances are compared to assess generalizability, transferability, and the effect of specialization on performance. In addition, we also compare two GCN architectures which respectively capture traffic demand at the vehicle and lane levels, to measure the flexibility of IG-RL in different data conditions, and its ability to exploit granularity.
In the second experiment, an IG-RL instance is transferred, with no additional training (i.e., zero-shot transfer), to the road network of Manhattan and its 3,971 TSCs. This evaluation constitutes, by far, the largest RL-ATSC experiment to date. This final transfer from small synthetic road networks to a large real-world network aims to demonstrate the scalability of learning an agnostic policy.

II. RELATED WORK A. Decentralization
The main thread of research this contribution draws on is decentralized RL (DEC-RL). In RL-ATSC, decentralization and distribution of control via multi-agent RL (MARL) is a popular approach [2], [9], [10], [11], [12], [13], [14], [15]. However, current MARL approaches used for ATSC suffer from the following caveats: • From the perspective of a given agent (e.g., controller), the evolving behaviors of other agents (e.g., controllers) in the global environment cause nonstationarity and instability during training [20], [21]. As a remedy, researchers include the recent policies of other agents as part of the state of every local agent which seems to help, but does not yet solve the problem [2]. • Without parameter sharing, though more scalable than previous approaches, computational resources and time required to train MARL approaches grow with every additional signalized intersection as it requires the training of additional agents. • The specialization of every agent on the particular environment experienced by its jurisdiction during training hinders generalization and transferability to new environments. Therefore, applying MARL on any new road network or after any modification to a road network already under RL control requires training from scratch while interacting with road users and pedestrians to gather experience. Scalability of MARL and RL-ATSC in general, has therefore typically been limited. In our work, instead of using MARL, we approach DEC-RL as a set of similar decision processes which can be controlled by a single agent and show results on a network of 3,971 lights, which is, to the best of our knowledge, the RL-ATSC setting involving the largest number of TSCs to date.

B. Graph-Based Representation
A second thread which we draw from is graph-neural networks, which have been shown to provide richer spatiotemporal representations of the road network and facilitate coordination [18], [19]. Instead of studying coordination however, we focus on leveraging the flexibility provided by graphneural networks.

C. Parameter sharing
Concurrent research [18] and [22] leverage parameter sharing to enable further scalability. However, our contribution differs significantly from the proposed methodologies in the following ways: • Our main contribution is an inductive method which generalizes and scales to massive networks at test time. Our parameter-sharing scheme enables zero-shot learningadaptation without additional training-to different road networks. In particular, we demonstrate that our method, trained on small random networks, can control the lights on much larger networks (e.g., with over 3,900 lights) with completely different structures (e.g., traffic lights on the whole of Manhattan). • Our method does not resort to (arbitrary) aggregations of local traffic statistics as state features for parameter sharing. The architecture we use is specifically built to enable full granularity exploitation of demand information sensed at the lane or vehicle levels.

III. BACKGROUND A. Reinforcement Learning (RL)
The optimization of a decision process under uncertainty can be framed as an MDP. An MDP is defined by a set of states S, a set of actions A that the agent can take, a transition function T defining the probabilities of transitioning to any state s given a current state s and an action a taken from s, and a reward function R mapping a transition (i.e., s, a, and s ) to a corresponding reward. Optimizing such a decision process means finding a policy π-a policy is a mapping from a state to the probability of taking an action-which maximizes the sum of the future rewards discounted by a temporal factor γ: where r t represents the reward at time step t. Reinforcement learning (RL) can be used to solve an MDP whose transition and value dynamics are unknown, by learning from experience gathered via interaction with the corresponding environment [23]. We introduce the key RL concepts that we will use in this paper: 1) Q-Learning a standard algorithm for optimizing a policy in RL problems, 2) the concept of adding parameterized noise for exploration, and 3) decentralized RL. This introduction is self-contained but not exhaustive (an indepth introduction to the field is available in [23]). 1) Q-Learning: The state-action-value function (Qfunction) under policy π is the expected value of the sum of the discounted future rewards, starting in state s, taking action a, and then following policy π: From the optimal Q-function Q * = max π Q π , we can derive the optimal policy π * (a|s) : a = argmax a Q * (s, a ). Q * can be obtained by solving the Bellman equation [24]: where r (s, a) is the immediate reward received after taking action a from state s. Q-Learning uses transitions sampled from the environment and dynamic programming to solve the Bellman equation with no initial knowledge of transition and reward functions.
There are standard approaches that enable Q-Learning to model environments with large discrete or continuous state spaces (as is the case in this work). One of them consists in fitting the Q-function using buffers of experienced transitions that are large enough to prevent excessive correlations between observed transitions. In continuous state spaces, tabular Q-Learning (shown above) is replaced by a parametric model Q θ such as linear regression or a deep neural network [17]. The latter approach is known as Deep Q-Learning.
To improve stability a common approach is to keep a lagging version of the model with recent parameters θ − , called target model Q θ − , and use it to compute the temporal difference error (TD error): r (s, a) + γ max a Q θ − (s , a ) − Q θ (s, a). The TD error is backpropagated to update the parameters θ following the standard Q-Learning update rule: where α is the learning rate. Two standard extensions of Q-Learning led to promising results in our early RL-ATSC experimentation, and we use them in all our experiments: • Double Q-Learning [25], which consists in reducing the overestimation of action values by decoupling action selection and action evaluation. • Dueling Q-networks [26], which consists in enabling some parameters to focus on the relative advantage of actions by using additional parameters to evaluate the average of Q-values. 2) Noisy Networks for Exploration: During training of RL agents, interaction with the environment involves a fundamental dilemma between exploiting what has been learned so far by the agent and gathering additional information [23, Chapter 1.1]. Trading off exploration-exploitation is critical to learning good policies in a fast and robust fashion, and it remains an active research area. Exploration via noisy networks refers to the use of parameterized noise, in the form of independent Gaussian noise added to the parameters of a neural network, in order to favor consistent and statedependent exploratory behavior [27]. In comparison to decaying ε-greedy exploration-which consists in decaying the probability of acting randomly during training-noisy-network exploration often leads to better performances, enables online learning since it does not involve limiting exploration as time passes, and does not require tweaking sensitive learning schedules [27]. This form of exploration resulted in considerable improvements during our experiments and we use it to train all models presented in this work.
3) Decentralized Reinforcement Learning: Decentralized reinforcement learning (DEC-RL) distributes the control based on the assumption that the global Q-function is decomposable [28]. Independant Q-Learning (IQL) is a straightforward DEC-RL method in which the global MDP is decomposed into DEC-MDPs which are solved using independent Q-Learners.

B. Graph-Convolutional Networks
Enabling machine learning model to learn from complex relationships between objects in graphs is challenging. Graph-Convolutional Networks (GCNs) are a recent and fruitful attempt to leverage neural-network architectures and backpropagation to address the complexity of graph data. GCNs consist in stacking k (a hyperparameter to be defined) convolutional layers as a neighborhood-information aggregation framework [29]. At every layer of a GCN, every node aggregates communications sent to it by both its neighbors and itself into an embedding. This form of embedding both exploits the graph structure and the features of the nodes belonging to its neighborhood (up to order k). Using different parameters for different types of relations [30] yields the following message propagation equation, applicable to every node: where n l i is the embedding of node i at layer l, f is a nonlinear differentiable function, E is the set of relation types, N e (i) is the 1st-order neighborhood of node i in the graph of relation type e, C i,j,e is a relation-specific normalization constant, and W le is the l th layer's weight matrix for message propagation corresponding to a relation of type e. Embeddings can be used to perform a supervised or unsupervised learning task, and an error signal corresponding to the task can be backpropagated through the entire model to perform gradient based optimization of its parameters.

IV. INDUCTIVE GRAPH REINFORCEMENT LEARNING (IG-RL)
We now introduce IG-RL, a scalable DEC-RL method. IG-RL models objects (e.g., lanes, traffic signals, vehicles) as nodes in a graph. Edges of this (dynamic) graph represent the physical connections between objects (e.g., a vehicle node is connected to its current lane node).
A GCN models this graph. Initial node embeddings encode observable features of the objects (e.g., the vehicle speed). Then, each inferred TSC-node-embedding is used to predict the Q-values of that TSC.
The key for generalization is this graph representation. By modeling road networks at the object-level (vehicles, lanes, traffic lights) parameters are independent of any intersection or road-network structure. Therefore, the GCN only instantiates a small number of parameters which are jointly learned across an entire road network or even across multiple road networks. The same parameters can then generalize and transfer, without additional training (zero-shot), to completely different intersections and road networks (e.g., different structures) with different states (e.g., different traffic conditions). In turn, this generalization translates into immediate scaling to larger networks independently of the number of intersections.
We first discuss how to decompose the road network into small MDPs and then introduce our GCN model.

A. Decentralized ATSC Decision Processes
The state and action spaces of the global ATSC MDP, which consists in the management of all controllers in a given road network, can be large and intractable in practice. We therefore decompose it into smaller decentralized decision processes. In this context, the management of every intersection by a TSC becomes an MDP.
Each MDP uses continuous states (which represent the state of nearby roads), discrete actions (e.g., whether or not to change phase), and discrete time steps (the length of which determines when the next action can be taken).
1) State: The state of every decentralized decision process consists in real-time information coming from local sensors and is said to be partially observable because it does not include all information about the global MDP. This state encodes both 1) current connectivity in the network, and 2) demand. Connectivity of a given intersection is defined by the current phase of the corresponding TSC, while demand can be sensed either at the level of lanes, or at the level of vehicles ( § IV-B1 and IV-B6).
2) Action: At every intersection of the road network, the flow of traffic is managed by a logical program, composed of a given number of phases, depending on the number of roads, the number of lanes, and the connectivity between lanes. This program cycles through phases, influencing connectivity between lanes, in a constant predefined order that is often known to the road users and pedestrians. In our studies, the agent must respect the program and chooses, every step, whether to switch to the next phase or prolong the current phase. The action space at every intersection is therefore binary. Compared to methods which enable complete freedom of the controller, with the agent being able to choose among any phase every time it picks an action [2] [7], this formulation of the ATSC problem is more constrained. The agent must learn transition dynamics ruled by a complex cycle influencing connectivity. It is however more adaptive than methods determining duration upstream of any given phase [31]. We leave the exploration of other policy constraints for future work, although the model we introduce in § IV-B is general and is not tailored to the aforementioned constraints. In fact, we discuss modelling an environment where the next action is arbitrary in § IV-B2.
3) Reward: The reward for a given agent is defined as the negative sum of local queues lengths q, which corresponds to the total number of vehicles stopped 1 on a lane leading to the corresponding intersection, at a maximum distance of 50 meters of the intersection. The reward at intersection j is where L j is the set of lanes leading to intersection j. This measure can both be obtained via lane or vehicular sensors. We choose this measure instead of alternatives such as wave [31] or average trip waiting time [9] because it is correlated to the local transition mechanisms of each DEC-MDP, and it is dense since actions tend to impact this measure quickly. These last two points translate into eased attribution of reward signals [2].

4)
Step: We discretize time into one-second time steps. In other words, actions are selected every second. 5) Episode: Depending on the experiment ( § V), an episode either ends as soon as all trips are completed or after a fixed amount of time.

B. Model
The control of all traffic signals is a family of decision processes. Each decision process in this family involves a controller, lanes, connections between lanes, and vehicles (e.g., nearby ones). A single GCN is used to model these entities and their dynamic relationships. IG-RL proposes systematic sharing of the parameters of the GCN among all objects and relationships of the same nature.
In addition, the flexibility of GCNs enables representing an arbitrary number of entities that changes over time (e.g., vehicles moving in and out of the network) in a detailed way (i.e., as nodes in a graph).
1) Architecture: Fig. 1 illustrates the GCN for modelling a traffic signal control. It includes 4 types of nodes: • TSC node, which represents the state of a controller. • Connection node, which represents the state of an existing link between an entry lane and an exit lane. A link (i.e., connection node) exists between an entry lane A and an exit lane B if, under at least one phase of the TSC program, a vehicle on A is allowed to continue its travel on B. • Lane node, which represents the state of a lane. • Vehicle node, which represents the state of a vehicle. The GCN uses the following edges and edge types: • An edge between every node and itself (4 types of edges). • Bidirectional edges between every TSC node and Connection nodes corresponding to connections the TSC can influence by changing its phase (2 types of edges). • Bidirectional edges between every Connection node and Lane nodes corresponding to its entry lane (2 types of edges) and its exit lane (2 types of edges). . Edges of the same type share the same color and parameters. The final embedding of the TSC node is then passed through the a fully connected layer to obtain its Q-values (one per possible action).
• Bidirectional edges between every Lane node and Vehicle nodes corresponding to vehicles located on the corresponding lane, given that the GCN exploits detailed vehicular data (2 types of links). Every layer of the GCN uses one set of parameters per edge type to perform message-propagation (W in Eq. 5). Our early experimentation with IG-RL and GCNs suggested that using normalization constraints (C in Eq. 5) was detrimental to performance. In contrast to Equation 5, we do not include a normalization constraint during message-passing The embedding n 0 j of a node j is initialized using its features ( § IV-B6). A fully connected layer, with a single set of parameters, maps the final node embedding of a TSC to the Q-values corresponding to its selectable actions.
Since preliminary experiments demonstrate that noisy exploration enables faster and more robust training, as well as better performance compared to ε-greedy exploration, parameterized Gaussian noise is added to the parameters of this final mapping. An initialization of 0.017 for the variance parameters performs well in multiple supervised [27] and reinforcement learning tasks [32], and so we also use that value. Fig. 1 is an illustration of the architecture of the full model for a simple road network with a single TSC and three vehicles.
2) Action Space Flexibility: Given the computational flexibility of GCNs, the model can be easily adapted to different action spaces. For instance, enabling the choice of an arbitrary phase (instead of a program that cycles through phases) is achieved by representing phases as nodes of a new type, and linking these nodes to both their corresponding TSC node and the nodes representing connections they influence. This way, one Q-value can be computed per phase node. Preliminary experiments using this formulation seem promising.
3) Parameter Sharing: Using a graph representation at the object level (TSC, connection, lane, car) enables sharing parameters between objects of the same type, both inside each decision process (e.g., two lanes on the same intersection), and across all decision processes belonging to the same family (e.g., two lanes on unrelated intersections). This parameter sharing is useful because: • It uses a constant number of parameters independent of the number of underlying processes. • During training, these shared parameters are updated using experience gathered from a variety of decision processes (a variety of intersection architectures belonging to a variety of road network architectures).

4) Generalization and Transfer via Inductive Learning:
Consistent parameter sharing enables training a unique set of parameters using experience gathered from a set of decision processes involving diverse intersection topologies, road network topologies, and traffic distributions. Enforcing such diversity can translate into better generalization to unseen MDPs from the same family of MDPs [33] [34]. In this work, the learning of general and transferable patterns is based on two key components: • The proposed architecture, based on a GCN, which enables parameter sharing without sacrificing granularity. • The sampling strategy over the family of MDPs, which in this context refers to the way decision processes are created. The sampling strategy should ensure that the training set is representative of the corresponding family of decision processes. 5) Marginalizing Nonstationarity: Nonstationarity, caused by the evolution of the behavior of neighbor agents during training, remains a critical challenge for DEC-RL ( § II-A). We hypothesize that emphasizing diversity in the training set, as described in § IV-B3, can limit nonstationarity and stabilize learning by marginalizing out the experience of a particular intersection with respect to its neighbor intersections. 6) Features: Our approach exploits data that can be accessed from sensors representing the various entities that make up the road network and traffic. At any time step in a given road network, every element defining the state is represented through a node in the GCN. The architecture of the GCN is built on the assumption that the current state of the controller (i.e., the current phase and its corresponding local connectivity) is known, and that traffic information coming either from lane sensors or vehicle sensors is also available. Other assumptions would result in the choice of a different GCN architecture or in the use of different features. The features used for different types of nodes are summarized in Table I Next open has priority defines whether the next opening of the connection will have priority or not. Time since last switch is the number of seconds since a traffic controller performed its last phase switch.

V. EXPERIMENTS
Using the SUMO traffic simulator [35], we evaluate the performance of several IG-RL instantiations using small synthetic road networks and a large real road network, and compare it to the results of several baselines.
Our study demonstrates that: • IG-RL outperforms all included baselines on the ATSC task in a traditional evaluation setting. • Training IG-RL under several varying initial conditions (including different road networks) yields the best performing method and enables zero-shot transfer. • Training IG-RL on small synthetic road networks with limited computational resources enables efficient zeroshot transfer to real road networks, and such a transfer translates into improved scalability. The design of a synthetic training set (of random road networks) before transferring the learned policy to a more realistic setting (Manhattan) is inspired by other zero-shot transfer approaches in reinforcement learning [36] [37].
A. General Setup 1) Network Generation: The structure of a road network is picked, at random, from a set which consists in the cross product of the number of intersections (typically between 2 and 6), the structure of every intersection, the length of every edge (between 100 and 200 meters), the number of lanes per route (between 1 and 4 per edge), and general connectivity. A given random seed will lead to the generation of the same network, enabling exact comparison of performance between policies. Examples of randomly generated networks from our procedure are shown in Fig. 2. 2) TSC-Programs: TSC-programs, or cycles, are generated by SUMO and typically include several phases, corresponding to the legal states at a given intersection. 3 Programs are therefore not restricted to abstracted axis connectivity such as North-South or East-West. Our synthetic network generation procedure ( § V-A1) leads, on average, to the creation of TSCs whose programs are composed of 5.5 phases. We leave experimentation with other types of TSC-programs to future work.
3) Traffic Generation: A given number of trips are to be generated per second, on average, during an episode. This value is referred to as the traffic regime. Every generated trip is assigned a trajectory. A trajectory is defined by a starting lane, a sequence of intermediary lanes, and a final lane. Every valid trajectory includes at least 2 lanes. To simulate the urban setting, a trajectory can start and end anywhere in the network. Parameters defining non-uniform distributions used for assigning trajectories to trips are re-sampled every 2-minutes to ensure non-stationary distributions. 4) Training: Models are trained on a set of 30 simulations running in parallel. During training, exploratory behaviors tends to lead to catastrophic congestion from which recovery becomes unlikely. This is a common RL challenge which makes training from long episodes inefficient in practice. For this reason, as in [2], every simulation runs 30,000 steps divided into independent short episodes of 500 steps (seconds) each. All simulations run on a single randomly generated road network. We also use this network for evaluation in our first experiment ( § V-D). Further, each simulation is initialized using a different random seed to ensure a variety of observed transitions. Models are trained using the exact same environment (network and traffic). The parameters of target Q-networks are updated every 100 parameters updates of the main Q-networks ( § III-A1) The learning rate is 0.001 and the batch size is 16. 4 5) Evaluation: All methods are evaluated using the same target networks and identical traffic. However, evaluation introduces traffic regimes (densities) never experienced during training, to evaluate how the methods generalize to new traffic distributions. We evaluate performance by measuring the evolution of the instantaneous delay: 3 "All red" phases are not included as part of SUMO's default TSC-Programs generation. 4 A batch is composed of networks. A network typically involves multiple intersections where V is the set of all vehicles in the road network, s v * is the maximum speed of the vehicle, s l is the maximum allowed speed on the lane the vehicle is on, s v is the current vehicle speed, and d t is the total instantaneous delay at time step t. Given that all models are evaluated on the same sets of networks and trips, we can pair the same trips together for hypothesis testing (using paired t-tests), and to study distributions of trips-durations differences.
6) Constraints: Phases involving yellow lights last exactly five seconds, as suggested by [31]. For all other phases, in order to maximize adaptiveness, instead of using a fixed time of 5 seconds between every action as suggested by [2], a TSC must spend at least 5 seconds in each phase before being allowed to switch. The parameters associated with these time constraints can be modified and their values are expected to influence achievable performance. We leave thorough experimentation with time constraints (e.g., minimum green time for pedestrians) to future work. For trainable methods, constraints are enforced by ignoring invalid actions selected by the agent. In other words, in a given time step, even though the agent can pick an action, its decision will have no influence on the MDP if its chosen action breaks a constraint. 7) Action-Correction: Ignoring an agent's invalid actions forces it to learn that the effect of the available actions on connectivity are dependent on other logical factors like the time since the last phase change and the nature of the current phase. To ease learning, and since it is preferable to focus on the actual influence an agent can have on its MDP, we replace, during experience replay, binary decisions which were chosen by the agent at every time step, by whether a light switch was performed. We also force the greedy policy, in the creation of the temporal difference target, to only consider actions which could be performed at the corresponding step. To adapt the proposed action-correction methodology to a policy which is able to choose an arbitrary phase, the selected action can simply be replaced by the action corresponding to the selection of the phase that is actually active at the next step. 8) Robustness: The seeds we use in the experiment influence the random generation of road networks, traffic distributions, initial weights of neural networks, and the Gaussian noise used for exploration during training. To ensure the robustness of our conclusions, every experiment, including both training and test, is repeated 5 times using different seeds. For instance, the training on 30 different simulations occurs 5 times for a total of 150 simulations. The same goes for evaluation. Reported test results are an aggregation over these 5 runs for both experiments.
B. IG-RL models 1) Architectures: Our complete GCN architecture, referred to as IG-RL-Vehicle (IG-RL-V), uses a 3-layer GCN and vehicle-level traffic information for both state 5

and reward ( § IV-A).
We also introduce a lighter GCN architecture, IG-RL-Lane (IG-RL-L) which uses a 2-layer GCN and lane-level traffic 5 Lane level traffic features are not included in IG-RL-V. information for both state 6 and reward ( § IV-A). Even though IG-RL-L does not exploit demand at the vehicle-level, it has the added benefit of requiring constant computational time with respect to the traffic demand.
For both IG-RL-L and IG-RL-V, the number of layers is chosen to be the minimum which ensures that information representing demand can reach the nearest TSC(s) nodes. For both GCN architectures, we use node embeddings of size 32 as further increasing this hyperparameter did not result in improved performances.
2) General versus Specialized: The target network refers to the road network used to test the performance of all methods after training and must be distinguished from the training set of road networks since the same IG-RL instance can be applied, and therefore trained on, different road networks. Two training sets of road networks are built for IG-RL.
The specialist set, used to train Specialist-IG-RL (S-IG-RL), aims to evaluate intentional specialization on a given network. It consists in training the model exclusively on the target road network, as described in § V-A4.
The generalist set, used to train Generalist-IG-RL (G-IG-RL), aims to study IG-RL's transfer abilities. This set is composed of independent road networks and does not include the target network. In such a context, performance reflects generalizability to new road network and intersection architectures. In addition, as different road networks imply different traffic distributions even when the regime ( § V-A5 and § V-D1) is left unchanged, performance reflects generalizability to new traffic distributions for all studied regimes, even the default one which is used during training.
Our experiments with hybrid training sets involving both the target road network and random road networks did not lead to improved performance compared to the generalist approach and are omitted for brevity.
3) Evaluating Action-Correction: To evaluate the importance of action-correction ( § V-A7), we introduce a version of IG-RL-L for which no action-correction is performed during training. This ablation study is referred to as IG-RL-nocorrection (IG-RL-n-c).

C. Baselines
We now describe the models used as baselines in both of our experiments.
1) Fixed-Time Baseline: This baseline follows the cycle of phases using predefined and constant durations generated by SUMO [35] based on the architecture of each intersection.
2) Max-moving-car heuristic (Greedy): This dynamic heuristic-based method aims at ensuring that as many vehicles as possible are moving on inbound lanes at any given time. At every intersection, the controller switches to the next phase if, on inbound lanes, the number of stopped vehicles is superior to the number of moving vehicles, and prolongs the current phase otherwise.
3) MARL-IQL: MARL-IQL is a learned baseline. In this model, every intersection is controlled by its corresponding agent (i.e., with its own unique parameters), which learns to solve its own local MDP. Every agent, parameterized by a deep Q-Learner network, takes the exact same variables that are available to IG-RL-L as inputs (Table I and  § V), with the exception of lanes' lengths, which is constant for each MDP. Every Q-Learner consists of a first hidden layer of 256 neurons, a second hidden layer of 128 neurons, and a last hidden layer of 64 neurons, which is the result of a gridhyperparameter search. While this model would constitute a "vanilla" multi-agent implementation, we chose to add both Double Q-Learning and a Dueling architecture to every Q-Learner ( § III-A1) as done in [38]. In addition, the introduction of noisy parameters ( § III-A2) and action-correction ( § V-A7), were the elements which led the the biggest improvements during training. These improvements have the two following objectives: • Create a strong baseline. • Enable an ablation study. The only difference between MARL-IQL and S-IG-RL-L is that the former uses one neural network per intersection to learn the local Qfunction and compute the local Q-values, while the latter uses the same GCN to do so at every intersection. Apart from this, all the information and methods they use for training are identical.
D. Experiment 1: Evaluating Generalization using Synthetic Road Networks 1) Description: In this experiment, we evaluate policies on a single road network. The same road network is used to train all methods except G-IG-RL. As MARL-IQL does not scale as efficiently as IG-RL methods, we picked a small road network. a) Evaluation setting: During evaluation, trips are generated during the first hour. An episode ends as soon as all trips are completed. Performance of all methods are evaluated using 30 different random seeds for traffic generation. Evaluation is run twice, with two distinct traffic regimes (denoted default and heavy), in order to evaluate the ability of models to generalize to traffic densities never experienced in training, as well as to study the performance in different density settings. The first evaluation traffic regime, which is the one used during training, introduces an expected one vehicle every 4 seconds in the network, while the second traffic regime introduces twice as many per second on average. b) Overcoming overfitting: While methods are evaluated during episodes of more than an hour, they are trained on shorter episodes of 500 seconds ( § V-A4). As more vehicles enter the network, congestion can therefore increase past levels experienced during training (which again implies new traffic distributions) even in the default traffic regime used during training. Since conditions differ between training and evaluation, the task is very challenging and the learning of generalizable patterns and behaviors (policies) is key to perform in both traffic regimes.
2) Results: This figure presents the distribution of trips durations during test in the same traffic regime used for training. The vertical axis is cut at MARL's median trip duration for readability. Since every method is evaluated using the exact same trips, for every method, we perform t-tests on the paired sample of: 1) The durations of the trips when using the given method and 2) The durations of the trips when using G-IG-RL-V. The results of these paired t-tests suggest that G-IG-RL-V significantly outperforms every other method, while the boxplots, means, and medians of trips durations suggest that IG-RL models using actioncorrection outperform all other methods in general.  (Fig. 3) a) IG-RL models: We now refer to the group of methods using both IG-RL and action-correction as IG-RL models, and highlight them in all following figures (green font). Means and medians of trip durations, reported in Figs. 3 and 4, are lower when using IG-RL models, compared to other methods. Instantaneous total delays, reported for every simulation step in Figs. 5 and 6, show that congestion increase is slower, and recovery 7 is faster using IG-RL models in both traffic regimes. In addition, in the default traffic regime, these methods reach the lowest steady-state in terms of congestion level.
G-IG-RL-V is the best performing method, with both the lowest trip-durations (Figs. 3 and 4), the slowest congestion  increase and the fastest recovery (Figs. 5 and 6). In addition, distributions of paired-trips-durations differences, between every method and G-IG-RL-V, reported in Fig. 7 and 8, show that a large majority of trips are completed faster using G-IG-RL-V. In the default traffic regime, some trips are delayed by up to a thousand seconds when using MARL or Fixed-Time. In comparison, no trips are delayed by more than a hundred seconds under the G-IG-RL-V policy. In the heavy traffic regime, the conclusions are similar, with even higher differences compared to other methods. Fig. 7 and 8 also Positive values indicate trips which were completed faster using G-IG-RL-V than the compared method. Negative values indicate trips which were completed faster using the compared method than G-IG-RL-V. Medians, means, and confidence interval for means of the differences are also reported. All these results show that when using G-IG-RL-V, trips tend to be completed faster . show that in addition to improving average performance, G-IG-RL-V tends to distribute delay in a more equitable way than any other method (numerous extreme delays are avoided and replaced by a few shorter delays).
b) G-IG-RL-L and S-IG-RL-L: A key finding emerges from comparing G-IG-RL-L and S-IG-RL-L. Even though their performance is similar in the default regime, G-IG-RL-L outperforms S-IG-RL-L in the heavy regime. This shows that G-IG-RL enables the efficient learning of transferable patterns, and that learning from a variety of road networks is key to enable generalization to new traffic regimes. This result further emphasizes the limits of training any policy, whether it is a single policy in the context of centralized RL or multiple policies in the context of MARL, on a single road architecture. (Recall that allowing for modelling different sets of train and test networks is one of our key contributions.) c) IG-RL-V and IG-RL-L: G-IG-RL-V outperforming other methods in both traffic regimes highlights the flexibility of the GCN to exploit a finer level of granularity (i.e., data sensed at the vehicle level). However, G-IG-RL-L and S-IG-RL-L being the second bests (ex-aequo) performing methods in the default traffic regime, and G-IG-RL-L being the second best performing method when generalizing to a heavier traffic regime, we can conclude that IG-RL can also perform well with demand sensed at the lane level. As the comparison between S-IG-RL-V and S-IG-RL-L leads to the same conclusion as the one between G-IG-RL-V and G-IG-RL-L, we did not include S-IG-RL-V in this paper to improve readability. d) Action-Correction: The performance of G-IG-RL-L(n-c) shows that removing action-correction ( § V-A7) has an important negative impact on performance and generalization to the heavy traffic regime, with trips lasting longer, congestion increasing faster, and recovering slower than every other IG-RL-based method. e) MARL-IQL: MARL-IQL is the worst performing model (it is even outperformed by the fixed-time baseline). We hypothesize that two reasons explain its performance: • Overfitting: As mentioned in § V-D1b, the current experiment is very challenging with respect to overfitting in both traffic regimes. During training, we observed that MARL-IQL largely outperforms the Fixed-Time policy, and its performance is both: 1) on par with the performance of the Max-moving-car heuristic (Greedy) policy, and 2) much closer to the performances of other learned models as compared to the gap in performance at test time. This observation suggests that policies learned by MARL-IQL do not generalize well to new demand and traffic distributions. • Nonstationarity: As described in § II-A, nonstationarity is a key reason for instability in MARL. It is no surprise that naively applying deep neural networks to MARL-IQL can lead to subpar performance. Note that this is the case even though we use an improved version MARL-IQL ( § V-C). In [2], for instance, a MARL-IQL method performed significantly worse than all other formulations, including one based on vanilla linear regression.
f) Conclusions: This experiment highlights that IG-RL's 1) parameter-sharing ( § IV-B3), 2) training schemes ( § IV-B4), and 3) vehicle-level-demand representation ( § V-B1) are key to its performance. The first is observed when comparing MARL-IQL to IG-RL models (parameter sharing distinguishes S-IG-RL-L from MARL-IQL as stated in § V-C3). The second is shown by the performance of the G-IG-RL models (especially in the heavy traffic regime). Further, training all parameters with a variety of experiences (from multiple intersections using S-IG-RL and even multiple road networks using G-IG-RL) and avoiding overparameterization helps IG-RL prevent overfitting (a key challenge in this experiment as stated in § V-D1b). We hypothesize that this variety also marginalizes out nonstationarity ( § IV-B5). Finally, comparison of G-IG-RL-V and G-IG-RL-L demonstrates the third highlight.
E. Experiment 2: Transfer and Scaling to Manhattan 1) Description: In this experiment the target network is Manhattan (Fig. 9), which involves 3,971 TSCs, some complex intersections being managed by several TSCs, and 55,641 lanes. This network provides a testbed for evaluating the generalization and scaling properties of G-IG-RL approaches. Training MARL based methods on that many intersections would imply massive time (running the corresponding simulations) and memory (storing and training one model per intersection) resources. Training S-IG-RL would enable the use of a constant number of parameters, but gathering the amount of experience required for RL on such a large road network would not be possible in a reasonable time with the simulator and computational resources that we use. Since it can leverage zero-shot transfer (i.e. no additional training) and keep the number of parameters constant, G-IG-RL, which is trained on a set of small networks as described earlier, is the only learned method usable in this large-scale context. For a given road network, G-IG-RL-L requires fixed computational time, while both the computational time and memory requirements of G-IG-RL-V increase linearly with the number of vehicles in the network. Considering the size of the experiment, even though we expect G-IG-RL-V to perform better than G-IG-RL-L considering the results of the previous experiment, we only evaluate the latter.
Evaluating methods on this large network is more computationally demanding compared to Experiment 1. Every method is only evaluated on 5 instances of randomly generated traffic for a fixed duration of one hour and run using a single traffic regime which introduces a vehicle every second in the network. Compared with what G-IG-RL experienced during training, and what was evaluated in Experiment 1, adding a vehicle every second in a network as large as Manhattan leads to light traffic density which increases in certain areas as vehicles enter the network following asymmetric distributions based on a given random seed. Waiting for all trips to end in a large network would mean prolonging the duration of some simulations in an unreasonable way, considering that some trips might require a lot of time to be completed. For computational concerns, episodes end after an hour. We report aggregated results, but trips cannot be compared via pairing, as done in Experiment 1, without introducing bias. As all simulations are running on the same road network, and the fixed time controls of many intersections are synchronized, the local delays are highly correlated across intersections and across simulations under this method, hence the periodic pattern.
2) Results: A first result is the bare fact that by using G-IG-RL-L, the 3,971 traffic signal controllers are operable, using a single GPU, and a model involving only 11,076 parameters. In addition, we can see that the total instantaneous delay, measured at every time step and reported in Fig. 10, shows that G-IG-RL outperforms baselines which can scale to Manhattan, in terms of the slowest congestion increase and the lowest congestion-level steady-state. Furthermore, it confirms the ability of IG-RL to generalize to different (here lighter) traffic regime than experienced in training. These results demonstrate that using synthetic randomly generated road networks for training is a viable and efficient way to learn policies that can transfer well to real road networks. This evaluation setting is similar in spirit to what is done in sim2real for robotics [33], [34].

VI. CONCLUSION
We introduce IG-RL, a reinforcement-learning trained method that leverages the flexible computational graphs of GCNs and their inductive capabilities [16] to obtain a single set of parameters applicable to the control of a variety of road networks. Experiments show that training using smallsynthetic networks is enough to learn generalizable patterns, which in turn enables zero-shot transfer to new road networks, as well as new traffic distributions. Experiments also show that IG-RL outperforms MARL-IQL, as well as both dynamic heuristics and fixed-time baselines. Further, generalizability over architectures, which emerges from a generalist IG-RL (G-IG-RL), helps improve performance and generalization to new regimes of traffic. In addition, we showed that the flexibility of GCNs enables a flexible representation of the road network. Demand and structure can be represented and exploited in various ways, at their finest level of granularity, no matter the evolution of the number of entities nor their respective locations in the road network. a) Future work: IG-RL opens a path for the following future works: 1) The flexibility of GCNs could be key to the optimization of multi-modal transportation. We represent vehicles as nodes in IG-RL-V and we could experiment with pedestrian nodes, cyclist nodes, and many more, to ensure that traffic signal control accounts for all road-users instead of prioritizing cars. 2) In this work, the reward function used for every MDP focuses on local queues (Equation 2). In such a setting, coordination between MDPs is limited, and a shallow GCN is able to perform well since local information is sufficient. Evaluating whether the use of additional coordination mechanisms can further improve global performance could be an interesting avenue for future work. One way of addressing coordination, in the context of more global reward functions, would be to perform message passing on longer network distances by adding more layers to the GCN, or by adding recurrent mechanisms to it as done in [39]. 3) The present work assumes data availability but some events (e.g. sensor failures) could generate various types of missing data. Evaluating IG-RL's robustness under several missing data mechanisms may constitute the object of future studies.

VII. SUPPLEMENTARY MATERIAL
Supplementary material, provided as a separate document, includes pseudo-code, a discussion on time complexity, and a complete version of Fig. 5.

VIII. ACKNOWLEDGEMENTS
We would like to thank the three anonymous reviewers for their interesting and constructive comments that lead to an improved version of this article. We would also like to thank Maxime Gasse for helpful discussions of deep learning on graphs which helped formulate GCN architectures included in this work.


extfieldsofstudy:
Computer Science
Medicine

text:
SiamMixer: A Lightweight and Hardware-Friendly Visual Object-Tracking Network

Siamese networks have been extensively studied in recent years. Most of the previous research focuses on improving accuracy, while merely a few recognize the necessity of reducing parameter redundancy and computation load. Even less work has been done to optimize the runtime memory cost when designing networks, making the Siamese-network-based tracker difficult to deploy on edge devices. In this paper, we present SiamMixer, a lightweight and hardware-friendly visual object-tracking network. It uses patch-by-patch inference to reduce memory use in shallow layers, where each small image region is processed individually. It merges and globally encodes feature maps in deep layers to enhance accuracy. Benefiting from these techniques, SiamMixer demonstrates a comparable accuracy to other large trackers with only 286 kB parameters and 196 kB extra memory use for feature maps. Additionally, we verify the impact of various activation functions and replace all activation functions with ReLU in SiamMixer. This reduces the cost when deploying on mobile devices.

Introduction
Visual object-tracking is a fundamental problem in computer vision, whose goal is to locate the target in subsequent video frames based on its position in the initial frame. Visual object-tracking plays an essential role in many fields such as surveillance, machine vision, and human-computer interaction [1].
Discriminative Correlation Filters (DCFs) and Siamese networks are the dominant tracking algorithm models presently. DCF emerged much earlier than Siamese network trackers. It uses cyclic moving training samples to achieve dense sampling and uses a fast Fourier transform to accelerate the learning and applying of the correlation filters. It has the advantage of high computational efficiency. However, the design of the feature descriptors requires expert intervention, and the circular sampling produces artifacts at the search boundary that can affect the tracking results. The emergence of Siamese networks provides an end-to-end solution and eliminates the tediousness of manually designing feature descriptors while exhibiting decent tracking performance.
The Siamese network tracker treats visual target tracking as a similarity learning problem. The neural network is used to learn the similarity descriptor function between the target and the search region. The Siamese network consists of two branches. The input 1.
We propose a novel lightweight and hardware-friendly visual object-tracking model based on the Siamese tracking scheme, namely SiamMixer.

2.
We design a compact backbone consisting of patch-based convolutions and mixer modules. The patch-based convolution reduces feature map memory use by processing each image patch individually. The mixer module enhances the accuracy by merging and encoding global information of feature maps.

3.
We verify the activation function impact on tracking accuracy and use ReLU as a satisfying alternative for exponential-based functions, which is favorable for Single-Instruction Multiple-Data (SIMD) operations.
Extensive experimental results demonstrate that the proposed method has comparable performance with many off-the-shelf Siamese networks, while the memory footprint is significantly lower.
The structure of this paper is as follows: Section 2 reviews the Siamese-networkbased trackers most relevant to our approach and the common approaches for building lightweight neural networks. Sections 3.1-3.3 introduce a description of the major components of the proposed network, including the convolutional layer for feature extraction, the mixer module for global encoding of the feature map, and the cross-correlation for target localization. The training setup and the loss functions design are described in Section 3.4. Section 3.5 introduces the datasets and evaluation metrics we used. Section 4.1 introduces our experimental results and compares them with the state-of-the-art algorithms. In Section 4.2, we analyze the storage overhead of SiamMixer for weights and feature maps. Section 5 concludes the paper.

Related Work
In this section, we review the visual tracker based on the Siamese network and popular methods for building lightweight networks to illustrate how our work differs from prior work.

Trackers Based on Siamese Network
The Siamese network tracker treats the tracking problem as a similarity discrimination problem. The basic tracking process starts with feature extraction of the target image and the search area using a neural network. The extracted feature information is then fed into the prediction head for target localization. The key to the success of the Siamese network model is to train the neural network offline using large amounts of labeled data. This allows the network to learn similarities between the target image and the search area. A well-trained Siamese network tracker maintains stable tracking even when the target undergoes complex transformations such as rotational distortion and illumination changes. A well-constructed backbone network enables the Siamese network to perform precise tracking while ensuring real-time speed, demonstrating a good balance of tracking accuracy and speed.
Because of the advantages above, trackers based on Siamese networks have been widely studied in recent years. SiamFC [12] is a pioneering work using Siamese networks for tracking tasks and has inspired a large amount of subsequent work. SiamFC proposes to use Siamese networks for feature extraction. The similarity scores between the target and the search region are calculated using cross-correlation. The location of the target is obtained from the similarity scores map. SiamFC provides a promising idea for tracking tasks, but its use of a multi-scale search scheme cannot accommodate large scale variations and aspect ratio variations.
SiamRPN [13] proposes to use a region proposal network (RPN) to estimate the target scale, avoiding the extraction of feature maps at multiple scales and achieving more accurate bounding box prediction. RPNs are widely used in object detection tasks, which use predefined anchor boxes with offsets to predict the location of targets. OCEAN further proposes the anchor-free method based on SiamRPN. Instead of using a predefined anchor box, the anchor-free method directly outputs the offset between the target and the ground truth, which improves the network accuracy and simplifies the network structure.
The backbone network directly impacts the performance of the Siamese-networksbased tracker. State-of-the-art trackers typically employ large pre-trained networks as backbone networks. SiamFC++ tests the performance of different backbone networks in the same network framework and demonstrates the impact of backbone networks on the performance of Siamese networks. SiamDW [14] analyzes the selection conditions of the backbone network in Siamese networks and presents a new residual module that allows Siamese networks to use deep networks as backbone networks.
Although the aforementioned work achieves significant performance improvements over SiamFC, its performance is entirely dependent on the generalization capability of the network. These models that are not updated online often lead to tracking failures when the appearance of the target changes significantly. ATOM, DIMP, DSiam [15] and ROAM [16] propose potential solutions for online learning of models and combine online learning with Siamese networks to achieve state-of-the-art performance. Although many techniques for model updating have been proposed, these methods usually significantly affect the speed of the network, making it impractical to meet the requirements of real-time tracking. Therefore, simply not employing online learning remains a robust and popular choice.
The subsequent work focused on improving the accuracy and robustness of the network and achieved significant improvements. However, it also brings extra computation and a large memory footprint, thus limiting its use in practical applications.

Lightweight Network Structure Design
Deploying neural network algorithms on edge computing platforms is a challenging task. These platforms are characterized by limited memory resources and low processor performance, thus making it impractical to deploy current state-of-the-art models and meet real-time requirements.
The work of building lightweight networks can be divided into two camps. One starts from existing high-performance networks, optimizes the network structure, compresses the network parameters, and finally makes the network meet the requirements of edge-end deployment. The representative work is deep compression [17], knowledge distillation [18], and low bit quantization [19]. These works require trade-offs between accuracy, frame rate, and the number of parameters, introducing excessive manual involvement. The others design lightweight structures directly and then combine pruning and quantization to eventually meet the requirements of edge-end deployment. Representative work is SqueezeNet and MobileNet. However, this work focuses mainly on reducing the weight parameters of the network while ignoring the memory overhead for feature maps.
Network architecture search (NAS) [20,21] is also widely adopted to build lightweight neural networks. Early NAS usually faced the problem of training a large number of neural networks from scratch, which required a significant amount of GPU resources. Subsequent NAS work narrowed the search space through manual intervention. However, limiting the search space makes the searched network structure suboptimal, and too much manual intervention also contradicts the main objective of network architecture search.

Proposed Algorithm
We propose to build lightweight target-tracking algorithms by constructing lightweight backbone networks, namely SiamMixer. The network can be divided into two parts, the backbone network for extracting image features and the correlation computation for object searching and locating. The diagram of the proposed tracker is shown in Figure 1.

Convolutional Layer
The main objective of the backbone network is to model the local and global information in the input image within a limited parameters budget. The input image is first encoded using a series of MobileNetV2 blocks. Formally, for a given input image with di-mension of C i nput × W i nput × H i nput, where C i nput denotes the image channels, W i nput denotes the image width, and H i nput denotes the image height.
We apply an n × n depth-wise convolutional layer followed by a pointwise (1 × 1) convolutional layer to conduct structural encoding. To preserve the simplicity of the network structure, the MobileNetV2 blocks used for structural encoding are implemented with the same kernel size. The architecture of the backbone network is shown in Table 1. To reduce the runtime memory cost, we conduct the convolutional layer in a patchby-patch order. During convolutional layer inference, one small image patch is processed at a time. Once the small image patches are processed, the memory space they occupy is freed so that the peak memory cost can be reduced. The main drawback of this method is that it is spatially constrained and unable to encode the global information of the input image. Lin [22] proposes perceptual field redistribution via NAS, thus solving the problem of constrained perceptual fields caused by patch-based inference. However, this requires an additional hyperparameter optimization in the already substantial search space. This will incur a considerable search cost.
Therefore, we propose to use the mixer module to globally encode the convolutional feature maps. A patch-based inference example is shown in Figure 2.

Patch2
Merged Feature Map Conv1 Conv2 Figure 2. Patch-based inference is adopted to reduce the peak memory.

Mixing Module
Following the convolutional layer, we use the mixer module to encode global information of the convolutional feature map. The mixer module is inspired by the network design of MLP-Mixer [23]. MLP-Mixer proposes to use a Multi-Layer Perceptron(MLP) as an alternative structure to ViT [24]. MLP-Mixer repeats the MLP-only operations on the spatial and feature channels, thus realizing the image local encoding and global encoding. A diagram of the Mixer layer is shown in Figure 3. We combine patch-based inference with the Mixer layer to save the network from the restricted perceptual field. According to our experimental results, the combination of patch-based convolution and the Mixer layer significantly improves the accuracy of the network. To simplify the computational process, we modify the basic module of MLP-Mixer as follows: 1.
Replace GELU activation function with ReLU activation function.
Exponential arithmetic in mobile devices is usually expensive. To reduce network deployment costs, we use ReLU to replace the activation function that involves exponential operations. LayerNorms are commonly used for normalization in RNN networks because the input to RNN networks usually varies with the length of the sequence. In addition, due to the large size of the RNN network, it is not practical to use large batch size training to reduce internal covariate shift. However, LayerNorms require hidden layer statistics during both training and inference, which can slow down the inference of the networks. Since the mixer module we use has fixed dimensional inputs and the network size is small enough to use a large batch size for training, we think it is reasonable to replace the LayerNorms with BatchNorms. The original mixer network uses feature map transposition and a fully connected layer to implement channel-mixing operations. However, the transposition process would introduce unnecessary memory access and bring no computational revenue. Therefore, we use one-dimensional convolution to implement the channel-mixing process equivalently.
The adjustments we made to the network architecture made deploying the network on mobile devices easier. Experimental data shows that the impact of these adjustments on network accuracy is acceptable.

Target Locating
Inspired by SiamFC [12], we use the correlation operation f (z, x) to compare the template z extracted in the first frame and the enlarged search region x centered at the previous object position. If the two images contain the same object, there will be a significant response. To find the location of the target in the new image, we search pixel by pixel for the candidate location most similar to the template image. To obtain the correct scale of the target, the search area is resized into multiple scales, and the scale with the highest classification score is chosen to be the final scale. Although no high-level modeling of the target is performed, it provides a reliable and straightforward method for target localization, which is beneficial for our evaluation of the backbone network as well as for deployment on edge computing devices.

Training Setup
The template input and search region input sizes of SiamMixer are 128 × 128 and 256 × 256, respectively. Since the image size used for training is varied, the training images need to be cropped, resized, and padded. We train the network with positive and negative image pairings and use logistic loss and triplet loss as joint loss functions [25].
where L l and L t denote logistic loss and triplet loss, respectively. v p i is the score of the positive sample candidate. v n i is the score of the negative sample candidate, and y ∈ [−1, +1], which corresponds to the ground truth.
where M, N are the number of positive and negative samples. y is the ground truth label. The parameters of the network can be obtained by stochastic gradient descent: where z, x and θ is the parameters of the network, target image and search region image, respectively. Image pairs are obtained from the annotated video dataset. Both images in the image pair contain the target. The class of the object is ignored during training. The dataset is enhanced using random horizontal flips, random rotations, and random luminance changes, where the probability of random horizontal flips is 0.5%, random rotations are from −2 • to 2 • . The center of rotation is the center of the image. The random luminance variation uses the brightness factor to jitter image brightness. The brightness factor is chosen uniformly from [0.7, 1.3]

Datasets and Evaluation Metrics
We evaluate our tracker on the target-tracking datasets OTB100 [26] and UAV123 [27]. The OTB100 [26] dataset contains 100 challenging videos. The tracker is not reinitialized when the tracker loses its target. The dataset uses two metrics to evaluate the performance of the tracker. The precision plot indicates the percentage within a given distance threshold between the center of the predicted position and the center of the ground truth. Success plot indicates the percentage of the intersection ratio between the predicted location and ground truth within a given threshold. After obtaining the precision plot and the success plot, the score at the 20-pixel threshold is designated as the precision score, and the area under the curve of the success plot is designated as the success score.
The UAV123 dataset contains 123 challenging aerial videos. Although the data sources are different, UAV123 also evaluates the performance of the tracker using both the precision plot and the success plot.

Ablation Analysis
To demonstrate the effectiveness of the proposed method, we test the performance of different network structures.
We set the hyperparameter range according to the possible application scenarios of SiamMixer. To enable the network to run on common edge devices, the number of parameters of the network needs to be kept within 1 MB, and the total computation should be within 600 M MACs [36]. Therefore, we parametrically adjust the depth of the mixer module as a way to adjust the computational and parametric values of the network. We test four structures SiamMixer-XS, SiamMixer-S, SiamMixer-M, and SiamMixer-L, corresponding to depths 1, 2, 4, and 8, respectively. We evaluate these structures on different computing devices, and the network frame rates are shown in Table 2. The success scores of these network results on the OTB100 dataset are shown in Table 3. Our algorithms can run at more than real-time speeds on common GPU devices while maintaining a low memory footprint. On the Nvidia Jetson Xavier development board, an edge computing device, our algorithms can run at quasi-real-time speeds.
As shown in Tables 2 and 3, the increase in the depth of the mixer module brings limited performance improvement while significantly slowing down the network and increasing the number of parameters in the network. In addition, overly deep networks degrade network performance, which is consistent with the phenomenon described in SiamDW [14]. Therefore, we believe that SiamMixer-XS should be the optimal candidate for deployment at edge computing devices. For the performance comparison, we focused on the performance of the SiamMixer-XS.
We record the success score of different structure networks on the OTB100 [26] dataset, calculate the information density (accuracy per parameters) [37,38], and compare it with the state-of-the-art models. The comparison results are shown in Table 4.
Information density [37,38] is a metric that can effectively evaluate the efficiency of using network parameters. We want to make the most of limited storage space for edge-side deployments, so we introduce this metric in the comparison.
As can be seen from the comparison results, our SiamMixer-XS has a 6.8× smaller number of parameters than LightTrack-Mobile [21], the state-of-the-art lightweight network, and an 8.16× smaller number of parameters than SiamFC [12], which has similar performance. At the same time, our SiamMixer-XS is state-of-the-art in the metric of information density, as only a minimal amount of weight storage is required.
SiamMixer is built up with MobileNetV2 blocks and mixer modules. The activation functions of these modules require exponential operations, which is expensive for embedded devices. Therefore, we explore the impact of activation functions on network performance. We replace all activation functions in the SiamMixer-XS structure and test them on the OTB100 [26] dataset. In this part of the experiment, SiamFC [12] is selected as a baseline since SiamFC [12] also uses correlation for target localization. A network without a Mixer layer is also chosen to be the baseline to demonstrate the effectiveness of the combination of the per-patch convolution and the Mixer layer. The average success plot and precision plot of the trackers on the OTB100 dataset are shown in Figure 4.  Figure 4. Experiment results on all OTB100 sequences. SiamFC_Pruned denotes a modified SiamFC whose memory cost is reduced to 600 kB by lowering the number of channels.
Our experimental results show that when using the ReLU activation function, the overall precision score of SiamMixer-XS is 0.76, and the success score is 0.56, which is only 2.56% and 1.75% lower than the SiLU+GELU version. The loss of accuracy from replacing the activation function is acceptable but significantly reduces the hardware deployment cost, favorable for implementation with SIMD instructions.
Since different challenges in tracking place different obligations on the tracker, it is crucial to study the tracker's tracking performance under these factors. The precision plots and success plots of SiamMixer-XS on the 11 challenge factors are shown in Figures 5 and 6.  As can be seen, SiamMixer-XS achieves favorable results in most cases and, in particular, achieves significantly better results than its competitors in the DEF, FM, OCC, OPR, OV, and SV challenge.
The test results of SiamMixer-XS on the UAV123 dataset and the comparison with other algorithms are shown in Table 5. It should be noted that our model is not optimized for specific challenge scenarios, nor does it use an online learning strategy. This indicates that our network has good generalization ability, and the training process is relatively simple. Moreover, our algorithm was trained only on the GOT10k dataset and was not fine-tuned with any images from the OTB100 and UAV123 datasets. This validates the robustness and accuracy of our algorithm. The snapshot of the SiamMixer-XS tracking result on OTB100 [26] dataset is shown in Figure 7a. The snapshot of the SiamMixer-XS tracking result on UAV123 [27] dataset is shown in Figure 7b.

Storage and Analysis
Typically, algorithm analysis focuses on the computational complexity and workload while ignoring the runtime memory requirements. However, for practical application scenarios, target-tracking algorithms are usually deployed to computing devices with limited memory space and computational resources. Therefore, in addition to the computational workload and weight parameter analysis, we also analyze the storage requirements of each network layer.
For the convolutional layer, its weight parameters can be calculated by: where k c , C in and C out are the kernel size, the input channel number and the output channel number, respectively. And the feature maps storage requirement can be calculated by: where H, W, and S are the input feature map height, width, and convolution stride, respectively. During patch-based inference, both H and W are equal to patch size. The patch size determines the perceptual field and computation load. A large patch size at the beginning of the network leads to a less effective memory reduction and overlapping perceptual fields in the later stages of the network. In SiamMixer, we set the patchsize = 16 and use the mixer module to increase the network perceptual field in the latter stages of the network. Since no residual blocks are adopted during the convolution process, the feature map of the previous layer can be overwritten after the computation of one convolutional layer, so that the storage requirements can be lower. The mixer module consist of fully connected layer so its weight parameters and the feature maps storage requirement can be calculated by : where D and P are the hidden dim and patch number of the Mixer layer, respectively. The actual storage cost of the SiamMixer-XS is shown in Figure 8.   Figure 8. Memory use distribution of SiamMixer-XS. Using per-patch inference, we can significantly reduce the peak memory cost of the SiamMixer. Other variants of SiamMixer simply change the depth of the mixer module, and per-patch inference has the same effect on these variants.
As shown in the figure, patch-based inference reduces its runtime memory by 4.26× and the peak extra memory for the feature map is only 196 kB, which expands the design space of the algorithm and makes it possible to deploy the algorithm on mobile devices or to work with other algorithms.

Conclusions
In this paper, a lightweight target-tracking network structure is proposed. We use a simple and efficient backbone network to extract features from the target and searching area. We use a patch-based convolutional layer to encode local features of the image. The mixer module is employed for global feature encoding. By combining the advantages of CNNs and mixer networks, our network achieves a good balance of performance, number of parameters, and runtime memory. Furthermore, we deploy this novel tracking algorithm on edge computing hardware and achieve real-time visual object-tracking.