extfieldsofstudy:
Computer Science
Mathematics

text:
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning

Recently many algorithms were devised for reinforcement learning (RL) with function approximation. While they have clear algorithmic distinctions, they also have many implementation differences that are algorithm-independent and sometimes under-emphasized. Such mixing of algorithmic novelty and implementation craftsmanship makes rigorous analyses of the sources of performance improvements across algorithms difficult. In this work, we focus on a series of off-policy inference-based actor-critic algorithms -- MPO, AWR, and SAC -- to decouple their algorithmic innovations and implementation decisions. We present unified derivations through a single control-as-inference objective, where we can categorize each algorithm as based on either Expectation-Maximization (EM) or direct Kullback-Leibler (KL) divergence minimization and treat the rest of specifications as implementation details. We performed extensive ablation studies, and identified substantial performance drops whenever implementation details are mismatched for algorithmic choices. These results show which implementation or code details are co-adapted and co-evolved with algorithms, and which are transferable across algorithms: as examples, we identified that tanh Gaussian policy and network sizes are highly adapted to algorithmic types, while layer normalization and ELU are critical for MPO's performances but also transfer to noticeable gains in SAC. We hope our work can inspire future work to further demystify sources of performance improvements across multiple algorithms and allow researchers to build on one another's both algorithmic and implementational innovations.

Recently, there has been a series of off-policy algorithms derived from this perspective for learning policies with function approximations [2,22,48]. Notably, Soft Actor Critic (SAC) [22,23], based on a maximum entropy objective and soft Q-function, significantly outperforms on-policy [52,54] and off-policy [36,18,13] methods. Maximum a posteriori Policy Optimisation (MPO) [2], inspired by REPS [50], employs a pseudo-likelihood objective, and achieves high sample-efficiency and fast convergence compared to the variety of policy gradient methods [36,54,7]. Similar to MPO, Advantage Weighted Regression (AWR) [48], and its variant, Advantage Weighted Actor-Critic (AWAC) [42], also employ the pseudo-likelihood objective weighted by the exponential of advantages, and reports more stable performance than baselines both in online and offline [35] settings.
While these inference-based algorithms have similar derivations to each other, their empirical performances have large gaps among them when evaluated on the standard continuous control benchmarks, such as OpenAI Gym [6] or DeepMind Control Suite [58]. Critically, as each algorithm has unique low-level implementation or code design decisions -such as value estimation techniques, action distribution for the policy, and network architectures -aside from high-level algorithmic choices, it is difficult to exactly identify the causes of these performance gaps as algorithmic or implementational.
In this paper, we first derive MPO, AWR, and SAC from a single objective function, through either Expectation-Maximization (EM) or KL minimization, mathematically clarifying the algorithmic connections among the recent state-of-the-art off-policy actor-critic algorithms. This unified derivation allows us to precisely identify implementation techniques and code details for each algorithm, which are residual design choices in each method that are generally transferable to other algorithms. To reveal the sources of the performance gaps, we experiment with carefully-selected ablations of these identified implementation techniques and code details, such as tanh-squashed Gaussian policy, clipped double Q-learning [13], and network architectures. Specifically, we keep the high-level algorithmic designs while normalizing the implementation designs, enabling proper algorithmic comparisons. Our empirical results successfully distinguish between highly co-adapted design choices and no-co-adapted ones 2 . We identified that clipped double Q-learning and tanh-squashed policies, the sources of SoTA performance of SAC, are highly co-adapted, specific to KL-minimization-based method, SAC, and difficult to transfer and benefit in EM-based methods, MPO or AWR. In contrast, we discover that ELU [8] and layer normalization [4], the sources of SoTA performance of MPO, are transferable choices from MPO that also significantly benefit SAC. We hope our work can inspire more future works to precisely decouple algorithmic innovations from implementation or code details, which allows exact sources of performance gains to be identified and algorithmic researches to better build on one another.

Related Work
Inference-based RL algorithms RL as probabilistic inference has been studied in several prior contexts [60,61,34,11,57,45], but many recently-proposed algorithms [2,22,48] are derived separately and their exact relationships are difficult to get out directly, due to mixing of algorithmic and implementational details, inconsistent implementation choices, environment-specific tunings, and benchmark differences. Our work organizes them as a unified policy iteration method, to clarify their exact mathematical algorithmic connections and tease out subtle, but important, implementation differences. We center our analyses around MPO, AWR, and SAC, because they are representative algorithms that span both 50,43,44,1,56,42] and 51,12,29,21,33,32] RL and achieve some of the most competitive performances on popular benchmarks [6,58]. REPS [50], an EM approach, inspired MPO, AWR, and our unified objective in Eq. 1, while Soft Q-learning [21], a practical extension of KL control to continuous action space through Liu and Wang [38], directly led to the development of SAC.
Meta analyses of RL algorithms While many papers propose novel algorithms, some recent works focused on meta analyses of some of the popular algorithms, which attracted significant attention due to these algorithms' high-variance evaluation performances, reproducibility difficulty [9, 28, 65], and frequent code-level optimizations [25,63,10,3]. Henderson et al. [25] empirically showed how these RL algorithms have inconsistent results across different official implementations and high variances even across runs with the same hyper-parameters, and recommended a concrete action item for the community -use more random seeds. Tucker et al. [63] show that high performances of actiondependent baselines [19,17,37,20,67] SAC [22], and other algorithms (see Appendix B). We follow the notation of Sec. 3 & 4. We characterize them with the algorithm family (EM or KL), how policies are updated (πq and πp; SG stands for stochastic-gradient-based, and TR stands for trust-region-based), the choice of G(·), how G is estimated, and the parameterization of the policy. While the advantage function just can be interpreted as Q-function with baseline subtraction, we explicitly write A π when the state-value function is parameterized, not Q-function. ( * Note that DDPG and TD3 are not "inference-based", but can be classified as KL control variants.) how code-level optimizations, instead of the claimed algorithmic differences, actually led more to PPO's superior performances. Andrychowicz et al.
[3] describes low-level (e.g. hyper-parameter choice, and regularization) and high-level (e.g. policy loss) design choices in on-policy algorithms affects the performance of PPO by showing results of large-scale evaluations.
In contrast to those prior works that mainly focus on a single family of on-policy algorithms, PPO and TRPO, and evaluating their implementation details alone, our work focuses on two distinct families of off-policy algorithms, and more importantly, presents unifying mathematical connections among independently-proposed state-of-the-art algorithms. Our experiments demonstrate how some implementation choices in Table 1 and code details are co-evolved with algorithmic innovations and/or have non-trivial effects on the performance of off-policy inference-based methods.

Preliminaries
We consider a Markov Decision Process (MDP) defined by state space S, action space A, state transition probability function p : S × A × S → [0, ∞), initial state distribution p 1 : S → [0, ∞), reward function r : S × A → R, and discount factor γ ∈ [0, 1). Let R t denote a discounted return ∞ u=t γ u−t r(s u , a u ). We assume the standard RL setting, where the agent chooses actions based on a parametric policy π θ and seeks for parameters that maximize the expected return E π θ [R 1 ]. Value functions for a policy π are the expected return conditioned by a state-action pair or a state, that is, Q π (s t , a t ) := E π [R t |s t , a t ], and V π (s t ) := E π [R t |s t ]. They are called the state-action-value function (Q-function), and state-value function, respectively. The advantage function [5] is defined as A π (s, a) := Q π (s, a) − V π (s). The Q-function for a policy π is the unique fixed point of the Bellman operator T π defined by T π Q(s, a) = r(s, a) + π(a |s )p(s |s, a)Q(s , a ) ds da for any function Q : S × A → R. We denote a trajectory or successive state-action sequence as τ := (s 1 , a 1 , s 2 , a 2 , . . . ). We also define an unnormalized state distribution under the policy π by d π (s) = ∞ t=1 γ t p(s t = s|π).
Inference-based Methods For simplicity, we consider a finite-horizon MDP with a time horizon T for the time being. As a result, a trajectory τ becomes a finite length: τ := (s 1 , a 1 , . . . , s T −1 , a T −1 , s T ).
As in previous works motivated by probabilistic inference [34,2], we introduce to the standard graphical model of an MDP a binary event variable O t ∈ {0, 1}, which represents whether the action in time step t is optimal or not. To derive the RL objective, we consider the marginal log-likelihood log Pr(O = 1|π p ), where π p is a policy. Note that O = 1 means O t = 1 for every time step. As is well known, we can decompose this using a variational distribution q of a trajectory as follows: is the evidence lower bound (ELBO). Inference-based methods aim to find the parametric policy which maximizes the ELBO J (p, q).
There are several algorithmic design choices for q and Pr(O = 1|τ ). Since any q is valid, a popular choice is the one that factorizes in the same way as p, that is, where π p is a prior policy, and π q is a variational posterior policy. We may also choose which policy (π p or π q ) to parameterize. As for Pr(O = 1|τ ), the most popular choice is the following one [21,22,2,34,48]: where η > 0 is a temperature, and G is a function over S × A, such as an immediate reward function r, Q-function Q π , and advantage function A π . While we employ exp(·) in the present paper, there are alternatives [46,47,55,66] Incorporating these design choices, we can rewrite the ELBO J (p, q) in a more explicit form as; In the following section, we adopt this ELBO and consider its relaxation to the infinite-horizon setting. Then, starting from it, we derive MPO, AWR, and SAC.

A Unified View of Inference-based Off-Policy Actor-Critic Algorithms
In Sec. 3, we provided the explicit form of the ELBO J (π p , π q ). However, in practice, it is difficult to maximize it as the expectation E q depends on π q . Furthermore, since we are interested in a finite-horizon setting, we replace t∈[T ] E q with E dπ(s) . Note that the latter expectation E dπ(s) is taken with the unnormalized state distribution d π under an arbitrary policy π. This is commonly assumed in previous works [2,22]. This relaxation leads to the following optimization:  Table 1. We first explain how these algorithms can be grouped into two categories of approaches for solving Eq. 1, and then we elaborate additional implementation details each algorithm makes.

Unified Policy Iteration: Algorithmic Perspective
Eq. 1 allows the following algorithmic choices: how or if to parameterize π p and π q ; what optimizer to use for them; and if to optimize them jointly, or individually while holding the other fixed. We show that the algorithms in Table 1 can be classified into two categories: Expectation-Maximization control (EM control) and direct Kullback-Leibler divergence minimization control (KL control). , and we term it EM control. At high level, the algorithm non-parametrically solves for the variational posterior π q while holding the parametric prior π p = π θp fixed (E-Step), and then optimize π p holding the new π q fixed (M-Step). This can be viewed as either a generic EM algorithm and as performing coordinate ascent on Eq. 1. We denote θ p and π q after iteration k of EM steps by θ (k) p and π (k) q , respectively.

Expectation-Maximization (EM) Control
In E-step at iteration k, we force the variational posterior policy π (k) q to be close to the optimal posterior policy, i.e., the maximizer of the ELBO J (π θ (k−1) p , π q ) with respect to π q . EM control converts the hard-constraint optimization problem in Eq. 1 to solving the following Lagrangian, We analytically obtain the solution of Eq. 2, where Z(s) is the partition function.
In M-Step at iteration k, we maximize the ELBO J (π (k) q , π p ) with respect to π p . Considering the optimization with respect to π p in Eq. 1 results in forward KL minimization, which is often referred to as a pseudo-likelihood or policy projection objective, (3) where we may approximate Z(s) ≈ 1 M M j=1 exp(η −1 G(s, a j )) with a j ∼ π θ (k−1) p (·|s) in practice.

Direct Kullback-Leibler (KL) Divergence Minimization Control
In contrast to the EM control in Sec. 4.1.1, we only optimize the variational posterior π q while holding the prior π p fixed. In this scheme, π q is parameterized, so we denote it as π θq . This leads to KL control [59,62,51,12,29,21].
Equivalent to E-step in Sec. 4.1.1, we force the variational posterior policy π θq to be close to the optimal posterior policy, i.e, the maximizer of J (π p , π q ) with respect to π q . The difference is that instead of analytically solving J (π p , π q ) for π q , we optimize J (π p , π θq ) with respect to θ q , which results in the following objective, max θq E dπ(s)π θq (a|s) η −1 G(s, a) − log π θq (a|s) π p (a|s) .

"Optimal" Policies of EM and KL Control
While we formulate EM and KL control in a unified framework, we note that they have a fundamental difference in their definition of "optimal" policies. KL control fixes π p and converges to a regularizedoptimal policy in an exact case [15,64]. In contrast, EM control continues updating both π p and π q , resulting in convergence to the standard optimal policy in an exact case [51]. We also note that EM control solves KL control as a sub-problem; for example, the first E-step exactly corresponds to a KL control problem, except for the policy parameterization.

Unified Policy Iteration: Implementation Details
In this section, we explain the missing pieces of MPO, AWR, and SAC in Sec. 4.1. Additionally, we also describe the details of other algorithms [30,49,50,41,56,36,13,52,54] from the EM and KL control perspective. See Appendix B for the details.

MPO
Algorithm MPO closely follows the EM control scheme explained in Sec. 4.1.1, wherein π p = π θp is parametric, and π q is non-parametric.
Implementation [π q Update] This corresponds to the E-step. MPO uses a trust-region (TR) method. Concretely, it replaces the reverse KL penalty (second term) in the Lagrangian (Eq. 2) with a constraint and analytically solves it for π q . As a result, η is also optimized during the training by minimizing the dual of Eq. 2, which resembles REPS [50]: [π p Update] This corresponds to the M-step. MPO uses a combination of Stochastic Gradient (SG) ascent and trust-region method based on a forward KL divergence similarly to TRPO [52]. Concretely, it obtains θ (k) p by maximizing the objective (Eq. 3) with respect to θ p subject to E dπ(s) [D KL (π θ (k−1) p (·|s) || π θp (·|s))] ≤ ε. MPO further decompose this KL divergence to two terms, assuming Gaussian policies; one term includes only the mean vector of π θp (·|s), and the other includes only its covariance matrix. Abdolmaleki et al.
[2] justify this as a log-prior of MAP estimation, which is assumed as a second-order approximation of KL divergence.
[G and G Estimate] MPO uses the Q-function of π θ (k−1) p as G in the k-th E-step. It originally uses the Retrace update [40], while its practical implementation [27] uses a single-step Bellman update.
[π θ ] For a parameterized policy, MPO uses a Gaussian distribution with state-dependent mean vector and diagonal covariance matrix. The trust-region method in MPO's M-step heavily relies on this Gaussian assumption. Since a Gaussian distribution has an infinite support, MPO has a penalty term in its policy loss function that forces the mean of the policy to stay within the range of action space.

AWR and AWAC
Algorithm AWR slightly deviates from the EM control. In the M-step, AWR simply set Z(s) to 1.

Implementation [π q
Update] This corresponds the E-step. AWR and AWAC analytically solves the Lagrangian (Eq. 2). In contrast to MPO, they don't use trust-region method.
[π p Update] This corresponds to the M-step. At iteration k, AWR uses an average of all previous policies π p k := 1 k k−1 j=0 π θ (j) p instead of π θ (k−1) p (cf. Eq. 3). In practice, the average policy π p k is replaced by samples of actions from a replay buffer, which stores action samples of previous policies.
[G and G Estimate] AWR uses the advantage function of π p k as G in the k-th E-step, and learns the state-value function of π θ (k−1) p with TD(λ) [53]. Due to its choice ofπ p k , this avoids importance corrections [40]. In contrast, AWAC estimates the advantage via the Q-function with TD(0).
[π θ ] For a parameterized policy, both AWR and AWAC use a Gaussian distribution with statedependent mean vector and state-independent diagonal covariance matrix (a constant one for AWR). As in MPO, they uses a penalty term to keep the mean of the policy within the range of action space.

SAC
Algorithm Contrary to MPO and AWR, SAC follows the KL control scheme explained in Sec. 4.1.2, wherein the variational posterior policy π q = π θq is parameterized, and the prior policy π p is fixed to the uniform distribution over the action space A. SAC uses as G a soft Q-function: with H(π θq (·|s t )) being −E π θq [log π θq (a t |s t )|s t ]. Implementation [π q and π p Update] SAC performs stochastic gradient (SG) ascent updates of only π q , where the objective function is shown in Eq. 1. SAC has no operation equivalent to M-step in MPO and AWR, since it keeps the prior policy π p to the uniform distribution.
Similar to the temperature tuning in MPO,Haarnoja et al. [23] consider the dual function of entropyconstrained Lagrangian and treat the temperature η as a Lagrange multiplier, which results in, whereH is the target entropy to ensure that the entropy of the policy should be larger than it.
[G and G Estimate] SAC uses Q π θq soft as G, and TD(0)-like algorithm based on this soft Q-function. In policy evaluation, clipped double Q-learning [13], which retains multiple Q-functions (typically two) and takes the minimum, is employed to suppress the overestimation of Q value.
[π θ ] The policy of SAC is a squashed Gaussian distribution: SAC first samples a random variable u from a Gaussian distribution N (µ θ (s), Σ θ (s)) with state-dependent mean vector and diagonal covariance; then, it applies the tanh function to u and obtain the action a ∈ [−1, 1] |A| . (If a dimension of the action space is not [−1, 1], an appropriate scaling and shift by an affine function is applied after the squashing.) Tanh squashing prevents out-of-bounds action.

Empirical Comparison
To evaluate the empirical performance of these off-policy algorithms (MPO, AWR, AWAC, and SAC), we compare their performances on Open AI Gym MuJoCo environments, namely, Hopper, Walker2d, HalfCheetah, Ant, Humanoid, and Swimmer, following Haarnoja et al. [23]. We reproduce all algorithms based on pytorch RL library [14], referring their original implementations [22,48,42,27]. Figure 1 shows that SAC outperforms others in four environments (Hopper, Walker2d, HalfCheetah, and Humanoid), while MPO in Ant and AWR in Swimmer achieves the best performance. Generally, SAC seems to perform consistently better. We extensively evaluate MPO, AWR, and SAC on the 28 tasks on DeepMind Control Suite and 3 MuJoCo manipulation tasks. See Appendix C and D for the details.

Evaluation on Implementational Choices
In Sec. 4.3, SAC shows notable results in most environments, while we revealed that the derivation and formulation of those methods resemble each other. To specify the effect of each implementational   or code detail on the performance, we experiment with extensive and careful one-by-one ablations on; (1) clipped double Q-learning, (2) action distribution for the policy, (3) activation and normalization, and (4) network size. The former two correspond to implementation details (the choice of G estimate and the parameterization of the policy), and the latter two correspond to code details (see Appendix G for further experiments on other implementation details, such as π p update or the choice of G). We also conclude several recommendations for the practitioners (Table 6).

Clipped Double Q-Learning
We hypothesize that clipped double Q-learning has a large effect on the notable performance of SAC, and could be transferable in EM control methods. To verify this, we test the effect of clipped double Q-learning. Instead of AWR, here we evaluate AWAC since it uses Q-function. Table 2 shows that single-Q SAC outperforms original one in HalfCheetah and Swimmer that do not have the termination of the episode, while struggles to learn in Hopper, Walker, Ant and Humanoid that have the episodic termination conditions. A hypothesis is that single-Q SAC obtains a more exploratory policy due to larger overestimation bias, which can help in environments where explorations are safe, but hurt in environments where reckless explorations lead to immediate terminations. In contrast, clipped double Q-learning does not help MPO or AWAC as significantly as SAC; most results do not change or lead to slight improvements over the originals. This suggests that clipped double Q-learning might be a co-dependent and indispensable choice to KL control methods.

Recommendation
Use clipped double Q-learning as a default choice, but you can omit it in EM control methods or non-terminal environments, such as HalfCheetah or Swimmer.

Action Distribution for the Policy
Another hypothesis is that the tanh-squashed policy (last column in Table 1) is an important and transferable design choice. We compare SAC without tanh transform (with MPO action penalty instead) to the original one, which results in drastic degradation (Table 3). This can be caused by the maximum entropy objective that encourages maximizing the covariance. These observations suggest that the high performance of SAC seems to highly depend on the implementational choice of the policy distribution. In contrast, MPO and AWR with tanh squashing don't enjoy such large performance gain, or achieve worse cumulative rewards. This also suggests that tanh-squashed policy might be highly co-adapted in SAC and less transferable to EM control methods. Note that for EM control, we must clip the actions to keep them within the supports of distributions; a ∈ [−1 + , 1 − ] |A| . We found that no clipping cause significant numerical instability. See Appendix H for the details.
Recommendation Use the original distributions for each algorithm. If you use the tanh-squashed Gaussian in EM control methods, clip the action to be careful for the numerical instability.

Activation and Normalization
The implementation of MPO [27] has some code-level detailed choices; ELU activation [8] and layer normalization [4]. We hypothesize that these, sometimes unfamiliar, code details in practical implementation stabilize the learning process, and contribute to the performance of MPO most.    Table 4 shows that layer normalization can contribute to a significantly higher performance of SAC in Ant, and replacing ReLU with ELU also improves performance a lot in Swimmer, where AWR is the best in Figure 1. In contrast, the performance of MPO drastically collapsed when we just removed ELU and layer normalization. This observation suggests that these code-level choices are not only indispensable for MPO, but transferable and beneficial to both KL and EM control methods. Additionally, we tested incorporating ELU and layer normalization to SAC in 12 DM Control tasks where MPO outperformed SAC, and observed that they again often benefit SAC performances substantially. See Appendix C for the details.
Recommendation It is worth considering to replace the activation function from ReLU to ELU and incorporate the layer normalization, to achieve the best performance in several tasks. Both of them are transferable between EM and KL control methods.

Network Size
While on-policy algorithms such as PPO and TRPO can use the common network architecture as in prior works [10,25], the architecture that works well in all the off-policy inference-based methods is still not obvious, and the RL community does not have agreed upon default choice. To validate the dependency on the networks among the inference-based methods, we exchange the size and number of hidden layers in the policy and value networks. We denote the network size of MPO ((256, 256, 256) for policy and (512, 512, 256) for value) as large (L), of SAC ((256, 256) for policy and value) as middle (M), and of AWR ((128, 64) for policy and value) as small (S) (see Appendix A for the details). Table 5 illustrates that SAC with large network seems to have better performance. However, as shown in Appendix F (Figure 8), the learning curve sometimes becomes unstable. While the certain degree of robustness to the network size is observed in SAC, EM control methods, MPO  and AWR, seem fragile and more dependent on the specific network size. This trend is remarkable in MPO. AWR (M) or (L) also struggles to learn in high-dimensional state tasks, such as Ant (111 dim), or Humanoid (376 dim). The results also imply that, in contrast to on-policy methods [3], the network size in the off-policy inference-based methods seems a less transferable choice.
Recommendation For SAC, use medium size network. Large size will also work, but the learning curve might be unstable. For MPO, we strongly recommend to stick to large size, because it is very sensitive to the network size. For AWR, using small size is a better choice, especially in high-dimensional state tasks, such as Ant, or Humanoid.  In this work, we present a taxonomy of inference-based algorithms, and successfully identify algorithm-specific as well as algorithm-independent implementation details that cause substantial performance improvements. We first reformulated recent inference-based off-policy algorithms -such as MPO, AWR and SAC -into a unified mathematical objective and exhaustively clarified the algorithmic and implementational differences. Through precise ablation studies, we empirically show that implementation choices like tanh-squashed distribution and clipped double Q-learning are highly co-adapted to KL control methods (e.g. SAC), and difficult to benefit in EM control methods (e.g. MPO or AWR). As an example, the network architectures of inference-based off-policy algorithms, especially EM controls, seem more co-dependent than on-policy methods like PPO or TRPO, which therefore have significant impacts on the overall algorithm performances and need to be carefully tuned per algorithm. Such dependence of each algorithmic innovation on specific hand-tuned implementation details makes accurate performance gain attributions and cumulative build-up of research insights difficult. In contrast, we also find that some code-level implementation details, such as ELU and layer normalization, are not only indispensable choice to MPO, but also transferable and beneficial to SAC substantially. We hope our work can encourage more works that study precisely the impacts of algorithmic properties and empirical design choices, not only for one type of algorithms, but also across a broader spectrum of deep RL algorithms. [29] Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, 2017.

References
[30] Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In Advances in neural information processing systems, 2008.
[31] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
[33] Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Advances in Neural Information Processing Systems, 2020.

Appendix A Network Architectures
In this section, we describe the details of the network architectures used in Sec. 4 and 5.
We mainly used 4 GPUs (NVIDIA V100; 16GB) for the experiments in Sec. 4 and 5 and it took about 4 hours per seed (in the case of 3M steps). Actually, we conducted exhaustive evaluations through the enormous experiments, and we hope our empirical observations and recommendations help the practitioners to explore the explosive configuration space.

B Relations to Other Algorithms
We here explain the relation of the unified policy iteration scheme covers other algorithms. While we mainly focused on AWR, MPO, and SAC in the this paper, our unified scheme covers other algorithms too, as summarized in Table 1: EM control algorithms: • PoWER [30]: πp (= π θ ) update is analytic. G = η log Q πp and Q πp is estimated by TD(1). π θ = N (µ θ (s), Σ θ (s)) . • RWR [49]: πp = π θ is updated by SG. G = η log r , and π θ = N (µ θ (s), Σ) . When the reward is unbounded, RWR requires adaptive reward transformation (e.g. u β (r(s, a)) = β exp(−βr(s, a)); β is a learnable parameter). • REPS [50]: πp is πq of the previous EM step (on-policy) or a mixture of all previous πq (off-policy), which is approximated by samples. G = A πp and estimated by a single-step TD error with a state-value function computed by solving a dual function. πq is assumed as a softmax policy for discrete control in the original paper. • UREX [41]: πp = π θ is updated by SG. G = Q πp and estimated by TD(1). π θ is assumed as a softmax policy for discrete control in the original paper. • V-MPO [56]: Almost the same as MPO, but a state-value function is trained by n-step bootstrap instead of Q-function. Top-K advantages are used in E-step.

KL control algorithms:
• TRPO [52]: πq = π θ = N (µ θ (s), Σ θ ). The KL penalty is converted to a constraint, and the direction of the KL is reversed. G = A πp and estimated by TD(1). πp is continuously updated to πq. • PPO with a KL penalty [54]: πq = π θ = N (µ θ (s), Σ θ ). The direction of the KL penalty is reversed. G = A πp and estimated by GAE [53]. An adaptive η is used so that DKL(πp || πq) approximately matches to a target value. • DDPG 3 [36]: πq = π θ is the delta distribution and updated by SG. G = Q πq and estimated by TD(0). η = 0 (i.e., the KL divergence and πp update are ignored). • TD3 2 [13]: It is a variant of DDPG and leverages three implementational techniques, clipped double Q-learning, delayed policy updates, and target policy smoothing. • BRAC [68] and BEAR [31] (Offline RL): When we assume πp = π b (any behavior policy), and omitting its update, some of the offline RL methods, such as BRAC or BEAR, can be interpreted as one of the KL control methods. Both algorithms utilize the variants of clipped double Q-learning (λ-interpolation between max and min). Figure 2: Benchmarking results on DeepMind Control Suite 28 environments. The performances are averaged among 10 random seeds. We use an action repeat of 1 throughout all experiments for simplicity.

C Benchmarks on DeepMind Control Suite
In this section, we show the benchmarking results on 28 tasks in DeepMind Control Suite (Figure 2). Each algorithm is run with 2.5M steps (except for humanoid domain; 14M steps), following Abdolmaleki et al. [2]. While the previous work mentioned that tuning the number of action repeats was effective [33], we used an action repeat of 1 throughout all experiments for simplicity. We also use the hyper-parameters of each algorithm presented in Appendix A. As discussed in Sec. 5.3, we incorporate ELU and layer normalization into SAC in several domains where SAC is behind MPO or AWR. ELU and layer normalization significantly improve performances, especially in pendulum_swingup and cartpole_swingup_sparse. Some of MPO results don't seem to match the original paper, but we appropriately confirmed that these results are equivalent to those of its public implementation [27].

D Benchmarks on MuJoCo Manipulation Tasks
We extensively evaluate their performance in the manipulation tasks ( Figure 3). The trend seems the same as the locomotion tasks, while AWR beats SAC and MPO in Striker, which means they fall into sub-optimal.

E Reproduction Results of AWR on MuJoCo Locomotion Environments
We re-implemented AWR based on PFRL, a pytorch-based RL library [14], referring its original implementation [48]. Figure 4 shows the performance of our implementation in the original experimental settings, also following hyper-parameters. We recovered the original results in Peng et al. [48] properly.

F Learning Curves
In this section, we present learning curves of the experiments in Sec. 5.    Table 3; ablation of Tanh transformation. A line that stopped in the middle means that its training has stopped at that step due to numerical error. We test SAC without tanh squashing, AWR with tanh, and MPO with tanh.  Table 5; experiment for finer network sizes. (S) stands for the small network size from AWR, (M) for the middle network size from SAC, and (L) for the large network size from MPO.

G Additional Experiments for Deeper Analysis of Implementation Details
We share the additional experimental results for deeper analysis of implementation details and co-adaptation nature. We report the final cumulative return after 3M steps for Ant/HalfCheetah/Walker2d/Swimmer, 1M steps for Hopper, and 10M steps for Humanoid. All results below are averaged among 10 random seeds. These extensive experimental observations below suggest not only the co-adaptive nature and transferability of each implementation and code detail (discussed in Sec. 6), but also the properties of each kind of algorithm (KLbased and EM-based); KL-based methods, such as SAC, shows the co-dependent nature in implementation details (clipped double Q-learning, Tanh-Gaussian Policy) but robustness to the code details related to neural networks. In contrast, EM-based methods, such as MPO and AWR, show the co-dependent nature in code details but robustness to the implementation details. We hope these empirical observations from our experiments are valuable contributions to the RL community.
π p Update We test different types of πp Update as summarized in Table 1 to investigate the effectiveness of implementation choices. We prepare 4 variants: (1) MPO or AWAC with a uniform prior, (2) SAC with target policy instead of a fixed uniform prior, (3) MPO without trust-region (only SG update). The details of (1) -(3) are described below: (1) We use the actions sampled from uniform distribution as well as the samples from the policy at the past iteration π θ (k−1) p for the M-step in EM-controls; aj ∼ αUnif. + (1 − α)π θ (k−1) p , α ∈ (0, 1]. These variants are much closer to SAC (using uniform distribution as πp). We test α = 0.25, 0.5, 0.75 for both MPO and AWAC.
(2) We copy the parameter of πq at a certain interval and use it as πp in the objective of KL Control, similar to MPO/PPO/TRPO. It seems "KL-regularized" actor-critic, rather than "soft" (entropy-regularized  Table 11 shows that SAC without soft Q degrades its performance over 5 tasks except for Ant, while it is not so drastic compared to clipped double Q or Tanh-Gaussian policy. In contrast, MPO with soft Q slightly improves the performance (over 4 tasks), and AWAC with soft Q slightly also does (over 3 tasks). These trends are similar to the clipped double Q or Tanh-Gaussian policy. We think these experiments support our empirical observation: KL-based methods, such as SAC, show the robustness to the code details, while EM-based methods, such as MPO and AWR, show the co-dependent nature in code details but robustness to the implementation details. Network Size for AWAC To investigate the co-dependent nature between implementation and code details more precisely, we add the network size ablation of AWAC, whose implementations stand between MPO and AWR. AWAC differs πp Update and network size (the default choice of AWAC is (M)) from MPO (in fact, MPO uses TD (0) in open-source implementation [27] and we assume the difference of π θ might be minor). Also, AWAC differs G and G estimate from AWR.
The results of AWAC (Table 12) show a similar trend to AWR in high-dimensional tasks (Ant, Humanoid); a larger network did not help. We may hypothesize that πp Update of AWR/AWAC, mixture+SG, is not good at optimizing larger networks, compared to SG + TR of MPO. In contrast, especially, Hopper and Walker2d show a

H Failed Ablations
This section provides the failure case of ablations on tanh-squashed distributions and exchanging network architectures, which shows the catastrophic failure during training, and unclear insights.

H.1 Action Distribution for the Policy: Without Action Clipping
We observe that naive application of tanh-squashing to MPO and AWR significantly suffers from numerical instability, which ends up with NaN outputs (Table 15 and Figure 9). As we point out in Sec. 5.2, the practical solution is to clip the action within the supports of distribution surely; a ∈ [−1 + ,    Table 15. We test SAC without tanh-squashed distribution, AWR with tanh, and MPO with tanh. SAC without tanh transform (using MPO action penalty instead) results in drastic degradation of the performance, which can be caused by the maximum entropy objective that encourages the maximization of the covariance. AWR and MPO with tanh squashing become numerically unstable. A line that stopped in the middle means that its training has stopped at that step due to numerical error.

H.2 Network Architecture: Whole Swapping
In contrast to prior works on TRPO and PPO, the network architecture that works well in all the off-policy inference-based methods is not obvious, and the RL community doesn't have an agreeable default choice. Since the solution space is too broad without any prior knowledge, one possible ablation is that we test 3 different architectures that work well on at least one algorithm.
To validate the dependency of the performance on the network architecture, we exchange the configuration of the policy and value networks, namely, the size and number of hidden layers, the type of activation function, network optimizer and learning rate, weight-initialization, and the normalization of input state (See Appendix A). All other components remain the original implementations.
However, this ablation study might end up the insufficient coverage and the unclear insights. We broke down the network architecture comparison into the one-by-one ablations of activation and normalization, and experimented with finer network sizes. These results suggest that these off-policy inference-based algorithms might be fragile with other network architectures and more co-dependent with architectures than on-policy algorithms. A line that stops in the middle means that training has stopped at that step with numerical error due to NaN outputs.      Table 21: Results in Swimmer-v2 environment ( † numerical error happens during training). All results are averaged over 10 seeds and we also show their standard deviations.


s2fieldsofstudy:
Computer Science
Business

text:
Deliberate Clipping and Iterative Distortion Recovery for UCA-Based OAM Multiplexing Systems

Deliberate amplitude clipping is a simple and well-known technique to reduce the peak-to-average power ratio of orthogonal frequency division multiplexing (OFDM) systems. In this paper, we propose a clipping technique for peak power reduction in orbital angular momentum (OAM) multiplexing systems with uniform circular array (UCA) antennas. In the proposed technique, clipping is performed on digital baseband signals prior to OAM beamforming and pulse-shaping filtering, which helps to avoid out-of-band radiation and affecting the orthogonality of the OAM modes. In addition, an iterative distortion recovery algorithm is proposed in order to mitigate performance degradation in bit error rate (BER) due to the clipping. The algorithm is derived by unfolding the clipping noise cancellation (CNC) algorithm for OFDM systems into layers and by introducing layer-wise learnable parameters. Simulation results show that for a realistic OAM multiplexing system with 256QAM signaling, the unfolded CNC exhibits excellent BER performance even when the conventional CNC suffers from a high error-floor. The combination of the proposed clipping and distortion recovery schemes provides a significant reduction in the peak power of the OAM signals at the cost of only a slight degradation in BER performance.

I. INTRODUCTION
Radio orbital angular momentum (OAM) multiplexing transmission [1], [2] has recently attracted attention for application to high-capacity wireless communication systems, especially to point-to-point (PTP) line-of-sight (LOS) millimeter wave radio systems for mobile fronthaul and backhaul links [3]- [8].
OAM multiplexing is realized with electromagnetic waves of different OAM modes, which are inherently orthogonal to each other. Several methods have been proposed to generate OAM waves by using special antennas, such as helicoidal parabolic antennas [2], spiral phase plates [3], and thin metamaterial plates [9]. Among them, the uniform circular array (UCA) antenna is considered to be especially promising because of its simple structure and low cost. In [10], it was shown that standard antennas arranged in circular arrays can be used to generate OAM and that only local The associate editor coordinating the review of this manuscript and approving it for publication was Li Zhang. measurements are required for detecting OAM-modes. Edfors and Johansson [11] theoretically investigated the UCA-based OAM channel and showed its equivalence in terms of channel capacity to a pre-coded MIMO channel. Moreover, several experimental studies have recently demonstrated the effectiveness and feasibility of OAM multiplexing systems utilizing UCAs [4]- [7].
The UCA-based OAM system presented in [11] is equipped with UCAs with N antenna elements at each of the transmitter (Tx) and receiver (Rx) ends, which face each other on the same beam axis in free space. The pair of Tx and Rx UCAs leads to a circulant channel matrix and thus N parallel channels are created by employing a discrete Fourier transform (DFT) and its inverse transform as respective preand post-processings. The UCA-based OAM system can thus be thought of as an N × N LOS-MIMO system with OAM beamforming processing. There are two major approaches to implementing the OAM beamforming processing functions; one is to use a Butler matrix [4], [12], [13], which is a passive analog beamforming circuit, and the other is to perform digital signal processing [5] on baseband signal samples. While each of these approaches has its own advantages and disadvantages, we focus in this paper on the peak-power issue of the digital beamforming approach, in which the OAM signals are generated from baseband signals obtained by precoding N independent signals using a DFT matrix. The precoded signal at each antenna element in the UCA exhibits high peak transmission power, which leads to a significant reduction in power efficiency of the power amplifier (PA). The UCA-based OAM transmitter with the digital beamforming function thus needs to reduce the peak power at each antenna element in order to improve the power efficiency. The issue concerned in this paper is the high peak power of the OAM signals, which is caused by signal multiplexing in digital domain.
A number of peak power reduction techniques have been presented for orthogonal frequency division multiplexing (OFDM) systems in the literature and references therein [14], [15], and many of them are applicable to the UCA-based OAM multiplexing systems. Among the existing techniques for OFDM systems, we focus on clipping and filtering (CAF) [16]- [18]. CAF is one of the simplest and most effective techniques, wherein the peaks of the signal are clipped and the out-of-band radiation caused by the clipping is filtered at the Tx side. The critical issue here is the clipping distortion that remains after the filtering, which leads to a significant degradation in bit error rate (BER) performance. As a way to improve BER performance, iterative distortion recovery algorithms have been proposed [19]- [22], in which the clipping distortion is reconstructed using a decision feedback approach and then it is subtracted from the received signals observed at the Rx end.
On the basis of the above CAF techniques developed for OFDM systems, we propose a clipping method and its corresponding iterative clipping distortion recovery algorithm for UCA-based OAM systems. To avoid out-of-band radiation and affecting the orthogonality of the OAM modes, the clipping process is performed on the baseband digital signals prior to the OAM beamforming and the pulse-shaping filtering. The resulting peak power of the signal at each antenna element depends on the clipping ratio, roll-off factor (ROF) of the pulse-shaping filter, and an internal parameter related to the in-band distortion. The iterative clipping noise cancellation (CNC) algorithm [21] for OFDM systems can be applied straightforwardly at the receiver side, where the number of antenna elements N is the counterpart of the number of subcarriers in OFDM systems. Unfortunately, however, simulations show that it performs poorly in a realistic situation [5] where N = 8 and 256 QAM is used as the modulation scheme. As in the OFDM case [23], the value of N has a significant effect on performance. It seems that, in this case, N is too small for the CNC to perform well. As a way to improve distortion recovery performance, we propose a learning-based algorithm that is derived by unfolding the CNC algorithm and by introducing learnable parameters.
The set of the learnable parameters contains the counterparts of the Bussgang coefficient that is used in the original CNC [21] and is computed analytically under the assumption that the distribution of the pre-clipped signal is Gaussian. Simulation results show that the CNC performs well with the parameters optimized through training rather than with a single Bussgang coefficient. The main contributions of our study are summarized as follows: • The statistical distribution is investigated for the instantaneous power of the UCA-based OAM signals. It is shown numerically that the peak power increases rapidly with increasing OAM modes N up to around 16 and then saturates. Also shown is the relationship between the peak power and the ROF of the pulse-shaping filter, which provides the optimal ROF in terms of minimizing peak power.
• A clipping method is presented for reducing the peak instantaneous power of the OAM signals, which neither causes bandwidth expansion nor has an impact on the orthogonality of OAM modes. The clipping processing is performed prior to the OAM beamforming processing and the pulse-shaping filtering.
• To compensate for the distortion caused by the clipping, a learning-based clipping noise cancellation algorithm is derived by unfolding the iterations of the conventional algorithm developed for OFDM systems into layers and by replacing the Bussgang coefficient with layer-wise learnable parameters. Simulations indicate that the proposed algorithm performs very well. The rest of this paper is organized as follows: In Section II, we briefly review the UCA-based OAM transmission model and investigate the statistical distribution of the instantaneous power of the OAM signals. Simulations show that the digital OAM beamforming process significantly increases peak instantaneous power. Section III presents a clipping method for the UCA-based OAM systems. Section IV describes the learning-based algorithm for mitigating the clipping noise distortion. Section V presents the simulated BER performance of UCA-based OAM transmission systems with the proposed clipping and distortion recovery algorithms. Section VI summarizes our results and concludes the paper.
Notation: We denote by R and C, respectively, the sets of real and complex numbers. Uppercase and lowercase boldface letters denote matrices and (column) vectors, respectively. The superscripts T , * , and † stand, respectively, for transposition, element-wise conjugation and Hermitian transpose; and and denote the real and imaginary part, respectively. Additionally, I denotes the identity matrix of an appropriate size, and • and ⊗ are the Hadamard and Kronecker products, respectively. For a vector x, diag(x) denotes the diagonal matrix with diagonal entries from x, and x 2

II. SYSTEM MODEL AND PEAK SIGNAL POWER A. OAM MULTIPLEXING TRANSMISSION SYSTEMS
We consider an OAM multiplexing transmission system equipped with UCA antennas at the Tx and Rx sides. For simplicity, we assume that the UCA consists of N antenna elements equidistantly arranged on a ring, and we refer it to as an N -UCA. Let s (k) [n] denote the n-th discrete-time signal transmitted from the k-th antenna element in the UCA and x (l) [n] denote the n-th QAM symbol transmitted through OAM mode-l channel. Then, s (k) [n] can be written as Eq. (1) can be written in matrix form as is upsampled and then filtered using a root raised cosine filter (RRCF) [24]. The signal sequence at each antenna element in the UCA is modulated on a single carrier frequency f c .
The free space transmission model (e.g., [11]) is used to simulate an UCA-based OAM channel, in which the transfer function between a pair of Tx and Rx antenna elements is given by where d is transmission distance, λ is wavelength, λ/(4π d) represents the free space loss, and G contains all relevant constants such as attenuation and phase rotation caused by antennas and their patterns on both sides. We denote by d (k,l) the distance between the k-th antenna element in the Rx N -UCA and the l-th antenna element in the Tx N -UCA. The N ×N channel matrix H UCA can be expressed as follows: Assuming the Tx and Rx UCAs are ideally aligned, H UCA can be regarded as a circulant matrix. Note that, in this case, ). The channel output signals received at the Rx N -UCA are filtered by RRCFs, down-sampled, and then post-processed by performing an N -point DFT. Consequently, the resulting post-processed signal y[n] = (y (0) [n], y (1) [n], . . . , y (N −1) [n]) can be written as where r (k) [n] denotes the n-th (downsampled) output signal of the k-th RRCF and

B. INSTANTANEOUS SIGNAL POWER OF UCA-BASED OAM SIGNALS
As explained above, the OAM signals are obtained by adding several independent signals via an IDFT, which may considerably increase the peak power of the transmitted signals.
In this section, we evaluate the complementary cumulative distribution function (CCDF) of the instantaneous power for the OAM signals and compare them with those for classical single carrier SISO signals. The instantaneous power considered here is normalized by its average power, i.e., |s (k) (τ )| 2 /P (k) , where for k = 0, 1, . . . , N −1, s (k) (τ ) denotes the output signal of the RRCF associated with the k-th Tx antenna, and P (k) denotes the average power. The CCDF of the normalized instantaneous power (NIP) is thus written as Shown for comparison are the simulated CCDFs of the NIP for conventional 256 QAM SISO signals. It can be seen that when ROF = 0.4, the NIP increases by more than 3.5 dB at a CCDF of 10 −5 by combining eight independent signals via the IDFT. Also, as the ROF increases from 0.1 to 0.4, the NIP of the OAM signal slightly increases while that of the SISO signal decreases by about 1.5 dB at a CCDF of 10 −5 . Fig. 2 shows the relationship between the NIP of the N -UCA-based OAM signals observed at a CCDF of 10 −5 and ROF, where 256 QAM was used as the modulation scheme and the number of antenna elements N in the UCA was varied from 1 to 64. When N = 1, the OAM signal can be thought of as a classical SISO signal. In this case, the NIP is minimized around an ROF of 0.4. In contrast, it can be seen that the NIP for N ≥ 8 increases monotonically as ROF increases from 0.1 to 1.0. When the ROF is set to 0.4 as in [5], the NIP increases by more than 3.5 dB by combining eight independent 256 QAM signals using IDFT. It is known that, due to the nonlinear characteristics of power amplifier (PA), a high peak normalized instantaneous power leads to signal distortion, which significantly reduces the power efficiency of the PA. Thus, peak instantaneous power reduction techniques become necessary in order to improve PA efficiency for OAM multiplexing.

III. DELIBERATELY CLIPPING FOR PEAK POWER REDUCTION
To reduce the peak instantaneous power of the OAM signals, we present a deliberate amplitude clipping method.
for k = 0, 1, . . . , JN − 1. Note that z[n] is the IDFT of a vector obtained by zero-padding x[n] to length JN and it can be regarded as a signal for OAM transmission with JN -UCAs. Each signal sample z (k) [n] is then clipped by the following soft-envelope limiter: where a tilde mark placed on top of a symbol represents a clipped version of that symbol. The amplitudes of the clipped samples are thus limited to a predetermined threshold value of A. In the following, we denote (8) as z (k) [n] = g(z (k) [n]) 1 In general, J does not need to be an integer as long as JN is an integer. for short. The clipping ratio γ of g is defined as It is customary to express the clipping ratio γ in units of dB as γ dB = 20 log 10 (γ where F JN ,N denotes an N ×JN matrix consisting of the first N rows of F JN and g(·) is applied in an element-wise manner. Fig. 3 is a block diagram of the proposed OAM multiplexing transmitter with the above clipping procedure for instantaneous peak power reduction. The original QAM signals are clipped via (10) and then pre-processed by N -IDFT, as described in (1) Fig. 4, the difference between the simulated CCDF curves of J = 1 and J = 2 becomes larger when ROF = 0.4, while there is no significant difference between the curves of J = 2 and J = 4. Fig. 5 shows the relationship between the clipping ratio γ dB and the NIP of the OAM signals observed at a CCDF of 10 −5 . Similarly to Fig. 4, there is no significant difference between the curves of J = 2 and J = 4. Note that when ROF = 0.4, the NIP for (J , γ dB ) = (1, 6.0) is almost the same as that for (J , γ dB ) = (2, 4.0). The difference in NIP between J = 1 and J = 2 at the same clipping ratio increases as the clipping ratio decreases. As we will show by simulations in Section V, the in-band distortion of the transmit signal for J = 1 is much larger than those for J = 2 and J = 4, while there is no significant difference between J = 2 and J = 4.

A. UNFOLDING THE CLIPPING NOISE CANCELLATION ALGORITHM
In this section, we present a method for estimating the originally transmitted QAM symbol vector x[n] from the post-processed received signal vector y[n]. For simplicity of notation, we will drop the index n in the following discussion. The received signal vector y can be written as where w = (w (0) , w (1) , . . . , w (N −1) ) is a complex white Gaussian noise vector. Let us consider the linear minimum-mean square error (LMMSE) estimate of z (k) given z (k) . Denoting the error in the LMMSE estimate by ε (k) , z (k) can be written as where can be considered not to depend on the antenna index k. In the following, we refer to it as the Bussgang coefficient and denote it by α. It is known that for sufficiently large N , the probability distribution of the magnitude of z (k) is well described by a Rayleigh distribution and that the following expression for α is obtained [17], [21], [25]: The clipped symbol vector x of (10) can then be rewritten as Since ε is a deterministic function of the originally transmitted QAM symbol vector x, we often denote it by ε(x), i.e., In accordance with (11) and (14), x can be estimated by solving the following optimization problem: where y ε = y − Hε(x) and Q denotes the set of the QAM constellation points. If the receiver can observe y ε (i.e., ε(x) is known at the receiver), the problem can be reduced to the OAM detection problem discussed in [7]. In a realistic scenario, however, x should be obtained even if the exact value of ε is not known at the receiver. The clipping noise cancellation (CNC) algorithm presented by Chen and Haimovich [21] for OFDM systems can be used to find a solution to the above problem. Here, the number of subcarriers in the OFDM system is a counterpart of the number of transmission modes (i.e., the number of antenna elements N ) in the OAM multiplexing system. In the CNC algorithm, the originally transmitted symbol vector x and the clipping noise vector ε are alternately and iteratively estimated. The CNC algorithm for the OAM multiplexing system can be described by the following iterations: where denotes the iteration number, Q (·) is the entry-wise projection onto the QAM constellation set Q, and D(H, ·) denotes a linear MIMO detector function. When the QAM constellation is square, Q (z) can be written as (x) + j (y) for z = x + jy, where for some nonnegative integer q, and sign(x) = +1 if x ≥ 0 and −1 otherwise. Furthermore, for simplicity, we use conventional zero-forcing (ZF) as the MIMO detector D(H, ·). In a practical system, the channel matrix H will be estimated using predetermined pilot symbols. Note that this is not affected by the clipping process when it is assumed that the pilot symbol power is the same level as the average data symbol power. If the Tx and Rx UCAs are ideally aligned, D(H, y − Hε ) can be simply written as Note that if the Tx and Rx UCAs are not ideally aligned, the OAM modes are not orthogonal to each other. In this case, (20) does not perform well, and as a result, an inter-mode interference cancellation function must be integrated into the MIMO detector, as discussed in [7]. At the first iteration of (17), x 0 is set to 0 as the initial condition, and thus, the transmitted signal vector is estimated as x 1 = Q ((αH) −1 y). Then at a subsequent iteration , the estimate ε +1 of the clipping noise vector is obtained from the given x by (17). The estimate x is then updated to x +1 by using ε +1 , as described in (18). Note that the iterations (17) and (18) contain the parameter α, which must be determined in advance. It was shown in [21], [23] that for the OFDM counterpart with a sufficiently large number of subcarriers, the clipping noise cancellation in [21] performs well with α determined by (13). For UCA-based OAM systems, however, a realistic number of transmission modes N seems to be not large enough for the iterations (17) and (18) to perform well with α of (13). Indeed, as we will show in Section V, when it is applied to a realistic system presented in [5], the clipping noise cancellation performs poorly with α given by (13), where the number of transmission modes is 8 and 256 QAM is used as the modulation format.
To optimize the unknown parameter and improve the clipping noise cancellation performance, we unfold the iterations (17) and (18) into multiple layers and introduce layer-wise parameters that can be trained using a stochastic gradient descent (SGD) method. Fig. 6 is a block diagram of the OAM receiver with the unfolded CNC, which consists of L layers each with the same structure. For = 0, 1, . . . , L − 1, the processing in the -th layer is as follows: where α is a learnable parameter, β and t are learnable parameter vectors, and ψ Q,t (·) is a parameterized counterpart of Q (·). Similarly to (19), for z = x + jy and a real vector t = (t i ), ψ Q,t (z) is given as ψ t (x) + jψ t (y), where and R(x) denotes the ReLU function. The following pseudocode summarizes the unfolded CNC algorithm.  4: x +1 ← ψ Q,t β •H −1 (y − Hε +1 ) 5: end for 6: x ← x L For each layer = 0, 1, . . . , L − 1 in the above algorithm, the learnable parameters are α ∈ R, β ∈ R N and VOLUME 9, 2021 t ∈ R 2q+1 , where we may set α 0 to 1 since x 0 = 0. The loss function we use for training these parameters is given by 1) ) is the originally transmitted symbol vector and x = ( x (0) , x (1) , . . . , x (N −1) ) is its estimate obtained from y by the unfolded CNC algorithm. Note that the total number of parameters to be learned in the above unfolded CNC algorithm is (2q + 2 + N )L − 1. The number can be reduced by imposing restrictions on the parameters, which may lead to a performance degradation while reducing the complexity of the training parameters. For example, when imposing the restriction β = (α −1 , α −1 , . . . , α −1 ) for 1 ≤ ≤ L − 1, the number is reduced to (2q + 2)L. Another restriction is ψ t (−x) = −ψ t (x), i.e., t −i = t i . In this case, the number of parameters to be learned is reduced to (q + N + 2)L − 1. Furthermore, by imposing t 0 = t i for all −q ≤ i ≤ q, the number becomes (N + 2)L − 1.
The computational complexity of each iteration in Algorithm 1 is the sum of the complexity of the DFT/IDFT in (21) and the MIMO detection in (22). The former complexity is O(JN 2 ), which can be reduced to O(JN log 2 (JN )) if JN is a power of 2. The latter complexity is O(N ) if the Tx and Rx UCAs are ideally aligned and otherwise O(N 2 ) as shown in [7]. The total complexity is thus at most O(L(J + 1)N 2 ).

B. OPTIMIZING PARAMETERS
We begin this section by introducing notation. For a given complex-valued matrix A, let A denote its real-valued counterpart defined by Similarly, for a given complex-valued column vector a, we denote by a its real-valued counterpart, i.e., Lines 3 and 4 of Algorithm 1 can be rewritten using this notation as follows: Furthermore, for a vector u = (u i ), ∂E/∂u denotes the gradient (column) vector of the loss function of (24) w. r. t. u, and for two vectors v = (v i ) and w = (w i ), ∂v/∂w denotes a matrix whose (i, j)-th entry is ∂v j /∂w i . Then, the gradient ∂E/∂α for = 0, 1, . . . , L − 1, can be computed as follows: where B = diag(β ) and u +1 = B H −1 (y − Hε +1 ). For u = (u i ) ∈ R 2N , ∂ψ t (u)/∂u becomes a diagonal matrix, whose diagonal entries are of the form, For the parameter vector β , we have Note that the (i, j)-th entry of ∂u +1 /∂β is given by where v +1,j is the j-th entry of H −1 (y − H · ε +1 ). For the parameter vector t , we have where for u j , t i ∈ R, ∂ψ t (u j )/∂t i is given as For z = x + jy, let g R (x, y) and g I (x, y) denote, respectively, the real and imaginary parts of the soft-envelope limiter function g(z). Let Dg(z) denote the Jacobian matrix of the map (g R , g I ) at (x, y), i.e., if |z| ≥ A and Dg(z) = I, otherwise. For a complex vector z = (z i ), we denote by diag(Dg(z)) a real block diagonal matrix whose diagonal entries are of the form Dg(z i ). Then, ∂E/∂x can be computed inductively for

V. SIMULATION RESULTS
This section provides simulation results that demonstrate the performance of the unfolded CNC algorithm. We consider as a typical example the UCA-based OAM transmission system in [5], [7]. In our simulations, unless stated otherwise, the number of antenna elements N was set to 8; the distance separating the Tx and Rx antennas was set to 40 m; the carrier frequency was set to 84.5 GHz; and the radius of the UCA was set to 0.265 m. These conditions were selected with reference to those of the field trial experiment reported in [5]. In addition, for simplicity, the channel matrix H was assumed to be known at the Rx side. In practice, this assumption may not be true, and the receiver needs to estimate H via pilot symbols [5]. Note that assuming that the pilot symbol power is the same level as the average data symbol power, the estimation of H will not be affected by the clipping process. Furthermore, equal power allocation over all transmission modes was assumed, and Gray-coded 2 2m QAM with constellation points Q = {x + jy | x, y ∈ {±1, ±3, . . . , ±(2 m − 1)}} was used for all modes. Note that, in this case, the number of entries of the parameter vector t in (23) is 2q + 1 = 2 m − 1. As noted in Section IV-A, for each = 0, 1, . . . , L − 1, we imposed a restriction on the parameter vector t = (t ,i ), i.e., t ,−i = t ,i , i = 1, 2, . . . , 2 m−1 −1. Accordingly, the total number of parameters to be learned was (2 m−1 +1+N )L −1, where the number of layers L was set to L = 2, 4, 8, 16. The Adam optimizer [27] with a learning rate of 10 −3 was used for training these parameters. The training and testing processes were implemented using TensorFlow [26] on an NVIDIA GeForce RTX 2080 with 8 GPU cores and 2 GPUs each with 8 GB RAM. The batch size and number of training epochs used for the training were set to 4,096 and 40,000, respectively. Fig. 7 shows the simulated BER performance of the unfolded CNC algorithm with the trained parameters set for L = 2, 4, 8, 16, where 256 QAM was used as a modulation scheme, the scale factor J for clipping processing was set to J = 2, and the clipping ratio γ was set to γ = 1.50 (left) and γ = 2.00 (right). In each epoch of the training process, the SNR per transmit symbol E s /N 0 was chosen randomly from a uniform distribution between 23 dB and 27 dB. For purposes of comparison, the figure also shows the simulated BER performance of the unclipped case and that of the conventional CNC algorithm described as (17) and (18) with the Bussgang coefficient α computed by using (13). We can see that a significant improvement in the BER of the unfolded VOLUME 9, 2021  CNC can be gained by increasing the number of layers to L = 16, while the BER of the conventional CNC saturates after iterations of L = 2 and suffers from a high error floor. When γ = 1.50, there is a gap between the BER performance of the unfolded CNC with L = 16 and that of the unclipped case, and the gap at a BER of 10 −3 is about 0.8 dB. When γ = 2.00, near optimal performance can be achieved by the unfolded CNC after a small number of iterations. At a BER of 10 −3 , there is negligible difference in performance between the unfolded CNC with L ≥ 4 and the unclipped case. Note that the conventional CNC suffers from a high error floor even when γ = 2.00. It can also be seen that in the low SNR region, the BERs of the CNC algorithms are slightly lower than that of the unclipped case. This is because the clipping of the OAM signals leads to a reduction in the transmit signal power.
Similarly, Fig. 8 shows the simulated BER performance of the unfolded CNC algorithm, where 64 QAM was used as a modulation scheme and the clipping ratio γ was set to γ = 1.26 (left) and γ = 1.50 (right). In each epoch of the training process, the SNR per transmit symbol E s /N 0 was chosen randomly from a uniform distribution between 17 dB and 21 dB. The other simulation conditions were the same as those of Fig. 7. It can be seen that a significant improvement in the BER of the unfolded CNC can be gained by increasing the number of layers while the conventional CNC suffers from a high error floor. When γ = 1.26, there is a gap between the BER performance of the unfolded CNC with L = 16 and that of the unclipped case, and the gap at a BER of 10 −3 is about 1.1 dB. When γ = 1.50, near optimal performance can be achieved by the unfolded CNC, and at a BER of 10 −3 , there is negligible difference in performance between the unfolded CNC with L ≥ 4 and the unclipped case. Fig. 9 plots the E s /N 0 required for the unfolded CNC algorithm to achieve a BER of 10 −3 versus the clipping ratio γ dB for L = 2, 4, 8, 16, where the modulation schemes used were 256 QAM (left) and 64 QAM (right). It can be seen from Fig. 9 (left) that for γ dB ≥ 3.5, the SNR loss relative to the unclipped case can be reduced to less than 1.0 dB after at most 16 iterations. Note that from Figs. 4 and 5, it is clear that applying the proposed clipping process with γ dB = 3.5 and J = 2 can reduce the NIP at a CCDF of 10 −5 by about 2.5 dB when ROF = 0.4. Similarly, it can be seen from Fig. 9 (right) that for γ dB ≥ 2.5, the SNR loss relative to the unclipped case can be reduced to less than 1.0 dB after at most 16 iterations. It can thus be concluded that the proposed clipping and distortion recovery schemes perform effectively with 256 QAM and 64 QAM signaling.
Next, we show the clipping noise cancellation performance of the proposed method for the cases of J = 1 and J = 4. Fig. 10 (left) plots the simulated BER performance of the unfolded CNC algorithm, where the transmit signals were clipped with J = 4; the clipping ratio was set to γ = 1.50; and the other simulation conditions were the same as those  of Fig. 7. For comparison purposes, Fig. 10 (left) also plots the performance for the case of J = 2 and γ = 1.50. We can see that the performance of J = 4 is superior than that of J = 2, especially when the number of layers L is small. Thus it seems better to set J to a large value if the computational complexity allows. Fig. 10 (right) plots the simulated BER performance for the case of (J , γ ) = (1, 2.00). The figure also plots the performance for the case of (J , γ ) = (2, 1.58) for comparison purposes. Note here that, when ROF = 0.4, the NIP of the transmitted OAM signals clipped with J = 1 and γ = 2.00 (γ dB = 6.0) is almost the same as that of the OAM signals clipped with J = 2 and γ = 1.58 (γ dB = 4.0). As can be seen from the figure, the unfolded CNC algorithm suffers from high error-floors when J = 1. The OAM signals clipped with J = 1 are not restored even using the unfolded CNC, while for the signals clipped with J = 2, they are successfully restored. The clipping process with J = 1 thus leads to a large amount of distortion compared with that with J = 2 even when the resulting NIPs are almost the same.
Next we show the simulated BER performance of N -UCA based 256 QAM OAM receiver with the proposed clipping method when the number of antenna elements N is 16, 32 and 64. In the simulation for the case of N = 16, the distance d between the Tx and Rx antennas, the carrier frequency f c , and the radius r of the UCA were set to (d, f c , r) = (40 m, 157 GHz, 0.271 m), and in the simulations for N = 32 and 64, they were set to (40 m, 157 GHz, 0.413 m) and (15 m, 157 GHz, 0.360 m), respectively. These conditions were selected so that they can maximize the channel capacity under practical constraints [7]. Fig. 11 shows the simulated BER performance of the unfolded and the conventional CNC algorithms for the cases of N = 16, 32, 64, where the clipping ratio and the scale factor J were set to γ = 1.50 and J = 2, respectively, and the number of layers was set to L = 8. Shown for comparison are the simulated performance for the case of N = 8 that were plotted in Fig. 7 (left). We can see that similarly to the case of N = 8, the proposed clipping and distortion recovery schemes can achieve good performance for N = 16, 32, 64.
Finally, while we assumed so far for simplicity that there is no inter-mode interference (IMI), we show by simulations that the proposed clipping and distortion recovery methods work well even when there is IMI due to antenna misalignment. As discussed in [7], when the Tx and Rx UCAs are not ideally aligned, the channel matrix H does not become diagonal anymore. For each OAM mode-k, the signal-tointerference ratio (SIR) due to the antenna misalignment can be evaluated by using the entries H (k,l) of the matrix H as SIR(k) = |H (k,k) | 2 / N −1 l=0,l =k |H (k,l) | 2 . We used the average of SIR(k) over k = 0, 1, . . . , N − 1 to measure the amount of IMI. Fig. 12 shows the simulated BER performance of the unfolded CNC algorithm, where instead of the ZF-based MIMO detection of (20), we employed an iteration of the learning-based OAM-MIMO detection algorithm presented in [7] in order to cancel the IMI. In the simulations of Fig. 12, the antenna misalignments were set randomly so that the resulting SIR becomes around 6 dB and 10 dB. The clipping ratio was set to γ = 1.78 (γ dB = 5.0) and 1.50 (γ dB = 3.5). The number of layers L was set to 16, and the other conditions were the same as those of Fig. 7. From Fig. 12 it can be seen that when γ = 1.78, the BER performances for the cases of SIR = 6 dB and 10 dB only slightly deteriorate compared with that of the IMI-free case (SIR = ∞). When γ = 1.50, while it can be seen that the slope of the BER curves for SIR = 6 dB and 10 dB decrease gradually as increasing the E s /N 0 , the required E s /N 0 to achieve a BER of 10 −3 deteriorates by only 0.5 dB even for SIR = 6 dB. It can be thus concluded that our clipping method works effectively, even in IMI environments.

VI. CONCLUSION
We investigated the statistical distribution of the instantaneous power of UCA-based OAM signals. It has been shown that the OAM signal at each antenna element exhibits high peak power and that a reduction in peak power is needed in order to improve power efficiency. To address this issue, we presented a clipping method and an iterative distortion recovery algorithm for UCA-based OAM systems. The algorithm can be regarded as an unfolded version of the clipping VOLUME 9, 2021 noise cancellation algorithm developed for OFDM systems, in which learnable parameters are introduced in place of the Bussgang coefficient used in the OFDM case.
We performed simulations for UCA-based OAM multiplexing systems with the proposed clipping and distortion recovery schemes. The simulation results showed that the combination of the proposed clipping and distortion recovery schemes provides a significant reduction in peak power of the OAM signals at the cost of only a slight degradation in BER performance. We also showed by simulations, the proposed clipping and distortion recovery methods work well in combination with the OAM-MIMO detection algorithm presented in [7] even when there is inter-mode interference due to antenna misalignment.