Neural Surface Maps

Maps are arguably one of the most fundamental concepts used to define and operate on manifold surfaces in differentiable geometry. Accordingly, in geometry processing, maps are ubiquitous and are used in many core applications, such as paramterization, shape analysis, remeshing, and deformation. Unfortunately, most computational representations of surface maps do not lend themselves to manipulation and optimization, usually entailing hard, discrete problems. While algorithms exist to solve these problems, they are problem-specific, and a general framework for surface maps is still in need. In this paper, we advocate considering neural networks as encoding surface maps. Since neural networks can be composed on one another and are differentiable, we show it is easy to use them to define surfaces via atlases, compose them for surface-to-surface mappings, and optimize differentiable objectives relating to them, such as any notion of distortion, in a trivial manner. In our experiments, we represent surfaces by generating a neural map that approximates a UV parameterization of a 3D model. Then, we compose this map with other neural maps which we optimize with respect to distortion measures. We show that our formulation enables trivial optimization of rather elusive mapping tasks, such as maps between a collection of surfaces.

Introduction
Maps are one of the most fundamental concepts in surface geometry: in differential geometry, a surface, i.e., a 2-manifold, is usually (locally) defined as the image of a (non-degenerate) map f : R 2 → R n .
Not surprisingly, maps are also used to define correspondences between different parts of surfaces in an atlas, to evaluate similarity between surface pairs, or across surface collections.
Accordingly, computing maps is central in most geometry processing tasks operating on surfaces. The ubiquitous f φ ψ h Figure 1. Two surfaces are respectively represented by two neural maps, φ and ψ, each mapping the unit square to 3D. A distortion-minimizing surface-to-surface map f is visualized by texture transfer. This surface map is achieved by a third neural map h between the square to itself, which yields the surface map by an implicit composition of the three neural maps. The distortion of the composed map f is trivially optimized by defining it as a loss in pytorch and optimizing with respect to h. Inset shows the initial random map of the hippo to the cow is a poor initialization, covering only a very tiny region of the cow model. Optimization results in a final map that covers the whole target surface, and reducing isometric distortion, resulting in an average Symmetric Dirichlet energy of 11 between these highly-nonisometric surfaces. As input, 4 keypoint constraints were used, each on one leg. concept of a UV map [54], mapping a surface into the plane, provides a local coordinate system on surfaces, and hence enables downstream tasks such as texturing, surface correspondence, remeshing, quad-meshing [11] to name only a few. Similarly, surface-to-surface maps [53] enable defining correspondences between surfaces, which are at the heart of shape analysis, transfer of properties, deformations, or defining morph sequences. Indeed, almost all shape processing tasks, including parameterization, surface correspondence, remeshing, and deep learning on surfaces, heavily rely on access to such surface maps.
However, many of the tasks related to maps and their computation become extremely hard to handle when the target domain is a surface, i.e., a 3D mesh (n = 3). This is due mostly to the fact that meshes are combinatorial representations, which in turn leads to a combinatorial representation of the surface maps, and taints the optimization task with a combinatorial nature as well. Although elegant solutions in the form of discrete differential geometry [50,52], meshing invariant spectral analysis [42,40], functional maps [45,35] have been proposed to work around the combinatorial representation, the diverse choices and different data representations inhibit easy end-to-end optimization and adaptation outside the specialized geometry processing community.
As an example, consider the problem of computing a mesh-to-mesh mapping in which a continuous map from one surface to the other is computed: one needs to account for the image of each source vertex, which lands on a triangle of the other mesh, and the image of a source edge may span several triangles of the target; this leads to extensive bookkeeping, and any attempt to optimize, e.g., the map's inter-surface distortion leads to combinatorial optimization of the choice of target triangle for each source vertex as in [53,31]. An alternative is to optimize proxy maps into a common base domain [4] in the hope that the resulting surface-to-surface map will be optimized by proxy. Such an approach, however, does not yield surface maps that are even a local minimizer of the energy they set to minimize. This is particularly problematic when optimizing inter-surface maps across shape collections.
In this work, we consider neural networks as a parametric representation of both individual surfaces as well as inter-surface maps. Specifically, we consider networks with parameters θ that receive 2D points as input and output points either in 2D or 3D, φ θ : R 2 → R n . While this definition is similar to, e.g., AtlasNet [25], we do not aim to perform any learning task, and our network does nothing more than map 2D points with the aim of performing one task: approximate a single surface map φ θ ∼ f : R 2 → R n , so we can work with neural networks instead of with, e.g., mappings of triangular meshes.
Specifically, we use a map φ θ ∼ f : R 2 → R 3 to directly characterize a given manifold shape (restricted to surface patches homeomorphic to a disc), and use another map ψ β ∼ g : R 2 → R 2 to update the surface map by restricting movements on the underlying 2-manifold. These neural networks are, by construction, differentiable and composable with one another, hence they lend us a simple model for defining a differentiable algebra of surface maps, enabling us to compose maps with one another and optimize objectives directly over their composition, rather than propose approximations via intermediate proxy domains.
We employ this concept in two ways that build on top of one another: first, we revisit the differential-geometry definition of a surface as a map from 2D to 3D, by overfitting a neural network to a given UV parameterization computed via a standard parameterization algorithm, such as Tutte's embedding [59] or SLIM [51]. Two such maps, φ, ψ, are shown in Figure 1. This gives us a parametric, differentiable representation of the surface, from a canonical domain. Second, we compose the overfitted map with other maps, either to optimize the distortion of the map, or to compute distortion-minimizing maps between two or more surfaces. Figure 1 shows an example of a distortionminimizing map f defined by composing h with φ, ψ.
We evaluate our method on a variety of triangular meshes with varying complexity and show their efficacy in computation of parameterizations, surface-to-surface distortionminimizing mapping, and also for mapping across collections of shapes. We also provide comparison to baseline methods. In summary, our main contribution is introducing neural surface map as a novel representation and utilizing it towards addressing a variety of geometry processing applications. We particularly stress the modular nature of the representation that enables harnessing the power of current deep learning frameworks to solve many (classical) shape analysis tasks in a uniform framework. Code available from the project page http://geometry.cs.ucl.ac. uk/projects/2021/neuralmaps/.

Surface maps
Mappings of surfaces (mostly, meshes) is an active research area. Usually, algorithms compute surface maps by striving for a specific type of map, such as a harmonic one [59,48,20,23,2]. In other cases the goal is to compute a map that minimizes or bounds some notion of distortion such as conformality -preservation of angles - [33,30,34], or isometry -preservation of local distances - [57,4,51,61]. Many surface mapping algorithms focus specifically on parameterizations, see [21,54] for a more detailed review. On the other hand, the task of computing mappings between two surfaces is a long-studied and notoriously hard problem. Our work uses the popular concept of a common base domain to which two surfaces are mapped, to define the surface map via the overlay of the two maps in the base domain, which can either be a coarse mesh [32,53,31,12] or a planar domain [5,61,6,3]. The resulting surface map can be optimized via direct mesh-mesh intersection at the price of yielding an optimization problem with a significant combinatorial part [53], due to the discreteness of the triangles. Otherwise, the properties of the resulting map are ignored at the hope that optimizing the maps into the common domain will be sufficient [4].
Soft notions of maps such as Functional Maps [46] enable a parametric definition of fuzzy maps which can be used in a deep learning context [35,19], however they define fuzzy correspondences and not a continuous surface map from one surface to another.
Cycle-consistency [44,29] is a trait of maps between a collection of surfaces, that ensures that the set of surfaces are in global correspondence by ensuring that mapping a point across random shapes until reaching the source shape maps the point back to itself. We present the first method to minimize the distortion of a cycle-consistent collection of maps.

Neural surface representation
Atlas-based representations have been prevalent in geometrical deep learning, mostly focusing on generative and analysis tasks (e.g. segmentation), but not on raw representation of surface mapping.
Many works consider UV-mapping a 3D surface, rendering surface functions as a 2D image, and applying a neural network to the images [55,41,10,27]. Representing surfaces as regularly-sampled image grids suffers from high distortions in the mapping, where large surface areas can be mapped to sub-pixel regions. Some works apply neural networks directly on meshes [28,37]. Generative techniques, such as FoldingNet [63] and AtlasNet [25] propose to alleviate this issue by using a neural network to approximate the atlas map from 2D to 3D as a continuous function conditioned on a latent shape code. Many extensions have been proposed including regularizing differential properties of the mapping [9], optimizing for elementary shape of the atlas [18], and forcing surfaces to align with the level set of shape's implicit function [49].
While our network architectures are inspired by these prior techniques, as we also train a neural module that maps 2D points to mesh surfaces, they have very different underlying objectives. The aforementioned methods aim at training a network to reconstruct shapes, conditioned on a latent code. Our work is distinct from them, in that it focuses on considering each neural network as a unique, single surface map, and using this representation to solve classic geometry processing problems in surface mapping.

Neural shape representation
In addition to surface-based representations many alternatives have been used, such as voxels [38,22,13,16], point clouds [1,58], meshes [15], and implicit functions [47]. From these techniques only the neural implicit representations do not suffer from discretization artifacts, since they use neural modules to represent continuous functions, mapping a point in 3D to an occupancy value. Similarly to surface-based methods they aim to create a shared latent space for all shapes, and various extensions have been proposed, such as enforcing unit gradient to satisfy the Eikonal equation [24], and sign-agnostic version of this normalization [8,7]. Littwin and Wolf in [36] introduce a meta-learning approach closely related to HyperNetworks [26] for implicit representations. A meta-model f regress weights θ g for the implicit function g, which describes the signed distance field for a specific shape. Although the approaches introduced above are very successful, they focus on generalizing over a plethora of models rather than represent a single one. Hence, they all present artefacts and thus can only be used as a rough proxy to the actual geometry and cannot be regarded as neural maps in a strict sense.
Davies et al. [17] overfit neural networks to implicit fields of individual shapes, as a compact representation for their geometry. Implicit fields have also been used for multi-view reconstruction, where Sitzmann et al. [56] optimize a neural network that represents a single shape, with the loss that favors this representation to be consistent with observed views of the object. Following the trends from [60,43], the proposed network learns to project the input into high-frequency features, hence learning in such domain. Although these techniques are similar to ours in that they use overfit networks to represent geometry, they focus on implicit surfaces and do not provide any mechanisms for inter-surface mapping. In contrast, we encode explicit surfaces via neural maps, and demonstrate that these maps can be composited and used for inter-surface mapping problems.
Lastly, a work related to our data generation method, [62] suggest overfitting an atlas to a point cloud as a method for surface reconstruction. They however focus on the task of inferring the surface topology via overfitting, while we simply estimate a given map which already inholds the topological data with a neural network, and instead focus on the differentiability and composability properties of it, for geometry representation and for optimizing other maps composed with the parameterization.

Method
We now define neural surface maps and how to compute and optimize them.

Neural Maps
We use the term neural surface map to refer to any neural network considered as a function φ : R 2 → R n , where the output dimension is 2 or 3. Indeed, this ensures the map's image is always a 3D surface, and, assuming the map is non-singular, also a 2-manifold.
Neural surface maps can be seen as an alternative method to represent a surface map that holds two main advantages: differentiability and ability to be composed with other neural maps. In short, this enables us to easily compose neural maps φ • ψ, and define an objective over the composition o (φ • ψ) which can be differentiated and optimized via standard (deep learning) libraries and optimizers without the need to write tailor-made code to handle new objective, work with combinatorial mesh representations, or deal with the notoriously-hard map composition problem.
Furthermore, we can choose any size, architecture and activation functions for our networks, and, thanks to the universal approximation theorem [14], know there always exists a network capable of approximating a given surface function.
We obtain and manipulate neural surface maps via two processes -overfitting and optimization, which we detail next.

Overfitting Neural Surface Maps
Let Ω ⊂ R 2 be the unit circle. All our neural maps will make use of Ω as a canonical domain. Given any map f : Ω → R n , we can approximate it via a neural surface map φ by using black-box methods to train the neural network and overfit it to replicate f . Namely, we optimize the least-square deviation of φ from f and the surface normal where n φp is the estimated normal at p, and n f p is the ground truth normal. In case f is indeed a continuous map, such as a piecewise-linear map mapping triangles to triangles, we can optimize this objective by approximating the integral in Monte-Carlo fashion by summing the integrand over a random set of sample points. Namely, to use neural surface maps to represent surfaces, we first compute a ground truth map f by overfiting to a UV parameterization of the mesh into 2D, computed via any bijective parameterization algorithm of our choosing -in this paper, we show results with SLIM [51], by which we achieve an injective map of the mesh into Ω ⊂ R 2 . We consider the inverse of this map, which maps Ω back into the 3D mesh S, as our input f : Ω → S, and overfit φ to it by minimizing Equation 1. Thus, we obtain a neural representation of the surface. More specifically, this is a mapping into the surface, endowed with specific UV coordinates, with point φ (x, y) having UV coordinates x, y. Fig. 2 shows several examples of such overfitted neural maps and their faithfulness to the original geometry. Our method can faithfully represent smooth shapes as well as those having sharp edges. Note that we assume that the objects are or have been cut open to be homeomorphic to a disc. Before progressing to discussing how can we compose maps and optimize them, we define the distortion measures we wish to optimize.

Surface Map Distortion
We wish to optimize several energies related to neural surface maps. Similarly to [9], for a neural map φ : Ω → R n , we denote by J p φ ∈ R n×2 the matrix of partial derivatives at point p ∈ Ω, called the Jacobian of φ. The Jacobian essentially quantifies the local deformation at a point. Letting M p = J T p J p , we subsequently can quantify the symmetric Dirichlet energy [51], where I is the identity matrix, added with a small constant , set to 0.01, to regularize the inverse. Likewise, we can define a measure of conformal distortion via We evaluate the integrals by random sampling of the function in the domain. Next, we show how to define surface-to-surface maps via various compositions of the maps and optimize their distortion, in the pairwise and in the shape collection setting.

Geometry-preserving optimization via composition
Our basic representation of 3D geometries is, as discussed above, via an overfitted neural surface map φ : Ω → R 3 that approximates a given map f . We now treat φ as our de-facto representation of the geometry. Our goal is to optimize various properties relating to the surface map, without affecting the geometry. However, optimization of the map is not trivial since it will immediately change our 3D geometry. We propose a solution to completely avoid this issue, next.
Assume we are given a neural surface map representing some surface φ : Ω → S; we wish to optimize the distortion D (φ) of the map. It is immediate to optimize φ itself with respect to our differentiable notion of distortion, however that will cause the map to change, and thus its image, the 3D surface, will change and could, for instance, flatten to the plane. To overcome this, we suggest introducing another neural surface map h : Ω → Ω. We can now define a new map, φ h = φ • h. As long as we solely optimize h and ensure it is onto Ω, we are guaranteed that the image of φ h is still the original image of φ, i.e., respects the original surface.
We can now optimize the distortion of φ h , by optimizing h and keeping φ fixed, thereby finding a map from Ω to S which is (at least a local) minimizer of the distortion measure of our choice: The distortion is a differentiable property of the map and hence is readily available, e.g., via automatic differentiation. In fact, composition, and minimization of distortion can be achieved in a mere few lines of code in Pytorch.
We can now consider composing more than two of these maps, to enable maps into more intricate domains.

Compositing Neural Maps
Map composition via common domains. One of the many advantages of our representation's composability is to enable representing maps between a pair of surfaces, using the classic method of a common domain, as depicted in Figure 1: we posses two overfitted neural maps, φ, ψ : Ω → R 3 , respectively representing two surfaces S, T , and we wish to define and optimize an inter-surface mapping between these two 3D surfaces, f : T → S.
To address the above problem, we define as before a map h : Ω → Ω and the composition ψ h = ψ • h. At first glance, it would seem that in this case, to map a point from T to S, we will need to consider the map ψ h • φ −1 , which includes an inverse of the entire map, that is of course not readily tractable.
However, we can define the map f via the following simple definition: for any point p ∈ Ω, f is implicitly defined as the map satisfying f • φ ψ h , or in simple words: for any point p ∈ Ω, f matches the image of p under φ h with the image of p, mapped through h and then through ψ (refer to Figure 1 for an illustration). This definition is known as the common domain definition of a map and has been used in many works [32,53,31,12,5,61,6,3]. It is easy to verify that this definition is identical to the one using the inverse, as long as the inverse exists, and can still provide a bijective map between the surfaces even in cases where it does not exist (cf., [61,4]).
Computing distortion in the common domain. Even though f itself is not tangible for optimization, as it is implicitly defined by h, luckily the only differential quantity we need from f to compute the distortion, is the Jacobian of f , denoted J q f at point q = φ (p). Using basic differential calculus arithmetic, J q f can be derived to be exactly which is composed of the Jacobian of ψ and the inverted Jacobian of φ at point p, both readily available. Hence to optimize the distortion of f , we can take (4), and plug it as the Jacobian used to define M in one of the distortion measures (2),(3), which we denote as D (f ).
Optimizing h for bijectivity.
In order for h to indeed be a well-define surface map, it needs to map exactly bijectively (i.e., 1-to-1 and onto) to the source domain of ψ, which is Ω. To ensure that, we only need to ensure that h has a positive-determinant Jacobian everywhere, and maps to the target boundary injectively. We optimize h to map the boundary onto itself, via the energy where σ is the squared signed distance function to the boundary of Ω. Note that the boundary map is free to slide along the boundary of Ω during optimization, enabling the boundary map to change. This is true for all points on the boundary, except those mapped to the four corners which are fixed to place and are essentially keypoint constraints between the two models. Further, we also optimize h to encourage its Jacobian's determinant to be positive, via Keypoint constraints. Lastly, in many cases, a sparse set of corresponding key points on the two surfaces are given, and it is required that the surface map f maps those points to one another. Given keypoints on S, we can, in a preprocess before optimization, find their preimages in Ω, to get a set of points P s.t. φ (P i ) maps to the ith keypoint. We likewise can find the preimages of the keypoints from T and their preimages Q under ψ. If these key points are required to be mapped to one another between the two surfaces by f , we can achieve that by requiring h (P i ) = Q i , which guarantees the induced f maps the points correctly. We optimize for that equality by reducing its least-squares error: To facilitate the optimization, we apply a rotation, R, to the input of h. R is pre-computed from the landmarks.
Optimization for surface-to-surface maps. To compute the surface map, we optimize the distortion of f with respect to h, while ensuring h respects the mapping constraints This yields a map h that maps onto the domain square, and represents a distortion-minimizing surface map f that maps the given sets of corresponding keypoints correctly, as shown for instance in Figure 1.
Cycle-consistent surface mapping. We also extend our method to discover inter-surface mapping among a collection of k surfaces S 1 , S 2 , ..., S k represented respectively via neural maps φ 1 , φ 2 , ..., φ k , we can define a cycle consistent [44,29] set of surface maps by considering k additional neural maps, h i : Ω → Ω, define the composition φ h i = φ i • h i , and then define the surface-to-surface maps This naturally allows extracting a set of mutually consistent maps while additionally optimizing for (all pairs) surface-to-surface maps, see Figure 6. Note that achieving similar qualities via classic methods is significantly challenging, and to the best of our knowledge, while previous methods could compute cycle consistency, none could optimize for true surface-surface distortion minimization over the entire collection.

Experiments
We use our neural-mapping representation in the context of various mapping problems, such as surface parameterization, inter-surface mapping, and mapping a collection of shapes. See the supplementary material for more examples.
Neural Mapping. For all surfaces shown in this paper, we render the reconstructions obtained with our neural mapping representation. Note how our overfitting procedure is able to capture even very detailed features of the original shape with a high fidelity. Figure 2 illustrates the difference between our reconstruction and the input mesh (highlighted in red). There are minor discrepancies between the models in regions like hairs of the bust and paws of the rhino. We observe that our reconstructions tend to be slightly smoother than the original shapes due to the use of softplus.
Surface Parameterization. The main advantage of neural mapping is not in representing the surfaces, but in representing the mapping. We now take the map φ : Ω → R 3 from Figure 2, and introduce another map h : Ω → R 2 , where we don't constrain its output domain. Similarly to the discussion in Subsection 3.5, we can define the map f from the 3D model implicitly via as f (φ (p)) = h (p) for all p ∈ Ω. We then minimize the isometry distortion of f (Eq. 2), using the method to extract the Jacobian discussed in Subsection 3.5. Note that this objective is different from the one that was used to produce φ, hence we undo the original parameterization's distortion by compositing the neural map with a newly optimized map in Figure 3. See the supplementary for more results.
In contrast to UV parameterizations of meshes, the complexity of our optimization for this composition is completely independent of the resolution of the geometry.
Surface-to-surface Maps. We can obtain a surface-tosurface map by compositing neural maps with a map between two atlases, as discussed in Subsection 3.5.
In Figure 4, we show the evolution of the map during optimization. Note how despite significant geometric differences between surfaces, the result is a bijective, low-distortion mapping. Please see more such maps in the supplementary.
Composition with Analytical Maps. Our method can optimize an inter-surface map f from φ, ψ just as well when ψ is not a neural map, but rather an analytical mapping defining some surface. Indeed, only h itself is required to be neural in our formulation of surface-to-surface maps. In Figure 5, we show mappings of Bimba into three such analytical surfaces. In this case, we optimize the conformal distortion (3) of f . Please refer to the supplementary for further visualizations.

Cycle-consistent Mapping for Collections of Surfaces.
Finally, we show that thanks to the compose-ability of neural surface maps, our method can be efficiently applied to cycle-consistent mapping problem for a collection of shapes. Furthermore, since we use a common domain, the maps are guaranteed to be cycle-consistent, as in [44,29]. We minimize the isometric distortion of the surface-tosurface maps between all pairs of surfaces in a collection of three models, following the method discussed in Subsection 3.5. Figure 6 illustrates that we were able to obtain cycle-consistent low distortion maps between all shapes in the collection. We used one keypoint on the nose and shoulders of each model to ensure correct alignment. See the supplementary for more collection-maps. Baseline comparison. To validate neural surface maps, we offer visual comparisons with the classic inter-surface method [53] and Mandad et al.

Implementation Details
In all our experiments, we use a neural network consisting of ten-layer residual fully-connected network, with 256 hidden units per layer, with a Softplus activation function. We use λ n = 0.01, λ B = 10 6 , λ inv = 10 2 , λ C = 10 3 in all experiments. We sample the initial mesh uniformly with 500k points. Since our goal is to fully-optimize the networks, they are trained until the gradient's norm drops below a threshold of 0.1. In all cases, we optimize the network with and RMSProp, and initialize the optimization procedure with a learning rate of 10 −4 and momentum 0.9, the step size is modulated with [39]. Similarly, maps used for surface mapping are four-layer fully-connected network of 128 hidden units, with Softplus. In general, overfitted networks converge in 3-7h based on the complexity of the model, while, surface-map and collection-map optimization take around 3h to reach a stable configuration.

Conclusions and Future Works
We introduced neural surface maps as a core representation for surfaces that is easily differentiable and composeable. Using the common domain approach, we can easily use these traits to optimize for different properties. Overfit to individual meshes allows encoding shapes as network weights, and subsequently optimize maps while keeping the surface approximation quality fixed. We demonstrated the universality of neural maps addressing a wide range of challenging classical tasks including parameterization, surface-to-surface distortion minimization, and extracting maps across a collection of shapes.
Our work has several limitations. For one, we only discussed representing disk-topology surfaces. Other topologies can be approached with cuts. Second, we relied on the assumption of h being bijective and mapping the keypoints correctly; in theory, we cannot guarantee that this requirement is upheld, however, in our experiments, it is rare for this condition to be violated.
We see many immediate uses to the differentiability and composability of our representation, such as applying differential geometry operators to the models as well as solving PDEs on them. Resorting to neural network generalization capabilities can bring large high-resolution dataset within our reach, exposing neural surface maps to applications like segmentation and classification. Figure A1. Surface to Surface maps. Red spheres depicts landmarks.

A. Supplementary material
A.1. Surface-to-surface Maps As described in the paper, we can represent and optimize surface-to-surface maps by composing two neural atlases φ and ψ with h that maps one atlas to the other.

A.2. Cycle-consistent Mapping for Collections of Surfaces
As discussed in the paper, thanks to our formulation it is possible to optimize all maps between a collection of shapes. As previously illustrated, a collection of neural maps is inherently cycle-consistent, and we can, for the first time, optimize the distortion of all maps in the collection. Fig. A2 depicts surface maps over a collection of heads (a) and a 4-legged models (b). In both cases we employ landmarks highlighted as red spheres.  Figure A2. Cycle-consistent maps over a collection. We minimize the isometric distortion of the maps between all pairs of shapes. Landmarks are visualized as red sphere.

A.3. Approximating Surfaces
As described in the main paper, a neural map ψ can approximate a given surface S by approximating the atlas f : R 2 → S accurately. To further demonstrate the effectiveness of our formulation, in Fig. A6 we show more overfitted models for categories such as Stanford Bunny (a), hand (b), heads (c to d), busts (e and f), animals (g and h), and human (i and j). Each neural surface map consists of a ten-layer residual fully-connected network with Softplus activation.
In Fig. A3 we compare different activation functions such as LeakyRelu and Relu. These non-smooth functions introduce artefacts, such as unwanted wrinkles Fig. A3 (b  and c). Overlooking this behavior might bear negative effects in map composition as these introduced details can be amplified or inject distortion in the final map. The use Softplus alleviates these artefacts, but biases the map towards smooth surfaces, hence the hairs in Fig. A6(d and f), will have minor discrepancies from the ground truth, as some areas will be smoothed out.

A.4. Surface Parametrization
We evaluate the efficacy of our method in optimizing different energies, by optimizing the distortion of the map φ • h as a function of h. We show a parameterization minimizing conformal-distortion in Fig. A4(b), reducing the conformal distortion from 1.74 to 1.52. Similarly, we show a parametrization minimizing isometric distortion in Fig. A4(c) reducing the median symmetric dirichlet from 11.58 to 9.92.

A.5. Composition with Analytical Maps
As discussed in the paper, we can directly compose a neural map with analytical functions to get surface maps into analytical surfaces. Fig. A5 illustrates several additional analytic surfaces we can map into. The neural map is optimized to minimize the conformal energy. Figure A4. Parametrization with different energies. Starting from a neural surface map (a), we optimize for conformal distortion (b) and isometric distortion boundary free (c). Figure A5. Surface mapping between David and a variety of parametric surfaces. Shape coloring is based on source model normals, i.e., David. Figure A6. Neural surface maps depicted are color-coded to highlight the error compared to the ground truth mesh. In general, the produced models have low deviation from the original surface.


Isadora: Automated Information Flow Property Generation for Hardware Designs

Isadora is a methodology for creating information flow specifications of hardware designs. The methodology combines information flow tracking and specification mining to produce a set of information flow properties that are suitable for use during the security validation process, and which support a better understanding of the security posture of the design. Isadora is fully automated; the user provides only the design under consideration and a testbench and need not supply a threat model nor security specifications. We evaluate Isadora on a RISC-V processor plus two designs related to SoC access control. Isadora generates security properties that align with those suggested by the Common Weakness Enumerations (CWEs), and in the case of the SoC designs, align with the properties written manually by security experts.

INTRODUCTION
Security validation is an important yet challenging part of the hardware design process. A strong validation provides assurance that the design is secure and trustworthy: it will not be vulnerable to attack once deployed, and it will reliably provide software and firmware with the advertised security features. A security validation engineer is tasked with defining the threat model, specifying the relevant security properties, detecting any violations of those properties, and assessing the consequences to system security.
Existing commercial design tools (e.g., Mentor Questa Secure Check, Cadence JasperGold Security Path Verification, and Tortuga Logic Radix) can verify security properties of a design, but the tools are only as strong as the provided properties. Defining these hardware security properties is a crucial part of the security validation process that currently involves a significant manual undertaking. We propose an automated methodology that combines information flow tracking with specification mining to create a human-readable information flow specification of a hardware design. The specification can be used as a set of security properties suitable for use with existing security validation tools, and it can also be studied directly by the designers to support their understanding of how information flows through the design.
Information Flow Tracking (IFT) is a powerful security verification technique that monitors how information moves through a hardware design. Recently, IFT has been demonstrated at the RTL [6,29] and gate level [7,30,41], and has been used to monitor implicit flows through digital side channels [5,8,36]. Existing verification engines that incorporate IFT capabilities can be used to confirm whether a given information flow property holds. However, it is up to the designer to specify the full set of desired flow behaviors.
The technique of specification mining offers an automatic alternative to manually writing properties. Specification mining can be applied to software [4] and hardware [24] and has recently been applied to system on a chip (SoC) designs [21,37]. Security specification mining focuses on developing the security goals of a design and has been developed for processors [17,18,45]. However, many important vulnerabilities violate security goals related to how information flows, goals that are not expressible as the trace properties that specification miners discover.
The insight that led to this research is that mining the traces produced by an IFT-instrumented design will generate trace properties that correspond to information flow properties over the original, uninstrumented design. The information flow tracking logic transforms the information-flow properties from the space of hyperproperties [12] -where trace-based mining does not apply -to the space of trace properties, where trace-based mining can apply.
A naive application of trace-based mining to an IFT-instrumented design quickly runs into issues of complexity: the instrumented designs are large and overwhelm the miner. Additionally, the miner will discover properties over tracking signals and original design signals that are meaningless and cannot be transformed back to the space of information flow properties in the original design. To handle these issues we separate the process of identifying source-sink flow pairs in the design from the process of mining for the conditions that govern those flows. The first can be done by leveraging existing information flow tracking tools and the second makes use of existing trace miners. The key to making the approach work is to synchronize the two parts using clock-cycle time.
The methodology we present here can inform an automated analysis of a hardware design by identifying flow relations between all design elements, including flow conditions and multi-source and multi-sink cases. The methodology requires no input from the designer beyond the design and testbench.
To evaluate our methodology, we developed Isadora, a fully automatic security specification miner for information flow properties. Isadora uses information flow tracking (IFT) technology from Tortuga Logic's Radix-S simulation based security verification engine [3] and is implemented on top of Daikon [20], a popular invariant miner.
To our knowledge, Isadora represents the first specification miner capable of extracting information flow security properties from hardware designs. Our results demonstrate: • Isadora characterizes the flow relations between all elements of a design. • Isadora identifies important information flow security properties of a design without guidance from the designer. • Isadora can be used to find undesirable flows of information in the design. • Isadora is applicable to SoCs and CPUs.
To measure our methodology and the usefulness of Isadora's mined specification, we evaluated Isadora over an access control module, a multi-controller and multi-peripheral system with a known security policy, and a RISC-V design. We evaluated the output of Isadora versus expected information flow policies of the design and found information flow specifications that, if followed, protect designs from known and potential future attack patterns.

PROPERTIES
Isadora generates two styles of information flow properties: no-flow properties, in which there is no flow of information between two design elements; and conditional-flow properties, in which there exists some flow of information between two design elements, but only when the design is in a certain state. Isadora can also generate unconditional-flow properties, but these tend to be less interesting for purposes of security validation.

Tracking Information Flow
IFT can precisely measure all digital information flows in the underlying hardware, including, for example, implicit flows through hardware-specific timing channels. Isadora uses IFT at the register transfer level [6] to track data flow between registers rather than considering individual bits, with 'registers' in this context referring to the Verilog notion of a register. Isadora may additionally be configured to consider Verilog wires, though doing so provided no observable improvements to generated specifications and considerably increased trace generation costs. The Isadora methodology can be applied to individual bits, as the underlying information flow tracking used within Isadora does consider individual bits.
However, bit level analysis would result in extraordinarily high trace generation costs for even small designs.
Tracking proceeds as follows: for each signal s in the design, a new tracking signal s is added along with the logic needed to track how information propagates through a design. Once the tracking signals and tracking logic are added to the design, one or more signals may be set as the information source by initializing their associated tracking signals to a nonzero value. All other tracking signals are initialized to zero. As the design executes, and information from a source signal propagates to a second signal, that second signal's tracking signal is updated from zero to nonzero.

Information Flow Restrictions
Using information flow tracking, we can express the property that information from register r 1 should never flow to register r 2 as a trace property: if r 1 is the only signal whose tracking signal r 1 is initialized to nonzero, then for all possible executions of the design, r 2 's tracking signal r 2 should remain at zero: This style of no-flow property can be useful for ensuring unprivileged users cannot influence sensitive state or for ensuring that sensitive information cannot leak through, for example, debug ports. However, it cannot capture conditional properties, for example that register updates are allowed only under certain power states.

Information Flow Conditions
Using information flow tracking, we can express the property that information from a register r 1 may flow to another register r 2 under some condition : if r 1 is the only signal whose tracking signal r 1 is initialized to nonzero then for all possible executions of the design, r 2 's tracking signal r 2 will only become non-zero if some predicate holds: This style of a conditional flow property can be used to express, for example, that register updates are allowable only under certain power states, or that memory accesses are allowable only when specific access control checks have succeeded.

Grammar of Properties
In order to produce properties that use only the signals in the original design, without including the tracking signals, we need an operator that expresses some notion of information flow. Both no-flow and conditional flow properties can be expressed using a no-flow operator, for which we use the notation =/=>. The grammar of Isadora properties is as follows.
The property r 1 =/=> r 2 states that no information flows from r 1 to r 2 . The property → r 1 =/=> r 2 states that information may flow from r 1 to r 2 only when ¬ . The symbol r is a register in the design, r ∈ { , , } means that r may take on any one of the values in a set of cardinality less than or equal to three, and prev(r) refers to the value of r in the previous clock cycle.

METHODOLOGY
Isadora analyzes a design in four phases: generating traces, identifying flows, mining for flow conditions, and postprocessing. An overview of the workflow is presented in Figure 1.
First, Isadora instruments the design with IFT logic and runs the instrumented design in simulation using the user-provided set of testbenches. The result is a trace set that specifies the value of every design signal and every tracking signal at each clock cycle during simulation.
Next, Isadora studies the trace set to find every flow that occurred during the simulation of the design. This set of flows is complete: if a flow occurred between any two signals, it will be included in this set. At the end of this phase, Isadora also produces the complete set of information flow restrictions: pairs of signals between which no information flow occurs.
Then, Isadora uses an inference engine (Daikon [20]) to infer, for every flow that occurred, the predicates that specify the conditions under which the flow occurred.
The final phase removes redundant and irrelevant predicates from the set and logically combines the predicates with the information flows to produce the conditional flow properties. These, along with the no-flow properties from the second phase, form the information flow specification produced by Isadora.

Generating Traces with Information Flow Tracking
To generate a trace set, the design is instrumented with IFT logic and then executed in simulation with a testbench or sequence of testbenches providing input values to the design. Let src = ⟨ 0 , 1 , . . . , ⟩ be the trace of a design instrumented to track how information flows from one signal, src, during execution of a testbench. The state of the design at time is defined by a list of triples describing the current value of every design signal and corresponding tracking signal in the instrumented design: In order to distinguish the source of a tainted sink signal, each input signal must have a separate taint label. However, tracking multiple labels is expensive [30]. Therefore, Isadora takes a compositional approach. For each source signal, IFT instrumentation is configured to track the flow of information from only a single input signal of the design, the src signal. This process is applied to each signal in a design. The end result is a set of traces for design D and testbench T: Each trace in this set describes how information can flow from a single input signal to the rest of the signals in the design. Taken together, this set of traces describes how information flows through the design during execution of the testbench T.

Identifying All Flows
In the second phase, the set of traces are analyzed to identify: (1) every pair of signals between which a flow occurs, and (2) the times within the trace at which each flow occurs.
Each trace src is searched to find every state in which a tracking signal goes from being set to 0 to being set to 1. In other words, every signal-values triple (s, , ) that is of the form (s, , 0) in state −1 and (s, , 1) in state is found and the time is noted. This is stored as the tuple (src, s, { 0 , 1 , . . .}), which indicates that information from src reached signal s at all times ∈ { 0 , 1 , . . .}. We call this the time-of-flow tuple. There can be multiple times-offlow within a single trace because the tracking value of signals may be reset to zero by design events such as resets.
Once all traces have been analyzed, the collected time-of-flow tuples (src, s, { 0 , 1 , . . .}) are organized by time. For any given set of times { 0 , 1 , . . .} there may be multiple discovered flows. For all traces src generated by a single testbench, the timing of flows from one source src can be compared to the timing of flows from a second source src ′ ; the value will refer to the same point in the testbench. At the end of this phase, the tool produces two outputs. The first is a list of the unique sets of times present within time-offlow tuples and all the corresponding register pairs for which flow is discovered at precisely the times in the set: The same src may flow to many sinks s ∈ {s 1 , s 2 , . . .} at the same times ∈ { 0 , 1 , . . .}, and the same sink s may receive information from multiple sources src ∈ {src 1 , src 2 , . . .} at the same times The second output from this phase is a list of source-sink pairs between which information never flows: The pairs in this set comprise the noninterference properties of the design, and can be specified using the no-flow operator, for example src =/=> s

Mining for Flow Conditions
In the third phase, Isadora finds the conditions under which a particular flow will occur. For example, if every time src flows to s, the register r has the value , Isadora infers the conditional information flow property: [20] on traces to infer design behavior using pre-defined patterns. In order to isolate the conditions for information flow between two registers, Isadora uses flows to find all the trace times at which information flows from src to s during execution of the testbench. The corresponding trace(es) are then decomposed to produce a set of trace slices that are two clock cycles in length, one for each time . Consider time-of-flow tuple (src, s, [ , , , . . .]), which as a notational convenience here uses distinct letters to denote time points rather than subscripts for clarity in the following expression. Given this tuple, Isadora will produce the trace slices ⟨ −1 , ⟩, ⟨ −1 , ⟩, ⟨ −1 , ⟩. These trace slices include only the signals of the original design, all tracking logic and shadow signals are pruned. Using trace slices, or trace windows of length two, allows dynamic invariant detection to generation predicates specifying design state both immediately prior to and concurrent with the occurence of some flow. Predicates match one of the four patterns for expressions given in the grammar of Isadora properties in Section 2.

Postprocessing
Finally, Isadora performs additional analysis to find invariants that may hold over the entire trace set by running the miner on the unsliced trace. Isadora eliminates any predicate that is also found to be a trace-set invariant. One such trivial example is the invariant The final output properties from postprocessing are the conditional flow properties. To ease readability, Isadora can express the conditional flow properties as multi-source to multi-sink flows, where all flows within the same property occur at the same time and under the same conditions. This produces comparatively few properties, which in practice were approximately as many as the number of unique source signals, and avoids redundant information. The conditional flow properties and the no-flow properties discovered in phase 2 (Section 3.2) make up the set of information flow properties produced by Isadora. Two examples of postprocessed properties are shown in Appendix A.

IMPLEMENTATION
Isadora uses the Tortuga Radix-S simulation-based security verification technology [3] to generate IFT logic for a hardware design, the Questa Advanced Simulator [2] to simulate the instrumented design and generate traces, and the Daikon [20] invariant miner to find flow conditions. A Python script manages the complete workflow and implements flow analysis and postprocessing.
Traces are generated for all signals within a design. An automated utility identifies every signal within a design and configures Tortuga Radix-S to build the IFT logic separately for each of these registers. We run Tortuga in exploration mode, which omits cone of influence analysis, and track flows to all design state using the $all_outputs variable. The resulting instrumented designs are simulated in QuestaSim over a testbench (see Evaluation, Sec. 5) to produce a trace of execution.
Phase two is implemented as a Python tool that reads in the traces generated by QuestaSim and produces the set of no-flow properties and the time-of-flow tuples. This phase combines the bit-level taint tracking by Radix-S into signal-level tracking. Each -bit signal in the original design is then tracked by a 1-bit shadow signal, which will be set to 1 at the first point in the trace that any of the component shadow bits where set.
The mining phase is built on top of the Daikon invariant generation tool [20], which was developed for use with software programs. Daikon generates invariants over state variables for each point in a program. We built a Daikon front-end in Python (411 LoC, including comments) that converts the trace data to be Daikon readable, treating the state of the design at each clock cycle as a point in a program. The front-end also removes any unused or redundant signals and outputs relevant two-clock-cycle slices as described in Sec. 3.3.

EVALUATION
We assess the following questions to evaluate Isadora: (1) Can Isadora independently mine security properties manually developed by hardware designers? (2) Can Isadora automatically generate properties describing Common Weakness Enumerations (CWEs) [1] over a design? (3) Does Isadora scale well for larger designs, such as CPUs or SoCs?

Designs
We assessed Isadora on two designs, the Access Control Wrapper (ACW) proposed within the AKER framework [39] and the PicoRV32 RISC-V CPU. An ACW wraps an AXI controller and enforces on it a local access control policy, which is setup and maintained by a trusted entity (e.g., a Hardware Root of Trust or a trusted processor). The ACW checks the validity of read and write requests issued by the wrapped AXI controller and rejects those that violate the configuration of the local access control policy. We used the AKER framework in two configurations: first implementing a single-controller AKER-based access control system; second, implementing a system with two traffic generators, each wrapped by an ACW, connected to three AXI peripherals though an AXI interconnect. This setup simulates the use of the ACWs in an SoC environment. In both cases, the input signals of the ACWs are dictated by the testbench, which initializes them with the access control policies and acts as the trusted entity. We refer to these two designs as the "Single ACW" and "Multi ACW" cases. They are shown in Figures 2 and 3, respectively.
PicoRV32 is a CPU core that implements the RISC-V RV32IMC Instruction Set, an open standard instruction set architecture based on established reduced instruction set computer principles.
The secure operation of the ACW and AKER-based access control systems has been verified through a property-based security validation process by the designers. We study the AKER framework to evaluate how Isadora's properties compare to a manually developed security specification. We use the PicoRV32 to evaluate how well Isadora automatically generates properties describing CWEs and to evaluate how well Isadora scales on a CPU design.

Time Cost
Isadora ran on a system with an Intel Core i5-6600k (3.5GHz) processor with 8 GB of RAM. Traces were generated on a Intel Xeon CPU  E5-2640 v3 @ 2.60GHz server. Trace generation dominated time costs, and scaled slightly worse than linear with number of unique signals in a design. Trace generation was suitable for parallelization though parallelization was not considered in the evaluation.
The design sizes are given in Table 1. For the Single ACW, trace generation took 9h33m. For the Multi ACW, trace generation exceeded 24 hours so we consider a reduced trace, which tracks sources for one of the ACWs, though all signals are included as sinks or in conditions. The reduced trace was generated in 6h48m. For PicoRV32, trace generation took 8h35m.

Theoretical Gains to Parallelization.
When parallelizing all trace generation and all case mining, Isadora could theoretically evaluate the Single ACW case fully in less than five minutes. Parallelizing the first phase requires a Radix-S and QuestaSim instance for each source register, and each trace is generated in approximately 100 seconds. Further, the trace generation time is dominated by write-to-disk, and performance engineering techniques could likely reduce it significantly, such as by changing trace encoding or piping directly to later phases. Parallelizing the second phase requires only a Python instance for each source register, and takes between 1 and 2 seconds per trace. Parallelizing the third phase requires a Daikon instance for each flow case, usually roughly the same number as unique sources, and takes between 10 and 30 seconds per flow case. The final phase, postprocessing, is also suitable for parallelization. Maximally parallelized, this gives a design-tospecification time of under four minutes for the single ACW and for similarly sized designs, including PicoRV32.

Designer Specified Security Properties
For the Single ACW we compared Isadora's output against security assertions developed by the AKER [39] designers using the Common Weakness Enumerations (CWE) database [1] as a guide. These assertions, the CWEs described, and the results of Isadora on the Single ACW are shown in Table 2. For each assertion Isadora mined either a property containing the assertion or found both a violation and the violating conditions for each assertion. We reported the observed violations to the designers who determined that the design remained secure but a conditional flow had been incorrectly specified as always illegal. Isadora also found the conditions for legality.
Only 9 Isadora properties, out of 303 total Isadora properties generated, were required to cover the designer-provided assertions, including conditions specifying violations. The Isadora output properties may contain many source or sink signals that flow concurrently and their corresponding conditions, whereas the designers' assertions each considered a single source and sink. For example, on the ACW nine distinct read channel registers always flow to a corresponding read channel output wire at the same time, so Isadora outputs a single property for this design state. This state included the reset signal and a configuration signal both set to non-zero values, which were captured as flow conditions, demonstrating correct design implementation. This single Isadora property captured 18 low level assertions related to multiple CWEs.

Case Study: Unintended Proxy.
In the Multi ACW case, we studied CWE 411: Unintended Proxy or Intermediary ('Confused Deputy'). The system contained two controllers (C), with two access control modules (ACW ), a trusted entity that configured each ACW (T ), and three peripherals (P). The ACWs each implemented an access control (AC) policy shown in Figure 3 and given as: Isadora discovered legal flows from the 2 write data to 3 read and write data, and 2 read data. Isadora also finds an illegal flow to 1 write data. The 2 -to-1 illegal flow has a flow condition specifying a prior flow from the relevant signals within 2 to 1 . While not constituting a precise path constraint, this captures an access control violation and suggests the confused deputy scenario because the flow profile from 2 is consistent with this path.

Automatic Property Generation
For the two designs with full trace sets, the Single ACW and Pi-coRV32, Isadora generates a specification describing all information flows and their conditions with hundreds of properties. To assess whether these properties are security properties, for each design we randomly selected 10 of the 303 or 153 total properties (using Python random.randint) and assessed their relevance to security.
We use CWEs as a metric to evaluate the security relevance of Isadora output properties. To do so, for each design, we first determine which CWEs apply to the design. For both the ACW and PicoRV32, we used the Radix Coverage for Hardware Common Weakness Enumeration (CWE) Guide [3] to provide a list of CWEs that specifically apply to hardware. We considered each documented CWE for both designs. CWEs, while design agnostic, may refer to design features not present in the Single ACW or PicoRV32 or may not refer to information flows. High level descriptions in multiple CWEs may correspond to the same low level behavior for a design and we consider these CWEs together.
Information flow hardware CWEs describe source signals, sink signals, and possibly conditions. CWEs provide high level descriptions, but Isadora targets an RTL definition. To apply these high level descriptions to RTL, we first group signals for a design by inspecting Verilog files and, if available, designer notes. With the groups established, we label every property by which group-togroup flows they contain. We also determine which source-sink flows could be described in CWEs, which often correspond to, or even match exactly, a signal group. We use these groups to find CWE-relevant, low-level signals as sources, sinks, and conditions in an Isadora property. We also use these groups to characterize the relative frequency of conditional flows between different groups, which we present as heatmaps in the following subsections.

ACW Conditional Information Flow.
Over the ACW we assess fourteen CWEs which we map to five plain-language descriptions of the design features, as shown in Table 3.

CWE(s) Description 1220
Read/write channel separation 1221-1259-1271 Correct initialization, reset, defaults 1258-1266-1270-1272 Access controls use operating modes 1274-1283 Anomaly registers log transactions 1280 Control checks precede access 1267-1269-1282 Configuration/user port separation  GLOB signals are clock, reset, and interrupt lines. S PORT represents the signals that the trusted entity T uses to configure the ACW. C PORT represents the signals which are used to configure the controller C to generate traffic for testing. M PORT carries traffic between the peripheral P and the ACW's control mechanism. CNFG represents the design elements which manage and store the configuration of the ACW. M INT carries the traffic between the ACW's control mechanism and the controller. If it is legal according to the ACW's configuration, the control mechanism will send M INT traffic to M PORT and vice versa. CTRL represents the design elements of the aforementioned control mechanism.
First consider the heatmap view of the Single ACW in Figure 4. In this view, all of the designer-provided assertions fall into just 3 of the 49 categories; these are outlined in red. Further, all of the violations were found with S PORT to CNFG flows, while all satisfied assertions were flows between M INT and M PORT. Another interesting result visible in the heatmap is the infrequent flows into S PORT, which is used by the trusted entity to program the ACW. Most of the design features should not be able to reprogram the access control policy, so finding no flows along these cases provides a visual representation of secure design implementation with respect to these features.
For the ACW, all ten sampled properties encode CWE-defined behavior to prevent common weaknesses, as shown in Table 4. In this table, the columns labeled by a CWE number and a '+' refer to all the CWEs given in a row of Table 3. Eight out of the ten properties provide separation between read and write channels, which constitutes the main functionality of the ACW module. CWEs 1267, 1269, and 1282 are not found within the conditional flow properties produced by Isadora as these are no-flow properties, so they are not present within the samples drawn from the numbered, conditional flow properties, but we were able to verify they are included in Isadora's set of no-flow properties.

PicoRV32
Conditional Information Flow. Over PicoRV32 we assess eighteen CWEs which we map to seven plain language descriptions of the design features, as shown in Table 5.
PicoRV32 had no designer-specified signal groups so we used comments in the code, register names, and code inspection to group all signals. We use lower case names to denote these groups were not defined by the designer.
• 'out' -Output registers • 'int' -Internal registers • 'mem' -Memory interface • 'ins' -Instruction registers • 'dec' -Decoder • 'dbg' -Debug signals and state • 'msm' -Main state machine The memory interface and the main state machine were indicated by comments in the code. The instruction registers, the decoder, and debug all appeared under one disproportionately large section described as the instruction decoder. Debug was grouped by name after manual analysis found registers in this region prefixed with 'dbg_', 'q_', or 'cached_' to interact with and only with one another. Instruction registers prefixed 'instr_' all operate similarly to each other and differently than the remaining decoder signals, which were placed in the main decoder group. Internal signals were the remaining unlabeled signals that appeared early within the design, such as program and cycle counters and interrupt signals, and the output registers were all signals declared as output registers.
First consider the heatmap view of PicoRV32 in Figure 5. An interesting result visible in the heatmap is the flow isolation from debug signals to the rest of the design. Many exploits, both known and anticipated, target debug information leakage, and the heatmap shows this entire class of weakness is absent from the design.
For PicoRV32 we find eight of ten sampled properties encode CWE defined behavior to prevent common weaknesses. We present these results in Table 6. The columns labeled by a CWE number and a '+' refer to all the CWEs given in a row of Table 5. The remaining two Isadora properties were single source or single sink properties representing a logical combination inside the decoder, and captured only functional correctness.

DISCUSSION
In this section, we discuss the threats to validity for properties produced using Isadora.
False positives may be introduced by insufficient trace coverage, by limitations of information flow tracking, or by incorrectly classifying functional properties as security properties. Sampling output properties found a 10% false positivity rate with respect to misclassification. This rate is discussed in greater detail in Section 6.2.
With regard to false negatives, they fall into two cases: known and unknown. Isadora captured all known assertions for the Single ACW (Section 5.3). For Single ACW and PicoRV32 properties (Section 5.4), the sampled properties partially addressed all CWEs  manually determined to be relevant, but no CWE was completely covered within the sampled properties. A manual inspection is needed to rule out the possibility of false negatives with respect to CWEs. Unknown false negatives can arise from limitations in trace coverage or in logical specificity.

Trace Reliance
As a specification miner, Isadora relies on traces. The second stage relies on traces with sufficient case coverage to drive information flow through all channels present in the design. The third stage relies on traces to infer flow predicates. Over buggy hardware, these predicates may form a specification describing buggy behavior. Traces may not cover all cases that can be reached by a design or even occur during normal design operation. Traces may not precisely describe some design features. For example, when considering property number 154 on the Single ACW, one of the sampled properties, Isadora found predicates that ARLEN_wire and AWLEN_wire are both set to be exactly 8 for any flow to occur. This property is shown in full in Appendix A.1.
The AxLEN_wire registers set transaction burst size for reads and writes. For transactions in write channels, the ARLEN_wire value should be irrelevant, and this clause within the broader property constitutes a likely false positive.
The AWLEN_wire is a different case. In a properly configured write channel supporting transcations, this register would necessarily be non-zero, and for wrapping bursts must be a power of two, but manual inspection of the code provides no indication the value must be precisely 8.
While Isadora is testbench reliant, testbench generation is an active area of research, and is more fully explored in related works such as Meng et al. [33], which studies concolic testing for RTL.

Functional Properties
When using CWE-relevance as the metric, Isadora does include functional properties in its output, as shown in Table 6. Sampling output properties found a 10% false positivity rate with respect to misclassification for the sampled properties from both designs, with 0 of 10 properties found to be false positives over the Single ACW version of AKER, and 2 of 10 properties found to be false positives over the PicoRV32 RISC-V CPU.
We attribute finding functional properties solely on RISC-V primarily to differences in design and testbench. The ACW studied was the target of validation efforts related to information flow, and the testbench we used was developed as part of those efforts. Further, as an access control module, by nature much of its functionality was relevant to secure access control.
With RISC-V, a minimal test bench was used that was intended only to run the design in an environment without access to the full RISC-V toolchain (such as our simulation environment for instrumented trace generation), and much of the design was devoted to behavior for which CWEs did not apply, such as logical updates during instruction decoding. One example of an Isadora property classified as functional is shown in Appendix A.2.

Measuring Interference
Isadora assumes the correctness of the information flow tracking used in trace generation. Information flow tracking is an active area of research, and is more fully explored in related works such as Ardeshiricham et al. [6], which studies IFT for RTL.

Specification Logic
Isadora does not define temporal properties beyond a single delay slot incorporated in the trace slices of length two. However, manual examination of output properties suggests information flow patterns during initialization, which is the first 4 cycles for AKER and first 80 for RISC-V, are highly dissimilar to later flows. During initialization, Isadora discovers flow conditions referencing registers with unknown states. Isadora also finds concurrent flows between elements for which no concurrent flows occur after reset. Because  conditions are inferred from comingled trace slices from during and after initialization, the output properties may be insufficiently precise to capture secure behavior related to this boundary.

RELATED WORK 7.1 Properties of Hardware Designs
Automatic extraction of security critical assertions from hardware designs enables assertion based verification without first manually defining properties [28]. IODINE looks for possible instances of known design patterns, such as one-hot encoding or mutual exclusion between signals. [25]. More recent papers use data mining of simulation traces to extract more detailed assertions [11,26] or temporal properties [31]. Recent work has focused on mining temporal properties from execution traces [13][14][15]31]. A combination of static and dynamic analysis extracts word-level properties [32].
The first security properties developed for hardware designs were manually crafted [9,10,27]. SCIFinder semi-automatically generates security-critical properties for a RISC processor design [45] and Astarte generates security-critical properties for x86 [19]. Recent hackathons have revealed the types of properties needed to find exploitable bugs for RISC SoCs [16].

Mining Specifications for Software
The seminal work in specification mining comes from the software domain [4] in which execution traces are examined to infer temporal specifications as regular expressions. Subsequent work used both static and dynamic traces to filter candidate specifications [42]. More recent work has tackled imperfect execution traces [44], and the complexity of the search space [22,23,38]. Daikon, which produces invariants rather than temporal properties, learns properties that express desired semantics of a program [20].
In the software domain a number of papers have developed security specific specification mining tools. These tools use human specified rules [40], observe instances of deviant behavior [20,34,35], or identify instances of known bugs [43].

CONCLUSION
We presented and implemented a methodology for creating information flow specifications of hardware designs. By combining information flow tracking and specification mining, we are able to produce information flow properties of a design without prior knowledge of security agreements or specifications. We show our implementation, Isadora, characterizes the flow relations between all elements of a design and identifies important information flow security properties of an SoC and a CPU according to Common Weakness Enumerations.  To consider the output properties of Isadora, Figures 6 shows an example of Isadora output, Case 154 of the 303 output properties over the ACW module. This a case that was sampled during evaluation. Here the condition predicates shown are register equality testing versus zero. Other predicates are captured within the workflow but not propagated to individual properties formatted for output.

A SAMPLE PROPERTIES
A visible difference between an Isadora output property and the property grammar of Section 2 is that at output stage Isadora properties may specify multiple source registers, may consider multiple sink registers though do not do so in this case, and may contain multiple invariants as conditions. Case 154 includes an example of a flow condition between internal and peripheral visible signals in addition to specifying other aspects of design behavior. This is similar to the example of write readiness from Section 2, but in Case 154, the flow is from the internal signal to the peripheral, though the power state predicate is identical. Of note, as in the case of write readiness, this flow occurs exclusively within the write channel, as denoted by the "W" present in ready wire and the data register. AWREADY_int =/=> WDATA unless (ARESETN ≠ 0) A.1.1 Security Relevance. Under the working definition of security properties for Isadora, where internal signals and peripheral signals should not flow to one another unless ACW is not undergoing a reset, this single source, single sink, single invariant description of behavior composed from an Isadora output property establishes Case 154 as a security property under the working definition. Case 154 describes signals marked as sensitive by designers, both labeled as such within the design using comments and present within security properties they specified, and differs from a designer provided property only in the specific pairing of registers.

A.2 Case 144: ACW Functional Property
One example of an Isadora property classified as functional, with truncated flow conditions, is presented in Figure 7, and captures a logical update to an internal decoder signal. This additional shows an example of a property over multiple sinks, a single source, and for which there are predicates capturing both equality and inequality to zero.


A Functional Clipping Approach for Policy Optimization Algorithms

Proximal policy optimization (PPO) has yielded state-of-the-art results in policy search, a subfield of reinforcement learning, with one of its key points being the use of a surrogate objective function to restrict the step size at each policy update. Although such restriction is helpful, the algorithm still suffers from performance instability and optimization inefficiency from the sudden flattening of the curve. To address this issue we present a novel functional clipping policy optimization algorithm, named Proximal Policy Optimization Smoothed Algorithm (PPOS), and its critical improvement is the use of a functional clipping method instead of a flat clipping method. We compare our approach with PPO and PPORB, which adopts a rollback clipping method, and prove that our approach can conduct more accurate updates than other PPO methods. We show that it outperforms the latest PPO variants on both performance and stability in challenging continuous control tasks. Moreover, we provide an instructive guideline for tuning the main hyperparameter in our algorithm.

Reinforcement learning, especially deep model-free reinforcement learning, has achieved great progress in recent years. The application of reinforcement learning ranges from video games [9], [14], board games [21] to robotics [12], [13], and challenging control tasks [5], [19]. Within these approaches, Policy gradient (PG) methods are widely used in model-free policy search algorithms, contributing significantly to the fields of robotics and high-dimensional control. PG methods update the policy with a step estimator of the gradient of the expected return and gradually converge to the optimal policy [16]. One of the most significant problems of PG-based methods is to determine the suitable step size for the policy update while an improper step size will either prolong the policy searching process or even degrade the policy, since the sampling data highly depends on the current policy. As a result, the trade-off between learning speed and robustness is a challenging problem for PG methods.
The associate editor coordinating the review of this manuscript and approving it for publication was Shagufta Henna .
The trust region policy optimization (TRPO) method [18] proposes an objective function (also called the ''surrogate'' objective) and aims to maximize the objective function while subjecting to a constraint on the size of the policy update by KL divergence. However, the subjecting item is computationally inefficient and difficult to scale up for high-dimensional problems when extending to complex network architectures. To address this problem, the proximal policy Optimization (PPO), which adopts a clipping mechanism on the probability of likelihood ratio, was introduced [20]. This clipping mechanism significantly reduces the complexity of the optimizing process. To deal with learning robustness PPO tries to remove the incentive for pushing the policy away from the old one when the ratio is out of a clipping range, being more efficient for both implementation and computation than the trust region-based TRPO.
Despite the success of PPO it also has some flaws in restricting the update step size, which means that PPO cannot truly bind the likelihood ratio within the clipping range [25]. Aiming towards a better step size update, the proximal policy optimization with rollback (PPORB) applied a rollback operation on the clipping function to inhibit the policy from being pushed away during training [26]. PPORB adopts a straight downward-slope function instead of the original flat function when the ratio is out of the clipping range, which hinders the natural incentive from PPO to seek a large policy update. However, this solution introduces new problems, such as when the ratio is extremely big and the clipped ratio will shoot to infinity or negative infinity, and thus contrast with its original aim. Additionally, the inner workings of PPORB's hyperparameter α, the slope of the rollback function, is still abstruse and poorly understood because PPORB only provides the empirical values for specific experiments instead of a tuning method or function.
Inspired by the insights above, we propose a novel PPO clipping method, named Proximal Policy Optimization Smoothed algorithm (PPOS) which combines the strengths of both PPO and PPORB. We apply a functional clipping mechanism to prevent large policy updates while keeping the convergence of the clipped ratio. We compare PPOS with PPO and PPORB in high-dimensional control environments. We also analyze the hyperparameter here introduced in relation to the dimensions of five problems and provide a useful guideline for readers to orient their hyperparametric choices according to the dimension of their problems.

II. PRELIMINARIES
This section discusses the background information needed to understand our policy search approach. We first describe the reinforcement learning (RL) framework, which is based on the Markov decision process (MDP). Then, we give an overview of policy search methods. We continue by introducing trust-region methods and actor-critic methods. Finally, we delve into the Proximal Policy Optimization (PPO) algorithm, which is currently one of the state-of-the-art algorithms in policy search.
A RL procedure consists of an agent and an environment, where the agent can interact with the environment and gain rewards during this interaction. We consider RL as a process running by a Markov decision process (MDP). An MDP is defined by the tuple (S, A, p, ρ 0 , r, γ ), where S ⊆ R d s is a finite set of states of the agent and A ⊆ R d A is a finite set of actions which can be conducted by the agent. p (s t+1 | s t , a t ) denotes the probability of transfer from state s t to s t+1 given the action conducted by the agent at time step t. p (s t+1 | s t , a t ) is usually unknown in high-dimensional control tasks, but we can sample from p (s t+1 | s t , a t ) in simulation or from a physical system in the real world. ρ 0 denotes the initial state distribution of the agent, and r(s t , a t ) denotes the valued reward in state s t when the agent executes action a t . The last γ ∈ (0, 1) is the discount factor. A policy in RL is defined as the map (or function) from the state space to the action space. The goal of RL is to find a stationary policy π that maximizes the long-term expected reward [18], where Q π (s t , a t ) = E s t+1 ,a t+1 ,...
in which ρ π (s) is the state distribution under π, i.e., the probability of visiting state s under π. Q π (s t , a t ) is called state-action value function which evaluate the state-action pair. s t+1 ∼ p (s t+1 | s t , a t ), a t ∼ π (a t | s t ) and T is the terminal time step. We could also compute the value function V π (s t ) = E a t ,s t+1 ,... T t=0 γ t r (s t , a t ) which is the evaluation of the state s t , and advantage function A π (s t , a t ) = Q π (s t , a t ) − V π (s t ) which indicates how reasonable the chosen action a t is given the state s t .
In policy search, the goal is to find a policy π that maximized Eq.1 In recent work, the policy usually adopts the neural network structure (e.g. [14], [15] and [24]) which simulates the network of neuron of human-being to some extent. A policy can be improved by update the policy by the following surrogate performance objective which is called policy gradients methods [22], where r s,a (π) = π(a | s)/π old (a | s) is the likelihood ratio between the new policy π and the old policy π old , A π old (s, a) is the advantage function of the old policy π old .

A. TRUST REGION POLICY OPTIMIZATION
The common way of conducting policy updates is using the samples from the old policy π old to improve the current policy π at each iteration. However, the update may not be valid for the new policy in practice because the estimates for ρ π (s) and Q π (s, a) are based on the old policy which could result in some extremely large updates. To address this problem, Trust-Region Optimization (TRO) methods are used for keeping the new policy not far away from the old policy, which was first introduced in the relative entropy policy search (REPS) algorithm [17]. Many variants of TRO methods were introduced after the REPS algorithm (e.g. [1], [2], [3]). Most of these TRO methods use a bound of the KL-divergence of the policy update which prevents the policy update from being unstable while keeping the update suitable. Trust region policy optimization (TRPO) [18] adopts this bound to optimize the policy. The surrogate objective of TRPO is maximized while being subjected to a constraint on the KL-divergence between the old and current policy, where δ is a constant hyper-parameter and ''.'' means any action a. In general, when δ is small enough, the update is VOLUME 9, 2021 valid since the difference between the new policy and the old policy is not substantial.

B. PROXIMAL POLICY OPTIMIZATION
TRPO has been theoretically instantiated as capable of ensuring monotonic performance improvements of the policy. Nonetheless, due to the KL-divergence constraint TRPO is computationally inefficient and difficult to scale up for high-dimensional problems. In order to simplify the implementation and computational requirements, the PPO algorithm [20] was introduced. PPO restricts the policy update by a clipping function F on the ratio r s,a (π ) directly, In PPO, F is defined as, (7) where 1 − and 1 + are called the lower and upper clipping range, respectively, and ∈ (0, 1) is a hyper parameter. The minimization operation in Eq. (6) is to keep the gradient when the r s,a is out of the clipping range. The illustration of the clipping mechanism as the red dash line can be seem in Fig.1.
Both TRPO and PPO adopt the actor-critic method [10], in which the Value function Q π (s, a) is estimated by a parameterized function approximator Q(s, a; ω), where the vector ω is the parameter of the approximator e.g. a linear model or usually a neural network. This estimator is called the critic which can have a lower variance than traditional Monte Carlo estimators [27]. The critic's estimate is used to optimize the policy π(a | s; θ) which is also called the actor parameterized by θ. The actor is usually represented by a neural network. After that, the policy parameters can be updated by the gradient of surrogate objective Eq. (6) as, where β is the step size and ∇L CLIP (θ t ) denote the gradient of L CLIP at θ t .

C. PPO WITH ROLLBACK
PPO has made progress in different control tasks, especially with high-dimensional observation and action space, but it also has raised concerns of researchers about whether this clipping mechanism can truly restrict the policy ( [8], [25]).
In reality, PPO is incapable of binding the likelihood ratio within the clipping range because its clipping mechanism still pushes the ratio out of its range, rendering the policy out of the bounds.
To address this problem, PPORB [26] proposed a rollback operation on the clipping function attempting to remove the incentive of moving far away, which is defined as where α is a hyperparameter to decide the force of the rollback operation. When the ratio r s,a is out of the clipping range, it will make negative effects on the output instead of keeping the value as a constant. PPORB has made vast progress from the original PPO algorithm, but at the same time, it leaves space for improvement. Firstly, the rollback operation cannot converge when the ratio is extremely large: In high dimensional tasks, it will make the policy search sway around the optimal policy. Secondly, the rollback operation introduces an extra hyperparameter which can be only tuned by experience of the same problem. However, the hyperparametric possibilities result in significantly different results, with the values for this parameter ranging from 0.02 to 0.3.

III. METHOD
In this section, we first describe our reinforcement learning method, and then compare it with other methods of proximal policy optimization algorithms. Finally, we discuss the stability and strength of our method in this paper.
As discussed in the section Preliminaries II-C, while the rollback operation can, to some extent, restrict the likelihood ratio, it also leaves unsolved problems: the convergence property and the choice of hyperparameters. We deploy our method on the structure of PPO with a general form for sample (s, a), L s,a (π) = min r s,a (π)A π old (s, a), F(r s,a (π), ·)A π old (s, a) where F is a clipping function which attempts to restrict the policy update and ''.'' means any hyperparameters of F. In PPO, F is a ratio-based clipping function F PPO as Eq. (7), which will be flat whenever out of the clipping range.
In PPORB, F PPORB has the rollback operation as Eq. (9) instead of keeping it flat. We modify this function to promote the performance and stability of restricting the likelihood ratio by a functional clipping method, which is defined as where α is a hyperparameter to decide the scale of the functional clipping. The decreasing slope will be more steep when α is larger. Combining the original PPO algorithm with the novel function clipping method, we obtain the Proximal Policy Optimization Smoothed algorithm as shown in Fig. 1 which plots L CLIP (π) as functions of the likelihood ratio r s,a (π). We utilize the gradient change of hyperbolic tangent activation function to obtain the smoothed effect. As the figure depicted, when r s,a (π ) is out of the clipping range, the slope of L PPOS (π ) starts with a large absolute value and generally converges to zero. This smoothing mechanism guarantees the convergence of the clipping function. In contrast to PPORB, PPOS will never have an extremely large clipping output which is unacceptable in policy search. When α < 0, the function is more natural and smoother in shape. However, the actual performance will not be better than α > 0 which can be viewed in section IV-C. At the same time, the hyperbolic tangent activation function could be replaced by other activation functions to explore more.

IV. EXPERIMENTS
In this section, we present experiments to assess the performance of our proposed algorithm in terms of performance and stability during policy searches. We compare our method with the aforementioned state-of-the-art PPO variants in different environments. We also take different hyperparametric choices into consideration and their corresponding results are shown, along with a quick guideline with tuning suggestions to quickly deploy our algorithm on different applications.

A. EXPERIMENTAL SETUP
We test the performance of our algorithm with the following benchmark: (a) PPO: the original PPO algorithm with the author recommended clipping range hyperparameter = 0.2, as seen in [20]. (b) PPORB: PPO with the rollback function. The rollback coefficient α is set to be 0.3 for all tasks (except for the Humanoid-v2 task) as suggested by the authors in [26] (c) PPOS: Our proposed method with an α coefficient set according to the Table 2.
The algorithms are evaluated on continuous control benchmark tasks implemented on OpenAI Gym [4], simulated by MuJoCo [23] in a Tianshou platform. 1 The chosen MuJoCo environment is the version-2 (v2) for all experiments. The actor and critic networks are neural networks with one hidden layer and 128 neurons per layer, and the adopted activation function is ReLU function [6]. Each algorithm was run with ten random seeds and 250,000 timesteps, which are obtained by 2500 steps per epoch and 100 epochs (except for the Humanoid-v2 task with 1 million timesteps, due to its higher dimensionality) using ADAM optimizer [11]. The details can be viewed on the Table 3 in the Appendix.

B. COMPARISON WITH OTHER METHODS
We conducted experiments in five Mujoco environments with continuous control tasks of both high and low dimensions, as shown in Fig. 2. In Fig. 3 we plot episode rewards during the training process on the tasks. Our PPOS method with the functional clipping mechanism significantly outperforms both PPO and PPORB on hard tasks characterized by high observation and action dimension (e.g., Humanoid-v2 with |O| = 376 and |A| = 17 and Ant-v2 with |O| = 111 and |A| = 8), both in terms of learning speed and final FIGURE 3. Episode rewards of the policy during the training process averaged over ten random seeds on MuJoCo tasks, comparing our methods with PPO and PPORB. The solid lines are mean values of ten trials while the shaded area depicts the 60% confidence interval. |O| represents the dimension of observation.  rewards. For the HalfCheetah-v2 with medium dimensionality (|O| = 17 and |A| = 6) our method achieves a better final reward in the later stage. The advantage given by the clipping mechanism is less salient in lower dimensionality, and our PPOS is comparable to both PPO and PPORB in such tasks (e.g., Swimmer-v2 with |O| = 8 and |A| = 2 and Reacher-v2 with |O| = 11 and |A| = 2). The stricter clipping mechanism from both PPORB and PPOS have a tendency to slow down these algorithms on their initial stage (especially in low dimensions, while making it faster in higher dimensionalities), with smaller policy updates, but eventually it enables these algorithms to asymptote at higher rewards.
The numerical analysis with mean and standard deviation is presented in Table 1. PPOS reaches a better performance in 80% of the adopted Mujoco tasks, and, as seen on Table 1, it was virtually tied with PPORB on the low dimensional Swimmer-v2 task (and presented a lower standard deviation). On average, PPOS reaches 8% higher rewards than PPO and 7% higher than PPORB, while the standard deviation of PPOS is 3% lower than both PPO and PPORB. A possible explanation for this is that the functional clipping mechanism successfully keeps the policy update within a reasonable range, while PPO and PPORB made the update either too large or too small. Besides, the nature of PPORB's clipping can generate very negative updates, which can be detrimental to the policy search.
We also compared three methods on entropy during all training procedures as shown in Fig. 4. In general, PPOS and PPORB have lower entropy than PPO in most of the tasks which means PPOS and PPORB can make the trained policy more deterministic. In Humanoid-v2, PPOS can even have lower entropy than PPORB which means the policy is more practical for implements on real robots.

C. CHOICE OF THE HYPERPARAMETER
Correct hyperparametric choices are crucial for policy search algorithms. For example, in PPORB the coefficient of the rollback operation α could vary from 0.3 to 0.05 for  different tasks. However, the suggested value was only provided by empirical methods for specific tasks in most of the former algorithms e.g. [18], [20], and [26]. With this in mind, we evaluate our choices of hyperparameters on varied tasks and provide a selection criterion of the hyperparameter value for different problem dimensionalities.
We conduct the experiment on Humanoid-v2 and Ant-v2 tasks, which have a higher dimensionality in their observation space, since in low-dimensional tasks the performance difference between PPO, PPORB and PPOS can be very subtle. As shown in Fig. 5, in the Humanoid-v2 task the coefficient α = 0.05 achieves the best performance among different values, while α = 0.2 is the best choice in the Ant-v2 task. As discussed in section III, the negative α, which enables a smoother clipping function, is incapable of improving the performance of the Ant-v2. A possible solution for this issue is decreasing the clipping range towards a lower value. For the other three tasks with relatively low dimensions, the default α = 0.3 can be regarded as the benchmark, which is presented in Table 2.
Once we empirically found the optimal parameters for all five environments, we use an exponential regression function α = 0.3333e −0.0048 * |O| to fit α value regarding the dimension of observation as shown in Fig. 6. In general, the proposed α is lower when the dimension of observation is higher. Our explanation for this is that a lower α value is effective in preventing unnecessarily large policy updates which are more likely to take place in high-dimensions. For low-dimensional tasks, default α = 0.3 can balance the performance and robustness. The exponential regression function has two key advantages compared with linear and polynomial regressions: Firstly, it converges to 0 as α grows with an elegant shape which meets the intuition that higher dimensions will result in lower α values. Secondly, the function is more efficient to express the same data than higher-order polynomial expressions.

V. CONCLUSION
Although PPO group algorithms have achieved state-ofthe-art performance on policy searches on high-dimensional control tasks, they also present flaws when restricting the step of each policy update. The original clipping method of PPO cannot prevent large policy updates, while the improved PPORB cannot guarantee the convergence of this very same update. Additionally, as hyperparameters were proposed according to specific tasks, users are left without guidance to retune those parameters when redeploying the algorithm to their needs. Based on these observations, in this paper we proposed the PPOS algorithm with a functional clipping mechanism combined to the original PPO. Our work significantly improves the ability to restrict policy updates and maintain stability, and we demonstrated those through extensive results. We also provide a helpful guideline for tuning the only hyperparameter used by our method, and thus resulting in efficient redeployment on novel environments in the future.
While in this paper we focused on the hyperbolic tangent function and the tuning of the coefficient, in the future we will consider the clipping mechanism with different functions, as other functions have different properties which will affect the performance of the policy search. We will also study the relationship between the clipping range and the clipping function. The default = 0.2 is widely used in different variants of PPO, so there is considerable space for exploration of the clipping range. Another possible improvement in the future is the trust-region method, which is another mechanism to restrict the policy updates [18]. Finally, the implementations of deep reinforcement learning algorithms suffer from the repetitive tuning of hyperparameters [7]. Although we proposed a tuning method for one hyperparameter, there are still many hyperparameters, as seen in Table 3, which may need to be explored in those frameworks. APPENDIX See Table 3.
WANGSHU ZHU received the bachelor's degree in computer science from ShanghaiTech University, in 2019, where he is currently pursuing the master's degree in computer science with the School of Information and Science Technology. He joined the Living Machines Laboratory, Shang-haiTech University, in 2018. His current research interests include model-based fault tolerance for manipulators and reinforcement learning method for robotics. He also focuses on the combination of model-based learning methods and robots. He was awarded by the Merit Student of ShanghaiTech University, in 2020.