GDCP: A custom package for flexible detector setups in Geant4

Geant4 Detector Construction Pattern (GDCP) is a C++ package containing a set of utility classes for developers to utilize during the detector construction phase of a Geant4 application. The main objective of this study is to provide an interface that helps users develop a highly flexible, understandable, and maintainable detector construction code. The package also contains a couple of helper classes facilitating the implementation of optical components in scintillation detectors. Finally, an advanced Geant application has been provided to demonstrate the package's utility. Program summary Program title: GDCP CPC Library link to program files: Licensing provisions: GNU General Public License 3 Programming language: C++ External routines/libraries: Geant4 Nature of problem: Geant4 provides an abstract class, G4VUserDetectorConstruction, that is used for the whole detector construction. This is sufficient for a simple detector setup but implementing a single class for building a complex detector leads to complexity in the code. To solve this issue it is recommended to split the implementation into more methods or classes; however, there is no standard solution published yet and it is entirely left to the users' ability. Solution method: To address this issue, I propose to apply the"Builder design pattern"in the detector construction phase. For this, the elements of the Builder design pattern have been introduced and made available for developers. Keywords: G4VUserDetectorConstruction, Builder design pattern, Detector construction, Component builder, Optical photon, Scintillator

Introduction
GEANT4 [1,2,3] is an object-oriented Monte Carlo simulation toolkit developed for simulating the passage of particles through matter. Although originally developed for nuclear and particle physics experiments, its application domain has spread to fields such as space science, accelerator, and medical physics. The toolkit is regularly updated, developed, and maintained by a worldwide collaboration of scientists. In recent years, it has become the most preferred detector simulation software in the high energy and nuclear physics communities.
The classes comprising the Geant4 software are categorized based on their logical content. Each category is designed as independently as possible. This allows each to be developed in parallel without serious interference. This work falls under the geometry and material category of Geant4 and proposes a new detector construction scheme for achieving more flexible detector setups. ponents, for instance, can be upgraded, replaced by other components, or totally removed from the system. More importantly, such change requests should be made by extending the code rather than breaking it, so that different versions of detectors become comparable. This necessitates a highly flexible detector production code.
In order to circumvent all these complications and have a clear implementation structure, this work proposes a component-based detector construction strategy for flexible detector setups. The strategy involves applying the "Builder design pattern", which is commonly used in object-oriented programming for creating complex objects step-by-step, to the detector construction phase. However, in the context of Geant4 geometry construction, I call it "Geant4 Detector Construction Pattern" 1 . The following section publicizes participants in this pattern and describes its implementation in detail.

Utility classes for geometry construction
The Builder design pattern is applied to the detector construction process in two stages. The first stage involves creating detector components as single or multiple parts, and the second involves assembling those components in a volume that represents the entire detector geometry. The following introduces elements of this pattern: • Abstract builder classes: VComponentBuilder and VMul-tiComponentBuilder. The first is used to build a single detector component, and the second is used to build a multiple detector component. These two classes declare product construction steps.
• User builder classes: These are the concrete classes to be derived by users. Two kinds of builder classes are derived, one for detector components and one for the detector itself. I further provide ready-to-use concrete component builder classes for the most commonly used shapes in Geant4: BoxComponentBuilder, TubsComponent-Builder and TrdComponentBuilder. When an object of one of these classes is created, a few user interface commands are defined automatically for the modification of the material and size of that component.
• Director class: DetectorDirector. This class defines the order in which to execute the building steps, while the detector builder class provides the implementation for those steps. The usage of this class is optional.
• Product class: SmartComponent. All builder classes produce the same type of product. A product is an object of type SmartComponent encapsulating G4LogicalVolume and G4VPhysicalVolume objects. So, depending on its content, it may represent a single detector component, a multi-component, or the detector itself. It also keeps a list of locations where it is positioned in the detector geometry. 1 This is also where the package gets its name. Fig. 1 summarizes in a UML [8] diagram the structure of the classes provided for applying the Builder design pattern to the detector construction process of a Geant4 application.
The following steps explain how to set up a complex detector in a highly flexible manner using the supplied classes: • Make a list of all the components to be employed in the detector construction.
• • Create a concrete material factory class as described in Sec. 2.3. If a material is employed just for one component, it can be defined simply within the CreateMaterial() method of the relevant builder class. If employed for more than one component, it can be defined either in a member method within the concrete material factory class or in a new material builder class created for that material.
In the second case, the object of the material is called from the CreateMaterial() method.
• After the above three steps are completed, proceed to the assembly phase of the components. For this, derive the detector builder class from VMultiComponentBuilder 2 and implement its construction steps. Listing 1 shows an example derivation of the detector builder class.
• The last step is the implementation of the Geant4 detector construction class. This class makes use of UserDetec-torBuilder and optionally DetectorDirector class for the construction of the detector. Listing 2 shows an example implementation of the detector construction class.
The advantages of using this pattern are summarized below: • It provides flexibility for change requests that may arise during the detector development process.
• It provides reusability of detector components. As detector components are treated as self-contained pieces, a component developed for a particular detector could be used in various detector constructions.
• It allows developers to create a messenger class per component.

Utility classes for scintillation detectors
This section concerns the construction of scintillation detectors. One of the crucial steps in the implementation process of scintillation detectors is to specify the optical properties of the materials employed and the optical surfaces defined. Geant4 provides G4MaterialPropertiesTable and G4OpticalSurface classes for this purpose. The former is used to assign new properties to pre-defined materials or surfaces, while the latter is used to describe optical properties of medium boundaries.
Optical properties are stored as entries in a G4MaterialPropertiesTable, and each entry consists of a key and a value pair. These properties may be energyindependent or energy-dependent. In the first case, the value is a constant, and in the second case, it is a list of energy-value pairs.
The usual approach is to put these energy-value pairs inside the detector construction file as elements of std::vector or C-style arrays. However, the presence of various datasets in a typical application and their direct embedding into the source file results in long lines of code and reduces code readability.
Moreover, the parameters defining the properties of materials and surfaces are frequently changed in studies where the effect of these parameters is investigated. This requires changing the source code repeatedly.
To overcome these challenges and perform the entire implementation process through user interface commands, I propose using two new classes. The first one is OpticalSurface class 3 , which is derived from G4OpticalSurface. The only functionality added to this subclass is that its implementation can be conducted by user interface commands. The second one is MaterialPropertiesHelper class, which is a subclass of G4MaterialPropertiesTable. This class provides additional methods to receive data from users. Fig. 2 summarizes, in a UML class diagram, the provided classes and their relationships with the Geant4 classes.
To illustrate, I take the code snippet shown in Listing 3 from the examples of Geant4 4 and compare it to the implementations of MaterialPropertiesHelper and OpticalSurface classes ( Listings 4 and 5), which fulfill the same task.

Utility classes for material construction
This section introduces two new abstract classes, VMateri-alBuilder and VMaterialFactory, that developers can utilize in the detector construction phase of a Geant4 application. The first offers an interface for developers to create custom material classes, while the second serves as interface for creating material objects from a single place in an application. Fig. 3 shows the UML class diagram of these two abstract classes.
To create a concrete material builder class, users inherit from VMaterialBuilder and implement its CreateMaterial() method. This class utilizes MaterialPropertiesHelper to assign additional properties to the materials to be defined. To get the resulting material object from the builder, the method GetProduct() is invoked. Listing 6 shows an example concrete implementation of VMaterialBuilder class.
To derive a concrete material factory class, users inherit from VMaterialFactory and implement its GetMaterial() method. This method is used to create objects of user-defined materials. However, it is not invoked directly by users to obtain a material object; instead, the FindOrCreateMaterial() method is invoked. This method first searches the desired material in GEANT4's material database. If the material is not found, it invokes the GetMaterial() to search among the user-defined materials. If it is not found there as well, it throws an exception. Listing 7 depicts an example usage of VMateri-alFactory class.
The benefits of these two classes can be summarized as: • Providing reusability of materials. Encapsulating a material construction code into a class makes it available to other developers. This is particularly significant for materials that are troublesome to implement. For example, to implement a scintillator, its atomic composition, scintillation character, and all-optical properties should be defined. This requires an extensive literature review followed by conversion of the collected data into the desired format. Once a scintillator is implemented by a developer, it can be used by others as well. VMaterialBuilder class acts as an interface for producing ready-to-use material classes.
• Improving code readability by encapsulating material construction code.
• Preventing code duplication by allowing users to access material objects from different sections of the code.

Structure of GDCP
The GDCP package is a collection of C++ classes developed for designing flexible detector setups in Geant4 applications. The provided classes are collected and organized in directories according to their dependencies and functionalities. The package consists of the following three directories: • gdcp/: This directory includes header and implementation files of GDCP.
• scntDetector/: This directory includes various utility classes for the implementation of scintillation detectors.
• g4Example/: This directory includes an advanced geant4 application that demonstrates the usage of the provided classes. The application also uses the NuSD framework [10] that we previously developed for modeling segmented scintillation detectors. The detector construction interface in this framework is updated using the GDCP in order to enhance the framework's flexibility and maintainability. The application includes the installation of PANDA [11,12] and SOLID [13,14,15] experiments as examples. uImanager -> A p p l y C o m m a n d ( " / control / execute path / to / macro / " + name + " . mac " ) ; // Implement your material definitio n here using the provided Geant4 classes . 17 // As an example , EJ200 s c i n t i l l a t o r is implemen t ed using data sheet provided by the m a n u f a c t u r e r .

Conclusions
In this study, a new custom package GDCP developed to create flexible detector installations in Geant4 applications is introduced. The classes in the package are grouped into three categories based on their roles and responsibilities. The first set of classes offers a detector construction interface based on independent detector components for flexible detector setups. This interface forces to apply the Builder design pattern to the detector construction phase in two steps: 1) for the construction of the detector components and 2) for the assembly of those components. For this, the elements of the Builder design pattern are introduced and made available for users.
The second set of classes provides an interface to obtain custom material classes. This is particularly significant for materials that implementation necessitates extensive data collection from the literature. Including a material construction code in a class allows other developers to easily use that material.
The third group of classes is related to scintillation detectors and makes a dual contribution to the implementation process of optical components. This both provides an easier interface for defining properties of optical materials and allows that interface to be implemented through the user interface commands.


QReg: On Regularization Effects of Quantization

In this paper we study the effects of quantization in DNN training. We hypothesize that weight quantization is a form of regularization and the amount of regularization is correlated with the quantization level (precision). We confirm our hypothesis by providing analytical study and empirical results. By modeling weight quantization as a form of additive noise to weights, we explore how this noise propagates through the network at training time. We then show that the magnitude of this noise is correlated with the level of quantization. To confirm our analytical study, we performed an extensive list of experiments summarized in this paper in which we show that the regularization effects of quantization can be seen in various vision tasks and models, over various datasets. Based on our study, we propose that 8-bit quantization provides a reliable form of regularization in different vision tasks and models.

Introduction
Deep neural network (DNN) quantization has been an active research topic in the past few years. Quantization is the task of approximating high precision (floating point) weights and activations of a model with lower bit resolution counterparts (usually 1 to 8 bit integers). Quantization can be done after (post training quantization) or during training (quantization aware training). Although both methods can compress the model and reduce overall execution time, at the cost of model accuracy, post training quantization usually takes less time to quantize a model. On the other hand, quantization aware training usually requires a full training session and with a moderate quantization level (int8) can produce quantized models with little to no accuracy loss compared to their full precision counterparts. In fact, as we will discuss in Section 2, there are many existing works 1 Ecole Polytechnique Montreal 2 Mila, Université de Montreal 3 Deeplite Inc., Montreal. Correspondence to: MohammadHossein AskariHemmat <m.h.askari.hemmat@gmail.com>.

Objectiveness Loss Value
Epoch Number We believe that the quantized model has better generalization performance due to its regularization effect.
claiming that quantization aware training produced accuracy improvements compared to some of the best full precision models. In these studies, the authors usually attribute such accuracy gains to the regularization effect of quantization. This has motivated us to explore how quantization affects training. In this work, we first hypothesize the regularization effect of quantization. Moreover, we hypothesize that this regularization effect is correlated with the quantization level. To confirm our hypothesis, we analytically explore how quantization noise propagates in a network at training time, and how the quantization level is correlated with the amount of regularization. We then empirically explore the regularization effect of quantization by applying different quantization aware training methods on different models and datasets with different quantization levels. In both analytical and empirical studies, we confirm our initial hypothesis.

Background
The regularization effect of quantization has previously been reported in the literature. We have categorized earlier works into three main lines of work, which we will review briefly in the following subsections.

Effect of Quantization on Accuracy Improvements
In (Courbariaux et al., 2015), one of the first quantization methods for DNNs, the authors empirically show that modern deep learning optimization techniques such as stochastic gradient descent are compatible with noisy weights, as they appear in quantization. They also empirically show that noisy weights provide a form of regularization. They confirm their hypothesis by achieving state-of-the-art results on MNIST, CIFAR-10 and SVHN datasets. Similar to this work, authors in (Hubara et al., 2017), argue that weight quantization is a form of noise injection to weights and as shown in DropConnect (Hou & Kwok, 2018), this noise injection can improve the generalization performance. However, both studies are only limited to small-scale classification tasks. Their studies are also limited to one quantization technique. Furthermore, their hypotheses lack analytical study.
In (Abdolrashidi et al., 2021) the authors study the effect of quantization on accuracy and provide an analysis of the regularization effect of quantization. Their study is limited to showing a generalization gap between a quantized model with 8 and 4 bit precision and full precision models on the Imagenet dataset on Resnet50. On the other hand, in addition to their work, they reported the generalization gap of different quantization methods. However, in their experiments, the regularization effect of quantization is not investigated. Furthermore, they did not explain whether this generalization gap is coming from quantization or whether they have used other regularization techniques that might have affected the overall results.
Apart from the papers above, there are many other works reporting accuracy improvement after quantization, (Xu et al., 2018a;Louizos et al., 2019;Chen et al., 2021;Xu et al., 2018b;Andri et al., 2018;Wang et al., 2019;Liu et al., 2020;Mishchenko et al., 2019). However, to the best of our knowledge, none of them provided analytical or empirical studies and the authors justified the boost in performance (accuracy) with a conjectured regularization effect of quantization.

Analytical Studies
As presented in 2.1, the regularization effect of quantization has been observed in previous works. In the following, we present three papers that provide an analytical study on how quantization affects training.
In (Merolla et al., 2016), the authors claim that applying a constraint on the weights can act as a regularizer. They investigate how DNNs are robust to different types of weight distortion. As an example, for quantization, they studied the effect of binarization proposed in (Courbariaux et al., 2015) on training. Using proximal methods, the authors showed that applying weight clipping during training can be modeled as a type of regularization that penalizes weights outside the unit ball of the ∝ norm. Their study is limited to applying binarization on weights and they do not explain how different levels of weight quantization affect training. Moreover, like (Courbariaux et al., 2015) , they only investigate the regularization effect of weight distortion on small models.
A more recent study in (Alizadeh et al., 2020) provides an analytical overview on how quantization can affect robustness. Although not directly relevant to our work, the authors in this paper provide an analytical study on how quantization affects training and the loss function. In this paper, the authors propose a regularization based quantization method that, unlike most quantization-aware methods, provides a selectable quantization bit-width after training. Like previous methods, the authors model quantization as a form of noise. Using first order Taylor approximation, they then provide an analytical study on how this noise affects the robustness of DNNs. With their method, they show that if the 1 -norm of the gradient is small, the perturbation (at least to the first-order approximation) can be effectively controlled for various quantization bit-widths. To guarantee the 1 -norm of the gradients, they propose a regularization term in the loss function. As we will show in 3, we used similar quantization noise modeling and approximation techniques to investigate the effect of quantization on the loss function.
In (Zhang et al., 2022), the authors provide an analytical study on how models with stochastic binary quantization can have a smaller generalization gap compared to their full precision counterparts. In this work, the authors propose a so-called "quasi neural network" to approximate the effect of binarization on neural networks. The authors then derive the neural tangent kernel for the proposed quasi neural network approximation. With this formalization, the authors show that binary neural networks have lower capacity, hence lower training accuracy, but also a smaller generalization gap than their full precision counterparts. Finally, the authors provide some empirical results to support their derivations. However, the authors fail to show the effect of different quantization levels on the generalization gap. As mentioned earlier, the authors only focused their analysis on statistical binary quantization and they did not explore how different quantization methods and precision levels affect generalization. Also, their empirical results are only limited to a two layer model.

Using Quantization for its Regularization Effect
Although very limited, the idea of using quantization solely for its regularization effect has been explored before. In (Hou et al., 2017), the authors studied the effect of binarization on the loss function. They then proposed an algorithm that uses diagonal Hessian approximation that directly minimizes the loss function with regard to the binarized weights.
For training their binarized model, the authors claimed that they only used regularization that is induced by weight binarization. Furthermore they further developed their method in (Hou & Kwok, 2018) where they show that the same techniques can be applied to weight ternerization.

Quantization Noise as a Regularizer
The authors in the previous works usually assume that weight quantization is similar to adding noise to a model's weights. However, it is not clear how this noise propagates through the network at training time. Also, it is not clear how different quantization levels and algorithms affect the magnitude of this noise. In this section, we will show how quantization noise can be modeled as a regularization term. We will then extend our study and show that this noise is directly related to the quantization level.
For simplicity, let us consider a regression problem where the mean square error loss is defined as, where y i is the target, andŷ i = f (x, w) is the predicted target of the network f parameterized by w. Weight quanti- zation can be characterized as noise perturbing the weights and activations. Such noise can be described as: Thus, a quantized neural network, effectively has the following loss,L whereŷ i q = f (x, w + δ).
Theorem 3.1. Given the mean square error loss defined in Eq. 3, assuming that the quantization noise follows a normal distribution, δ ∼ N (0, σI), applying a first-order Taylor approximation around the weights of the full precision model, results in the following approximation on theL: This means that minimizingL is equivalent to minimizing the loss of a non-quantized neural network with gradient norm regularization.

Derivation in Appendix A.
Direct application of noise on the weights as a form of regularization has been studied in the literature. In (Goodfellow et al., 2016) authors argue that noisy weights result in forcing the model to find minima that are insensitive to small perturbations of the weights. In other words, noisy weights encourage finding minima in flat regions and thus lead to better generalization (Hochreiter & Schmidhuber, 1994).

Experimental Studies
In this section, we will describe empirically how quantization has a regularization effect. Unlike previous works that was described in Section 2, we tried to explore the regularization effect of quantization over different quantization techniques, levels, models, vision tasks and datasets. In the following subsections, we briefly describe our test setup and results.

TRAINING SETUP
As we described in Section 3 we hypothesize that quantization has a regularization effect that is correlated with quantization level (precision). In order to test this hypothesis, we needed a quantization technique that could be used to quantize a model to any quantization precision. For our experiments, we picked Learned Step size Quantization (LSQ) (Esser et al., 2020) and Parameterized Clipping Activation (PACT) (Choi et al., 2018). Since PACT applies quantization only to activations, we used DoReFa (Zhou et al., 2016) quantization for weights. For PACT, DoReFa and LSQ, we used learning rate of 0.1 with momentum of 0.9 for the classification experiment. In our experiments, we initially used quantization as the only regularizer in the network. However, both full precision and quantized models were unable to achieve a good accuracy. Hence, we used weight decay of 0.0001 for both full precision and quantized models in our experiments. We trained both full precision and quantized models for 200 epochs. We used a multi-step learning scheduler with milestones of 50, 90, 130 and 180 with gamma of 0.2.

MODELS AND DATASETS
In our experiments, we explored the regularization effect of quantization in different vision tasks, models, quantization levels and datasets. We explored two popular vision tasks, object detection and classification.  Figure 2 illustrates part of the experiments that we ran to test our hypothesis. Figure 2 shows results of nine different test configuration in nine squares. Each square consists of 20 cells, which as we discussed in Section 4.1.2, corresponds to the test accuracy difference between full precision model and quantized models when different augmentation is used. From left to right, each column shows the quantization levels (2-bit, 4-bit and 8-bit) that was used for our tests. From top to bottom, we have tested our hypothesis on Resnet20 on cifar10, Resnet18 on cifar100 and YOLOv5n on VOC dataset respectively. As it can be seen (i.e. the overall cell colors in the right column squares are green), regardless of model, dataset, augmentation or quantization technique, 8-bit quantized models generally have better generalization compared to full precision models. As we discussed in 2.3, we believe that this performance improvement corresponds to the regularization effect of quantizing the models. As we predicted, this regularization effect is correlated with the quantization level. Moving from left to right columns, the overall color codes for each cell changes from red to green. Which indicates that quantized models are generalizing better compared to their full precision counterparts as we use higher precision quantized models (from 2-bit to 8-bit). Regardless of the model, dataset, data augmentation or quantization technique, the generalization difference shrinks and even get worse as we use less precision for quantization. Another way to evaluate how quantization is helping with better generalization, is to compare the error of quantized and full precision models. Table 1 shows the performance improvement relative to model error. We averaged the accuracy of each square in Figure 2. We then, calculated the performance improvement relative to error using the following formula: Where f val and qval are the average accuracy of full precision and quantized model over 19 different augmentation setups. Since models on small datasets like cifar10 generally have small errors compared to models on more complex datasets (like cifar100 or VOC), we used a logarithm in Equation 5. Once again, according to Table 1, in all cases 8-bit quantized models reduce the error more (i.e, generalizing better) compared to full precision models. On all three models tested over 19 different augmentation setups, all 8-bit quantized models reduce the error when compared to full precision models (i.e last column for 8-bit quantization levels is green for all models). This effect (reducing error) is less clear as we use lower precision models .
Our results presented in Figure 2, Table 1 and Appendix B empirically confirm our hypothesis. Unlike (Courbariaux et al., 2015), we believe that the regularization effect of quantization is in fact correlated with the quantization level. Our empirical study shows that moderate quantization (8bit) generally helps models generalize better. In addition to these results, specially for more complex models, we observe that quantized models overfit less at training time. Figure 1 illustrates the validation loss of 8-bit, 4-bit quantized and full precision for the YOLOv5n model applied to the VOC dataset. Due to regularization effect of quantization, the quantized models overfit much less compared to full precision model. This, confirms our hypothesis. For the full list of experiments, please refer to Appendix B.

Conclusion
In this paper, we explored the regularization effect of quantization. We hypothesized that this regularization effect is correlated with the quantization level. To confirm our hypothesis, we provided an analytical study as well as an extensive list of experiments. Our experiments show that the regularization effect of quantization exist regardless of model, dataset, vision task and quantization technique. Finally, we analytically and empirically confirm that this regularization effect is correlated with quantization level. Our empirical study shows that moderate quantization (8-bit) helps models generalize better.

A. Analytical Study
A.1. Derivation of Theorem 3.1 Theorem A.1. Given the mean square error loss defined in Eq. 3, assuming that δ ∼ N (0, σI), applying a first-order Taylor approximation around the weights of the full precision model, results in the following approximation on theL: This means that minimizingL is equivalent to minimizing a non-quantized neural network while regularizing the norm of the gradients.
From Eq. 3, we have,L , we can apply a first order Taylor approximation, Thus,ŷ i q ≈ŷ i + δ∇ wŷi .
Re-writing, the expectation onL, Note that since δ ∼ N (0, σI), we have E p(x,y,δ) [2δ(∇ wŷi )(ŷ − y)] = 0.  Figure 3. Figure 3(a) illustrates the positional location of augmentation that was applied on the original test dataset. In addition to test images for each dataset, in our experiments, we applied 19 different augmentations on the test dataset. We used the augmentation that is provided in (Hendrycks & Dietterich, 2019). Figure 3(b) illustrates the generalization improvement between quantized and full precision Resnet18 on CIFAR100 model when different augmentation is applied. The value in each cell shows the difference and the position of each cell corresponds to the data augmentation technique listed in Figure 3(a).


RepParser: End-to-End Multiple Human Parsing with Representative Parts

Existing methods of multiple human parsing usually adopt a two-stage strategy (typically top-down and bottom-up), which suffers from either strong dependence on prior detection or highly computational redundancy during post-grouping. In this work, we present an end-to-end multiple human parsing framework using representative parts, termed RepParser. Different from mainstream methods, RepParser solves the multiple human parsing in a new single-stage manner without resorting to person detection or post-grouping.To this end, RepParser decouples the parsing pipeline into instance-aware kernel generation and part-aware human parsing, which are responsible for instance separation and instance-specific part segmentation, respectively. In particular, we empower the parsing pipeline by representative parts, since they are characterized by instance-aware keypoints and can be utilized to dynamically parse each person instance. Specifically, representative parts are obtained by jointly localizing centers of instances and estimating keypoints of body part regions. After that, we dynamically predict instance-aware convolution kernels through representative parts, thus encoding person-part context into each kernel responsible for casting an image feature as an instance-specific representation.Furthermore, a multi-branch structure is adopted to divide each instance-specific representation into several part-aware representations for separate part segmentation.In this way, RepParser accordingly focuses on person instances with the guidance of representative parts and directly outputs parsing results for each person instance, thus eliminating the requirement of the prior detection or post-grouping.Extensive experiments on two challenging benchmarks demonstrate that our proposed RepParser is a simple yet effective framework and achieves very competitive performance.

Introduction
Multiple human parsing (MHP) aims to segment body parts for each person in an image, which is a fundamental yet challenging task in the human-centric intelligence system. Compared to many dense predictions tasks, such as object detection (Ren et al. 2017) and instance segmentation Tian, Shen, and Chen 2020;Yanwei Li and Jia 2021;Yu et al. 2022), it is the arbitrary number of finegrained body parts that have made MHP much more chal- lenging. In particular, the success of MHP models depends on two key aspects: 1) whether the model can make a correct instance separation, and 2) whether this model can decide what semantic behind each image pixel. Inspired by the success of the human-centric recognition, such as pose estimation (Wang et al. 2021a(Wang et al. , 2022Sun et al. 2019;Wang et al. 2021b), existing methods of multiple human parsing adopt a two-stage strategy, which consists of top-down and bottomup pipelines. In particular, the top-down pipeline ( Fig. 1.(a)) starts with person detection responsible for the first aspect, then an RoI operation is adopted to crop the person from the feature maps or the original image. After that, single-person human parsing is performed for addressing the second aspect. Instead, the bottom-up pipeline ( Fig. 1.(b)) firstly seg-ments instance-agnostic body parts responsible for the second aspect, then groups them into instance-aware results for addressing the first aspect. Despite the great progress, previous state-of-the-art methods for multiple human parsing still encounter several challenges, as analyzed below: The strong coupling of the second stage with the first stage in the two-stage framework significantly hampers high-quality multiple human parsing. Specifically, top-down methods (Ji et al. 2020;Yang et al. 2019;He et al. 2017;Yang et al. 2020) are highly dependent on person detection results while bottom-up methods rely on instance-agnostic part segmentation results. Since person bounding boxes are rectangular, they may contain irrelevant contents such as body parts belonging to other persons. In this way, the performance of human parsing will drop significantly if the person detection performance decreases dramatically. In terms of bottom-up pipeline, existing methods (Gong et al. 2018(Gong et al. , 2019Li et al. 2018;Zhao et al. 2018) predict redundant instance-agnostic body parts. In this way, some body parts may be removed during grouping post-processing due to their low-quality confidential scores. Besides, the processing of assembling body parts (e.g., Hungary algorithm) is often heuristic, making these methods complicated yet inefficient. Overall, the bottleneck of the two-stage frameworks lies in the first stage, as the performance of a model in the first stage decides the upper bound of the entire algorithm.
The above challenges motivate us to rethink two problems: 1) how to design a single-stage pipeline for multiple human parsing, and 2) how to equip this pipeline with the ability to establish a direct mapping from an image to various instance-specific body parts. To handle the above two problems, we present an end-to-end multiple human parsing framework using representative parts, termed RepParser. As illustrated in Fig. 1.(c), the proposed RepParser is designed in an end-to-end manner without resorting to person detection or post-grouping. To this end, RepParser decouples the parsing pipeline into instance-aware kernel generation and part-aware human parsing, which are responsible for instance separation and instance-specific part segmentation, respectively. The core idea is that we empower the parsing pipeline with representative parts, since they are characterized by instance-aware keypoints and can be utilized to dynamically parse each person instance. Specifically, representative parts are obtained by jointly localizing centers of instances and estimating keypoints of body part regions. After that, we dynamically predict instance-aware convolution kernels through representative parts, thus encoding person-part context into each kernel responsible for casting an image feature as an instance-specific representation. Furthermore, a multi-branch structure is adopted to divide each instance-specific representation into several part-aware representations for separate part segmentation. In this way, RepParser accordingly focuses on person instances with the guidance of representative parts and directly outputs parsing results for each person instance, eliminating the need for person detection or body part grouping. In summary, our work has the following contributions: (1) We propose a novel multiple human parsing pipeline termed RepParser, which eliminates the dependence of prior person detection and avoids heuristic postgrouping operations.
(2) The RepParser is designed in a flexible fashion, as it dynamically encodes person-part contexts into corresponding convolution kernels. To our knowledge, this is the first single-stage method for multiple human parsing and it can inspire related research on fine-grained recognition.
(3) Extensive experiments conducted on two challenging benchmarks demonstrate the effectiveness and generalizability of the proposed method. Moreover, it significantly outperforms most two-stage methods and variants of single-stage instance recognition methods.

Multiple Human Parsing
To date, methods of multiple human parsing are based on the two-stage pipeline. Most of them can be divided into two categories: 1) bottom-up paradigm and 2) topdown paradigm. As mentioned above, the bottom-up methods (Gong et al. 2018(Gong et al. , 2019Li et al. 2018;Zhao et al. 2018) regard multiple human parsing as a segment-thengrouping pipeline.  (Ren et al. 2017) to predict a bounding box for each person and extracts region-of-person from detector's features for performing instance-specific part segmentation. Following this idea, Parsing R-CNN ) and RP R-CNN (Yang et al. 2020) devote to solving the problem of the single-person human parsing and propose new variants of Mask R-CNN via contextual modeling or part re-scoring. Different from two-stage methods, our work is devoted to designing a novel single-stage pipeline and focuses on instance-aware body part segmentation with representative parts.

Single-stage Instance-level Recognition
Traditional solutions try to build an instance-specific model for instance-level recognition. For example, Tian et al. (Tian, Shen, and Chen 2020) adopt conditional convolutions for one-stage instance segmentation, where each convolution kernel is dynamically generated from a center point of person instance. This design improves the instance segmenta-   Figure 2: The overall architecture of RepParser without resorting to prior detection or post-grouping, where it decouples the multiple human parsing pipeline into instance-aware kernel generation and part-aware human parsing. In particular, it firstly estimates several representative parts, which are dynamically responsible for instance-aware feature generation. Then, a multibranch structure is adopted to divide instance-aware features into part-aware features for separate part segmentation.
tion performance while maintaining high efficiency. Moreover, Li et al. (Yanwei Li and Jia 2021) propose a locationaware kernel generation for panoptic scene understanding. Mao et al. (Mao et al. 2021) propose to dynamically generate a keypoint-aware estimator for multi-person pose estimation. Although these different approaches vary in tasks, they all share a characteristic: they focus on the instance-specific convolution kernel generation. However, each convolution kernel generated through existing methods encodes sparse content of an instance (i.e., object center only). Therefore, the generated kernels severely ignore the person-part context which is essential for accurate human parsing, thus leading to suboptimal results as demonstrated in the experimental results.
As a supplement to them, we extend instance-specific modeling to solve the multiple human parsing. Instead of directly deriving from single-stage frameworks that are used in other instance recognition tasks, we propose to parse multiple human instances through representative parts, and encode person-part context into each instance-aware convolution kernels as well as part representation. As a result, the proposed method significantly outperforms variants of single-stage methods applied in other instance recognition tasks.

Methodology
The pipeline of our RepParser is presented in Fig. 2. Given an input image I , the goal of multiple human parsing is to localize N person instances and segment C body parts for each localized person. In particular, it needs to address two issues: 1) how to distinguish each person instance from other instances or background; and 2) how to perform instanceaware parsing without extra operations (i.e., RoI cropping or part grouping). To address these, we propose to parse multiple persons using representative parts. Specifically, Rep-Parser firstly utilizes a backbone network (e.g., ResNet) to obtain an image-level feature F with a size of H × W × D, where D indicates the number of channels and {H, W } denotes the spatial size. Next, a detection branch, which is an FCOS (Tian et al. 2019) head with an object center estimator and a location regressor, is adopted to localize person instances. With the location of person centers, an instanceaware kernel generation branch is used to estimate the representative parts of each person and accordingly generate convolution kernels for each instance. With instance-aware kernels and representative parts, a part-aware parsing module, which is a multi-branch structure, is utilized to generate partaware features for accurate human parsing. In the following, we describe the details of RepParser.

Representative Parts
As discussed before, bounding boxes and object centers are often used to represent person instances in two-stage methods and single-stage methods. Due to the rectangular shape, a bounding box has a rough global context of a person but cannot account for the semantically important local areas. Instead, the object center only accounts for small local areas, thus ignoring the interrelation among body parts and the global context of an instance. To overcome these limitations, our core idea is that each person instance in an image is represented by representative parts. It is expected that the representative parts can encode the characteristics of each person instance and only focus on the pixels of corresponding body parts. Motivated by this, we propose to dynamically construct representative parts of an instance through keypoints of body parts, as they can reflect the global context of a person (e.g., posture or shape) and semantically salient part areas. Formally, let R = {(x k , y k )} C k=1 denotes representative parts of a person, where (x k , y k ) is the keypoint of the k-th part (e.g., face, left-arm, right-arm, and so on), C is the number of part categories (e.g., C=20 for CIHP dataset). Thus, we parse person instances conditioning on their representative parts, as they not only present characteristics of pose and shape but also reflect person-part relations.
To construct representations of representative parts, we need to localize them by object centers. As shown in Fig. 2, we firstly adopt a feature pyramid network (Lin et al. 2017) to produce multi-scale feature maps from levels 3 to 7. Following FCOS (Tian et al. 2019), we treat each location on the feature maps as a potential instance. Thus, for each location (x h , y h ) on the feature maps, we estimate the confidential score being a person center and the offsets to representative parts. Based on this, representative parts R is calculated by Eq. 1.
are the normalized offsets from the center of a person instance to the center of the representative parts. After that, we construct an initial representation of representative parts by sampling pixel points from the image-level feature F . Formally, we denote f h as the feature of sampled instance point and {f k h } C k=1 as the features of representative parts. Next, we employ these sampled representative parts for instance-aware kernel generation and part-aware human parsing.

Instance-aware Kernel Generation
To obtain high-quality instance representations for accurately human parsing, it is expected that the instance-aware convolution kernels are dynamically generated by relying on the characteristics of instances. To achieve this, we propose to generate instance-aware kernels by representative parts, since they encode potential contexts about personpart relations. Instead of directly applying initial representative parts to predict instance-aware kernels, we first readjust the representation of the representative parts according to the person-part relations, aiming to dynamically encode the person-part context into the corresponding kernel. Specifically, the re-adjusted representative parts are obtained through Eq. 2: where W a and W m are learnable parameters that are respectively responsible for relation estimation and feature re-adjustment. σ(·) is the standard sigmoid function and ⊕ means a concatenation operation. The α is the estimated relation matrix, where each element in α denotes a part being relevant to a person with a confidential score. The f p denotes the representation of representative parts, which is dynamically generated via person-part interaction. Note that the estimated relational score α is continually updated by directly supervised training so that it becomes more accurate from time to time. Given re-adjusted representative parts f p and instance representation f h , we generate two types of convolution kernels through Eq. 3.
where V 1 and V 2 are linear matrices for kernel generation. W f is used to project an image feature to an instance-aware feature without resolution reduction, while the W o is responsible for predicting part masks from many part-aware features. Notably, the generated kernels are very compact, as they are designed in a 1 × 1 convolution layer with less channels (e.g., 32 for W f and C for W o ). We project the image-level feature F to the instance-aware feature for human parsing by Eq. 4.
where is the convolution operation. Compared with topdown methods, such as Parsing-RCNN consisting of eight 3 × 3 convolution layers with 256 channels for instance feature extraction, the generated kernels are much more lightweight.

Part-aware Human Parsing
After handling the issue of person separation by representative parts, we would like to predict an accurate mask for each part. Instead of predicting masks from the instanceaware feature, we first build multiple branches for separate parsing. In each branch, we construct a part-aware representation that can be used to only fire on the pixels of the corresponding part. Specifically, we first divide the instanceaware feature F p into C groups, each of which is responsible for part-specific segmentation. For each group, we construct a geometry map that records relative distances from all pixels to the corresponding representative part, suggesting the salient area of the corresponding part. Formally, we denote all geometry maps as F s and compute the part-aware representation by Eq. 5.
where W p is the learnable transformation matrix. Next, we utilized dynamically generated kernel W o to predict mask for each part, which is formalized by Eq. 6.
where M is the predicted parsing maps and φ(·) is the standard softmax function. Although it is possible to predict masks by instance-aware feature, we empirically find that using part-aware representation performs better.   (Gong et al. 2018). MHP-v2 is a commonly used dataset for instance-level human parsing. It is split into 15k/5k/5k images for train/val/test. Each image contains an average of three people with 58 body part categories. In addition, the CIHP dataset is currently the largest multiple human parsing dataset, which covers 19 part categories and involves many crowded scenes. It is split into 28k/5k/5k images for train/val/test.

Metrics:
For evaluation, we use many standard metrics to measure the performance of all parsing models, including the Average Precision based on part (AP p ) and Percentage of Correctly parsed semantic Parts (PCP). We report AP p vol and AP p 50 . AP p vol is the average of AP p at different IoU thresholds ranging from 0.1 to 0.9. In particular, AP p 50 means that AP p is calculated at an IoU threshold of 0.5. In terms of instance-agnostic parsing, we report mean Intersection-Over-Union (mIOU) for model evaluation.  (Lin et al. 2017) is used as the feature extraction network. The weights of all backbones are pre-trained on Im-ageNet, while the remaining weights are randomly initialized. We train models using SGD/AdamW for convolution-/transformer-based models, respectively. A mini-batch size of 16 is used. Other details are identical to FCOS (Tian et al. 2019).

Main Results
In this section, we compare proposed RepParser with stateof-the-art multiple human parsing methods and report evaluation results summarized from two datasets. In addition to existing two-stage methods that are either based on topdown paradigm or bottom-up paradigm, we also compare RepParser with other representative single-stage methods that are applied for other instance recognition tasks, including DETR (Carion et al. 2020), Deformable DETR (Zhu et al. 2021) and condInst (Tian, Shen, and Chen 2020). Moreover, we re-implement these single-stage methods and train them under the same settings for a fair comparison, since these methods are not evaluated for multiple human parsing in original papers. In addition to standard metrics, we also measure inference time per image for each method on the same hardware if possible. Furthermore, we also report the relative proportion of time costs reduction for investigation of the efficiency of each single-stage model. MHP-v2: As shown in Table 1 Table 2: Comparison with state-of-the-art methods on CIHP validation set. The symbol " * " means that model is a reimplemented version. find that RepParser achieves very competitive parsing results, which are comparable to or higher than that of stateof-the-art methods. It is worth noting that many non-unified top-down methods such as SemaTree (Ji et al. 2020) adopt an isolated object detector to detect persons and then crop the RoIs on the original images, thus leading to high inference costs (i.e., from the input image to the parsing results). As illustrated in Fig. 3, the inference time of top-down methods, such as Mask RCNN, Parsing RCNN, and PR-RCNN, dramatically increases as the number of persons linearly increases. However, the RepParser keeps almost constant inference time, since it eliminates the prior detection and each generated kernel is very compact (i.e., 32 × 32 × 1 × 1). This suggests that the RepParser could be applied to many complex real-world scenes, such as crowded scene, while keeping stable yet high efficiency. Moreover, another major merit of the RepParser is that it does not rely on bounding boxes. As illustrated in Fig. 4(left), top-down methods only perform human parsing inside the predicted bounding boxes. As a result, some body parts cannot be parsed if the detector yield inaccurate bounding boxes. However, RepParser can well handle the body parts even outside the box (see Fig. 4(right)).

RepParser (ours) Mask R-CNN
In terms of the comparison with single-stage parsing methods, the RepParser achieves higher parsing performance than that of other single-stage methods, while maintaining competitive efficiency. This indicates that directly deriving single-stage methods from other instance tasks leads to suboptimal results since they severely ignore the instance-part contexts that are essential for accurate multiple human parsing. CIHP: Similar to experiments conducted on MHP-v2 dataset, we compare RepParser with state-of-the-art methods on CIHP validation set. Corresponding results are listed in Table 2. In line with findings from Tab. 1, RepParser also achieves competitive performance on CIHP validation set. For example, RepParser with ResNet-50 backbone has a better performance than that of bottom-up models and significantly outperforms other single-stage models under the same setting. Moreover, it also performs comparable to

Ablation Experiments
The effect of representative parts. As discussed before, the representative parts separately contribute to instance-aware kernel generation and part-aware feature generation. Thus, we choose the condInst (Tian, Shen, and Chen 2020) with ResNet-50 backbone as the baseline and gradually incorporate representative parts into the pipeline. The experimental results are summarized in Table 3. From the results, we have the following observations: First, compared with baseline, applying representative parts to predict convolution kernels leads to a significant improvement, achieving 2.8% of mIoU, 3.6% of AP p vol and 5.9% of PCP 50 higher than that of baseline. This indicates that encoding person-part context into convolution kernel is particularly important for instanceaware feature generation. Second, constructing part-aware representation via representative parts brings a stable improvement, e.g., improving the score of AP p vol from 50.7% to 51.9%. This suggests that focusing on salient areas derived from representative parts is particularly beneficial for accurate human parsing. The effect of the kernel scale. In this section, we investigate the effect of the kernel scale. Here, we consider two factors: the width of each generated convolution kernel and the depth of generated convolution kernels. Our baseline consists of two 1x1 convolutions with 32 channels and performs convolution on the 1/8 down-sampling ratio feature maps. We conduct experiments by adjusting the number of channels or varying the number of convolution layers. As reported in Table 4, the performance improves as the width increases, but it seems to be saturated when the width is set as 32. However, increasing depth has a negligible effect on parsing performance. Thus, one can conclude that simply enlarging model capacity has reached the performance bottleneck. Qualitative results. As shown in Fig. 5, RepParser can produce good parsing results which are comparable to those of two-stage methods. Furthermore, two-stage methods fail to handle identical parts appearing in an intersection of two bounding boxes (see column 4). In contrast, this has a minor effect on RepParser, as it does not rely on bounding boxes. On the other hand, estimated representative parts tend to be located on semantic parts of persons, thus benefiting the instance-aware human parsing. For more details, we refer the reader to supplementary materials.

Conclusion
In this paper, we propose a new single-stage multiple human parsing method termed RepParser, aiming at breaking the limitation of the two-stage pipeline. To achieve this goal, we utilize representative parts to generate instance-aware kernels as well as part-aware representations, thus facilitating instance-aware human parsing. Extensive experiments on two benchmarks prove the effectiveness of our method.

Details of training schedule
In general, a good initialization of models leads to better performance. Thus we explore the impact of initialization methods on multiple human parsing. As shown in Tab 6, human parsing methods pre-trained on the COCO keypoint dataset can improve AP p vol by 1%. It indicates that a good initialization will lead to better parsing results.


Fuse and Mix: MACAM-Enabled Analog Activation for Energy-Efficient Neural Acceleration

Analog computing has been recognized as a promising low-power alternative to digital counterparts for neural network acceleration. However, conventional analog computing is mainly in a mixed-signal manner. Tedious analog/digital (A/D) conversion cost significantly limits the overall system's energy efficiency. In this work, we devise an efficient analog activation unit with magnetic tunnel junction (MTJ)-based analog content-addressable memory (MACAM), simultaneously realizing nonlinear activation and A/D conversion in a fused fashion. To compensate for the nascent and therefore currently limited representation capability of MACAM, we propose to mix our analog activation unit with digital activation dataflow. A fully differential framework, SuperMixer, is developed to search for an optimized activation workload assignment, adaptive to various activation energy constraints. The effectiveness of our proposed methods is evaluated on a silicon photonic accelerator. Compared to standard activation implementation, our mixed activation system with the searched assignment can achieve competitive accuracy with $>$60% energy saving on A/D conversion and activation.

INTRODUCTION
Deep neural networks (DNNs) have received an explosion of interest due to state-of-the-art inference accuracy in a myriad of artificial intelligence tasks. In parallel, the rapidly escalating model size and data volume raise a surging need for more efficient computing solutions. However, as Moore's law winds down, it becomes increasingly challenging for conventional digital counterparts to meet the computational demands of DNN workloads. A slew of new processor architectures employing analog techniques is keenly sought to reduce power dissipation and improve computational speed. Crossbarbased processing-in-memory (PIM) architectures [2,10,18,19,26] and integrated optical neural networks (ONNs) [6,8,9,20,21,28] are two prominent examples in this direction.
However, analog computing is mainly in a mixed-signal manner: digital inputs are transformed into analog signals for computation, and then computing results are converted back to the digital domain Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ICCAD ' Figure 1: Power breakdown of (a) a crossbar-based CNN accelerator ISAAC [19] and (b) a silicon photonic accelerator Mars [17]. WDACs means DACs for weights, and VDACs means DACs for vector inputs. In Mars, it chooses a more costly DAC configuration than the ADC configuration, and most of the power consumption in digital part comes from data movement.
for the downstream operations, e.g., activation. Tedious digital/analog (D/A) and analog/digital (A/D) conversion overhead exists and hinders the overall system energy efficiency [7,22,27]. Figure 1 shows two case studies of energy breakdown in two representative analog convolutional NN (CNN) accelerators, where the A/D conversion achieved by costly analog-to-digital converters (ADCs) counts for a significant portion of overall power consumption. The A/D conversion overhead concern is further escalated in ONNs due to its high-speed ADC requirements.
Based on the above analysis, we observe a solid demand for seeking another efficient and low-latency alternative to costly ADCs to bring analog signals back to the digital domain. A recently proposed memristor-based analog content-addressable memory (ACAM) [13] shows up to be a promising candidate with picosecond-level latency and femtojoule-level energy consumption. BRAHMS [22] has explored a RRAM-based accelerator to use ACAM to implement nonlinear activation, pooling, and A/D conversion successively, thus eliminating the usage of ADCs. However, the separate implementation of the three operations requires routing the analog signals back to ACAM three times, raising a signal noise concern. Moreover, a severe accuracy drop has been seen in [22] when the precision of A/D conversion implemented by ACAM is insufficient. These obstacles need to be overcome before viable ACAM can be utilized in analog computing to reduce the A/D overhead.
In this work, we devise an efficient analog activation unit based on MTJ-based ACAM (MACAM), simultaneously implementing nonlinear activation and A/D conversion in a fused manner. To compensate for the limited representation capability of MACAM, we propose a mixed activation system that integrates the proposed low-energy analog activation and the conventional high-precision digital activation datapaths. With a given activation energy constraint, a SuperMixer training flow is proposed to automatically learn how to assign activation workloads on the mixed activation system, aiming at balancing the expressiveness and energy cost.
Our main contributions are as follows, • We propose a novel analog and mixed activation system for energy-efficient neural network acceleration. • We devise a fused analog activation unit based on MACAM that can simultaneously achieve nonlinear activation and A/D conversion with significant energy reduction. • We propose a mixed activation system that integrates both analog and digital activation dataflows to balance expressiveness and energy efficiency. • We develop a SuperMixer framework to automatically learn how to assign activation workloads on our mixed activation system adaptive to various energy constraints. • A learnable nonlinearity threshold is proposed with an enhanced training recipe to boost accuracy under limited MACAM resolution, allowing practical application of very nascent technologies to use their benefits and mitigate their present challenges. • We evaluate our methods on a photonic accelerator. Regarding the energy cost of A/D conversion and activation, experiment results show that using fully analog activation units gives ∼65% energy saving, and the searched assignment on the mixed activation system can achieve >60% energy reduction with comparable accuracy.

PRELIMINARIES 2.1 Analog Content-addressable Memory
Memristor-based analog content-addressable memory is recently proposed in [13]. Fig. 2(a) shows the single ACAM cell design that supports search in a continuous analog interval. The lower bound (LB) and upper bound (UB) of the match interval are represented by tuning the resistance of two memristor devices, M1 and M2. The ACAM cell can take an analog voltage as the input, which is applied to the data line (DL). DL is connected to the gate of two switching transistors, S1 and S2. When the input is smaller (larger) than the LB (UB), both S1 and S2 are OFF (ON), making T1 (T2) ON. Then the match line (ML) will be pulled down, leading to a mismatch. If the input is within the interval, the ML will not be pulled down, resulting in a match. The prior study [13,22] has proved the ACAM can achieve the search functionality with picosecond-level latency and femtojoule-level energy consumption. We can cascade multiple ACAM cells to form an ACAM array, where each cell represents one specific interval such that all intervals can consist of a large search interval. The maximum search range is decided by the minimum and maximum resistance of the memristor device. It is obvious that the number of implementable intervals is decided by the number of available resistances of the memristor device.
Regarding the choice of the memristor device, since the ACAM acts as a role of memory with potential frequent access, it inherently requires the chosen memristor to be long-endurance. The magnetic tunnel junction (MTJ) turns out to be a more suitable choice with excellent endurance (10 15 cycles) compared with PCM (10 7 cycles) and RRAM (10 5 cycles) [18]. Among MTJs, Figure 2 Figure 2: (a) Schematic of the ACAM cell [13]. (b) The adopted three-terminal domain wall-MTJ [12].
better endurance while also providing analog resistance levels with excellent stability of the resistance levels over numerous cycles [12]. Hence, in this paper, we choose the DW-MTJ as the memristor device of ACAM. But, this is a very new MTJ type, and currently the number of resistance levels implemented on DW-MTJ is rather limited, where 5 stable resistance levels are demonstrated in [12] as the state-of-the-art (SOTA) implementation.

Related Works and Motivation
Several prior works have explored reducing the A/D conversion overhead in PIM. In [16], building fully analog circuits is proposed such that the signals are transmitted between layers without A/D conversion. This fully analog manner has been proved to be accuracy unfriendly. PRIME [2] chooses to use sense amplifiers (SAs) to do A/D conversion instead of ADCs. But a SA can only convert one bit at a time, leading to a long latency to get the whole output. CASCADE [5] proposes to accumulate the partial sums in the analog domain by connecting the outputs of multiple crossbars via an additional buffer RRAM array. However, the last-mile A/D conversion is kept to convert the sum back to the digital domain for downstream tasks, e.g., activation. Typically, in analog computing, only matrix multiplications (MMs) are conducted in the analog domain, while nonlinear activation function and pooling are implemented in the digital domain. Thus, A/D conversion is necessary. BRAHMS [22] proposes to put all the MMs, activation, and pooling in the analog domain, where ACAMs implement the latter two in two stages. In this way, only those analog values which are kept after activation and pooling need to be converted to the digital domain. The A/D conversion is still done by ACAMs. In this case, costly A/D conversion can be reduced by using the efficient ACAM. However, it is prone to signal noise since the analog signals need to be routed through ACAM three times to do nonlinear projection, pooling, and AD conversion. Moreover, severe accuracy drops are observed when the precision of A/D conversion implemented by ACAM is insufficient. Therefore, in this paper, we devise a MACAM-based analog activation unit. It is capable of implementing nonlinear activation and A/D conversion at one time in a fused manner. It can replace the traditional digital activation path, i.e., costly ADCs and digital activation units. As a result, the overhead of A/D conversions can be reduced. Since choosing MTJ as the memristor device will raise the precision issue as well, a slew of efforts are dedicated to improving the expressiveness of this device type.

MACAM-ENABLED FUSED ANALOG ACTIVATION UNIT
Nonlinear activation functions in analog computing are normally implemented by digital logics or lookup tables in the digital domain.
Since the computation is conducted in the analog domain, analogto-digital conversion overhead exists to bring analog signals to the digital domain for activation. We denote this conventional activation implementation as electrical digital activation, which includes the needed ADCs and the following digital logics or lookup tables. Recently, BRAHMS [22] has explored the idea of using ReRAMbased ACAM to implement nonlinear activation functions and A/D conversions, benefiting from the high-speed possessing and low power consumption of ACAM. However, the nonlinear projection and A/D conversion are separate. The analog signals being nonlinearly projected need to be routed back to the ACAM to do the A/D conversion, which may suffer from signal noise. Moreover, severe accuracy drops are observed when the precision of ACAM-based A/D conversion is not sufficient.
In this paper, we propose to implement the nonlinear activation function using MACAM, where nonlinear activation and A/D conversion are simultaneously achieved by the same MACAM array in a fused fashion. The fundamental idea here is to utilize the analog search functionality of MACAM to digitize the analog input signals while introducing in-situ nonlinearity. We call this fused nonlinear activation unit an analog activation unit. In this way, activation energy cost can be largely eliminated, as most ADCs can be replaced by energy-efficient MACAM.
Considering the precision issue, instead of supporting general nonlinear functions as in [22], we choose to implement the ReLUfunction. ReLU-is widely used in quantized models that have limited bit-widths to represent weight and activation. The ReLUfunction with a clipping threshold works as follows, where X is the pre-activation feature map and X is the final output feature map. The clipping threshold can bound the output of the ReLU function, therefore, a small bit-width is capable of representing the bounded value range of feature maps.
For ReLU-, since the input values can be either positive or negative, we need two MACAM arrays to handle positive and negative inputs, respectively. The negative part of ReLU-is easy to implement by generating a constant digital result '0'. Fig. 3 illustrates the implementation for the positive part of ReLU-in a single MACAM array. In the -th row of the MACAM array, by tuning the resistance of the MTJ, one MACAM cell represents an acceptance search interval [ −1 , ) for a match. All the intervals of MACAM cells in the top rows consist of a searchable range of [ 0 , ). is decided by the number of implementable resistances of the MTJ. To achieve the search interval [ , ∞], we encode ∞ similar to [22] by using a resistor with considerably larger resistance than the resistance upper bound of the MTJ. It is sufficient to cover the possible maximum voltage output from the analog engine as it cannot be arbitrarily large. The match result of all match lines consists of a one-hot vector. After obtaining the one-hot match result, we then transfer it into a binarized index using a digital priority encoder such that it can be stored at a minimized memory storage cost. The digital priority encoder can also handle corner cases when input is on the bounds of interval. Each binarized index represents a group of inputs that falls into one specific interval. Similar to quantization, we set the binarized index to correspond to one shared value, in which we use the mid-point of the interval's lower bound and upper bound in our implementation. For interval [ , ∞], the corresponding value is set to . This mapping relationship can be defined as a codebook Q as shown in Fig. 3. Within this process, we actually digitize the analog input signals. For example, in Fig. 3, the input falls into [ 1 , 2 ), the one-hot match result corresponds to the binarized index '1', which is treated as 1 + 2 2 . Since the minimum voltage 0 cannot be 0, the input needs to be biased to support the search of inputs starting from 0. In the following, for simplification of illustration, we assume 0 = 0 and = .
Now we explain how the projection of the MACAM array can realize in-situ nonlinearity of ReLU-. Figure 4(a) show the transformation behavior of the MACAM array. The range [0, ] is divided into four intervals, represented by the MTJ with 5 resistance levels using the measured data from [12]. The inputs within each interval are treated as the same value after projection based on the Q. If we draw the red dashed line through the interval's midpoint, which is labeled by the star symbol, it implements = . Actually, we can treat this as a nonlinear quantized version of a ReLU-function.
The MACAM array can only support a fixed searchable range of [0, ]. In order to implement any arbitrary ReLU-, we need to first scale the X with to fit the search range, feed the scaled input into ACAM, and then scale it back to [0, ], as shown in Fig 4(b). The first scaling operations before the MACAM array can be done using OpAmps. We can use shared OpAmps at the output of the computation units without extra overhead [23]. The off-chip computers can do the second scaling operation.
All in all, the behavior of MACAM-based activation can be modeled as follows, where Π Q indicates the discretized projection function of the ACAM within [0, ] following the codebook Q. ⌊·, ⌉ denotes clipping to the range of [0, ].

MIXED ACTIVATION SYSTEM 4.1 Proposed Mixed Activation System
As the DW-MTJ can currently only demonstrate a small number of resistance levels, the number of implementable intervals in MACAM is restricted, further limiting the representation capability of the MACAM-based analog activation unit. Hence, we are motivated to provide a mixed activation system, which integrates both the fused analog activation and the traditional digital activation datapaths. The traditional digital activation datapath first uses high-precision ADCs to do A/D conversion, then performs nonlinearity in dedicated digital activation units. The output feature map can choose either of the two paths for nonlinear activation and A/D conversion. With the mixed activation system, we can jointly utilize the lowenergy analog activation datapath and the high-precision digital datapath to balance expressiveness and energy efficiency. Figure 5 shows the system architecture overview with the tensor computing system and the proposed mixed activation system. We represent the fundamental tensor computing unit as vector-dotproduct (VDP) unit, with each unit supporting the dot-product between two length-vectors. Multiple VDPs implement a largesize vector product due to the limited single VDP size, generating multiple partial sums (PS). Note that nonlinear activation is not applied to any intermediate PS but to the final computation results, thus applying our MACAM-based analog activation requires inplace partial sum accumulation in the analog domain, which is doable by partial sum summation units [5,22,24]. Then, the final results are assigned to either the analog activation or the digital activation datapath to do nonlinear activation and A/D conversion.
We elaborate a weight-stationary dataflow to meet the requirement of the overall system following [22]. Take a convolutional layer as an example, where the 2-D weights ∈ R ×( 2 ) and 2-D output feature map X ∈ R ×( ′ ′ ) . The size of vector-dotproduct in its computaion is 2 , which is distributed onto ⌈ 2 ⌉ VDPs. This is doable since the largest vector product in modern ResNet and VGG models is only (512 × 3 × 3). At each cycle, we obtain the convolution result of the whole dot-product, ensured by in-place partial sum summation units. Fig. 6 illustrates our dataflow on a simple filter with 2 input channels. The weights can be stationary in the VDP units to continually complete the convolutions of adjacent sliding windows. Instead of frequently fetching inputs from costly memory, inputs can be reused in the convolutions of adjacent sliding windows. After obtaining the whole result, it can be sent to the activation datapath to do A/D conversion and activation.

Fully Differentiable SuperMixer Training
Our mixed activation system provides a more costly but higher precision digital activation datapath to compensate for the expressiveness due to analog activation datapath. This intuitively raises the question, "how do we assign the activation tasks of output feature maps onto the two different activation datapaths to balance expressiveness and energy efficiency?" Problem Formulation. Considering the energy gap between the two activation datapaths, our target is to assign the activation task of each value of output feature map ∈ X to either of the two activation paths with high expressiveness under a given activation energy constraint. We define the assignment as A. In this way, the problem can be formulated as follows, The binary selection variable , decides that each output feature value ∈ X in the -th layer is passed through either 1 (·) (MACAM-based analog activation datapath) or 2 (·) (digital activation path). The selection variables consist of the assignment A, which is our primary search target with a activation energy cost (A) satisfying its constraints [ ]. Search Space Specification. Considering the on-chip routing issue and computation regularity, it is not realistic to tediously assign individual output feature map value to different activation units. Instead, in this work, we propose to assign the workloads in the filter level. Concretely, take a convolutional layer as an example. Suppose it contains filters and each filter has input channels and kernel size , where its 2-D input X −1 ∈ R ×( × ) and 2-D output feature X ∈ R ×( ′ × ′ ) . The entire -th output channel of the output feature map X is passed through either the analog activation path 1 (·) or the electrical activation path 2 (·), decided by a binary selection variable , . The behavior is given as follows, In this way, our search space is defined as the assignment A that assigns each channel of output feature maps to either the analog or the electrical activation path. For layer , the number of total combinations can be 2 . Thus, the total search space for a model with layers is extremely large, which is O ( 2 ). Fully Differentiable SuperMixer Training. Considering the enormous search space of A and its discreteness, we propose a differentiable SuperMixer training flow as shown in Fig. 7.
In the SuperMixer training, we need to optimize weight , clipping threshold in ReLU-, and assignment A. It is highly    ill-conditioned to jointly optimize all those continuous and discrete variables. Aware of this, in this work, we divide our SuperMixer flow into two phases. The first SuperMixer Warmup phase aims at stabilizing our following search stage, where only and are optimized to obtain a good initial point. The second SuperMixer Search phase aims at searching for the assignment A to boost the expressiveness. It optimizes (1) , and (2) A alternatively to avoid prohibitive co-optimization difficulty. We periodically enter the optimization of (1) and (2) with a ratio of 2:1. Moreover, during the SuperMixer Search phase, we need to satisfy the activation energy cost constraint, thus, an energy cost penalty L E is added besides the original loss L. After SuperMixer training, we fix the searched optimal assignment A ℎ and conduct variationaware training to improve the model's robustness regarding on-chip variations, e.g., the resistance variation of memristors in MACAM. Now we explain how to optimize assignment A in a differentiable way. As shown in Eq. (4), the selection variable , is a binary variable. Instead of searching A in such a discrete space, we relax the optimization problem by constructing a stochastic mixed activation unit. During the inference, the activation of the -th channel of feature in layer , X , is using either the analog activation path 1 (·) or the digital activation path 2 (·) with the sampling probability of Equivalently, the output of the stochastic mixed activation, X , can be expressed as, where , is a random variable in {0, 1} and is evaluated based on the sampling probability in (5). Therefore, through parameterizing the probability distribution of activation unit choices by the sampling coefficient , we can relax the problem as the optimization of probability of . However, we cannot propagate the gradient back through the discrete variable , to , . To sidestep this issue, Gumbel-Softmax (GS) trick is adopted to relax , to be a continuous varible as follows, , follows the Gumbel distribution Gumbel(0, 1) as a Gumbel noise. A temperature parameter is used to control the GS function. As is close to 0, the Gumbel-Softmax function approximates categorical samples based on (5). A larger introduces randomness to encourage exploitation of the assignment A. Therefore, during the SuperMixer Search phase, we gradually decay such that SuperMixer can automatically exploit the search space and learn the optimal assignment to augment A the models' expressiveness. Activation Energy-Constrained Optimization. The assignment A directly impacts the activation energy cost, as it defines the mixed way of using low-precision analog activation units and highprecision electrical activation units. Fully using analog activation datapath results in an ADC-free system to convert computation results to the digital domain, while fully using electrical activation datapath raised a serious A/D conversion cost. With the assignment A, we can get the activation energy cost (A) as follows, # and # denote the number of output feature maps being passed through the analog (anlg) activation datapath and the digital (digi) activation datapath, respectively. The and represent the energy cost of two datapaths. The former indicates the MACAM array's cost, and the latter contains ADC and digital activation unit costs. As the ADC cost is far larger [19] and the digital activation unit cost is dependent on the digital part's frequency, we use ADC cost to represent during the search phase. To honor the energy cost constraint [ , , , ], we add a probabilistic activation energy cost penalty term L E , We set the margin to 5% to tighten the constraint. Note that , in Eq. (8) is differentiable guaranteed by the GS trick. Precision-Adaptive Optimization. Implementing ReLU-can ease the issue of insufficient precision of MACAM. However, it is not enough to address this since the SOTA MTJ can only provide five stable resistance levels, as mentioned before. The precision of ReLU-implemented by ACAM is only around 2-bit. Thus, a solution is in great need to further tolerate the precision issue.
Instead of choosing a fixed in Eq. (1), inspired by PACT [3], we propose to adopt a learnable with an enhanced training recipe to accommodate the low resolution of MACAM for accuracy boost. This can be done by configuring the gain of the OpAmps [23]. In [3], during the learning process of , the gradient to is computed by X = Sign(X) when X ≥ . Activation X that is smaller than cannot contribute to the gradient, resulting in inaccurate gradient estimation to . Instead of updating in the same way of [3], we re-formulate the gradient to based on Eq.(2) as follows, where our scaling operation works as a reparameterization trick to preserve the gradient contribution from X ∈ [0, ). Thus, it can correct the inaccurate gradient estimation to , which is proved to get a significantly larger accuracy boost than [3] in our experiments.

CASE STUDY: PHOTONIC ACCELERATOR
We demonstrate our design principle on a photonic accelerator as a case study. We focus on a SOTA incoherent photonic design based on micro-ring (MR) resonators [24,25]. Other accelerators can also benefit from our methods. For example, we can simply replace the ACAM design in a RRAM-based accelerator [22] with ours but own better tolerance to signal noise and the precision of ACAM. Fig. 8 shows the adopted photonic vector-dot-product (VDP) engine [24,25] and photonic in-place partial sum (PS) summation unit [24] for our system in Fig. 5. The computation of the convolutional (CONV) layer and fully-connected (FC) layer are unfolded to matrix multiplication, where each vector dot product is implemented by the vector dot product (VDP) units based on micro-ring (MR) resonators. To support two length-vector multiplication within each VDP unit, the size vectors are decomposed into small-sized vector chunks. Each small-sized vector dot-product is performed using MRs in each arm of the VDP unit. In this way, a large × vector dot product can be achieved in one unit, e.g., 100 × 100 in [24]. Across VDP units, photonic partial summation units are used to accumulate the partial sums from multiple VDPs. The partial sums from multiple VDPs are converted from the analog domain to the photonic domain by VCSELs, multiplexed into one single waveguide, and summed via another photodetector. Energy Modeling of the A/D Conversion and Activation Cost. Here, we model the energy cost of the A/D conversion and activation, / + based on O-HAS [14]. Our mixed activation system provides two datapaths to do nonlinear activation and A/D conversion, with an extra photonic summation unit overhead. Consider one convolutional layer with filters and output feature map X ∈ R ×( ′ × ′ ) . Each filter has input channels and a kernel size of . The energy model of / + can be modeled as follows, denotes the energy of the analog activation datapath, , and , denote the A/D conversion and activation energy of the digital activation datapth. and denote the energy for VCSEL and photodetector in the photonic PS summation unit.
In conventional implementation, the A/D conversion is done right after obtaining partial sum and the digital activation datapath is fully used. The energy cost can be modeled as, 6 EXPERIMENTAL RESULTS

Experiment Setup
Models and Datasets. We evaluate our methods on two modern CNNs (VGG13 and ResNet18) and CIFAR100 [11], with 6-bit weight precision. In VGG13, we replace the last three FC layers with one to avoid over-fitting. In ResNet18, we move the residual path after activation such that no extra addition is needed before activation. Training Settings. In SuperMixer training flow, we train for 90 epochs using an SGD optimizer with an initial learning rate (lr) of 0.02, a momentum of 0.9, and a cosine lr scheduler. The Gumbelsoftmax temperature exponentially decreases from 5 to 0.5. is set to 0.6 for the activation energy cost penalty. The initial of the adaptive ReLU-is set to 8. We set 10 epochs for SuperMixer Warmup phase and 80 epochs for search phase. We sample an assignment A from the learned distribution , then enter the variation-aware training. We train all models for 200 epochs during variation-aware retraining using an SGD optimizer with an initial learning rate (lr) of 0.02, a momentum of 0.9, and a cosine lr scheduler. MACAM and ADC Designs. We consider two MACAM designs with two MTJ devices in this paper. MACAM-1 uses the DW-MTJ with 5 resistance levels with around 2-bit precision. MACAM-2 uses the DW-MTJ with 3 resistance levels with around 1-bit precision. We choose two 6-bit ADC designs shown in Table. 1. Noise Injection. We set the variation of MRs following a Gaussian distribution N (0, 0.05 2 ). For MACAM, we set the MTJ resistance device-to-device variation as N (0, 0.128 2 ) [15], run Monte Carlo simulation 10 4 times to capture the noisy distribution of MACAM's intervals, and equivalently add it to the input following [22]. A/D and Activation Energy Simulation. The performance and power dissipation of MACAM are evaluated with Cadence ADE and spectre simulations. We estimate the A/D conversion and activation energy by Eq. (11) based on O-HAS [14]. The size of VDP is set as 128. Since the final FC layer is the output layer without activation, for a fair comparison, we don't include its energy consumption.

Main Results
Evaluation of Our Precision-adaptive ReLU-. In Table 2, we validate the expressiveness of our proposed precision-adaptive ReLU-on VGG13. We compare it with the adaptive ReLU-in PACT [3] and the ReLU-with a fixed . For the ReLU-with fixed , the commonly used ReLU6 is adopted, while ReLU2 is adopted in the extremely low activation bit-width. Under 6-bit weight bitwidth, our implementation achieves the highest accuracy, especially on low activation bit-width cases. This attributes to our proposed precision-adaptive optimization scheme, which learns the to accommodate the low resolution. Compared to PACT [3], our enhanced training recipe can correct its inaccurate gradient, resulting in a better accuracy boost. Hence, our adaptive ReLU-can address the accuracy drop issue in [22] due to the low activation bit-width. It is essential to boost the expressiveness of the analog activation unit so as to achieve competitive model accuracy.
Evaluation of Our SuperMixer. We search the assignment A with the proposed SuperMixer flow on different MACAM designs, ADC designs, and activation energy constraints. We denote searched assignments as searched A 0 to searched A 3 . Table 3 and 4 show the test accuracy of searched assignments on VGG13 and ResNet18. Our searched assignment series show improved expressiveness in terms of accuracy with constrained activation energy cost. Especially, given a large activation energy cost budget on MACAM-1, our SuperMixer can find an assignment with comparable accuracy to the case fully using high-precision digital activation. In conclusion, our SuperMixer flow successfully utilizes the provided mixed activation system to boost the expressiveness. Energy Saving on AD Conversion and Activation. We simulate the energy cost of A/D conversion and activation cost on VGG13.
Configurations of ADC-1 and ACAM-1 are adopted. The conventional implementation of using ADCs for A/D conversion of partial sums and using digital activation datapath consumes 35.7 J. With photonic summation units to eliminate A/D conversion needs of partial sums, fully using digital activation datapath consumes 17.24 J with a 51.7% reduction, and fully using analog activation datapath uses 12.44 J with a 65.2% reduction. Our SuperMixer enables the mixed use of electrical and analog activation with a trade-off between energy cost and expressiveness, where the searched-A 3 consumes 14.2 J with 60.2% reduction but comparable accuracy. Searched Layer-wise Assignment A. We further validate the success of SuperMixer to learn a good assignment A for boosting expressiveness. Fig. 9(a) and Fig. 9(b) visualize the ratio of feature map channels being assigned to digital activation path in each layer of VGG13 and ResNet18. Under different activation energy cost constraints, SuperMixer learns a similar tendency to assign the workloads. In former layers, the size of each channel of the feature map is larger, thus fewer channels are assigned to   the electrical activation units to avoid violation of the energy constraint. In contrast, more channels are assigned to the electrical activation units in the latter layers to boost the model accuracy. This demonstrates our SuperMixer flow can automatically learn the optimized assignment with boosted expressiveness under energy constraints. Energy Penalty Curve. In Fig. 10(a), the activation energy (A) is visualized along the SuperMixer training process.
(A) is well-bounded in the given constraint. It continues the exploitation of the search space and converges with the temperate of the Gumbel Softmax approaching 0. Noise Robustness of Searched Models. In Fig. 10(b), we evaluate the variation-robustness between searched models and model using fully analog activation units. With the increasing variation on MACAM, our searched models show better noise robustness since of the involvement of electrical activation units.

CONCLUSION
In this work, we propose a novel analog and mixed activation system for energy-efficient neural network acceleration. We first devise a fused analog activation unit based on MACAM that is capable of achieving nonlinear ReLU-and A/D conversion simultaneously, with superior energy efficiency to conventional digital activation implementation. We further integrate both the analog and digital activation dataflows to create a mixed activation system. A SuperMixer training flow is developed to automatically learn how to assign activation workloads to the low-energy analog activation datapath and high-precision digital activation datapath, aiming at a balance of expressiveness and energy efficiency. Our proposed methods are evaluated in a silicon photonic accelerator case study. Compared to the standard activation implementation, our mixed activation system with the searched assignment can achieve competitive accuracy with >60% energy saving on the overall A/D conversion and activation energy cost. Our MACAM-enabled analog and mixed activation system is viable to break through the curse of A/D conversion overhead in analog computing.