extfieldsofstudy:
Computer Science
Medicine

text:
Generative Convolution Layer for Image Generation

This paper introduces a novel convolution method, called generative convolution (GConv), which is simple yet effective for improving the generative adversarial network (GAN) performance. Unlike the standard convolution, GConv first selects useful kernels compatible with the given latent vector, and then linearly combines the selected kernels to make latent-specific kernels. Using the latent-specific kernels, the proposed method produces the latent-specific features which encourage the generator to produce high-quality images. This approach is simple but surprisingly effective. First, the GAN performance is significantly improved with a little additional hardware cost. Second, GConv can be employed to the existing state-of-the-art generators without modifying the network architecture. To reveal the superiority of GConv, this paper provides extensive experiments using various standard datasets including CIFAR-10, CIFAR-100, LSUN-Church, CelebA, and tiny-ImageNet. Quantitative evaluations prove that GConv significantly boosts the performances of the unconditional and conditional GANs in terms of Inception score (IS) and Frechet inception distance (FID). For example, the proposed method improves both FID and IS scores on the tiny-ImageNet dataset from 35.13 to 29.76 and 20.23 to 22.64, respectively.

Introduction
Generative adversarial network (GAN) [8], which allows to synthesize data distribution from a given prior distribution, has achieved great success for various applications such as text-to-image translation [27,13], image-to-image translation [15,4,46], and image inpainting [42,29,32]. In the original setting, GAN consists of two different networks called generator and discriminator: the generator is trained to construct the real data distribution to fool the discriminator, whereas the discriminator aims to classify real samples from synthetic ones which are generated by the generator [38]. Since the generator and discriminator are trained to minimize opposing goals, i.e. min-max game between them, training GAN stably is more difficult compared to supervised learning-based convolutional neural network (CNN) [45]. Therefore, a recent line of study focuses on moderating the unstable training problem. Several works [43,16,44,2] presented novel generator and discriminator architectures. These methods generate high fidelity images that easily fool humans, but they cannot fully overcome the unstable training issue of GAN [26].
The sharp gradient space of the discriminator is the most popular reason for the unstable GAN training [38]. To mitigate this issue, some papers [23,9,18,45] proposed regularization or normalization techniques that prevent the discriminator from making sharp gradients during the training procedure. As a regularization strategy, conventional methods [9,37,35,18,28] usually add regularization terms into the objective functions of adversarial learning. For instance, the gradient penalty-based regularization methods [9,37,35], which compute the gradient norm as the penalty term, have been widely used for regularization techniques. On the * Corresponding author spark.cbnuh@gmail.com (S. Park); ygshin@hnu.kr (Y. Shin) ORCID(s): other hand, normalization-based methods [23,1,20] also have been widely studied. Spectral normalization (SN) [23] is the most popular one that imposes the Lipschitz constraint on the discriminator around one by dividing weight matrices with the largest singular value. Since these regularization and normalization techniques successfully make the training procedure stably, most recent studies use these techniques in their applications.
On the other hand, some researchers explored architectural modules to enhance the GAN performance. Miyato et al. [24] proposed a conditional projection module that employs the conditional information by projecting the conditional vector to the feature vectors. This approach significantly improves the quality of the class-conditional image generation. Zhang et al. [43] introduced a selfattention unit which guides the generator and discriminator where to attend. The self-attention unit can improve the GAN performance, but it requires high computational costs to produce the self-attention map due to the matrix multiplication operation. Yeo et al. [40] introduced a simple yet effective technique, called a cascading rejection module, which produces dynamic features in an iterative manner at the last layer of the discriminator. Recently, Park et al. [25] proposed a novel residual block, called a generative residual block (GRB), which contains an additional side-residual path. By emphasizing the essential feature and suppressing the unnecessary one in the side-residual path, the GRB effectively decodes the latent feature and produces a highquality image.
However, there have been only a few attempts to redesign the standard convolution (Conv) to be effective for GAN training. Sagong et al. [30] proposed a novel conditional convolution (cConv) which incorporates the conditional information into the convolution operation. This approach effectively boosts the conditional GAN (cGAN) performance, Figure 1: The illustration of the major problem of the generator. The target image data is composed of two different modals, i.e. cat and dog images.
but it has a limitation: cConv only can be used for cGAN scheme since it is designed for replacing the conditional batch normalization. Recently, Park et al. [26] proposed perturbed convolution (PConv) that not only improves the GAN performance but also prevents the discriminator from falling into the overfitting problem. However, since PConv is developed for the discriminator, it is hard to apply to the generator.
Indeed, most state-of-the-art GAN studies [23,24,2,43] construct their generator architectures by stacking multiple residual blocks [10,11] composed of Conv, batch normalization (BN) [14], and activation function. More specifically, these methods usually employed the latent vector such as the Gaussian random noise vector as the input of their generators. Thus, to produce different images according to the input latent vector, the generator should learn dynamic features that consider the input latent vector. However, since Conv uses the shared convolution kernel without considering the input latent vector, it is difficult to learn the dynamic features. We agree that even though Conv has this problem, it has shown fine performance in the GAN study. However, we believe that there would exist a novel convolution technique that is more suitable for the generator.
To prove our hypothesis, in this paper, we present a novel convolution method, called generative convolution (GConv), which effectively improves the GAN performance by learning the dynamic features considering the input latent vector, i.e. latent-specific features. Different from Conv, GConv contains a simple kernel converting module that produces kernels representing the latent-specific features. More specifically, the kernel converting module first chooses useful kernels which are compatible with the input latent vector, and then linearly combines the selected kernels to produce latent-specific kernels. This approach is simple but surprisingly effective. First, with slight additional hardware costs, the GAN and cGAN performances are significantly improved. Second, since GConv has the same external structure with Conv, it can be easily implemented to the existing generators without altering the network architecture. To prove the superiority of GConv, we provide extensive experiments using various datasets such as CIFAR-10 [19], CIFAR-100 [19], LSUN-Church (LC) [41], CelebA (CA) [21], tiny-ImageNet (TIN) [7,39], and CelebA-GQ [21,16]. Quantitative evaluations show that GConv significantly boosts the GAN and cGAN performances in terms of Inception score (IS) and Frechet inception distance (FID).
Key contributions of our paper are summarized as follows: First, we propose a novel convolution operation, i.e. GConv, which is a direct replacement of the standard convolution. Second, extensive experiments on various datasets reveal that GAN and cGAN trained with GConv outperform those trained with Conv in terms of FID and IS. For instance, the proposed method improves FID and IS scores on the TIN dataset from 35.13 to 29.76 and 20.23 to 22.64, respectively.

Preliminaries
This section briefly introduces GAN and points out the limitations of existing methods. In general, the adversarial learning between the discriminator D and the generator G is formulated as where and are the objective functions for the and , respectively. In addition, x means a sample from the real data ( ) and z indicates a latent vector sampled from latent distribution ( ) such as Gaussian normal distribution. In this setting, is guaranteed to generate ( ) if D is always optimal. However, due to various issues such as gradient vanishing or explosion problems, this objective function is often fail to train both networks stably. To alleviate this problem, various studies have reformulated Eqs. 1 and 2. Mao et al. [22] proposed a least square errorbased objective function (LSGAN), whereas Arjovsky et al. [1] employed Wasserstein distance between real and generated samples to build the objective function (WGAN). The current widely-used practice is training the GAN with the hinge adversarial loss [40,24,23,2,26,29,32,3]. The hinge adversarial loss is formulated as To further improve the GAN performance as well as training stability, many studies have explored novel network architectures or normalization techniques. However, these methods still have a common issue with the generator. In detail, the generator is trained to synthesize the target data distribution, i.e.
( ). Thus, if ( ) is a single-modal distribution, the generator makes samples easily following ( ). However, if ( ) consists of multiple modals, the generator is difficult to successfully generate all modals in ( ). A major reason causing this problem is that the generator employs the shared convolution kernels without considering the input latent vector, i.e. .
For ease of understanding, we prepare an example [25]. Let us consider where ( ) is a distribution that is composed of two classes of images, i.e. dog and cat images, as shown in Fig. 1. In our example, we assume that the left and right sides of the latent space are mapped to the dog and cat images, respectively. To produce the visually plausible images, the generator should learn features specialized for each class. However, since the generator used the shared convolutional kernels, it is difficult to produce dynamic features specialized for each class. That means it is needed to learn features that consider for generating the diverse images well. To alleviate this problem, this paper proposes a novel convolution technique which is effective to GAN.

Generative Convolution (GConv)
This paper presents a novel convolution operation, i.e. GConv, which produces latent-specific features. In GConv, there is a simple kernel converting module that modulates convolution kernels for learning the latent-specific features. = ℎ × × in which ℎ , and indicate the kernel height, width, and input feature dimension, respectively, and n means the number of kernels. In the kernel converting module, the network first selects kernels by multiplying the scaling value which is computed as follows: where ∈ ℝ 1× is the input latent vector with dimension and ∈ ℝ 1× is a mean vector obtained by applying channel-wise global average pooling in the input tensor.
∈ ℝ ( + )× is a weight matrix trained for inferring the scaling parameters S = { 1 , ..., } ∈ ℝ 1× . In addition, and ⊙ indicate a sigmoid function and concatenation operation, respectively. Using Eq. 5, each GConv individually computes the scaling parameters in the range zero to one. By multiplying to each kernel , the generator pays less attention to the useless kernels needed for producing images. In other words, by using S, the generator could build a set of selected kernelsK ∈ ℝ × that considers the latent vector. After buildingK, the latent-specific kernelsK ∈ ℝ × can be produced by linearly-combining theK, which is defined aŝ where W L ∈ ℝ × is the weight matrix trained for producinĝ K. It is worth noting that all operations in Eqs. 5 and 6 are simple linear algebraic computations; the proposed method requires a small number of additional computational costs, i.e. W s and W L . Using the latent-specific kernels, the output features of GConv is naturally computed as follows: where X ∈ ℝ ×ℎ× × and Y ∈ ℝ ×ℎ× × are input and output features, and b, h, and w indicate the sizes of batch, height, and width, respectively. Here, Eq. 7 can be reformulated by replacing thê term as follows: where Diag(⋅) is a function creating a diagonal matrix from the given vector. It is worth noting that the second term in Eq. 8, i.e. X * KDiag(S)W L , can be easily represented with the neural network operation. More specifically, the Diag(S) multiplication acts the same role with element-wise multiplication, whereas the W L multiplication is equivalent to 1 × 1 convolution operation. Based on these observations, we draw a block diagram of GConv as depicted in Fig.3. This block diagram has the mathematically same meaning with Eq. 8. Therefore, it is easy to implement GConv by using deep-learning libraries such as Tensorflow or Pytorch.
To clarify the effectiveness of the proposed method, we trained a GAN with a simple network architecture consisting of multiple fully-connected layers. In our experiments, eight two-dimensional (2D) Gaussian mixture models (GMMs) are used as the real data distribution. For a fair comparison, in the generator, we only replaced the standard fullyconnected layers with the GConv. Fig. 4 illustrates the experimental results. The generator with standard fully-connected layers suffers from the mode collapse problem, whereas the generator with the GConv generates all the GMMs successfully. These results demonstrate that the proposed method is effective to train GAN. More extensive experiments will be presented in Sec. 4.2.
Indeed, one may anticipate that GConv is similar to the conventional channel-attention (CA) methods [36,34], which emphasizing the selected channels of the input feature. However, there is a major difference between the proposed method and conventional ones: the proposed method is an alternative operation of convolution, but the CA technique is a post-processing algorithm used after conducting the convolution operation. More specifically, existing CA methods are usually located after performing the convolution operation to enhance the specific channels of output feature. In contrast, the proposed method directly selects and combines kernels in the convolution layer. That means it is able to use the existing CA method after performing GConv. The detailed explanations will be described in the section 4.2.

Implementation Details
To prove the superiority of GConv, we conducted extensive experiments utilizing the various datasets which are widely used for evaluating the GAN performance: CIFAR-10 [19], CIFAR-100 [19], LC [41], CA [21], TIN [7,39], and CelebA-HQ [21,16]. The resolutions of the CIFAR-10 and CIFAR-100 datasets are 32 × 32, whereas the images of LC, CA, and TIN datasets are resized to 128 × 128 pixels. Also, the CelebA-HQ dataset is used for training the networks that produce the 256 × 256 images. To train GAN, we used the hinge-version loss in Eqs. 3 and 4 as the objective function. Since all parameters in the discriminator and generator including GConv can be differentiated, we employed the Adam optimizer [17] and set the user parameters of Adam optimizer, i.e. 1 and 2 , to 0 and 0.9, respectively. For training the CIFAR-10, CIFAR-100, and CelebA-HQ datasets, we set the learning rate as 0.0002, and the discriminator was updated 5 times using different minibatches when the generator is updated once. In contrast, for training the LC, CA, and TIN, we employed a twotime scale update rule (TTUR) [12] that the learning rates of the generator and discriminator are set to 0.0001 and 0.0004, respectively. Note that the TTUR technique updates the discriminator a single time when the generator is updated once. For all datasets, we decreased the learning rate linearly in the last 50,000 iterations. For the CIFAR-10 and CIFAR-100 datasets, we set a batch size of the discriminator as 64 and trained the generator for 50k iterations. In our experiments, following the previous papers [24,23,25], the generator was trained with a batch size twice as large as when training the discriminator. That means the generator and discriminator were trained with 128 and 64 batch size, respectively. For the LC, CA, and TIN datasets, both batch sizes of the discriminator and the generator are set to 32. The network is trained to 300k iterations on LC and CA datasets and 1M iterations on TIN dataset. For the CelebA-HQ dataset, the network is trained to 100k iterations, while setting the batch sizes of the generator and discriminator to 32 and 16, respectively.
To quantify the importance of GConv, we employed a strong baseline in [23] following the state-of-the-art studies [24,2,43,25]. More specifically, we employed the generator and discriminator architectures constructing with multiple residual blocks (RBs) as our baseline models; we just replaced Conv in the generator with GConv and did not alter the architecture of the discriminator. Note that we used   Fig. 5. For training the CIFAR-10 and CIFAR-100 datasets the spectral normalization (SN) [23] is only used for the discriminator, whereas the SN is applied to both generator and discriminator for training LC, CA, and TIN datasets [43]. In the discriminator, the feature maps are down-sampled by utilizing the average-pooling after the second convolution. In the generator, the up-sampling (a nearest-neighbor interpolation) operation is located before the first convolution. A detailed descriptions of the network architectures for the generator and discriminator are described in Tables 1 and 2. To train the network in the conditional GAN (cGAN) framework, following the most representative cGAN scheme, we replaced the BigBN in the generator with the conditional BigBN [2] and added the conditional projection layer in the discriminator [24]. More specifically, in the RB of the generator, we concatenated z with the one-hot vector which represents the input class, and used the concatenated vector to conditional BigBN and GConv. Note that other network architectures were the same as the models used for experiments of unconditional GAN.

Evaluation Metric
In the field of GAN, FID [12] and IS [31] are most widely used assessments that evaluate how well the generator produces image. More specifically, FID is defined as follows: where { , } and { , } are the mean and covariance of the samples with distributions of real and generated images, respectively. Lower FID scores mean better quality of the generated images. To measure the performance using FID, in this paper, we generated 50,000 images for each dataset.
On the other hand, IS is expressed as where l is the label predicted by the Inception model [33] trained for classifying ImageNet dataset [7] having a thousand classes, and ( | ) and ( ) represent the conditional class distributions and marginal class distributions, respectively. That means it measures how the generator produces diverse images with high-quality. However, IS has an one major drawback: IS is no meaning when evaluating the performance in a single class dataset such as CA, since it measures the score using the KL-divergence between ( | ) and ( ). Therefore, in this paper, we did not use IS for evaluating the GAN performance for CA and LSUN datasets.

Experimental Results
Before proving the superiority of GConv on various datasets, we first conducted extensive ablation studies on CIFAR-10 and CIFAR-100 datasets. In our experiments, the network was trained three times from scratch to prove that the performance improvement was not due to the lucky weight initialization. First, we compared the GAN performances between Conv and GConv. As described in Table 3, the proposed method exhibits better performance than Conv in terms of FID and IS. It is worth noting that the proposed method shows superior performance to Conv in the cGAN as well as GAN. That means the GConv could be used for both studies for improving the performance. In addition, the proposed method consistently outperforms its counterpart, in both CIFAR-10 and CIFAR-100 datasets. These observations indicate that the latent-specific features produced in GConv effectively support the generator to produce highquality images. Indeed, owing to W s and W L in Eqs. 5 and 6, GConv needs a slightly more network parameters than Conv. For instance, when training the CIFAR-10 and CIFAR-100 datasets, the generator with Conv uses about 3.54M convolution weights, whereas that with GConv utilizes about 4.37M convolution weights. Thus, one may anticipate that the performance improvement is caused due to the increased number of network parameters. To mitigate this issue, we conducted ablation studies that match the computational costs of Conv with that of GConv. More specifically, we increased the number of output channels in Conv to match the number for convolution weights with GConv. In this paper, we referred the Conv with increased output channels as Conv*. As described in Table 3, Conv* shows performance either slightly improved or failed to improve even increasing the number of network parameters. In particular, on the CIFAR-10 dataset, Conv* trained with the cGAN framework shows a poor FID score compared with Conv. In addition, GConv still shows better performance than Conv*. These results reveal that even if the number of network parameters is increased, it is difficult to further improve the performance using Conv. In contrast, GConv significantly boosts the GAN performance successfully in both GAN and cGAN. Based on these observations, we concluded that GConv has a fine trade-off between improved performance and increased computational costs, which makes GConv to be fine for practical use.
On the other hand, one may have a query that the difference between GConv and conventional CA modules. As mentioned in Section 3.1, the major difference between the two methods is their locations in the network: GConv is a replacement of Conv, whereas the CA module is the post-processing method used after performing convolution operation. In other words, the existing CA techniques can be used after GConv. To clarify this claim, we conducted ablation studies that attach the CA module after Conv and GConv. In our experiments, we used the CA technique introduced in [36], which is the most popular one among the CA techniques. In this paper, we denoted Conv and GConv with CA module as Conv † and GConv † , respectively. The illustrations about Conv † and GConv † are depicted in Fig. 6. As described in Table 3 Conv † shows better performance than Conv, which means the exiting CA module is effective to improve the GAN performance. It is worth noting that GConv † also shows better performance than GConv. In addition, GConv † exhibits superior performance to Conv † in both GAN and cGAN schemes. These results contain two important meanings. First, GConv can be generally used as the replacement of Conv since it is compatible with the existing techniques designed for Conv. Second, GConv and the CA module act different roles in the networks; they can be used together for improving the GAN performance. Based on these observations, we concluded that it does not need to compare the effectiveness between GConv and the CA module. Therefore, in the remainder of this paper, we only compared the performance of GConv with that of Conv.
Furthermore, to show the generalization ability of GConv, we conducted other ablation studies in which the networks are trained with different experimental settings. First, we trained the networks using various adversarial objective functions: the objective function based on the cross-entropy (CE) theorem (Eqs. 1 and 2) and the objective function proposed in the least square GAN (LSGAN) paper [22]. As shown in Table 4, the proposed method outperforms Conv even trained with various objective functions. That means GConv can be generally applied to train the GAN regardless of the objective function. In addition, to check the compatibility of GConv with various activation functions, we changed the activation function in the RBs of the generator. In our experiments, we employed the exponential linear unit (ELU) [5] and gated linear unit (GLU) [6] as the activation function. As summarized in table 5, the proposed method exhibits superior performance to Conv in the various activation functions. Based on these results, we expected that the GConv can be generally used for the GAN-based image generation techniques having various experimental settings.
In order to validate the ability of GConv when synthesizing images in challenging datasets, we trained the GAN using the LC, CA, and TIN datasets. Specifically, the LC and CA datasets are trained by following the experimental settings for GAN, whereas the TIN dataset is trained by following that for the cGAN. In Tables 3,5, and 4, we already proved that the performance improvement is not due to the lucky weight initialization. Thus, we trained the network a single time from scratch. Instead, to clearly show the superiority of GConv, we drew a graph representing the FID and IS scores over training iterations, as shown in Fig. 7. The FID and IS scores of Conv are converged earlier than those of GConv; the proposed method shows better performance than Conv. We summarized the final FID and IS scores of GConv and those of Conv in Table. 6. The scores in the bracket indicate the minimum FID and the maximum IS scores during the training procedure. As described in Table 6, even when comparing the minimum FID score, GConv exhibits better performance than Conv. In particular, the proposed method significantly improves the GAN performance on the challenging dataset, i.e. tiny-ImageNet. That means GConv is effective to produce images in various datasets. Fig. 8 shows the samples of the generated images. As shown in Fig. 8, the proposed method effectively leads the generators to produce high-quality images. Moreover, to show the effectiveness of GConv for generating high-resolution images, we conducted additional experiments using the CelebA-HQ dataset [21,16]. In this paper, we trained the networks to produce 256 × 256 images. The experimental results are presented in Table 7 and Fig. 9. It is worth noting that the GConv shows much lower FID scores compared with Conv, and it produces the convincing images. Based on these results, we expected that the proposed method can utilized to produce high-resolution images. Indeed, this study does not intend to produce the design of an optimal generator and discriminator architectures for GConv. There might be another network architecture that improves the GAN performance and produces more high-quality images. Instead, this paper focuses on verifying whether it is possible to achieve the better GAN performance by simply replacing Conv with GConv.

Conclusion and Future Works
This paper has proposed a novel convolution technique for the generator, called GConv, which significantly improves the GAN performance. By simply replacing Conv with GConv, the generator is able to synthesize the visually plausible image, which results in the performance improvement of GAN. The main advantage of GConv is that it can be integrated into any generator architectures seamlessly with marginal overheads, while considerably improving the performance. In addition, this paper deeply investigated the effectiveness of GConv in various aspects by conducting  many-sided ablation studies. Thus, we expect that the proposed can be applied to various GAN-based image synthesizing techniques. However, this paper aims at introducing a novel convolution method specialized to the GAN. We agree that the GConv might work well for the GAN-based image generation technique, but might cause some issues in other networks used for different applications. Although this paper only covers the effectiveness of GConv in the field of GAN-based image generation technique, we have shown that the proposed method could improve the GAN performance significantly with various aspects. As our future work, we plan to further investigate a novel convolution method that could cover various applications.

Declaration of Competing Interest
The authors declare no conflict of interest.


Research on Synchronization Mechanism of Artificial Intelli-gence Model Based on Belief Measure and Its Application in Pumped Storage Power Station

For the security monitoring of pumped storage power station, a model synchroniza-tion mechanism for cloud edge cooperation framework is proposed. The method uses the belief function to describe the threshold and uses the ping-pong operation strategy to update the model alternately, which solves the problem of artificial intelligence model synchronization and update of edge equipment. The cloud is based on Baidu BML platform, the edge uses customized servers, and the average model update cycle is about three months.

Introduction
With the implementation of the double carbon policy, the construction of energy Internet has accelerated. Pumped storage power station(PSPS) is an important part of energy Internet. In China, about more than 60 PSPSs have been built or are under construction. "Few people on duty" is a feature of PSPS. Therefore, artificial intelligence technology has a wide application prospect in PSPS. This work proposes a model synchronization mechanism for cloud edge cooperation framework, which solves the problem of artificial intelligence model synchronization and update of edge devices.

Basic ideas
The artificial intelligence models used in the security monitoring of PSPS can be divided into two categories: (1) applications less related to the environmental background, such as not wearing safety helmets and smoking, etc.; (2) Applications related to environmental background, such as plant scaffold construction, etc. For (1), all PSPSs adopt the general model; for (2), personalized model is adopted for different PSPS. In the cloud, the basic sample warehouse and model warehouse are realized. At the same time, personalized samples are saved for different PSPS, and the corresponding models are trained. At the Edge, the customized equipment is used to realize the real-time image detection, and the model synchronization and update mechanism and sample return mechanism are designed and implemented.

Introduction to system architecture
Edge AI has the potential to benefit both the data and the network infrastructure itself. At a network level, it could be used to analyze the flow of data for network prediction and network function management, while enabling edge AI to make decisions over the data itself offers significantly reduced backhaul to the cloud, negligible latency and improved security, reliability and efficiency across the board. Another key function of edge AI is sensor fusion: combining the data from multiple sensors to create complex pictures of a process, environment or situation. Consider an edge AI device in an industrial application, tasked with combining data from multiple sensors within a factory to predict when mechanical failure might occur. This edge AI device must learn the interplay between each sensor and how one might affect the other and apply this learning in real-time.
In this paper, Baidu BML platform is used as the bottom support [1], and customized server is used to realize the edge computing support.

Near edge device
In the implementation of Near Edge device, the heterogeneous computing [2] mode is adopted, the main FPGA accelerates the common model, and the h.265 decoding card is used to improve the speed of video coding and decoding, and GPU is used to realize the image recognition based on deep learning. Furthermore, in the airborne equipment of the project, the Yolo V4 [3] framework is adopted. A large

PCI-E communication module
PCI-E is a high-speed serial point-to-point dual channel high bandwidth transmission. In many applications, PCI-E switching is the main implementation way in heterogeneous systems. Its main advantage is high data transmission rate. The highest 16X2 version now achieves 10GB/s. In fact, in general applications, high speed Ethernet technology can be used in data communication, and the advantage of using PCI-E is that the real time is guaranteed.

FPGA module
FPGA is more efficient than CPU even GPU, mainly due to its architecture without instruction and no memory sharing. There are two functions of memory in the von Neumann structure. One is the save state, and the two is the inter cell communication. Because memory is shared, we need to do access arbitration. In order to make use of the locality of access, every executive cell has a private cache, which means we need to maintain consistency between components. For the needs of saving state, registers and on chip memory (BRAM) in FPGA belong to their own control logic, without unnecessary arbitration and caching. For communication needs, the connection between each logical unit of FPGA and surrounding logic units has been determined during reprogramming, and no communication is required through shared memory. In this work, Xilinx Virtex-6 is used.

Performance optimization
In order to improve the performance, in the case of engineering application precision, this paper made three changes during the implementation process.
The decode image is completed by using the H.265 decoding card.  Fig.4 Image decoding process

Communication in the microservice
In the near edge server, it is a better choice to support endpoint devices with microservice architecture. in the microservice architecture, the patterns of communication between clients and the application, as well as between application components, are different than in a monolithic application. There are two main approaches to inter-process communication in a microservice architecture. One option is to a synchronous HTTP-based mechanism such as REST(Representational State Transfer) or SOAP (Simple Object Access Protocol). This is a simple and familiar IPC(Inter-Process Communication) mechanism. It's firewall friendly so it works across the Internet and implementing the request/reply style of communication is easy. The downside of HTTP is that it doesn't support other patterns of communication such as publish-subscribe.
In this work, we use one FPGA and six GPUs (each about 20Tflops), which can support about 128 channels of 1080p video monitoring at the same time.

3.Ping-pong operation
Ping Pong operation is a data buffer optimization design technology in FPGA development [4], which can be regarded as another form of pipeline technology. This paper uses its design idea for reference, combines it with reliability function and applies it to model updating.  Strategy of Ping Pong operation resolver (1)Define the belief function BF (which can be replaced by Bayesian formula). In case of simplification, F1 score can be used instead of this function [5].
(2)For the same target problem, two yolov4 models are trained with different sample sets without losing generality, model a and model b (3)According to the performance of model a and model b in the test set, the priority is defined (4)Model a and model b are deployed on edge devices (5)In the process of edge target recognition, when the results of model a and model b are different, the difference image is uploaded to the cloud for manual confirmation, and the confirmed results are used as the evaluation of model a and model b and accumulated.
(6)When the difference reliability function is greater than the threshold, the model with low evaluation score is updated

4.Conclusion
Based on the results and discussions presented above, the conclusions are obtained as below: (1)Ping Pong operation strategy can effectively improve the accuracy of artificial intelligence model (2)The performance of customized server is better than general server in cloud side collaborative application.


extfieldsofstudy:
Medicine
Computer Science

text:
Syntax Customized Video Captioning by Imitating Exemplar Sentences

Enhancing the diversity of sentences to describe video contents is an important problem arising in recent video captioning research. In this paper, we explore this problem from a novel perspective of customizing video captions by imitating exemplar sentence syntaxes. Specifically, given a video and any syntax-valid exemplar sentence, we introduce a new task of Syntax Customized Video Captioning (SCVC) aiming to generate one caption which not only semantically describes the video contents but also syntactically imitates the given exemplar sentence. To tackle the SCVC task, we propose a novel video captioning model, where a hierarchical sentence syntax encoder is firstly designed to extract the syntactic structure of the exemplar sentence, then a syntax conditioned caption decoder is devised to generate the syntactically structured caption expressing video semantics. As there is no available syntax customized groundtruth video captions, we tackle such a challenge by proposing a new training strategy, which leverages the traditional pairwise video captioning data and our collected exemplar sentences to accomplish the model learning. Extensive experiments, in terms of semantic, syntactic, fluency, and diversity evaluations, clearly demonstrate our model capability to generate syntax-varied and semantics-coherent video captions that well imitate different exemplar sentences with enriched diversities.

INTRODUCTION
V Ideo captioning [1], [2], [3], which aims to automatically generate natural language sentences to describe video contents, has aroused great interest recent years. Inspired by the success of the sequence-to-sequence model [4] in neural machine translation, the encoder-decoder architecture [5] leveraging CNNs and RNNs to encode video semantics and utilizing RNNs to decode sentences has become a common and effective configuration for most video captioning approaches.
Previous methods [3], [6], [7], [8], [9], [10] are mostly trained with the (conditional) maximum likelihood objective, which encourages the use of the n-grams that appeared in the training samples. Consequently, the generated sentences will bear high resemblance to training sentences in detailed wording [11]. For example, the generated sentences by the captioning models trained on MSRVTT [12] often present similar syntactic structures like "A is doing B at/on/when C". Although such generated captions can describe the video contents, the monotonous and plain sentence forms are of very limited linguistic diversity and expression variability. Compared with those captioning models, people can express things more freely and vividly in daily life. Such an ability, on the one hand, is due to that we have the basic syntactic knowledge about sentence organization. On the other hand, we can acquire abundant corpora during daily reading and talking, and learn how to express things. Therefore, is it Manuscript received XXXX; revised XXXX. possible to learn and leverage sentence syntactic information to guide the video caption generation? Or, more intuitively, can we use the human-edited sentences to customize video captions so as to strengthen the captioning variability in expression?
To answer the above questions, we propose a novel task, namely Syntax Customized Video Captioning (SCVC) in this paper. As shown in Figure 1, given a video and any syntax-valid exemplar sentence, we aim to generate one caption, which should not only express the video semantics but also follow the syntactic structure of the given exemplar sentence. Since exemplar sentences are easy to acquire with no restrictions, our proposed task provides a promising way to strengthen the diversity and expressiveness of video captioning.
In order to solve the SCVC task, our proposed video captioning model consists of three fully coupled components. Firstly, one hierarchical sentence syntax encoder is proposed to capture the syntactic information of the exemplar sentence, in specific a character-level LSTM and a word-level LSTM are stacked to characterize the local lexical features and the global syntactic structures, respectively. Then, one video semantic encoder is introduced to represent the video semantics. Finally, we design a syntax conditioned caption decoder realized in a two-layer LSTM, in which one layer relies on the exemplar syntactic information to modulate the LSTM for customizing the caption syntax, and the subsequent layer relies on the video features to modulate the LSTM for generating the syntactically structured caption describing video semantics. As we do not have the syntax customized groundtruth captions, we propose a new training strategy, which fully leverages the syntactic and semantic information residing in both conventional video captioning data and our collected auxiliary exemplar sentences to train the overall arXiv:2112.01062v1 [cs.CV] 2 Dec 2021 Father tying shoe laces of his son traveling in train.
Woman preparing food ingredients with her audience watching on TV.

Female patient watching tv and remoting control in hand in hospital bed. Woman cook slicing egg and mixing salad in bowl on wooden table.
Water splashes when a coin dropped in a glass. Egg scatters when a knife cuts at a board.

RELATED WORKS
Video captioning has been extensively studied in the past few years. From pioneering template-based methods [2], [13], [14], [15] which defined special grammar rules to compose captions, to recent sequence-to-sequence architectures which leveraged RNNs to encode videos and then decode captions sequentially [3], [6], [7], [8], [9], [10], [16], numerous improvements have been achieved. However, most of these previous approaches are trained to select words with the maximum probability sampled from the learned distribution of the training corpus, resulting in a monotonous set of generated captions bearing high resemblance to the training data. The above works mainly aim at generating precise captions to describe visual contents for keeping fidelity, inspired by the research of text style transfer [17], [18], [19], [20], [21] in the natural language processing community, some recent works for video captioning further put more emphasis on improving the expressiveness or diversity of the predicted video descriptive sentences [11], [22], [23], [24], [25], [26], [27], [28], [29]. Specifically, Gan et al. proposed a StyleNet to control the style in the captioning process so as to produce attractive captions with the desired style like romantic or humorous [25]. You et al. supplied the sentiment label as one additional dimension of the input feature, so as to make the model learn to control the injection of sentiment words in captions for sentiment-conveying [29]. Wang et al. proposed to sample a Part-of-Speech (POS) sequence based on the video representation. By manually altering the sampled POS, the subsequently predicted captions can also be modified [27]. Chen et al. proposed fine-grained control of image caption generation with an abstract scene graph, which makes the level of details such as attributes or relationships can be captured in the caption generation process [22].
Compared with using the style label/sentiment label/POS tag/scene graph to control caption generation, directly leveraging the exemplar sentences to customize the generated captions is more intuitive and natural. Moreover, since the exemplar sentences are easy to obtain and of various syntactic structures, the expressiveness and diversity of video captions can thereby be enriched through imitating those syntax-varied exemplars.

THE PROPOSED MODEL
In this section, we propose one novel model to tackle the SCVC task, with the model overview illustrated in Figure 2. Concretely, the proposed model consists of three components, namely the hierarchical sentence syntax encoder, the video semantic encoder and the syntax conditioned caption decoder. We will detail the model and demonstrate our training strategy in the following.

Hierarchical Sentence Syntax Encoder
To make one video caption imitate the syntactic structure of the given exemplar sentence, how to extract the exemplar sentence syntaxes rather than its semantics is an essential problem. Hence, as shown in the top-right part of Figure 2, we propose a hierarchical sentence syntax encoder as follows.
Specifically, suppose an exemplar sentence S = [w 1 , ..., w n , ..., w N ] with N words, in which the n-th word consists of L characters w n = [w n,1 , ..., w n,l , ..., w n,L ]. A character-level LSTM, namely LSTM c , is first performed on the sequential characters of each word: Here w c n,l is the embedding of the character w n,l in the word w n , and the corresponding hidden vector h c n,l will encode the context in the word to characterize the subword features of w n . We average the L character-level hidden vectors, and obtain the syntactic word feature w n = Avg(h c n,1 , ..., h c n,L ). As words with similar syntactic behaviors, such as part-ofspeech (POS), often have similar subword characteristics like suffix or prefix [30], our character-based word feature w n will capture the local lexical features to support the sentence level syntax encoding. Subsequently, another word-level LSTM, namely LSTM w , is stacked on LSTM c to aggregate the syntactic word features:  sentence S is obtained, which will be used to control the sentence syntax of the predicted video caption. Word Replacement Mechanism based on POS Tags. When conducting experiments, we observe that the sentence syntax encoder often tends to remember the exact words instead of learning the sentence syntactic structure. In order to make the syntax encoder put more emphasis on the sentence syntaxes and lexical word features, we propose a word replacement mechanism based on their POS tags. Specifically, we firstly use the Stanford NLP toolkit [31] to get the POS tag of each word in the sentence. Then, when feeding the sentence in the syntax encoder, we randomly replace the word in the sentence with the word of the same POS tag, and thus discourage the syntax encoder from memorizing the exact word in the exemplar sentence.

Video Semantic Encoder
For encoding video semantic features, we follow the conventional approaches which first use one pretrained CNN to encode each video frame in the input video V , and get a video feature sequence V = [v 1 , ..., v m , ..., v M ]. Then, an LSTM is leveraged to encode the video contexts, with the aggregated hidden vectors denoting the video semantic representation, which will further be utilized to provide semantic meanings of the generated video caption.

Syntax Conditioned Caption Decoder
In this section, we propose to condition the caption generation process on the encoded exemplar sentence syntaxes to achieve the caption syntax customization. The prediction of each word in the caption is on the basis of the global syntax "backbone", and the video semantics are taken as the "flesh" to replenish the caption with semantic meanings. Based on this consideration, we establish a two-layer LSTM to decode the caption, with the first layer modeling sentence syntaxes while the second layer fulfilling sentence semantics.
When decoding the t-th word in the caption, the first layer decoding LSTM, which we denote as LSTM syn , takes the concatenation of the embedding of the previous word e t−1 and the attentively summarized syntactic representation a s t = Att(h syn t−1 , H s ) as input, outputting the hidden vector h syn t to control the sentence syntax at this timestep: (3) In order to incorporate the syntactic information a s t and make it better control the recurrent word decoding procedure, we propose a new Conditional Layer Normalization (CLN) mechanism to modulate LSTM syn as shown in the bottomleft part of Figure 2. Specifically, taking a s t as the conditional guidance signal, the computation flow of Eq. (3) proceeds as: Here CLN performs the modulation on the LSTM gates which leverages a s t to conditionally scale and shift the layer normalized vectors: The scaling and shifting vectors f γ (a s t ) and f β (a s t ) are generated by two independent multi-layer perceptrons (MLPs) with a s t as the input, and the three CLNs in Eq. (4) are independent with each other and do not share weights. By equipping CLNs, the syntactic representation a s t is expected to affect the update procedure of LSTM syn , thereby achieving the caption syntax customization.
The second layer LSTM in our syntax conditioned caption decoder, which we denote as LSTM sem , has the same architecture as LSTM syn . Specifically, LSTM sem takes the concatenation of the syntax hidden vector h syn t and the attentively summarized video semantic representation as inputs, and is conditionally con- t is taken as the condition guidance signal to intensify the video contents in the word decoding procedure, making the predicted caption better preserve the video semantics. Through the above two-layer LSTM modeling, the words in the caption will be sequentially predicted based on the

Training Strategy
As we do not have the syntax customized groundtruth video captions, we propose to leverage the pairwise video captioning data in available datasets as well as our collected auxiliary exemplar sentence corpus to train our proposed model.

Training with Pairwise Video Captioning Data
As illustrated in Figure 3(a), the pairwise video captioning data is utilized for training the proposed model, with the groundtruth caption also being taken as the exemplar sentence. As such, the sentence syntax of the caption is extracted and then coupled with the video semantics to predict the caption. For the training of this syntax customized video captioning procedure, two loss terms are introduced as the learning objective, i.e., the syntactic loss and the semantic loss. Syntactic Loss. In our syntax customized video captioning, the syntactic structure of the generated caption should resemble that of the exemplar sentence. We use the Stanford NLP toolkit [32] to extract the constituency parse tree of the exemplar sentence (here is the input video caption), which is presented as a bracket syntactic token sequence. Then we collect the hidden state vectors of the LSTM syn as H syn =[h syn 1 , ..., h syn t , ..., h syn T ], feed them into a basic decoding model such as [10], and predict the corresponding syntactic token sequence of the input exemplar sentence. Accordingly, the syntactic loss can be defined as: L syn v,c is realized by the typical negative log-likelihood loss. V denotes the input video, and C denotes its accompanying caption in the dataset, which is also used as the exemplar sentence. C syn indicates the extracted syntactic token sequence of the input caption C.
Semantic Loss. Since the generated caption should express the semantic meaning of the video, we collect the hidden vectors H sem = [h sem 1 , ..., h sem t , ..., h sem T ] of the LSTM sem that directly serves for caption decoding, and adopt the negative log-likelihood loss for semantic supervision with C:

Training with An Exemplar Sentence Corpus
Since the syntactic structures of the captions in existing video captioning datasets are simple and monotonous, only relying on them to train our model will limit its ability on expression variability. To this end, we collect a large number of syntaxvaried exemplar sentences to enhance our model's learning ability. Meanwhile, as shown in Figure 3(b), we propose a sentence semantic encoder to replace the video semantic encoder in our model. Doing so will make one single sentence can also be autoencoded with the yielded architecture, and further help the training of the shared hierarchical sentence syntax encoder and syntax conditioned caption decoder. Concretely, the sentence semantic encoder relies on one fully-connected layer to encode the glove embedding [33] of each word, and the obtained word feature sequence will be taken as the sentence semantic representation. By feeding the semantic and syntactic representations of the sentence to the syntax conditioned caption decoder, we follow the same procedure in Sec. 3.3 to reconstruct the input sentence. The corresponding syntactic and semantic losses can also be defined: L syn s,s = −logP (S syn |H syn ; S, S), L sem s,s = −logP (S|H sem ; S, S).
Here S denotes the sentence we feed to the sentence autoencoding procedure, which can be our collected exemplar sentence E or the caption C in the existing video captioning datasets. S syn denotes the parse tree token sequence of S.

Overall Training Objective
The above training objective considers either feeding a (video, caption) pair or one single sentence to our model. However, there is also one case of taking one exemplar sentence E and one video V as the inputs to our model. Although we do not have groundtruth caption to support the L sem loss in this case, the L syn loss can still be involved in training since the parse tree sequence E syn of the exemplar sentence can be pre-extracted. Hence, we can obtain an additional syntactic loss as follows: L syn v,e = −logP (E syn |H syn ; V, E).
Summing up the above terms, the overall training objective of our model is defined as:

Dataset and Exemplar Sentence Collection
We conduct our syntax customized video captioning experiments on the MSRVTT [12] and ActivityNet Captions [34] datasets. We pair each video in these two datasets with 20 different syntax-varied exemplar sentences that are originally collected in the Shutterstock Image Description Corpus [35]. Accordingly, 20 different syntax customized video captions will be generated by our proposed model. The details of the datasets and our exemplar sentence collection are as follows.

Evaluation Metrics
To comprehensively evaluate the predicted syntax customized video captions, we refer to the research in the relating text style transfer task [17], [18] and conduct the objective evaluation from the syntactic, semantic and fluency aspects as follows. Syntactic Evaluation. To evaluate whether our generated captions comply with the syntactic structures of the exemplar sentences, we directly compare their constituency parse tree by computing the syntactic Tree Edit Distance (TED) [36] between them after removing word tokens. Smaller TED value means higher syntactic similarity. Semantic Evaluation. For each sentence, we first remove the stop words and then take the average glove word embeddings of the remaining words as its sentence semantic features. In the semantic feature space, the average cosine similarity (COS) between the syntax customized video caption and all the originally groundtruth video captions in the dataset is used to evaluate their semantic coherence. Sentence Fluency. Besides syntactic and semantic evaluation, another important aspect is the sentence fluency of the generated syntax customized video captions. We use a pre-trained language model to measure the perplexity of generated captions as the fluency score. A state-of-the-art BERT model [37] trained on a large scale lower case English dataset is used for the evaluation.

Implementation Details
To represent videos, we leverage the Inception-Resnet-v2 network [38] pretrained on the ILSVRC-2012-CLS image classification dataset [39] to extract a 1,536 dimensional feature vector for each frame. Videos in the MSRVTT and ActivityNet Captions datasets are represented with evenly spaced 30 and 100 features, respectively. Shorter videos of less than 30 or 100 features are padded with zero vectors. The word embedding size and all the LSTMs' hidden sizes are set as 256. Adam [40] optimizer is used for training. For the word replacement mechanism, the probability to replace the word with their same POS-tagged word is 0.7. The model is implemented with PyTorch [41]. For getting the syntactic token sequence, we first extract the constituency parse tree of the input exemplar sentence with Stanford NLP parser [32]. For example, the parse tree of the sentence " a short clip of news on a white background" is " (ROOT (NP (NP (NP (DT) (JJ) (NN)) (PP (IN) (NP (NN)))) (PP (IN) (NP (DT) (JJ) (NN)))))". We regard the sentence parse as a syntactic token sequence, where each element such as "ROOT", "NN" or "(" is an independent token in the sequence. Such a syntactic token sequence will be used for the syntactic loss as stated in Sec. 3.4.

Compared Methods
Since the proposed syntax customized video captioning is a new task, there is no baseline method for direct comparisons. In this section, we compare our proposed model with the following methods: • The ExemplarOnly method which directly outputs the given exemplar sentence as the video caption while does not consider the video contents. • The conventional sequence-to-sequence (Seq2Seq) video captioning method [10], which only predicts one single caption for one video while does not consider the sentence syntax. • The Template method, which firstly detects visual concepts [42] from each video to generate a bag of videorelated words. Then every content word in the exemplar sentence is simply replaced by one video-related word from the bag that has the same part-of-speech as the removed word. • The Gated Fusion Network (GFN) for video captioning with POS sequence guidance [27]. We replace their intermediately sampled POS sequences from videos with the POS sequences of the given exemplar sentences, and attain the corresponding syntax customized video captions. Table 1 compares our proposed model with the baseline methods in terms of the objective evaluation metrics introduced above. The ExemplarOnly method directly outputs exemplar sentences without considering video contents, making the exemplar and output sentence structures exactly the same and TED scores as 0.0, whereas the semantic coherence to the video is much lower. The Seq2Seq model mainly focuses on describing video contents while neglecting the given exemplar sentence syntaxes, so it gets higher semantic COS scores but inferior TED scores. The Template method simply fills the detected content words from videos in the exemplar sentences while does not consider the global sentence meanings. Doing so makes the generated captions fit the original exemplar syntaxes very well but the captions are not fluent and get high (poor) perplexity scores. Meanwhile, the semantic scores are also lower. Although GFN attempts to leverage POS features to guide video captioning, its controllability of caption syntax is weak and results in high TEDs between exemplar sentences and captions. The main focus of GFN is still to improve the semantic accuracy of video captions and keep their fidelity with the training corpus.

Quantitative Evaluation
Since the Seq2Seq and GFN models often generate simple sentence descriptions with common sentence forms, their perplexity scores are correspondingly lower. The collected exemplar sentences are human-edited, which often take more  various and complex syntaxes and are free in grammar. Thus, the perplexity scores of the ExemplarOnly method are relatively higher. Based on these complicated exemplars, the yielded Template and Ours method will also inevitably present higher perplexity scores. Our proposed model achieves much better TED scores and comparable COS scores than the Seq2Seq and GFN models, which shows that our method can comply with the exemplar sentence syntaxes while describing the video semantics. Also, imitating the complex exemplar sentences does not significantly affect the sentence fluency of our generated captions, with the perplexity scores increasing in a reasonable range compared to the ExemplarOnly method. Figure 4 shows the qualitative results for the syntax customized video captioning task. The Seq2Seq model can generate only one simple caption for one video, which is monotonous and bears high resemblance to the training corpus. The Template method which simply replaces main content words in the exemplar sentences make the generated captions lack of fluency. The GFN model cannot capture sentence syntaxes and produces similar or the same captions for different exemplar sentences. Our generated captions comply with the exemplar sentence syntaxes well, and more concrete video contents like "snow helmet" and "green grass" can also be described in the captions more expressively. Meanwhile, the two different syntax customized video captions further provide us the intuition of diverse video captioning.

Qualitative Evaluation
To better demonstrate the effectiveness of our proposed method on syntax customized video captioning, we further provide more qualitative results of Ours method in Figure 7 and Figure 8 at the end of the paper. It can be observed that on different types and contents of videos, our generated captions can not only present the video semantics accurately, but also well imitate the syntaxes of the given exemplar sentences.

Human Evaluation
Besides the quantitative and qualitative evaluations, we also conduct a human assessment of the generated captions. The Groundtruth caption: Two snowboarders preparing for a decent on top of a snowy mountain. Seq2Seq generated caption: A man is riding a bike on a mountain.

Exemplar Sentence 1: Little rat has been caught inside the mouse trap.
Template syntax customized caption 1: Snow man has parked caught inside the woman person. human evaluators were asked to rate the three aspects of the generated captions -syntax similarity with the exemplar sentence, semantic coherence with the video, and sentence fluency. Each rating aspect is graded in three scales {0.0, 0.5, 1.0}, and the higher, the better. We randomly chose 200 captions generated by each compared method from the test set and invited 10 evaluators (5 males and 5 females) to grade them. The average rating scores are presented in Table 2. These scores are in agreement with the above quantitative evaluation. Although the complex exemplar sentence syntaxes may influence the sentence fluency of our generated captions, they are still realistic and can describe the video semantic contents meanwhile imitating the exemplar sentence syntaxes.

Captioning Diversity Evaluation
Since we collect 20 different exemplar sentences for each video, we will accordingly get 20 different syntax customized captions with the proposed model. To measure the diversity among these generated captions, we follow the evaluation metrics introduced in [43], and report the LSA and Self-CIDEr based diversity scores in Table 3. The collected exemplar sentences are independent of each other, and it is evident that the diversity scores of the ExemplarOnly method are fairly high and can be seen as upper bounds in this diversity evaluation experiment. The conventional Seq2Seq model can only generate one single caption for one video, and thus it cannot achieve diverse video captioning and gets all the scores as 0.0. The GFN method can vary the video captions to some extent, while the captioning diversity remains limited. The Template method and Ours model can generate diverse captions to describe the video contents, and Ours model even gets comparable LSA and Self-CIDER scores to other state-of-the-art methods reported in [43]. Hence, the results verify that our proposed model can indeed strengthen the diversity of video captioning.

The Influence of Exemplar Sentence Length
In our experiments, we find that given different exemplar sentences, there can be different object descriptions coming from the video. Therefore, it is also interesting to study the effect of exemplar sentence length to the number of different object words in the generated captions. Therefore, we first count the different verb and noun words in each generated syntax customized video caption of our model. Then, we plot 6 scatter maps in Fig. 5, where the top three scatters are for MSRVTT dataset and the bottom three are for ActivityNet Captions dataset. The x coordinate of each point in the scatter map means the length of the given exemplar sentence, and the y coordinate means the number of different noun, verb, and noun+verb words (we call them objects in general) in the corresponding syntax customized video caption, respectively. The pearson correlation coefficient between the exemplar sentence length and the object number is also presented in the top-right part of each subfigure. It can be observed that the object number is positively correlated to the exemplar sentence length, no matter for verbs, nouns or their union.
Such results indicate that when we provide longer exemplar sentences, more objects will be incorporated in the generated syntax customized captions, making more representative and diverse video descriptions.

Discussions on the Conventional Metrics
The generated syntax customized video captions one the one hand should semantically describe the video contents, on the other hand should syntactically follow the given exemplar sentences, thus making the caption syntactic structures be greatly changed in the syntax imitation procedure. As such, our generated syntax customized video captions will be greatly different from the original groundtruth captions (in the MSRVTT and ActivityNet Captions datasets) on ngram characteristics and detail wording. Since conventional captioning evaluation metrics, such as BLEU, take the n-gram similarity between the predicted and groundtruth captions as a basic measurement, they are not very appropriate for evaluating our syntax customized video captioning task.
In this section, we still provide the evaluation results in the conventional video captioning metrics in Table 4 for reference. As expected, our proposed model does not achieve high performances in these metrics, while the Seq2Seq and GFN models get good results because they are still focusing on fitting the detail wording patterns in the training corpus. The different evaluation results between the conventional captioning metrics and our human ratings in the main paper also indicate the limitation of the conventional metrics in evaluating syntax-varied captions.

Model Contribution Examination
In this section, we perform three groups of ablation studies on the MSRVTT dataset to examine the contributions of our proposed model, with the results shown in Figure 6. Each point in this figure represents the performance of the model on the validation set during the training process. The xaxis indicates the 10/TED scores, and the y-axis denotes the COS scores. Since lower TED and higher COS means better performance, models at the top right are more desirable.
Loss Components. Comparing the model with only the L v,c =L syn v,c +L sem v,c loss and the model additionally considering the L s,s =L syn s,s +L sem s,s loss in Figure 6(a), we can observe that training with an exemplar sentence corpus as introduced in Sec. 3.4.2 can augment the training data and thereby improve the model performance. By considering both L v,c and L syn v,e terms, the performance curve rises steadily without fierce fluctuation. It shows that even without paired groundtruth syntax customized captions, merely providing supervision on sentence syntaxes can also help the model imitate the syntactic structures of the given exemplar sentences. By combining all the above terms, our proposed model achieves the best performances.
Sentence Syntax Encoding. Ours-WordSyntaxEmbed model in Figure 6(b) drops the character-level LSTM, and only keeps the word-level LSTM to encode the sentence syntactic structure. Ours-NoWordReplace model does not adopt the word replacement mechanism as we stated in Sec. 3.1. The performance superiority of the full model (Ours) over these two ablation models verifies the benefits of the character-level LSTM in encoding subword features and the word replacement mechanism. These designs can help our model capture exemplar sentence syntaxes effectively and filter out unnecessary sentence semantics, which is very crucial for syntax customized video captioning.
Caption Decoding. Ours-Concate model in Figure 6(c) simply concatenates sentence syntactic and video semantic representations and takes them as the inputs to the decoding LSTM without the proposed CLN processing. We can observe that the full model (Ours) achieves higher COS scores than Ours-Concate, which shows that equipping CLN in the decoding LSTM is beneficial to video semantic preservation when performing syntax customization. In Ours-Semantic1Syntax2 model, we feed video semantic representation to the first layer LSTM and sentence syntactic representation to the second. Such a semantic-first syntaxfollow architecture is significantly inferior to our proposed model. The reason mainly dues to that using syntactic information to guide the word prediction in the second layer LSTM will hinder it from choosing appropriate words coherent to the video semantics.

The Influence of Word Replacement Probability
For the word replacement mechanism proposed in Sec. 3.1, we have mentioned that we will randomly replace the word     Table 5.
It can be observed from Table 5 that if the replacement probability is small (e.g., 0.1, 0.3, 0.5), the models get lower COS semantic scores and smaller TED scores. The reason is that in these cases, the models tend to remember the exact words in the exemplar sentences instead of learning their POS tag information and sentence syntaxes, and thus causes the output captions just copy the words from the exemplar sentences. If we increase the replacement probability to higher value (e.g., 0.9), the TED score increases and the COS score decreases a little compared to those of Ours-0.7, while these two settings are comparable to each other in perplexity. Considering the overall better performance of Ours-0.7 in balancing the three metrics, setting replacement probability as 0.7 is suitable for our proposed model.

CONCLUSION
In this paper, we proposed a novel syntax customized video captioning task by imitating different exemplar sentences to strengthen the diversity and expressiveness of video captioning. To solve this task, a hierarchical sentence syntax encoder was proposed to capture both the local subword features and global sentence syntaxes of the exemplar sentence, based on which a two-layer syntax conditioned caption decoder was devised to generate the syntax customized caption expressing video semantic meanings. Comprehensive experiments verify that the generated captions by our model can vividly describe video contents while complying with different exemplar sentence syntaxes, thus indicating our contributions to enrich the video captioning diversity.
Groundtruth caption: A person is drawing a picture of a squidward. Exemplar Sentence: Happy business team with arms crossed at the office. Ours syntax customized caption: Animated cartoon character with glasses isolated at the picture.  (e) Groundtruth caption: She continues speaking to him and has him watch her while she steps on the machine. Exemplar Sentence: Beautiful carrot cake with golden nuts and fresh berries on a rustic board. Ours syntax customized caption: Several people stop with two legs and small shorts on a elliptical bike.
(a) Fig. 8: More qualitative results on the ActivityNet Captions dataset. For each video, we provide one groundtruth caption, one exemplar sentence, and the syntax customized video caption predicted by our proposed model.


Feedback Control of Injection Rate of the Injection Molding Machine Based on PID Improved by Unsaturated Integral

In conventional proportional–integral–derivative (PID) control, the integral term occupies a significant amount of controller memory, which prolongs the calculation time. (e integral term easily leads to overshooting and oscillation, while the derivative term reduces the controller’s anti-interference ability. In view of these problems, a PID expression with a recurrence relationship was derived, and the integral and differential terms of the conventional PID model were improved using an unsaturated integral and passivation differential to achieve a good control effect. (en, the improved PID was applied to the injection speed control of an injection molding machine, which is usually controlled using conventional PID control that featured difficulty in mathematical modeling, a nonlinear relationship between the input and the output, and high system complexity. Taking an injection molding machine as the control object, the transfer function of the injection system was constructed. (en, the improved PID was simulated using Matlab/Simulink. Lastly, the improved PID was verified using experiments. (e simulation and the experimental results showed that the control model had a rapid response, no overshoot, and a high precision.

Introduction
To make the parameters in the controlled object behave in an ideal way, it is necessary to use corresponding control methods, such as feedforward control [1], proportionalintegral-derivative (PID) control [2,3], robust control [4,5], adaptive control [6,7], and compound control [8,9]. Among the control methods, feedforward control is a type of openloop control. It uses a hysteresis model, which is easy to implement and does not require a sensor. However, it cannot completely eliminate control errors and it is difficult to use to achieve high-precision control. PID control, robust control, and adaptive control are types of feedback control that can theoretically eliminate control errors and meet the requirements of high-precision control. However, the cost of the control system is increased due to its need for highprecision sensor feedback. Compound control is a control method combining feedforward control with feedback control that can obtain good comprehensive performance, although the control algorithm is often complex. PID control technology is advanced, and it does not require an accurate model of a controlled object. Moreover, it features easy parameter tuning, a simple algorithm, and good stability. erefore, PID control is widely used. In conventional PID control, however, the integral term occupies excessive memory for the controller, which results in a series of problems, such as overshoot and oscillation being easily caused by the integral term, as well as the decrease in the anti-interference ability of the controller caused by the differential term. To solve the problem of unsatisfactory parameter tuning, different forms of improvement have been tried. Han et al. [10] used a search algorithm using particle swarm with H2 control standards objective function to optimize PID parameters, and the improved PID was used to control the water hydraulic position servo system which is intent to be used in remote control robot for fusion reactor.
Compared with conventional parameter tuning, the control effect of the PID significantly improved. Ajmeri and Ali [11] established a new cascade control structure for the PID of a first-order integral system. Due to the use of the PID controller in the forward channel, although the response of this control structure was faster, there were problems such as the increase of overshoot and the degradation of the disturbance suppression performance. Nunn and Grolman [12] proposed to control the injection speed by time periods under different injection molding environments and different raw materials, based on the characteristics of the injection speed, However, this control strategy still does not solve the injection accuracy, stability, and the problem of overshoot. In order to control the quality of plastic parts, Kamal et al. [13] proposed a cascade control system design scheme. By constructing the transfer function of the injection system and the transfer function of the temperature system, the injection speed and the body temperature between cycles are controlled.
Based on the research of the above scholars, the control of injection speed has its own advantages and disadvantages. At present, traditional PID control is still the most commonly used control strategy in actual production for injection speed control. erefore, this paper studies the highorder system model of injection speed, based on the analysis of the shortcomings of traditional PID controllers: slow response speed, large overshoot, poor robustness, and weak anti-interference ability.
rough the modification of the integration and differentiation term of traditional PID, the improved PID has the advantages of rapid response, strong anti-interference ability, small overshoot, and good robustness.

The Problem of Injection Speed Control of Injection Molding Machine
Injection molding of the injection molding machine can be divided into four stages: preheating, injection, packing, and cooling. As shown in Figure 1, in the second stage of injection molding, the screw in the barrel injects the preheated and melted raw materials into the cavity of the mold through the nozzle. During this stage, the mainly controlled parameters are injection speed and injection pressure. After the melted raw material fills the mold cavity, the injection hydraulic cylinder needs to maintain pressure for a period of time to prevent the raw material from flowing back until the raw material is cooled and solidified. e ejector pin then ejects the solidified part from the mold cavity and the process continue to repeat from the first stage. In the injection molding process, the control of the injection speed directly determines the quality of the molded product. e characteristics of the entire injection process are analyzed. e injection speed has the following characteristics of repeated operation, simple and stable, after the analysis of the entire injection process. e shortcomings of the traditional PID control of injection speed are improved and applied to the control of the injection speed.

PID of Recurrence Relationship.
e PID controller performed proportional adjustment, integral adjustment, and differential adjustment on the difference e between the target value and the actual value, and the adjusted combination was used as the input value u of the system. e equation could be expressed as follows: where u (k) is the system input variable of the k th sampling period, T is the sampling period and its corresponding k denotes sampling sequence number, kT represents the k th sampling period, e (k) and e (k−1) are the deviations between the target value and the actual value at the k th and k−1 th sampling, respectively, K P is the proportional coefficient, T I is integral time constant, and T D is the differential time constant.
To obtain the output of the PID control given in (1) at a certain time, it was necessary to integrate all the deviations of the past time and the current time, which not only occupied a significant amount of computer memory but also took a long time to calculate, making it difficult to achieve fast control. For this reason, a PID with a recurrence relationship was adopted in this study. According to (1), the output of the PID controller at time (k−1) T was (2) Using the difference between (1) and (2), the PID difference could be expressed as

(4)
According to (4), there were only four parameters that determined the system input variables u (k), namely, e (k), e (k−1), e (k−2), and u (k−1), which greatly reduced the calculation time of the controller, thus greatly improving the control speed and making fast control easier.
After performing a Z transformation on (4), the transfer function of the PID could be obtained: 3.2. Integral Term Improvement of the PID Controller. e integral term in the PID controller was used to eliminate the steady-state error in the control. However, when the target value changed drastically in a short period of time, a huge fluctuation deviation e is produced, leading to a large overshoot or oscillation of the output value that affect the control effect seriously. erefore, the integral term was improved to eliminate the steady-state error and control the overshoot.
In order to enable the integral term to achieve improvement described above at the same time, an unsaturated integral was used to improve the conventional integral in the PID controller. Its algorithm is shown in Figure 2. When calculating the control quantity u(k), the first step was to determine whether the control quantity u(k − 1) was greater than its maximum set value u max . If u(k − 1)u max , only the positive deviation was integrated; if u(k − 1) > u max , only the negative deviation was integrated.

Differential Term Improvement of the PID Controller.
Although the differential term in the PID controller could accelerate the response speed of the system, it would lower the controller's anti-interference ability. In the control process, the external environment was inevitably interfered with. Excessive external interference could cause significant changes in the value of the input variable u and rapid changes in the frequency. It had a great impact on the control accuracy of the system, especially for the actual production system. It would not only damage the production system but also harm the personal safety of production personnel. e conventional differential term was improved based on its shortcomings.
In the actual control process, the external interference signals were mostly high-frequency signals, and the low-pass filter was an effective means for removing high-frequency signals. erefore, in order to eliminate the adverse effect of the disturbance on the differential link, the passivation differential was used to improve the conventional differential in the PID controller, as shown in Figure 3. e lowpass filter was introduced into the conventional PID controller to filter out the interference signals so that the differential term could improve the dynamic characteristics of the system while reducing its sensitivity to interference.

Runner Nozzle
Heater Barrel Hopper Hydraulic cylinder Shock and Vibration 3 erefore, the discrete transfer function after passivation could be obtained:

Mathematical Model of the Injection System of the Injection Molding Machine
Ajmeri and Ali [11] reviewed the controllers of injection molding machine. Although the intelligent control strategy is well-researched, its stability is the most critical aspect for an actual production machine. For example, in order to ensure the stability of production, the injection speed of an injection molding machine is mainly controlled using the conventional PID control. In order to verify the effectiveness of the improved PID controller to improve the control effect of the injection speed of the injection molding machine, the injection system of the machine with complex mathematical model and production conditions and a tendency to produce overshoot was selected as the control object, based on one of the world's best-selling Haitian Tianlong IAIIS series injection molding machines. Figure 4 is the schematic diagram of

Mathematical Model for the Servo Valve.
e servo valve power transmission system consisted of a servo motor and an internal gear. It was a second-order system with the current as the input value and the valve flow that was collected using the flow sensor as the output value. e transfer function could be obtained: where K v � Q n ����� � P s /P sn /I sn is the flow gain, P s denotes the actual oil pressure value, P sn represents the rated pressure drop, Q 0 is the no-load flow, Q n is the rated flow, ω v represents the valve angle frequency, I sn is the rated current, and ς v is the damping coefficient.

Mathematical Model of the Injection Cylinder.
e injection cylinder was hydraulically driven. Its working principle is shown in Figure 5. According to Newton's Second Law of Motion, Force, and Acceleration, ignoring other nonlinear effects, the differential equation of motion of the injection cylinder could be obtained: where F l is the external load, v is the speed of the injection piston, A is the effective area of pressure, P is the internal pressure driving the movement of the hydraulic cylinder, and B is the damping coefficient. e injected liquid Q drove the injection rod to move by changing the volume on both sides of the cylinder. According to the conservation of the total liquid flow, the differential equation could be obtained:

Start
Calculate the proportion and differential term where C l represents the sealing coefficient, β denotes the equivalent volume of the liquid, V a represents the actual volume of the liquid, Q is the total flow in the cylinder, and (V c /β)(dP/dt) is the volume change rate of the liquid in the cylinder when the screw was moving.
After the Laplace transformation of (9) and (10), the function in the complex field could be obtained: After substituting (11) into (12), the following equation could be obtained: If load F l was 0, the transfer function of the hydraulic cylinder was where the natural frequency is ω n � �������������� (A 2 + C l B)/mV a , and the damping ratio is ς n � (ω n /2β) · (V c B + mβC l /A 2 + C l B).

Transfer Function of the Injection System.
According to the transfer functions of the servo valve and the injection cylinder, the transfer function of the injection speed could be obtained: e identification of the key parameters in the transfer function faced cumbersome and complicated problems with different parameters due to the nonlinear characteristics of the input and the output. For the above problems, the Taylor Formula was used for analysis, and the ideal state assumptions were as follows. (1) e servo motor was a firstorder link, (2) during the injection process, liquid leakage was negligible, and (3) the working range of the servo valve spool was narrow and controllable.
When identifying the mathematical model of an injection system, different methods are often used to identify the servo valve model, such as the pulse response method and the parameter optimization method. e mathematical model constructed in this study was highly consistent with the fourth-order linear model for the injection speed of the injection molding machine established by Wang and Shen [14]. Hence, the mathematical model used in the simulation described below was

Simulation
e mathematical model of the injection speed was constructed as described above, and it was combined with the improved integral term and the differential term. e diagram of the injection speed in Simulink using improved PID control is shown in Figure 6.
In order to verify the control effect of the improved PID controller, a target speed of 5 mm/s was input to the controller, and the injection speed rise time was 0.5 s, as shown in Figure 7(a). e system steady-state time was 1.1 s. e speed response had no overshoot and the steady-state error was 0. As shown in Figure 7(b), the maximum value of the output speed was 17 mm/s. e combined signal gradually decreased, and the error of the injection speed was almost zero. e simulation showed that the improved PID controller not only had a higher response speed to the injection speed, but also effectively reduced errors.

Experimental Verification
e performance of the improved PID controller was verified using experiments, the experiments are conducted in IA1200, Haitian Co., Ltd., as shown in Figure 8. e working process was as follows. e difference between the actual output speed and the desired speed was processed using the improved PID control strategy, the results of which were then used as the control signal. e digital signal was converted into an analog signal, which was taken as the input Reference input  voltage of the motor. e injection speed curve of the injection port was detected using a flow sensor. Next, the A/D converter converted the continuous signal into discrete signals and sent it to the controller. e obtained signal was compared with the desired value, and the difference between the two was used as the error value [11,12]. In order to make the experiment closer to the actual production, the step signal, which is the most commonly used signal in actual production, is selected as the input signal. A complex arbitrary amplitude signal is used as the input signal to further verify the control effect of the improved PID controller.

6.1.
Step Signal Test. Figure 9(a) shows the expected speed curve of the injection system of the injection molding machine with a duration of 10 s and a steady-state value of 5 mm/s, the results of traditional PID control and improved PID control. In traditional PID control, the response time of injection speed is 0.02 s. Compared with the expected speed curve, the steady-state error center line is about 0.2 mm/s, and the injection speed curve has an upward trend with the increase of time (i.e., the larger error), as shown in Figure 9(b). In the improved PID control, the injection speed response time is 0.2 s. Compared with the expected speed curve, the steady-state error center line is almost 0, and the control error does not increase with time, as shown in Figure 9(c). In actual production, the steady-state error and stability of the injection speed are more important parameters than the response time. erefore, the improved PID control strategy can significantly improve the injection accuracy of the injection speed. Figure 10(a) shows the arbitrary injection speed curve (duration 18 s, speed range 0 ∼ 16.5 mm/s), which is composed of sine signal, ramp signal, constant signal, and so on, result of the traditional PID control and the improving PID control. As shown in Figure 10(b), the actual output injection speed curve of traditional PID control is compared with the expected speed curve, and the error center line range is −1.72 ∼ 0.75 mm/s, and the followability is poor, which Shock and Vibration 7 cannot meet the requirement of the more complicated injection speed. As shown in Figure 10(c), when the improved PID controller is performed, the steady-state error center line is almost 0 mm/s. e improved PID controller also has better followability. It can be observed that the improved PID controller can track the injection speed well and the error is almost zero for any desired curve of amplitude.

Conclusions
First, the shortcomings of the conventional integral and differential terms of PID control were analyzed and the unsaturated integral algorithm was used to replace the integral term in the conventional PID algorithm. en, the differential term was passivated by adding a low-pass filter in the conventional PID controller in order to improve the conventional PID. Next, the improved PID was used to improve the control accuracy of the injection speed of the injection molding machine, and Matlab/ Simulink was used to simulate and analyze the control effect.
e simulation results showed that the injection speed rise time was 0.5 s and the system steady-state time was 1.1 s. e speed response had no overshoot and the steady-state error was 0. Finally, the control effect was verified using experiments, the results of which showed that for the 5 mm/s step target speed curve, the injection speed response time was 0.2 s without overshoot and the steady-state error midline was almost reduced to 0 mm/s from 0.66 mm/s for the PID control before improvement. For the target speed curve with an arbitrary amplitude, the error midline of the injection speed was reduced to almost 0 from the range of −1.72 mm/s to 0.75 mm/s for the PID control before improvement. erefore, with the improved PID controller, the injection speed of the injection molding machine could rapidly and accurately track the desired speed curve.
Data Availability e raw/processed data required to reproduce these findings cannot be shared at this time as the data also form part of an ongoing study.