Spatial-Temporal Transformer for 3D Point Cloud Sequences

Effective learning of spatial-temporal information within a point cloud sequence is highly important for many down-stream tasks such as 4D semantic segmentation and 3D action recognition. In this paper, we propose a novel framework named Point Spatial-Temporal Transformer (PST2) to learn spatial-temporal representations from dynamic 3D point cloud sequences. Our PST2 consists of two major modules: a Spatio-Temporal Self-Attention (STSA) module and a Resolution Embedding (RE) module. Our STSA module is introduced to capture the spatial-temporal context information across adjacent frames, while the RE module is proposed to aggregate features across neighbors to enhance the resolution of feature maps. We test the effectiveness our PST2 with two different tasks on point cloud sequences, i.e., 4D semantic segmentation and 3D action recognition. Extensive experiments on three benchmarks show that our PST2 outperforms existing methods on all datasets. The effectiveness of our STSA and RE modules have also been justified with ablation experiments.

Introduction
Point cloud sequences provide both spatial and temporal information of a scene over a period and can be applied in various vision tasks such as 4D semantic segmentation and 3D action recognition. With the rapid development of dynamic point cloud acquisition technologies (such as LiDARs and depth cameras), point cloud sequences are becoming increasingly available and have been investigated for different applications, such as self-driving vehicles, robotics, and augmented reality.
4D semantic segmentation is challenging due to several reasons. First, different from regular and dense 2D image sequences, raw point cloud sequences usually have a regular order in the temporal domain but are unordered in the spatial domain. Besides, point clouds are usually highly sparse. Therefore, it is extremely difficult to model the spatiotemporal structure in a point cloud sequence. Second, it is hard to predict the motion of points within a point cloud sequence. However, the construction of spatio-temporal neighborhoods rely on tracking the motion of points across different frames. It is therefore, difficult to effectively extract and aggregate the spatio-temporal context information of points within inter-frame neighborhoods.
Recently, several methods have been proposed to capture the spatio-temporal dynamics of points from point cloud sequences by utilizing cross-frame information. These methods can be roughly divided into voxel-based methods [5,15] and point-based methods [20,2,19,8]. A voxel-based method converts a 3D point cloud sequence into regular sparse voxels and then extracts pointwise features via a network with 3D convolutions or sparse convolutions. Due to the introduction of quantization error, the performance of this approach is limited. A point-based method constructs a spatial-temporal neighborhood for each seed point (e.g., sampled by Farthest Point Sampling (FPS)), and then applies Multi-Layer Perceptron (MLP) to extract point-wise features. It then passes these features backward and aggregates them with the features of subsequent frames using an inter-frame feature fusion strategy.
Although existing point-based methods can avoid the quantization error by directly processing raw point cloud sequences, they still face two major challenges. First, existing methods usually adopt attention operations [2,19] or Recurrent Neural Network (RNN) models [15,8] to fuse inter-frame features. However, since these methods reply on long-term dependency, the inter-frame information fusion conducted at a frame depends on the fusion of the information of all its previous frames, leading to information redundancy. It is necessary for an inter-frame feature fusion strategy to capture spatio-temporal point context information while reducing the redundancy of previous frames. Besides, current semantic segmentation methods mainly adopt an encoder-decoder architecture. The encoder usually consists of multiple hierarchically stacked feature extraction layers (e.g., the set abstraction layer [24]) and extracts rich semantic features by reducing the resolution of its feature maps. Therefore, information loss is introduced and the Figure 1. The architecture of our P ST 2 model. Our P ST 2 model is employed for two tasks: 4D semantic segmentation (as shown in (a)) and 3D action recognition (as shown in (b)). In the 4D semantic segmentation task, we apply two set abstraction layers to extract features, and then employ the Resolution Embedding (RE) module to recover the lost geometric information of the extracted features. Next, we perform convolution on the features of adjacent frames to obtain spatial-temporal patches, and then fed these patches into a Spatio-Temporal Self-Attention (STSA) module to capture spatial-temporal context information. Finally, we use multiple Feature Propagation (FP) layers to produce point-wise semantic predictions. In the 3D action recognition task, we insert our STSA module into the MeteorNet [20] model to achieve action recognition. segmentation performance is decreased.
In this paper, we propose a novel Point Spatial-Temporal Transformer (P ST 2 ) network to tackle the above two challenges. First, we introduce a self-attention based module, i.e., Spatio-Temporal Self-Attention (STSA), to capture inter-frame spatial-temporal context information. More specifically, our P ST 2 model adaptively aggregates these correlated inter-frame neighborhood features based on the self-attention operation instead of fusing all previous frames. Consequently, the redundancy is reduced, the training speed is increased, and the robustness is improved (with residual connection and layer normalization). Second, we introduce a Resolution Embedding (RE) module to enhance the resolution (which is lost in the encoderdecoder architecture). By aggregating inter-neighborhood features with the attention weights, the resolution can be enhanced and the segmentation performance can be improved significantly. To further demonstrate the effectiveness of our point spatial-temporal transformer, we introduce a new framework to leverage the proposed STSA module for action recognition from 3D point clouds. Experiments on the Synthia [25], SemanticKITTI [1], and MSR-Action3D [18] datasets demonstrate that our P ST 2 model outperforms existing methods [20,2].
The contributions of this paper are summarized as follows: 1) We propose a self-attention based module (namely, STSA) to capture point dynamics in point cloud sequences. It is shown that this module can encode the spatio-temporal features from point cloud sequences.
2) We introduce an RE module to enhance the resolution of feature maps. This module clearly improves the segmentation performance.
3) Our P ST 2 model is tested with two different tasks: 4D semantic segmentation and 3D action recognition. Experiments on three datasets show that the proposed STSA and RE modules can easily be plugged into exiting static point cloud processing pipelines to achieve improved performance.

Related Work
In this section, we briefly review those works that are highly related to our method, i.e., deep learning method on static point clouds, deep learning method on 3D point cloud sequences, and the transformer.

Deep Learning on Static Point Clouds
Deep learning has been widely used in many point cloud tasks, such as classification [23,24,13], part segmentation [37,14,36], semantic segmentation [35,17,31,12], reconstruction [6,39], and object detection [4,22,11]. Existing methods for deep learning on static point clouds can be roughly divided into multi-view based, volumetricbased, and point-based methods. Multi-view based meth-ods [27,34] first project a point cloud into multiple views and extract view-wise features, and then fuse these features for static point cloud processing. Volumetric-based methods [21,32] convert a point cloud into regular 3D voxels and then employ 3D Convolution Neural Network (CNN) or sparse convolution on the volumetric representation. These two approaches introduce quantization errors and exhibit performance declination, while point-based methods directly work on raw point clouds and reduce the quantization error.
PointNet [23] is a pioneering work of point-based methods. Qi et al. [23] used several MLP layers to extract point-wise local features and adopted a max-pooling layer to aggregate these features. The local features and the aggregated global features are then concatenated to predict the category of each point. In follow-up work, Qi et al. [24] introduced a hierarchical structure and multi-scale feature learning mechanism into PointNet++, thereby improving the performance of scene semantic segmentation. Subsequent works have been proposed based on the improvement of PointNet++ [24]. However, these methods cannot directly process dynamic point cloud sequences due to the lack of the inter-frame feature fusion strategy.

Deep Learning on 3D Point Cloud Sequences
In recent years, two major mainstream methods focusing on 3D dynamic point cloud sequences have been explored.
Several voxelization based methods have been proposed to handle dynamic point cloud sequences. MinkowskiNet et al. [5] first converted 3D point cloud sequences into 4D occupancy grids, and then employed sparse 4D convolution to process 4D occupancy grids. In 3D Sparse Conv LSTM, Huang et al. [15] voxelized 3D point cloud sequences into sparse 4D voxels and convolved them on a sparse grid, then passed the memory and hidden features through a sparse convolution inside the LSTM network. The LSTM network produces hidden and memory features that will be passed to the next frame and fused with features of subsequent frames.
Several methods directly working on raw point cloud sequences have also been explored. MeteorNet [20] is a leading deep learning method for dynamic 3D point cloud sequences. The structure of MeteorNet [20] is inherited from PointNet [23]. By proposing two grouping approaches (including direct grouping and chained-flow grouping) to construct spatio-temporal neighborhoods, inter-neighborhood features are aggregated to capture point dynamics in the point cloud sequence. Cao et al. [2] proposed an Attention and Structure Aware (ASAP) model, which contains a novel grouping approach (i.e, Spatio-Temporal Correlation) to construct spatio-temporal neighborhoods and a novel attentive temporal embedding layer to fuse the related inter-frame local features in a recurrent fashion. Our Point Spatial-Temporal Transformer P ST 2 is also directly performed on raw point cloud sequences, but aims at enhancing the resolution when capturing spatio-temporal dynamics across entire point cloud sequences.

Transformer
Transformer is a typical deep neural network based on the self-attention mechanism and particularly suitable for modeling long-range dependencies.
Transformer was first proposed in [28] for the sequenceto-sequence machine translation task, and then extended to various computer vision tasks. In computer vision tasks, image features are obtained from a backbone network and converted into sequences, and then a transformer-based network is employed to process sequence-form features. DEtection TRansformer (DETR) [3] is a landmark work that considers object detection as a set prediction problem. Specifically, an image is first divided into several candidate regions, and then the object in candidate regions is classified. Recently, transformer has been used in various computer vision applications, including image classification, high/mid-level vision, low-level vision, and video processing. In these applications, Transformer excels to process video sequences and has extended many video tasks, such as video retrieval [26], action recognition [9], and video object detection [38]. The Video Instance Segmentation Transformer (VisTR) [33] extends DETR [3] for video object instance segmentation (VIS). Specifically, frame-level features are concatenated and fed into a Transformer, and then instance predictions are produced.
Very recently, Transformer is extended from video tasks to point cloud tasks with promising results. Guo et al. [40] proposed a novel transformer-based framework (namely, Point Cloud Transformer (PCT)) for static point cloud learning. They used self-attention to design an offsetattention layer to model irregular point clouds. Zhao et al. [10] proposed a Point Transformer (PT) layer to fuse point cloud features.
However, there are few prior Transformer applications to model point cloud sequences. Intuitively, Transformer has the advantage of modeling long-range dependencies and should be an ideal candidate for point cloud sequences tasks. Therefore, we propose the P ST 2 model to model spatial-temporal information in point cloud sequences with Transformer.
Our P ST 2 model is different from the transformers designed in PCT [10] and PT [40]. First, our STSA module introduces skip connection and layer normalization operations based on the self-attention operation. Besides, our STSA module divides the input features of the adjacent frame into spatial-temporal patches for further processing, while the input feature is directly processed in [10,40]. Consequently, our module is better at capturing the spatial-temporal context information than [10,40].

Point Spatial-Temporal Transformer P ST 2
In this section, we first use 4D semantic segmentation as an example to describe our Point Spatial-Temporal Transformer network P ST 2 , including the overall architecture, the step for spatial-temporal neighborhood construction, the Resolution Embedding (RE) module, and the Spatio-Temporal Self-Attention (STSA) module. We then adapt the proposed module for 3D action recognition.

Overview Architecture
For 4D semantic segmentation, which is a point-level classification task, our P ST 2 model takes a multi-frame point cloud sequence as its input, and predicts the category of each point in the point cloud sequence. Our P ST 2 model adopts an encoder-decoder architecture. Specifically, the encoder consists of a backbone, an RE module and an STSA module, and the decoder includes multiple Feature Propagation (FP) layers. The FP layer is originally introduced in PointNet++ [24] to interpolate the semantic features extracted by the encoder upwards and predict the semantic category of each point.
As illustrated in Fig. 1 (a), given a point cloud sequence, three steps are adopted to capture point dynamics in our P ST 2 model. First, we use a backbone network (including two set abstraction layers [24]) to construct the spatialtemporal neighborhoods and extract point features in each frame (Sec. 3.1.2). Then, we adopt the proposed RE module to enhance the resolution of features in each frame (Sec. 3.1.3). Finally, we employ the proposed STSA module to fuse the inter-frame features and produce the predictions (Sec. 3.1.3).

Spatial-Temporal Neighborhood Construction
Given a point cloud sequence j } m j=1 can be sampled from S t by the FPS method [23,24], where m is the number of seed points and m < n.
We first sample the seed points in the first frame S 1 , and groups all points within a certain radius around these seed points to form a neighborhood. Next, we use the same seed points in subsequent frames {S 2 , ..., S T } to construct spatial-temporal neighborhoods with stable spatial locations at different times. Then, we apply two set abstraction layers [24] to extract local features h

Resolution Embedding (RE)
Given the feature representation h (t) i , we propose a novel resolution block in the RE module to enhance the resolution of point features in each frame. The RE module consists of a feature block and a resolution block. The feature block was designed to further extract semantic features, while the resolution block extracts the spatial context information from the input features, and treats these context information as an enhanced resolution to incorporate with the features produced by the feature block.
(1) Feature block: As shown in the Fig. 2(a), we perform a set abstraction layer on h i . The set abstraction layer is used in the encoder-decoder architecture to hierarchically extract local (even global) features.
(2) Resolution block: As shown in Fig. 2 1 and m (t) 2 along the spatial dimension, and then concatenate them along the feature dimension to obtain g (t) i . We employ a learnable multi-layer perception f to update the feature dimension of g i . This process aims at effectively extracting the spatial interneighborhood related information.
Once n is calculated based on these two attentions: where the attention weights a 1 and a 2 are calculated as: Figure 3. The architecture of our STSA module. SA, Norm, Smax, T and FF stand for the self-attention operation, the layer normalization function, the softmax operation, the matrix transpose operation, and the feed-forward network, respectively.

Spatio-Temporal Self-Attention (STSA)
Given spatial features I (t) i of each frame, the STSA module is used to incorporate inter-frame features along the temporal dimension and capture the spatio-temporal context information. Intuitively, different inter-frame spatialtemporal neighbors contribute differently to the final results. Specifically, for objects with slow-moving speeds, interframe spatial-temporal neighbors that are closer in space contributed more than those neighbors that are far apart. In contrast, for objects with fast-moving speeds, inter-frame spatial-temporal neighbors that are far in space should also be considered. Transformer-based methods have the potential to learn the correlation within inter-frame spatialtemporal neighborhoods.
As shown in Fig. 3, our STSA module consists of two blocks: spatial-temporal patch division, and self-attention.
(1) Spatial-Temporal Patch Division This block is designed to correlate multiple frame features and generate spatial-temporal patches. Inspired by [7], we first divide spatial features I (t) i into patches and then align these inter-frame spatial patches along the temporal domain based on the same seed points to form spatialtemporal patches. These spatial-temporal patches F input are adjacent along the temporal dimension, and neighborhoods in each patch is consistent along the spatial dimension.
(2) Self-Attention The self-attention mechanism measures intra-term semantic affinities within a data sequence by mapping a query and a set of key-value pairs. Following this mechanism, the self-attention block is designed to learn the similarities of the above spatio-temporal patches. Given query matrix Q, key matrix K, and value matrix V , the query, key and value are generated by linear transformations based on the input features F input ∈ R N ×d : where Q, K, V ∈ R N ×d , W q , W k , W v ∈ R d×d are the shared learnable linear transformation implemented by an MLP function, and d is the dimension of the query and key vectors. We then use the query and key matrices to calculate the attention weights A 1 : These weights are scaled by √ d to produce A 2 = (a 2 ) i,j : A 2 is further then normalized to produce A 3 = (a 3 ) i,j : Then, the features F sa out with self-attention are obtained as the weighted sums of the value vector using their attention weights: Finally, the features produced by self-attention F sa out and the input features F input are further concatenated to obtain the feature F output through residual connection, a feed-forward network F eedF orward(), and a layer normalization function LayerN orm(): The self-attention operation introduces randomness in the initial weights and tends to make the output features different from the input feature semantics. To reduce the influence of this randomness and accelerate the model fitting process, skip connection is used to incorporate the input feature and the output feature. Moreover, the difficulty to converge is a generally recognized problem with the original transformer [28]. Layer normalization is employed to accelerate the model training speed and to improve the robustness of the model by normalizing the output features.

Adaption to 3D Action Recognition
To further evaluate the ability of P ST 2 for modeling dynamic point cloud sequences, we also employ P ST 2 to perform 3D action recognition.
In a sequence-level classification task (e.g., action recognition), our P ST 2 model takes a multi-frame point cloud sequence as its input, and predicts the label of the sequence (e.g., action category). Since the RE module is specifically designed to enhance the resolution of a segmentation network (implemented with an encoder-decoder structure), we remove the RE module in the classification task. In addition, to intuitively show the effectiveness and easy portability of our STSA module, we improve an existing model MeteorNet [20] with our STSA module for action recognition. As shown in Fig. 1(b), we insert the STSA module into the encoding layer of MeteorNet. The remaining settings are the same as those in [20]. Note that, the meteor module proposed in [20] adopts a direct grouping method to roughly group the points in a spatial-temporal neighborhood and further extracts features, which ignore the correlation between these spatial-temporal neighborbors. After inserting our STSA module, more spatial-temporal context information can be obtained and performance can be improved. The effectiveness of the STSA module for a long sequence is evaluated with experimental comparison (see Sec. 4.4).

Experiments
In this section, we first introduce the Synthia [25], Se-manticKITTI [1], and MSRAction3D [18] datasets and their experimental settings in Sec. 4.1. Then, we report our results on these datasets and compare our model with several existing methods (Secs. 4.2 ∼ 4.4). Finally, we provide ablation analyses on two spatial-temporal correlation strategies (i.e., STSA and RE) proposed in this paper and justify their contributions in Section 4.5.

Datasets and Settings
We evaluate our P ST 2 model on the Synthia and Se-manticKITTI datasets for 4D semantic segmentation, and the MSRAction3D dataset for 3D action recognition.

4D Semantic Segmentation
We conducted two 4D semantic segmentation experiments on the large-scale synthetic dataset Synthia [25] and the largest publicly available real LiDAR dataset Se-manticKITTI [1]. Then, we compare our P ST 2 model with the MeteorNet [20], ASAP-Net [2], and MinkNet [5] baselines and perform an ablation study on the SemanticKITTI dataset.
Synthia The dataset consists of six sequences of driving scenarios in nine different weather conditions. Each sequence includes 4 stereo RGB-D images from four viewpoints captured from the top of a moving car. Following [20], we reconstruct 3D point clouds from RGB and depth images, and then employ the FPS method to downsample the point cloud to 16,384 points per frame. We employ the mean Intersection over Union (mIoU) and the mean Accuracy (mAcc) as the evaluation metrics in our experiments.
SemanticKITTI The dataset provides 23,201 frames for training and 20,351 frames for testing. To compare our method with the state-of-the-art method ASAP-Net [2], we follow the multi-frame experiments proposed in [2]. We only use the mean Intersection over Union (mIoU) as the evaluation metric in our experiments.

MSR-Action3D
The MSR-Action3D dataset consists of 567 depth map sequences acquired by Kinect v1. The depth map sequences include 10 different people, 20 categories of actions, and 23,797 frames. We reconstruct point cloud sequences from these depth maps, and use the same train/test split as previous work [20,30]. The overall accuracy is used as the evaluation metric in this experiments.

Implementation Details
The main hyper-parameters adopted in the experiments are reported in Table 1. In our P ST 2 model, we set the number of sampling points in the stacked feature encoding layers as [2048,512,128,64] in the Synthia experiment. Other implementation details, including those for the experiments on Synthia and SemanticKITTI, are set the same as in ASAP-Net [2]. For 3D action recognition, we improve the Me-teorNet [20] baseline with our STSA module, while other settings of the network are the same as MeteorNet [20]. Moreover, we follow the implementation of FPS provided by PointNet++ [24], which is implemented in GPU. All experiments are conducted on an NVIDIA RTX 2080Ti GPU using the ADAM optimizer. Table 2 reports the semantic segmentation results on the Synthia dataset. Our P ST 2 consistently outperforms MinkowskiNet [5], MeteorNet [20], and ASAP-Net [2]. It establishes a new state-of-the-art result with a mIoU of 81.86 and a mAcc of 87.03. Further, our method achieves the best performance under 5 out of 12 categories. This is mainly caused by the fact that our P ST 2 model can capture the spatial-temporal context information in a point cloud sequence.

Evaluation on SemanticKITTI
In the SemanticKITTI experiment, the state-of-the-art method ASAP-Net [2] adopts PointNet++ (PNv2) [24] as its backbone network to be incorporated with our ASAP module. For rigorous comparison, we employ the same encoder as ASAP-Net [2] to incorporate with the proposed STSA and RE module in our P ST 2 . Table 3 shows the semantic segmentation results on the SemanticKITTI dataset. As shown in Table 3, it can seen that our P ST 2 outperforms PNv2 by a large margin (16.4% in mIoU). Further, our P ST 2 surpasses the state-of-the-art method ASAP-Net [2] by 3.2% in mIoU and achieves a consistent performance improvement in 15 out of 19 categories. To provide an intuitive comparison, we visualize two segmentation results from the SemanticKITTI dataset, as shown in Fig. 4. It can be seen that our P ST 2 can accurately segment most objects, even in some complex scenes. For example, as can be seen from the first row of Fig. 4, most points on the objects of the motorcyclist category (c8) and the trunk category (c16) are misclassified as unlabeled category by ASAP-Net [2], but correctly classified by our P ST 2 . This is mainly because our RE module can enhance the resolution of the features in each frame, and our STSA module can better incorporate the spatial-temporal context information between adjacent frames. Table 4 presents the classification results on the MSRAc-tion3D dataset. MeteorNet [20] achieves a significant improvement under the different number of input frames in a sequence by plugging our STSA module. Specifically, given 16 frames in a sequence, our classification pipeline achieves the best performance among these methods, with a mAcc of 89.22. Given 8 frames in a sequence, the most remarkable improvement of 5.39% is achieved by our method. Moreover, it is observed that better classification results can be achieved with more frames in a sequence. That is because, our STSA module can better capture the inter-frame spatial-temporal context information. Given more frames in a sequence, more information can be encoded by our method.

Ablation Study
In this section, we analyze the impacts of two proposed modules on the semanticKITTI dataset.

Method
param (M)  nframe  mIoU  c1  c2  c3  c4  c5  c6  c7  c8  c9  c10  c11  c12  c13  c14  c15  c16  c17  c18   tion of the RE strategy. First, it can be seen from Table 5 that the PNv2 [24] network achieves better performances by incorporating our RE-module (i.e., PNv2+RE). That is, the mIoU is improved by 18.5%. Moreover, with the RE strategy, PNv2+RE outperforms PNv2 in all categories, and the performance is improved by up to 38.9% in the terrain category, and 31.7% in the car category. Second, the RE strategy contributes 4.1% in terms of mIoU, and achieves better performance in all categories. Specifically, the mIoU gains introduced by the RE strategy on all these categories vary from 0.1% to 10.5%. Large improvements can be observed on parking, building, and motorcyclist.

STSA Strategy
The STSA strategy not only improves the static point cloud pipeline with the ability to process point cloud sequences, but also makes the model achieve better performance. The qualification results are reported in Table 5. It can be seen that PNv2+STSA outperforms PNv2 on all categories with our STSA module. Specifically, the performance is improved by 12.3% in terms of mIoU. The improvement is up to 36.5% in the terrain category, and 29.7% in the car category.

Conclusion
In this paper, we proposed a point spatial-temporal transformer framework to effectively modeling raw 3D point cloud sequences. Our P ST 2 model consists of two core modules: the spatio-temporal self-attention module and the resolution embedding module. We apply the STSA module to capture inter-frame spatial-temporal context information, and the RE module to aggregate inter-neighborhood features to enhance the resolution. Extensive experiments demonstrate that our P ST 2 model outperforms existing methods on the tasks of 4D semantic segmentation and 3D action classification. Further, these two modules can be easily plugged into existing static point cloud processing frameworks to achieve remarkable performance improvements.


An smart toilet integrated performance detection device combined with machine vision

The traditional smart toilet comprehensive detection device proves unsatisfying on integration, intelligence, time and effort cost, and precision. In order to improve the detection accuracy and efficiency of smart toilets test, and overcome the poor integration of equipment and low intelligence, image processing technology is applied to the comprehensive performance detection equipment for smart toilets in this paper. Lenovo all-in-one computer is used as the upper computer to realize the collection and analysis of key data. Siemens PLC, as a lower-level computer, controls multiple temperature water tanks, frequency converters and pipeline valves, and collects and transmits data on temperature and cleaning power. Meanwhile, CCD camera is innovatively introduced to locate the seat temperature test points on the smart toilet. The test results show that the detection accuracy, efficiency and integration of the device are improved obviously.

Introduction
With the continuous development of society, smart toilets have received widespread attention, various products have emerged one after another, and standards have also emerged and are constantly updated [1].Standard requirements are constantly improving, so testing equipment needs to keep pace with the times. At present, there are many standards related to smart toilets in China. such as: JIS A 4422:2011 "Warm water wash toilet seat", GB/T 34549-2017 "Sanitary ware smart toilet", GB/T 23131-2008 "Electronic Toilets", JG/T 285-2010 "Seat Bidet", CBMF 15-2016 "Smart Toilets"; Energy efficiency and water efficiency standards: GB 38448-2019 "Smart Toilet Energy and Water Efficiency Limit Values and Grades " [2].
At present, when detecting the heating function of the seat in the standard of smart toilet at home and abroad, it is necessary to locate 6 to 30 temperature measuring points on the surface of the seat. For seats with different shapes and sizes, manual drawing line positioning is time-consuming and labor-intensive, and the accuracy is not high. When testing the life of smart toilet, there are different kinds of smart toilet. The control methods include buttons, touch screen, etc. the trigger directions are different. The trigger on the horizontal plane and the trigger in the vertical direction require more stations and larger area. The fixed cleaning power detection tool has poor applicability. When the water temperature stability test is performed, the time-temperature curve collection frequency is not high. The existing equipment comprehensive detection device proves unsatisfying on integration, intelligence, time and effort cost, and precision [3].
Based on the above problems, this paper designs an integrated detection platform for smart toilet, which uses machine vision technology to locate quickly and efficiently, uses six-degree-of-freedom manipulator to detect all-round life of products, uses high-speed acquisition card to collect cleaning force value, a multi-compatible mechanical tooling is designed for clamping different forms of products. The equipment has high precision, wide standard coverage and good stability. It is suitable for comparison and research of various detection methods. With the help of new equipment, it explores better quality detection and evaluation methods, which continuously improves the detection ability and fills the blank of intelligent equipment in the field of smart toilet detection.

Design and implementation
This device uses an integrated machine as the host computer to realize the collection and analysis of key data; the manipulator system is used to realize the clamping and positioning of the relevant performance fixtures; the vision system provides coordinate positioning for the manipulator; the Siemens PLC is used as the lower machine to control multiple temperature water tanks, frequency converters and pipe valves, as well as to collect and transmit temperature and cleaning force data. Figure 1 shows the design of the system.
The device is mainly composed of two parts: the control unit and the water supply unit. The control unit completes the hardware control, data processing and human-computer interaction of the whole system, including the integrated machine, the PLC、 manipulator and the visual control system. The visual system provides position data through RJ45.By PLC，the water supply unit realizes constant temperature and constant pressure water supply and field data acquisition.

control unit
The integrated machine is used as the upper computer of the whole system, which is connected with Siemens S7-200smartPLC、 vision system, Turing robot controller and temperature checker JK-32U. Among them, Siemens S7-200smartPLC externally connects analog quantity acquisition module, analog quantity output module, thermocouple module, Turing robot controller, and so on.
Siemens AE08 analog data acquisition module and AT04 temperature thermocouple module respectively realize the acquisition of the toilet water supply water pressure, outlet water temperature, air temperature and wind speed, flow rate, cleaning force and cold and hot water supply pipe temperature and other key data. Siemens AQ04 analog output module realizes the frequency conversion control of water supply water pressure. Temperature checker JK-32U is used to realize the data collection of the temperature points of the toilet seat; the six-axis manipulator grips the fixture for the performance test. The coms camera obtains the position of each temperature measurement point of the toilet seat. The coms camera and the six-axis manipulator combine to accurately and efficiently draw the position of each temperature measurement point.  Figure 2. System control structure diagram

the water supply unit
The water supply unit is mainly composed of multiple water tanks with different temperatures, water pumps, supporting frequency converters, heating pipes, refrigerators, pressure stabilizer tanks, various on-site data acquisition sensors, pipelines and solenoid valves. The structure diagram of the water supply unit is shown in Figure 3. This paper designed a comprehensive performance testing project that meets multiple standards such as GB 38448-2019 " Smart toilet energy efficiency and water efficiency limit value and grade", GB/T 34549-2017 " Smart toilet for sanitary ware", GB/T 23131-2019 "Electric toilet seat for household and similar purposes", CMBF 15-2019 "Smart toilet" and so on. All the water supply system is collected by PLC through real-time water pressure and pipe temperature, the pump is controlled by frequency converter, combined with solenoid valve, the constant pressure water supply is realized by steady pressure tank; As the standard requires multiple temperature points for the inlet water temperature, in order to ensure the stability of the inlet water temperature, this device uses a multi-temperature loop pipeline. At the beginning of the performance test, the solenoid valve is controlled to open and the inlet pipeline The water and the water tank circulate in real time. When the test conditions are reached, the circulating solenoid valve is closed to realize the water circulation in the pipeline and the water tank, and realize constant temperature water supply. The temperature stability of the whole water supply system is ±0.5℃and the pressure stability is ±0.005 MPa, which improves the stability obviously.

Seat ring detection system based on machine vision
The seat ring temperature measurement control system of this device is composed of a vision system and a 6-axis manipulator. The vision system is composed of a CCD camera, a light source, a vision controller, and a display. The physical map is shown in Figures 4 and 5. This vision system adopts advanced series of CA series cameras with dense resolution layout, and has two data interfaces: network port and U port; VB2200 vision controller is a central processing device for integrated control and data processing of machine vision. The vision system has visual interfaces such as network port, USB3.0、 light source interface, IO interface and so on. At the same time, it provides rich API interface function library to realize diversified control. the manipulator is connected to the vision controller through the Ethernet TCP protocol. After the controller receives the position coordinates fed back by the vision system, it accurately locates the corresponding point on the toilet seat through internal calculations. The structure diagram of the system is shown in Fig. 6. Machine vision is a visual function system that replaces human eye with machine vision products, which consists of light source, camera, vision controller and so on [4]. The camera takes and generates video images and transmits them to the camera image acquisition device in real time. The image acquisition device converts the received video image signal into a digital image signal which is easy to ICAMMT 2021 Journal of Physics: Conference Series 1885 (2021) 022008 IOP Publishing doi:10.1088/1742-6596/1885/2/022008 6 process by computer and sends it to the visual controller. The controller (image scaling, filtering, enhancement, edge extraction) processes the output results after the operation, at the same time guides the manipulator to accurately locate a certain point in three-dimensional space for the operation [5].
The manipulator communicates with the visual controller by modbus tcp protocol and is effectively connected by configuring the IP address, the specific settings are as follows: the manipulator teach pendant finds the "wired" control button and sets it. The camera captured pictures are processed by visual controller algorithm to obtain the accurate coordinates of all temperature measuring points in three-dimensional space, and feedback to the manipulator controller. The manipulator receives the coordinate signal of the space position and runs to the coordinate point in turn. According to the same way above, the manipulator completes the drawing of multiple temperature measuring points.

Experimental analysis
The stability and efficiency of the whole device are verified by testing the stability of sitting temperature and effluent temperature of several smart toilet devices on the market. It can be seen from Table 1 that the temperature fluctuation of the single-point sitting temperature detection is within ±0.5°C; Table 2 shows that when the inlet water temperature meets the standard, the water temperature fluctuation is within ±0.5°C; From Table 3, it can be seen that the manipulator greatly shortens the time for sitting temperature detection. Tests show that the detection accuracy, efficiency, and integration of this device have been significantly improved.

Conclusion
The smart toilet is a product of interdisciplinary integration with various performances. The integrated performance testing device of the smart toilet based on the industrial computer designed in this paper improves the temperature and water pressure stability of the device through the redesigned pipeline. At the same time, combining machine vision and manipulators, a set of automatic sitting temperature positioning system is designed, and a CCD camera is introduced innovatively to realize the detection of the sitting temperature of the smart toilet and improve its position accuracy. The experimental results show that the detection accuracy, efficiency, and integration of the device have been significantly improved, which solves the detection of multiple comprehensive performances of the smart toilet, and has high application value.

Project funded:
Shanghai Market Supervisory Authority projects 《 Development and application of intelligent comprehensive testing device for smart toilet》 (2019-39)


extfieldsofstudy:
Physics
Computer Science

text:
An approach to multi-satellite TT&C resource scheduling based on multi-agent technology and comprehensive weighted priority determination method

The purpose of satellite tracking telemetry and control (TT&C) resource scheduling is to scientifically allocate limited ground TT&C resources, and to maximize the scheduled tasks of satellite TT&C through a certain scheduling method. Due to the sharp increase in satellite numbers, Traditional scheduling methods face challenges with more TT&C tasks, more resource competition and more emergencies. As the agent is independent, flexible, intelligent and distributed, we introduced agent technology to solve the problem. Combining characteristics of multi-satellite TT&C resource schedule with description method of attribute, interaction, rule and state on agent technology, we established four agent models: manager agent, station agent, equipment agent and task agent. Then, by analysing the requirements of satellite TT&C engineering, we proposed a comprehensive weighted priority determination method to determine TT&C resource allocation. Last, we simulated real application scenarios to validate this method. And it proves to be effective.

Introduction
TT&C resources scientific scheduling is one of the key elements to guarantee the TT&C engineering and security operating management of spacecraft. The research of Multi-Satellite TT&C Resource Scheduling (MSTCRS) has great engineering significance. It can reduce the huge cost of spacecraft TT&C management and TT&C resource construction management.
In this paper, we study how to schedule TT&C resources to make the limited ground TT&C resources play maximum application efficiency, so that the TT&C requirements of growing satellite numbers can be met. In recent years, the contradiction of TT&C station resources, satellites and TT&C requirements is intensified, which increases the complexity of MSTCRS. It also increases the probability of emergencies and the difficulty of manually adjusting scheduling plan. Meanwhile, the scheduling system is urgently required to respond quickly and output new scheduling plan with minimum changes and take the original scheduling plan constraints into account. Therefore, a flexible, intelligent and efficient resource scheduling method is needed.
Now there are many studies on MSTCRS such as constraint programming method, heuristic algorithm, AI algorithm and agent technology. Although National Aeronautics and Space Administration (NASA) and Air Force Institute of Technology (AFIT) [1][2][3] proposed constraint programming methods which can accurately describe the problem, the constraints they considered are very complex. European Space Agency (ESA) proposed a Lagrangian-based heuristic algorithm [4]. It a) Manager agent collects new tasks, and forwards them to task agent; b) Task agent preprocesses these tasks and sends them back to manager agent. c) Manager agent sends pre-processed tasks to station agents. d) Station agents send pre-processed tasks to equipment agents. e) Manager agent sends a resource allocation start signal to station agents. f) Station agents send the signal to their equipment agents. 3 g) Equipment agents get the signal and send optimal resources and corresponding tasks to station agent they belong to. h) Station agents make a preliminary judgment on these feedbacks and send evaluation results to manager agent. i) Manager agent makes a final judgment and feedback the evaluation result to station agents. j) Station agents forward it to equipment agents. k) Equipment agents update resources and send updating results to station agents. l) Station agents send them to manager agent.

Agent description
These agents are described as follows.

Manager agent.
Manager agent is the core role of MSTCRS, and manages TT&C resource scheduling and allocation. It is responsible for the unified reception and distribution of TT&C tasks, beginning of resource allocation, and the evaluation of optimal resources. The model of manager agent is described as follows: Manager={MID, MName, MbeforeTasks, MafterTasks}, Manager refers to manager agent which is unique. Its attributes are as follows: MID is identifier; Mname refers to name; MbeforeTasks refers to tasks to be scheduled; MafterTasks refers to tasks which have been preprocessed and filtered by task agent.

Task agent.
Task agent pre-processes tasks send from manager agent. Based on the approach we proposed, the pre-processing operations include deleting invalid tasks, converting periodic tasks and separable tasks into some one-time tasks based on their periodicity or task split constraints. Besides, since geostationary earth orbit (GEO) satellites are visible all day to ground TT&C equipment, we allocate resources for GEO satellite tasks during the preprocess stage. The model of task agent is as follows: Task={TID,TName, beforeTasks, afterTasks, hightTasks, failTasks, sucessTasks}. Task refers to task agent which is unique; TID is its identifier; Tname refers to task agent's name; beforeTasks refers to tasks to be scheduled; afterTasks is tasks filtered by the task agent; hightTasks is tasks of GEO satellites; failTasks is tasks failed to allocate resources and sucessTasks refers to tasks that successfully allocate resources.

Station agent.
TT&C Station is used to track, measure, telemeter, remote control and communicate with the spacecraft. It includes all the equipment in a certain area. Station agent manages the attribute information of the equipment agents belonging to it, receives and forwards tasks to be scheduled. It selects the optimal couple of tasks and matching resources sent by equipment agents. The station agent is described as follows: Station={Station 1 ,Station 2 ,…, Station i }. i refers to the number of station agent. Station i ∈Station denotes the i-th station agent. Station i ={ID i , Name i , Type i , Status i }, ID is a unique station agent identifier; Name refers to the station's name; Type is the category of TT&C station; Status is availability status of station.

Equipment agent.
TT&C equipment includes TT&C antennas, BBE, converter, etc. Equipment agent provides autonomous and intelligent management of equipment using status, TT&C capability, and TT&C resource allocation. The intelligent management of TT&C resource allocation refers to find all possible resources for tasks according to the constraints of MSTCRS and make up a two-tuple; Then selecting the optimal one from the two-tuple. Finally, updating the two-tuple and attribute information of the equipment agent according to the TT&C resource allocation result sent by manager agent. The management of TT&C equipment using status means that if the equipment is unavailable due to equipment failure or other reasons, all tasks not performed in the equipment are set as pending tasks and sent to the manager agent for replanning. The equipment agent is described as: Equipment={Equipment 1 ,Equipment 2 , …, Equipment j }, where j refers to the number of equipment agent. Equipmentj ∈Equipment is the j-th equipment agent. Equipmentj={ID j , Name j , DF j , Ur j , Status j , Starttime j , Endtime j , Station j , Pr j , St j , Servce j ,Etasks j , Ett j , Estt j }.ID is an equipment agent identifier; Name refers to the name of equipment; DF is frequency band; Ur is equipment's resource utilization; Status is using status; Starttime is start time for equipment availability; Endtime is end time for equipment availability; Station is the information of the TT&C station the equipment belongs to; Pr is equipment's priority; St is switch time; Servce refers to service capabilities. Etasks refers to all tasks of equipment. Ett refers to the two-tuple of tasks and their possible resources. Estt refers to the twotuple of tasks and their successful scheduled resources.

Entities.
In the above agents, the following entities are used: 1) Satellite which describes satellite attribute information, such as orbit type, lifetime, TT&C frequency band, operation phase, equipment preference, use attributes (manned, military, civilian), static priority; 2) Task that describes the attribute information of TT&C tasks, such as start time and end time of a task, task type (injected, TT&C's, orbital, general), task nature (normal, emergency), task category (combat, joint debugging), task periodicity, task duration; 3) User Preference which describes the preferences in the process of TT&C scheduling, such as time preference, equipment preference. 4) Visible Time Window describes the visible time segments of ground stations to satellites, during which the satellite can be measured and controlled. Its attributes are: start time, end time, duration, owning satellite, owning station, priority value, type of laps (rising, falling).

Analysis of the characteristics of MSTCRS
The MSTCRS problem is characterized by high complexity, high conflicts, and multiple constraints. It is mainly manifested in the following aspects: a) More satellites and fewer stations. b) Satellite, TT&C equipment and TT&C requirements are various, and have their own characteristics. c) There are many constraints in the scheduling process, such as time constraints, TT&C task constraints, TT&C equipment constraints, satellite constraints, etc. Based on the above analysis, the constraints of MSTCRS are as follows: a) A set of equipment tracks one satellite at a time until the end of the visible time window. b) One TT&C task can only select one visible time window to execute. c) Telemetry tasks and telecontrol tasks of a satellite can be performed in parallel. d) The frequency band of the TT&C equipment and the satellite must match with each other. e) There must be a time interval determined by the satellite's characteristics between two consecutive tasks of a same satellite. f) When two tasks in different satellites are executed continuously on a piece of equipment, the interval time between them is greater than or equal to the switching time of the equipment. g) The key issue of MSTCRS is to schedule visible time window to corresponding task. The scheduling target aims to maximize the satisfaction of TT&C tasks. Based on the constraint, the MSTCRS problem model can be described as: where, vt donates the set of visible time windows, task donates set of tasks, p i donates priority of task i , x ij donates whether visible time window vt j is allocated for task i , that is : and each task can only be executed once, that is ∑ ∈ ≤ 1 and ∈ ∩ ∈ ≠ Ø where, satellite is the set of satellites. Sf refers to frequency band of satellite. DF refers to frequency band equipment.
where tstart is the start time of task, tend is the end time of task. vtstart is the start of visible time widow. vtend is the end of visible time widow. tdur is the duration of task. St is the switch time of equipment.

Algorithm design
This section describes the algorithms in TT&C resource scheduling method proposed. The first algorithm is called Comprehensive Weighted Priority Determination Method (CWPDM) which determines the order of resource scheduling. CWPDM runs in running rules of agent. Then we describe the overall process of resource scheduling which is called Multi-Satellite TT&C Resource Scheduling Algorithm Based on Multi-Agent Technology and Comprehensive Weighted Priority Determination Method (MSTCRSA). In the method, agents are used as architecture. They are responsible for transmitting information such as task set, scheduling result, agent status, etc. CWPDM is one of running rules of agent and can select an optimal couple of task and resource. Before CWPDM begins, we assign priority values to factors that can affect resource scheduling results. The priority value is determined by the degree of impact on resource scheduling. The higher the priority is, the bigger the priority value is. Generally speaking, a spacecraft with the characteristic of LEO, short-life, military, new, etc., has higher priority. Then we calculate the comprehensive priority value of factors of each element in the two-tuple DATA which consists of tasks and possible resources allocated to them, and select the resource with the highest priority value to allocate for a task; if there are elements with the same priority value, we choose the one with fewer resource conflicts between visible time windows. The priority value and the items of influencing factors can be adjusted according to the requirements of person and resources. The factors that affect the static priority of resources in step (4) of CWPDM include the attributes of the tasks, such as task type, task nature, task category. And the attributes of the spacecraft, such as orbit type, lifetime, using attributes, the priority of equipment, etc.
The factors of dynamic resource priority in the step (4)  Output: SCHEDULE which is the result of TT&C resource scheduling. Prerequisites: Each Agent has been started and initialized.

Begin
(1)Manager agent gets MbeforeTasks from batch tasks that are newly created according to the TT&C requirements, tasks that need to reallocate resources due to equipment or satellite failures, and emergency tasks; (2)If the number of elements in the TT is not zero, and the number of MbeforeTasks is large, the tasks that have not been executed in the TT are integrated into MbeforeTasks to reallocate resources; (3)Manager agent sends MbeforeTasks to task agent to preprocess; (4)Manager agent sends MafterTasks processed by task agent to station agents; Station agents send them to their equipment agents; (5)According to the task constraints, time constraints, resource constraints, preferences and other factors, equipment agents allocates TT&C resources for these tasks to make up TTA, a two-tuple of tasks and resources allocated to tasks. Then, they traverse all visible time windows in the equipment agent and calculate the number of conflicts between visible time windows; (6)Traversing each element in the TTA and calculating its priority value according to CWPDM; (7)The elements in the TTA are compared with the elements in the TTB, a two-tuple likes TTA. Besides, when tasks have been allocated resources successfully and have not yet been executed in station, if resource contention occurs and the elements in the TTA have a higher priority, tasks are need to re-allocate resources. Then updating TTA, TTB, and merge them into TTA'; (8)While there are tasks that have not been allocated resources do (9) Manager agent sends the resource allocation signal to station agents, and station agents send it to equipment agents; (10) Each equipment agent receives the signal, and selects the element with the highest priority in the TTA' and feeds back it to the station agent it belongs to; (11) Each station agent selects the optimal element from its equipment agents according to CWPDM and equipment's resource utilization, and feeds it back to manager agent; (12) Manager agent selects the optimal one from the resource allocation results sent by all station agents, and feeds back the result to all station agents. Then station agents return it to equipment agents.
Each equipment agent receives the result, and then updates the elements in the TTA' and the priority of corresponding elements. After that, sending the message that resource update has occurred to the station agent it belongs to. Station agent feeds it back to the manager agent; (14) After receiving the messages from station agents, manager agent completes the resource allocation of a task, stores the element made up of task and the resource which is allocated to into SCHEDULE, and updates MafterTasks.

Experimental scenarios
The experimental scenarios are divided into two categories: Scenario 1: High-volume tasks from TT&C requirements; Scenario 2: some TT&C tasks that occur during the scheduling process. a) Data Preparation Satellite: 100 satellites which are simulated by STK according to real satellites' parameter information. Station: 20 ground stations. TT&C Task: the TT&C tasks of each satellite are basically 2 laps for ascending laps and 2 laps for descending laps every day. There are about 400 TT&C tasks every day, which simulates the supplydemand relationship of "more satellites and fewer stations".
Station's visible time window to satellite: STK produces visible time windows based on the satellite and station parameter information. b) Experiment Procedure Scenario 1: We prepare 4 groups of TT&C tasks. They are tasks of one day, three days, five days, and seven days. Then we use these data to do experiment to verify the efficiency of MSTCRSA, and record resource schedule results and running times.
Scenario 2: First, we prepare TT&C tasks of one day for 3 groups, and call MSTCRSA to allocate resources for them; Then we prepare 30 unexpected TT&C tasks for each group, and respectively call the traditional algorithm (TA) and MSTCRSA; Finally recording running time. TA is the method of multi-satellite TT&C resource scheduling without using agent technology.

Experimental Results
We analyze the experiment results from the following three aspects: the weighted satisfaction of TT&C tasks, the weighted resource utilization, and running time. 4 groups of data are used, and the experimental results are shown in the table below.   As shown above, Table 1 and Figure 2 are the experimental results of Scenario 1. It can be seen that task satisfaction is above 99%, and the resource utilization rate is above 80%. And as the number of tasks increases, the running time is line-like. MSTCRSA is suitable for the scene of high-volume TT&C tasks with "more satellites and fewer stations". When encountering unexpected situations as shown in Scenario 2, it can be seen from table 2 and figure 3 that the scheduling time of MSTCRSA is much less than TA and is only a quarter of TA. Figure4 is the gantt chart of scheduling results in scenario 1. It displays the time and equipment distribution. In summary, MSTCRSA proposed in this paper is highly efficient to solve the problem of MSTCRS.

Conclusion
In this paper, we analyze the characteristics and engineering requirements of MSTCRS, establish the agent optimization model, propose MSTCRSA to complete TT&C resource allocation, and simulate the real application scenario of "more satellites, fewer stations" to do experiments. Through large amount of experiments, MSTCRSA is proved to be efficient. It improves the efficiency, adaptability and the satisfaction degree of TT&C resource allocation. And it can solve the scheduling plan adjustment caused by perturbational factors during the execution of resource scheduling plan, and quickly output the new scheduling plan that takes into account the constraints of original scheduling plan.
In the future, we plan to further study influencing factors of MSTCRS and these factors' priority values.