summaries: This paper aims to improve adversarial robustness of the classifiers in a different perspective than the existing works. Usually, the networks are trained using adversarial examples to improve robustness (adversarial training). This work extend this line of thought and make an input robust to adversarial attacks. Instead of updating the network, they make updates to the input to gain robustness. In other words, this work explore the existence of safe spots near the input samples that are robust against adversarial attacks. Results on CIFAR-10 and ImageNet reveals that there exists such safe spots which are resistant to adversarial perturbations and improve adversarial robustness when combined with adversarial training (the authors term it as safe-spot aware adversarial training). Based on this approach, the authors also propose out-of-distribution detection method that outperforms previous works.

summaries: This paper proposes a new adversarial framework where the defender could preemptively modify classifier inputs to find safe spots that are robust to adversarial attacks. They then introduce a novel bi-level optimization algorithm that can find safe spots on over 90% of the correctly classified images for adversarially trained classifiers on CIFAR-10 and ImageNet datasets and show that they can be used to improve both the empirical and certified robustness on smoothed classifiers. Besides, they propose a new training scheme based on their conjecture about safe spots for out-of-distribution detection which achieves state-of-the-art results on near-distribution outliers.

summaries: The authors argue that there are some safe "spots" in the data space that are less prone to adversarial attacks. The authors propose a technique to identify such "safe spots". They then leverage them for robust training and observe higher robust accuracy than baseline. Finally, they leverage this observation to identify out of distribution data.

abstractText: Recent advances on adversarial defense mainly focus on improving the classifier’s robustness against adversarially perturbed inputs. In this paper, we turn our attention from classifiers to inputs and explore if there exist safe spots in the vicinity of natural images that are robust to adversarial attacks. In this regard, we introduce a novel bi-level optimization algorithm that can find safe spots on over 90% of the correctly classified images for adversarially trained classifiers on CIFAR-10 and ImageNet datasets. Our experiments also show that they can be used to improve both the empirical and certified robustness on smoothed classifiers. Furthermore, by exploiting a novel safe spot inducing model training scheme and our safe spot generation method, we propose a new out-of-distribution detection algorithm which achieves the state of the art results on near-distribution outliers.

authors:
Anish Athalye
Nicholas Carlini
David Wagner

title: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples

authors:
Yair Carmon
Aditi Raghunathan
Ludwig Schmidt
Percy Liang
John Duchi

title: Unlabeled data improves adversarial robustness

venue: In NeurIPS,

authors:
Jeremy M Cohen
Elan Rosenfeld
J Zico Kolter

title: Certified adversarial robustness via randomized smoothing

venue: In ICML,

authors:
Francesco Croce
Matthias Hein

title: Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

authors:
Chelsea Finn
Pieter Abbeel
Sergey Levine

title: Model-agnostic meta-learning for fast adaptation of deep networks

authors:
Ian J Goodfellow
Jonathon Shlens
Christian Szegedy

title: Explaining and harnessing adversarial examples

venue: In ICLR,

authors:
Chuan Guo
Mayank Rana
Moustapha Cisse
Laurens van der Maaten

title: Countering adversarial images using input transformations

authors:
Dan Hendrycks
Kevin Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks

authors:
Dan Hendrycks
Kimin Lee
Mantas Mazeika

title: Using pre-training can improve model robustness and uncertainty

authors:
Dan Hendrycks
Mantas Mazeika
Thomas Dietterich

title: Deep anomaly detection with outlier exposure

authors:
Dan Hendrycks
Mantas Mazeika
Saurav Kadavath
Dawn Song

title: Using self-supervised learning can improve model robustness and uncertainty

authors:
Alex Krizhevsky
Geoffrey Hinton

title: Learning multiple layers of features from tiny images

venue: Technical report, Citeseer,

authors:
Mathias Lecuyer
Vaggelis Atlidakis
Roxana Geambasu
Daniel Hsu
Suman Jana

title: Certified robustness to adversarial examples with differential privacy

venue: In IEEE Symposium on Security and Privacy (SP),

authors:
Kimin Lee
Kibok Lee
Honglak Lee
Jinwoo Shin

title: A simple unified framework for detecting out-of-distribution samples and adversarial attacks

venue: NeurIPS,

authors:
Xuanqing Liu
Minhao Cheng
Huan Zhang
Cho-Jui Hsieh

title: Towards robust neural networks via random self-ensemble

venue: In ECCV,

authors:
Aleksander Madry
Aleksandar Makelov
Ludwig Schmidt
Dimitris Tsipras
Adrian Vladu

title: Towards deep learning models resistant to adversarial attacks

authors:
Yuval Netzer
Tao Wang
Adam Coates
Alessandro Bissacco
Bo Wu
Andrew Y Ng

title: Reading digits in natural images with unsupervised feature learning

venue: NeurIPS-W,

authors:
Anh Nguyen
Jason Yosinski
Jeff Clune

title: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

venue: In CVPR,

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga

authors:
Chongli Qin
James Martens
Sven Gowal
Dilip Krishnan
Krishnamurthy Dvijotham
Alhussein Fawzi
Soham De
Robert Stanforth
Pushmeet Kohli

title: Adversarial robustness through local linearization

authors:
Leslie Rice
Eric Wong
J. Zico Kolter

title: Overfitting in adversarially robust deep learning

authors:
Olga Russakovsky
Jia Deng
Hao Su
Jonathan Krause
Sanjeev Satheesh
Sean Ma
Zhiheng Huang
Andrej Karpathy
Aditya Khosla
Michael Bernstein
Alexander C. Berg
Li Fei-Fei

title: ImageNet Large Scale Visual Recognition Challenge

venue: In IJCV,

authors:
Hadi Salman
Jerry Li
Ilya Razenshteyn
Pengchuan Zhang
Huan Zhang
Sebastien Bubeck
Greg Yang

title: Provably robust deep learning via adversarially trained smoothed classifiers

authors:
Chandramouli S. Sastry
Sageev Oore

title: Detecting out-of-distribution examples with gram matrices

authors:
Yang Song
Taesup Kim
Sebastian Nowozin
Stefano Ermon
Nate Kushman

title: Pixeldefend: Leveraging generative models to understand and defend against adversarial examples

authors:
Christian Szegedy
Wojciech Zaremba
Ilya Sutskever
Joan Bruna
Dumitru Erhan
Ian Goodfellow
Rob Fergus

title: Intriguing properties of neural networks

authors:
Florian Tramer
Nicholas Carlini
Wieland Brendel
Aleksander Madry

title: On adaptive attacks to adversarial example defenses

authors:
Yisen Wang
Difan Zou
Jinfeng Yi
James Bailey
Xingjun Ma
Quanquan Gu

title: Improving adversarial robustness requires revisiting misclassified examples

authors:
Eric Wong
Leslie Rice
J. Zico Kolter

title: Fast is better than free: Revisiting adversarial training

authors:
Fisher Yu
Ari Seff
Yinda Zhang
Shuran Song
Thomas Funkhouser
Jianxiong Xiao

title: Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop

authors:
Sergey Zagoruyko
Nikos Komodakis

title: Wide residual networks

authors:
Haichao Zhang
Jianyu Wang

title: Defense against adversarial attacks using feature scattering-based adversarial training

authors:
Jingfeng Zhang
Xilie Xu
Bo Han
Gang Niu
Lizhen Cui
Masashi Sugiyama
Mohan Kankanhalli

title: Attacks which do not kill training make adversarial learning stronger

heading: 1 INTRODUCTION

text:
Deep neural networks have achieved significant performance on various artificial intelligence tasks such as image classification, speech recognition, and reinforcement learning. Despite the results, Szegedy et al. (2013) demonstrated that deep neural networks are vulnerable to adversarial examples, minute input perturbations designed to mislead networks to yield incorrect predictions. There have been a large number of studies to improve the robustness of networks against adversarial perturbations (Song et al., 2017; Guo et al., 2018), while many of the proposed methods have been shown to fail against stronger adversaries (Athalye et al., 2018; Tramer et al., 2020). Adversarial training (Madry et al., 2017) and randomized smoothing (Cohen et al., 2019) are some of the few methods that survived the harsh verifications, each focusing on empirical and certified robustness, respectively.
To summarize, the study of adversarial examples has been an arms race between adversaries, who manipulate inputs to raise network malfunction, and defenders, who aim to preserve the network performance against the corrupted inputs. In this paper, we approach the adversarial robustness problem from a different perspective. Instead of defending networks from already perturbed examples, we assume the situation where the defenders can also influence inputs slightly for their interest before the adversaries’ incursion. The defenders’ goal for this manipulation will be to improve robustness by searching for spots in the input space that are resistant to adversarial attacks, given a pre-trained classifier. We explore methods for finding those safe spots from natural images under a given input modification budget and the degree of robustness achievable by utilizing these spots, which we denote as preemptive robustness. Ultimately, we tackle the following question:
• Do safe spots always exist in the vicinity of natural images?
One practical example of the proposed framework is the case where a user uploads his or her photo from local storage (e.g., mobile device) to social media (e.g., Instagram), as illustrated in Figure 1. Suppose there is an uploader (A) who posts a photo on social media, a web user (B) who queries a search engine (e.g., Google) for an image, and a search engine that crawls images from social media, indexes them with a neural network, and retrieves the relevant images to B. Our threat model considers an adversary (M) that can download A’s image from social media, perturb it maliciously, and re-upload the perturbed image on the web, where the search engine may crawl and index images from. The classifier on the search engine will wrongly index the perturbed image, causing the search
engine to malfunction. Suppose an African-American uploader (A) posts a photo of him or herself on social media, and a racist adversary (M) perturbs it to be misclassified as “gorilla” by the search engine. When another person (B) searches “gorilla” on Google, the perturbed image would appear, though the image content shows a photo of A. This attack fools both A and B since the perturbed image is used contrary to A’s purpose and is not the image B wanted. To prevent this, the social media company, cooperating with the search engine company, could ask if A agrees that the images are slightly changed when uploaded to make them robust to such attacks. The purpose of the modification process, corresponding to the “safe spot filter” in Figure 1, will be to ensure that the uploaded images are used under A’s intention and provide more accurate search results to B.
We develop a novel optimization problem for searching safe spots in the vicinity of natural images and observe that over 90% of the correctly classified images have safe spots nearby for adversarially trained models on both CIFAR-10 (Krizhevsky & Hinton, 2009) and ImageNet (Russakovsky et al., 2015). We also find that safe spots can enhance both empirical and certified robustness when applied on smoothed classifiers. Furthermore, we propose a novel safe spot inducing model training scheme to improve the preemptive robustness. By exploiting these safe spot-aware classifiers along with our safe spot search method, we also propose a new algorithm for out-of-distribution detection, which is often addressed together with robustness (Hendrycks et al., 2019a;c). Our algorithm outperforms other baselines on near-distribution outlier datasets such as CIFAR-100 (Krizhevsky & Hinton, 2009).

heading: 2 RELATED WORKS

text:
Adversarial training Goodfellow et al. (2015) first show that the robustness of a neural network can be enhanced by generating adversarial examples and including them in training set. PGD adversarial training improves the robustness against stronger adversarial attacks by augmenting training data with multi-step PGD adversarial examples (Madry et al., 2017). Some recent works report performance gains over PGD adversarial training by modifying the adversarial example generation procedure (Qin et al., 2019; Zhang & Wang, 2019; Zhang et al., 2020). However, most of the recent algorithmic improvements can be matched by simply using early stopping with PGD adversarial training (Rice et al., 2020; Croce & Hein, 2020). Other line of works achieve performance gains by utilizing additional datasets (Carmon et al., 2019; Wang et al., 2020; Hendrycks et al., 2019a).
Randomized smoothing Injecting random noise during the forward pass can smooth the classifier’s decision boundary and improve empirical robustness (Liu et al., 2018). Using differential privacy, Lecuyer et al. (2019) give theoretical guarantees for `1 and `2 robustness of classifiers smoothed with Gaussian and Laplacian noise. Cohen et al. (2019) provide a tight bound of `2 robustness of networks smoothed with Gaussian noise via the Neyman-Pearson lemma. Another proof of the robustness bound was given in Salman et al. (2019) using Lipschitz property of smoothed classifiers, where they also propose a new adversarial training scheme for building robust smoothed classifiers.
Out-of-Distribution detection with deep networks Although deep networks achieve high performance on various classification tasks, they also tend to yield high confidence in out-of-distribution samples (Nguyen et al., 2015). To filter out the anomalous examples, Hendrycks & Gimpel (2017) use the maximum value of a classifier’s softmax distribution as a score function, while Lee et al. (2018)
propose Mahalanobis distance-based metric which spots out-of-distribution samples using hidden features. Hendrycks et al. (2019b) show that leveraging auxiliary datasets disjoint from test-time data can improve the detection performance. Recently, Sastry & Oore (2020) characterize activity patterns of hidden features by Gram matrices and use the matrix values to identify anomalies.

heading: 3 METHODS

heading: 3.1 GENERAL DEFINITION OF SAFE SPOT AND PREEMPTIVE ROBUSTNESS

text: We first establish a formal definition of safe spot and preemptive robustness. Let c : X → Y be a classifier which maps images to class labels. We define the safe region of the classifier c as the set of images that c can output robust predictions in the presence of slight adversarial perturbations. Definition 1 ( -safe region). Let c : X → Y be a classifier and ∈ R+ be the perturbation budget of an adversary. The -safe region of the classifier c is defined by S (c) := {x ∈ X | c(x′) = c(x), ∀x′ ∈ B (x)}. Here, B (x) = {x′ ∈ X | d(x, x′) ≤ } is the x-centered -ball. In this paper, we assume `p threat model, i.e., d(x, x′) = ‖x − x′‖p, which is the most common setting on adversarial robustness literature, and consider p ∈ {2,∞}. Now, suppose a defender can preemptively manipulate a natural image xo under a small modification budget, knowing its ground-truth label yo. We denote the modified output image as xs. Then, the defender’s objective is to make xs be correctly classified as yo and locate in the safe region S (c) to improve the robustness against adversarial attacks. If xs satisfies these two conditions, then we say xo is preemptively robust and xs is a safe spot of xo. Definition 2 (Preemptive robustness). Let c : X → Y be a classifier and δ, ∈ R+ be the modification budgets of the defender and the adversary, respectively. A natural image xo with its ground-truth label yo is called (δ, )-preemptively robust on the classifier c if there exists a safe spot xs ∈ Bδ(xo) such that (i) c(xs) = yo and (ii) xs ∈ S (c).

heading: 3.2 SAFE SPOT SEARCH ALGORITHM

text:
In this subsection, we develop an algorithm for searching a safe spot with a natural image. Given a classifier c, finding a safe spot xs from a natural image xo can be formulated as the following problem, which is directly from the definition of safe spot:
minimize xs 1c(xs)6=yo + 1xs /∈S (c)
subject to ‖xs − xo‖p ≤ δ,
where 1 is the 0-1 loss function.
Note in this formulation the defender requires the ground-truth label yo for the safe spot search. However, images in the real-world (e.g., social media) are usually unlabeled, unless uploaders annotate labels to their images by hand. So, it is natural to assume that the defender cannot access the ground-truth label yo. In this case, we utilize the classifier’s prediction c(xo) instead of yo:
minimize xs 1c(xs)6=c(xo) + 1xs /∈S (c)
subject to ‖xs − xo‖p ≤ δ.
As xs /∈ S (c) implies there exists an adversarial example xa ∈ B (xs) such that c(xa) 6= c(xs), we can reformulate the optimization problem as
minimize xs 1c(xs)6=c(xo) + sup xa 1c(xa)6=c(xs)
subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ .
Since the 0-1 loss is not differentiable, we employ the cross-entropy loss ` : X × Y → R+ of the classifier c as the convex surrogate loss function:
minimize xs `(xs, c(xo)) + sup xa `(xa, c(xs)) (1)
subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ .
Let h(xs) denote the objective in Equation (1). Instead of minimizing h(xs), we minimize h̃(xs) = supxa `(xa, c(xo)), since it upper bounds h(xs) when sufficiently minimized by Lemma 1.
Algorithm 1 Finding a safe spot input An image and its prediction (xo, c(xo)), the cross-entropy
function ` xs = xo for i = 1, . . . ,MAXITER do
Generate N adversarial examples for n = 1, . . . , N do x (0) a,n = xs + ηn where ηn ∼ U(B (0))
for t = 1, . . . , T do x (t) a,n = Πxs, ( f(x (t−1) a,n ; c(xo), `) ) end for
end for xs ← Πxo,δ ( xs − β · 1
N N∑ n=1 ∂`(x (T ) a,n, c(xo)) ∂xs ) end for
output xs
Lemma 1. If h̃(xs) ≤ − log(0.5) ' 0.6931, then h(xs) ≤ 2h̃(xs).
Proof. See Supplementary A.1.
Finally, we have the following optimization problem:
minimize xs sup xa `(xa, c(xo)) (2)
subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ .
To solve Equation (2), we first approximate the inner maximization problem by running T -step PGD (Madry et al., 2017) whose dynamics is given by
x(0)a = xs + η (random start) x̃(t)a = f ( x(t−1)a ; c(xo), ` ) (adversarial update)
x(t)a = Πxs, ( x̃(t)a ) , (projection)
where η is a random noise uniformly sampled from the `p zero-centered -ball, f is FGSM (Goodfellow et al., 2015) defined by
f(x; y, `) =  x+ α · sgn (∇x`(x, y)) if p =∞
x+ α · ∇x`(x, y)‖∇x`(x, y)‖2 if p = 2,
and Πxs, is a projection operation to B (xs). Then, we iteratively solve the approximate problem given by replacing xa to x (T ) a in Equation (2). To update xs, we need to compute the gradient of `(x (T ) a , c(xo)) with respect to xs expressed as
∂` ( x (T ) a , c(xo) ) ∂xs = ∂f̃ ( x (0) a ) ∂x ᵀ · · · ∂f̃ ( x (T−1) a ) ∂x ᵀ · ∇x` ( x(T )a , c(xo) ) ,
where ∂f̃/∂x is the Jacobian matrix of f̃ = Πxs, ◦ f which is easily computed via back-propagation. After computing the gradient, we update xs by projected gradient descent method:
x(i+1)s = Πxo,δ ( x(i)s − β · ∂`(x (T ) a , c(xo))
∂xs
) .
Note that the loss `(x(T )a , c(xo)) is now a random variable dependent on η. Therefore, we generate N adversarial examples {x(T )a,n}Nn=1 with different noises and optimize the sample mean of the losses instead. Algorithm 1 shows the overall safe spot search algorithm and Figure 2 illustrates our optimization process.

heading: 3.3 COMPUTING UPDATE GRADIENT WITHOUT SECOND-ORDER DERIVATIVES

text:
Computing the update gradient with respect to xs involves the use of second-order derivatives of the loss function ` since the dynamics f contains the loss gradient ∇x`(x, y). Standard deep learning libraries, such as PyTorch (Paszke et al., 2019), support the computation of higher-order derivatives. However, it imposes a huge memory burden as the size of the computational graph increases. Furthermore, for the case of p = 2, computing the update gradient with the second-order derivatives might cause exploding gradient problem if the loss gradient vanishes by Proposition 1. Lemma 2. Suppose ` is twice-differentiable and its second partial derivatives are continuous. If p = 2, the Jacobian of the dynamics f is
∂f ∂x = I + α ·
( I − ( g
‖g‖2
)( g
‖g‖2
)ᵀ) H
‖g‖2 ,
where g = ∇x`(x, y) and H = ∇2x`(x, y).
Proof. See Supplementary B.1.
Proposition 1. If the maximum eigenvalue of H in absolute value is σ, then∥∥∥∥∂f∂x ᵀ · a ∥∥∥∥ 2 ≤ ( 1 + α · σ‖g‖2 ) ‖a‖2.
Proof. See Supplementary B.2.
As we update xs, the loss gradients of xs and its adversarial examples xa get reduced to zero, which might cause the update gradient to explode and destabilize the update process. To address this problem, we approximate the update gradient by excluding the second-order derivatives, following the practice in Finn et al. (2017). We also include an experiment in comparison to using the exact update gradient in supplementary B.3. For the case of p =∞, the second-order derivatives naturally vanish since we take the sign on the loss gradient ∇x`(x, y). Therefore, the approximate gradient is equal to the exact update gradient.

heading: 3.4 FINDING A SAFE SPOT FOR CLASSIFIERS WITH RANDOMIZED SMOOTHING

text:
To further enhance the robustness of our safe spot framework, we can leverage the randomized smoothing technique along with our algorithm. Given a base classifier c : X → Y , the smoothed classifier g : X → Y is defined by
g(x) = argmax y∈Y
P (c(x+ η) = y) ,
where η ∼ N (0, σ2I). To find a safe spot xs of a natural image xo, we have to find an adversarial example xa of xs that maximizes the cross-entropy loss `(xa, c(xo)) for solving the inner maximization problem in Equation (2). However, crafting adversarial examples for the smoothed classifier is ill-behaved since the argmax is non-differentiable. To address the problem, we follow the approach in Salman et al. (2019) and approximate the smoothed classifier g with the smoothed soft classifier G : X → P (Y) defined as
G(x) = E η∼N (0,σ2I) [C(x+ η)] ,
where P (Y) is the set of probability distribution over Y and C : X → P (Y) is the soft version of the base classifier c such that argmaxy∈Y C(x)y = c(x). Finally, the adversarial example xa is found by maximizing the cross-entropy loss of G instead:
maximize xa
− log ( G(xa)c(xo) ) (3)
subject to ‖xa − xs‖p ≤ ,
which can be approximated by T -step randomized PGD with M restarts, where random noises are sampled from Gaussian distribution to compute the sample mean of the objective at each step. By replacing the inner maximization problem in Equation (2) by the randomized PGD, we can update xs similarly.

heading: 3.5 SAFE SPOT-AWARE ADVERSARIAL TRAINING

text:
In Section 3.2, we investigated how the defender can find a safe spot from a natural image, given a pre-trained classifier. In this subsection, we explore the defender’s training scheme for a classifier on which data points are preemptively robust. Suppose the defender has a labeled training set, which is drawn from a true data distribution D. To induce a classifier to have safe spots in the vicinity of data points, the defender’s optimal training objective should have the following form:
minimize θ E (xo,yo)∼D
[`(x∗a, yo; θ)]
subject to x∗a = argmax xa∈B (x∗s) `(xa, yo) and x∗s = argmin xs∈Bδ(xo) sup xa∈B (xs) `(xa, yo),
where θ is the set of trainable parameters. Concretely, the defender finds a safe spot candidate x∗s of a datapoint xo and generates an adversarial example x∗a from x ∗ s . Then, the defender minimizes the cross-entropy loss `(x∗a, yo; θ) so that x ∗ s becomes an actual safe spot. Note that the ground-truth label yo is used instead of the prediction c(xo), since we assume that the defender can access the ground-truth label during training.
The most direct way to optimize the objective would be to find x∗s from xo using our safe spot search algorithm and perform k-step PGD adversarial training (Madry et al., 2017) with x∗s . However, since the safe spot search algorithm requires running T -step PGD dynamics per each update, the proposed training procedure would be more computationally demanding than PGD adversarial training. To ease this problem, we consider replacing the inner maximization supxa `(xa, yo) in the safe spot search by `(xs, yo):
x∗s = argmin xs∈Bδ(xo) sup xa∈B (xs) `(xa, yo) =⇒ x∗s = argmin xs∈Bδ(xo) `(xs, yo).
Then, x∗s can be easily computed by running targeted FGSM or k-step PGD on xo towards the ground-truth label yo. We denote this training scheme as safe spot-aware adversarial training.

heading: 3.6 OUT-OF-DISTRIBUTION DETECTION

text:
The safe spot-aware adversarial training method induces the learned data distribution to have safe spots near its data points. Thus, we can naturally conjecture that the samples from the learned distribution will have a higher probability of having safe spots compared to the out-of-distribution (OOD) samples, as shown in Figure 3. We leverage this conjecture to propose a new out-of-distribution detection algorithm that jointly utilizes our safe spot generation method and safe spot-aware adversarial training.
Following the framework of Hendrycks et al. (2019b), which use auxiliary outlier data to tune anomaly detectors, we consider there are three types of data distributions, Din, Dtrainout , and Dtestout . Din refers to the learned distribution, also called the in-distribution. Dtrainout is the given distribution of outliers used to tune the detection algorithm, which is orthogonal to Dtestout . Dtestout is the distribution we want to detect as OOD during inference, which is unknown. We include the auxiliary outlier data to
our safe spot-ware training procedure and adapt the training objective as below:
minimize θ E (xo,yo)∼Din [`(x∗a, yo; θ)] + E x̂o∼Dtrainout [γ ·DKL(ȳ ‖ C(x̂o; θ))− λ · `(x̂∗a, c(x̂o); θ)] (4) subject to x∗a = argmax xa∈B (x∗s) `(xa, yo) and x ∗ s = argmin xs∈Bδ(xo) sup xa∈B (xs) `(xa, yo)
x̂∗a = argmax x̂a∈B (x̂∗s) `(x̂a, c(x̂o)) and x̂ ∗ s = argmin x̂s∈Bδ(x̂o) sup x̂a∈B (x̂s) `(x̂a, c(x̂o)),
where ȳ is the uniform distribution and C(x̂o; θ) is the softmax probability of x̂o. Since x̂o is unlabeled, we use the prediction c(x̂o) instead for safe spot searching. Note that if ≥ δ, the first term in Equation (4) also maximizes the confidence of the original in-distribution samples, since xo ∈ Bδ(x∗s) ⊆ B (x∗s) and therefore `(xo, y; θ) ≤ `(x∗a, y; θ). Similarly, the second and third terms minimize the prediction confidence and the probability of safe spot existence of the outlier samples, respectively.
With the trained classifier, we measure the safe spot objective value from Equation (2) along with the maximum softmax probability (MSP) and use the values as indicators to detect OOD samples. Concretely, we define the score function as a linear combination of the two indicators. Considering they have a different range of possible values, we replace the safe spot objective value with the MSP of the adversarial example for the safe spot solution. Finally, the score function is formulated as
D(xo) := µ ·max y∈Y C(xo)y + (1− µ) ·max y∈Y
C(x∗a)y,
where x∗s ∈ Bδ(xo) is the optimal solution of the safe spot algorithm for xo and x∗a ∈ B (x∗s) is the adversarial example of x∗s . We filter inputs with low scores as OOD.

heading: 4 EXPERIMENTS

text: As it is natural to assume that the defender and the adversary have the same modification budget, we set δ = for all experiments. We evaluate our methods by measuring clean and adversarial accuracies, where adversarial accuracy refers to the prediction score under 20-step untargeted PGD attack with a step size of /4. In the experiment tables, None column indicates using original images as inputs, and S-Full uses safe spot images from Algorithm 1. We also evaluate safe spot search via targeted FGSM and 20-step PGD towards the class inferred from the classifier, each denoted as S-FGSM and S-PGD. Detailed settings are listed on supplementary C.

heading: 4.1 CIFAR-10

text:
We use Wide-ResNet-34-10 (Zagoruyko & Komodakis, 2016) and consider two threat models, `∞ with = 8/255 and `2 with = 0.5. We run our experiments on four differently trained models. The natural model is trained in a standard manner without considering adversaries. ADV is a PGD adversarially trained model. S-FGSM+ADV and S-PGD+ADV are safe spot-aware adversarially trained models, with safe spot search approximated by FGSM or 10-step PGD with a step size of δ/4.
The `∞ threat model result in Table 1 (left) shows our methods can find safe spots on over 85% of the test set images, except for the natural model. This performance is near the upper bound, which is the classifier’s clean accuracy since we use predicted labels for safe spot search. We also observe that safe spot search via targeted FGSM or PGD is also feasible for ADV, S-FGSM+ADV, and S-PGD+ADV models, but they still miss on about 10% of correctly classified images. When jointly used with our safe spot search method, the safe spot-aware training achieves the highest adversarial accuracy, along with a clean accuracy much higher than PGD adversarial training.
The `2 threat model result in Table 1 (right) shows similar results as the `∞ experiment, except that the adversarial accuracy of safe spots generated by S-Full on the natural model is much higher. However, we note that the adversarial accuracy of safe spots on the natural model may go down to about 20% when the attack gets stronger, for example, by increasing PGD iterations. The results on stronger PGD attacks and other types of attacks are considered in supplementary D.3 and D.4.

heading: 4.2 IMAGENET

text:
We use ResNet-50 and consider three threat models: `∞ with ∈ {4/255, 8/255} and `2 with = 3.0. For safe spot-aware adversarial training experiments, we utilize “fast” adversarial training (Wong et al., 2020) and train the safe spot-aware model S-FGSM+Fast to reduce the training cost.
Table 2 (left) shows results on `∞ attack under = 4/255. Similar to results on CIFAR-10, our methods are capable of finding safe spots near to original images that are correctly classified on the robust classifiers. Also, our proposed safe spot-aware classifier outperforms the original robust classifier by a large margin in both clean and adversarial accuracies. Table 2 (right) shows results on `∞ on = 8/255. In this setting, we also apply our algorithm to the ADV model trained with train = 4/255. Note that by changing only the train value on adversarial training, we get a 10% gain on our safe spot’s adversarial accuracy. Surprisingly, classifiers adversarially trained with smaller train performs substantially better in terms of preemptive robustness compared to using more robust classifiers. This implies that the conventional notion of robustness does not necessarily translate to preemptive robustness. Experiments on `2 attacks show similar results and can be found in Supplementary D.1.

heading: 4.3 RANDOMIZED SMOOTHING

text:
We also evaluate our algorithm for classifiers with randomized smoothing. Here, we consider the `2 threat model, where = 0.5 for CIFAR-10 and = 3.0 for ImageNet. We run experiments on the smoothed classifiers based on the natural and the Gaussian-noise augmented model, which are considered certifiably robust (Lecuyer et al., 2019; Cohen et al., 2019). We measure empirical robustness against the randomized PGD on both models and measure certified robustness on the Gaussian model. Detailed settings such as noise level σ and randomized PGD are listed in supplementary C.2.
Table 3 shows our results on the empirical robustness against the randomized PGD. We observe that our algorithm can find safe spots for the natural model with randomized smoothing on 57% and 37% of correctly classified images of CIFAR-10 and ImageNet, respectively. Furthermore, as shown in supplementary D.3, the adversarial accuracy of the smoothed natural model does not suffer from accuracy drop when the attack becomes stronger, in contrast to the natural model. Also, the smoothed Gaussian model, whose training cost is comparable to standard training much less than PGD adversarial training, achieves higher clean and adversarial accuracy compared to the ADV model. Certified robustness results of smoothed classifiers can be found in Supplementary D.2, where our safe spot algorithm also improves the certified robustness on both the datasets.

heading: 4.4 OUT-OF-DISTRIBUTION DETECTION

text:
We evaluate the performance of our proposed detection algorithm on models trained with CIFAR-10. We consider various OOD datasets including CIFAR-100, SVHN (Netzer et al., 2011), TinyImageNet (Johnson et al.), LSUN (Yu et al., 2015), and synthetic noise. Following the experimental protocol of Hendrycks et al. (2019b), we evaluate the OOD detection methods on three metrics: area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPR), and the false positive rate at 95% true positive rate (FPR95). We compare our method’s performance to Mahalanobis (Lee et al., 2018), OE (Hendrycks et al., 2019b), and Gram (Sastry & Oore, 2020). Since Lee et al. (2018) utilizes a subset of the Dtestout data for tuning the detection procedure while our method and OE do not, we modify Mahalanobis to tune with Dtrainout for fair comparison. For detailed descriptions of the datasets and the experiments, refer to Supplementary E.1 and E.2.
Table 4 shows the evaluation results. While Mahalanobis and Gram works slightly better on synthetic datasets such as Gaussian noise, on more near-distribution outliers such as CIFAR-100, TinyImageNet, and LSUN, our method outperforms these baselines by a large margin, which leads to a gain in overall performance. Our method also outperforms OE on most metrics including the Gaussian noise.

heading: 5 CONCLUSION

text: Parting from recent studies on adversarial examples, we present a new adversarial framework where the defender preemptively modifies classifier inputs. We introduce a novel optimization algorithm for finding safe spots in the vicinity of original inputs as well as a new network training method suited for enhancing preemptive robustness. The experiments show that our algorithm can find safe spots for robust classifiers on most of the correctly classified images. Further results show that they can be used to improve empirical and certified robustness on smooth classifiers. Finally, we combine the new network training scheme and the safe spot generation method to devise a new out-of-distribution detection algorithm that achieves the state of the art performance on near-distribution outliers.


summaries: This paper introduces a new convolutional approach to directly process raw spatiotemporal (ST) point cloud data. The proposed point spatio-temporal (PST) convolution operates on "point tubes" and decouples space and time through a shared spatial convolution at each timestep, followed by a temporal convolution. It also introduces a transposed PST to enable point-wise predictions in an encoder-decoder framework (PSTNet). The presented experiments demonstrate the effectiveness of these convolutions by using PSTNet for action recognition and semantic segmentation on point cloud sequences, showing improvement over relevant recent work.

summaries: The paper introduces point spatio-temporal convolutions, which are used for the feature extraction of point cloud sequences. A trainable kernel is used which is applied locally as a continuous convolution. An important aspect is that the temporal dimension is processed separately, with an additional convolution, instead of simply using a 4D convolution. The authors claim that in this way the network will achieve a better understanding of the dynamics of the input.

summaries: This paper aims to process the point cloud data in a convolution manner. The authors propose the PST convolution and deconvolution operations to handle the different tasks such as the classification and segmentation on point cloud. The extensive experiments verify the effectiveness of the proposed method and achieve state-of-the-art results on multiple benchmark datasets. Overall, the paper is well-written and organized.

abstractText: Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension. Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner. Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.

authors:
Paul J. Besl
Neil D. McKay

title: A method for registration of 3-d shapes

venue: IEEE Trans. Pattern Anal. Mach. Intell.,

authors:
Carlos Caetano
Jessica Sena de Souza
François Brémond
Jefersson A. dos Santos
William Robson Schwartz

title: Skelemotion: A new representation of skeleton joint sequences based on motion information for 3d action recognition

venue: In 16th IEEE International Conference on Advanced Video and Signal Based Surveillance,

authors:
João Carreira
Andrew Zisserman

title: Quo vadis, action recognition? A new model and the kinetics dataset

authors:
Xiaozhi Chen
Huimin Ma
Ji Wan
Bo Li
Tian Xia

title: Multi-view 3d object detection network for autonomous driving

authors:
Christopher B. Choy
JunYoung Gwak
Silvio Savarese

title: 4d spatio-temporal convnets: Minkowski convolutional neural networks

authors:
Angela Dai
Angel X. Chang
Manolis Savva
Maciej Halber
Thomas A. Funkhouser
Matthias Nießner

title: Scannet: Richly-annotated 3d reconstructions of indoor scenes

authors:
Hehe Fan
Yi Yang

title: PointRNN: Point recurrent neural network for moving point cloud processing

authors:
Hehe Fan
Xiaojun Chang
De Cheng
Yi Yang
Dong Xu
Alexander G. Hauptmann

title: Complex event detection by identifying reliable shots from untrimmed videos

authors:
Hehe Fan
Zhongwen Xu
Linchao Zhu
Chenggang Yan
Jianjun Ge
Yi Yang

title: Watching a small portion could be as good as watching all: Towards efficient video classification

authors:
Hehe Fan
Yi Yang
Mohan S. Kankanhalli

title: Point 4d transformer networks for spatio-temporal modeling in point cloud videos

authors:
Hehe Fan
Tao Zhuo
Xin Yu
Yi Yang
Mohan S. Kankanhalli

title: Understanding atomic hand-object interaction with human intention

venue: IEEE Transactions on Circuits and Systems for Video Technology (TCSVT),

authors:
Kensho Hara
Hirokatsu Kataoka
Yutaka Satoh

title: Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet

authors:
Alexander Kläser
Marcin Marszalek
Cordelia Schmid

title: A spatio-temporal descriptor based on 3d-gradients

venue: In BMVC,

authors:
Junnan Li
Yongkang Wong
Qi Zhao
Mohan S. Kankanhalli

title: Unsupervised learning of viewinvariant action representations

authors:
Maosen Li
Siheng Chen
Xu Chen
Ya Zhang
Yanfeng Wang
Qi Tian

title: Actional-structural graph convolutional networks for skeleton-based action recognition

authors:
Wanqing Li
Zhengyou Zhang
Zicheng Liu

title: Action recognition based on a bag of 3d points

venue: In CVPR Workshops,

authors:
Yangyan Li
Rui Bu
Mingchao Sun
Wei Wu
Xinhan Di
Baoquan Chen

title: Pointcnn: Convolution on x-transformed points

authors:
Jun Liu
Gang Wang
Ping Hu
Ling-Yu Duan
Alex C. Kot

title: Global context-aware attention LSTM networks for 3d action recognition

authors:
Jun Liu
Gang Wang
Ling-Yu Duan
Kamila Abdiyeva
Alex C. Kot

title: Skeleton-based human action recognition with global context-aware attention LSTM networks

venue: IEEE Trans. Image Processing,

authors:
Jun Liu
Amir Shahroudy
Mauricio Perez
Gang Wang
Ling-Yu Duan
Alex C. Kot

title: Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding

venue: IEEE Trans. Pattern Anal. Mach. Intell., 2019a. doi: 10.1109/TPAMI.2019.2916873

authors:
Jun Liu
Amir Shahroudy
Gang Wang
Ling-Yu Duan
Alex C. Kot

title: Skeleton-based online action prediction using scale selection network

authors:
Lu Liu
Tianyi Zhou
Guodong Long
Jing Jiang
Lina Yao
Chengqi Zhang

title: Prototype propagation networks (PPN) for weakly-supervised few-shot learning on category graph

venue: In IJCAI,

authors:
Lu Liu
William Hamilton
Guodong Long
Jing Jiang
Hugo Larochelle

title: A universal representation transformer layer for few-shot image classification

authors:
Mengyuan Liu
Junsong Yuan

title: Recognizing human actions as the evolution of pose estimation maps

authors:
Xingyu Liu
Charles R. Qi
Leonidas J. Guibas

title: Flownet3d: Learning scene flow in 3d point clouds

authors:
Xingyu Liu
Mengyuan Yan
Jeannette Bohg

title: Meteornet: Deep learning on dynamic 3d point cloud sequences

venue: In ICCV,

authors:
Yanbin Liu
Juho Lee
Minseop Park
Saehoon Kim
Eunho Yang
Sung Ju Hwang
Yi Yang

title: Learning to propagate labels: Transductive propagation network for few-shot learning

authors:
Wenjie Luo
Bin Yang
Raquel Urtasun

title: Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net

authors:
Joe Yue-Hei Ng
Matthew J. Hausknecht
Sudheendra Vijayanarasimhan
Oriol Vinyals
Rajat Monga
George Toderici

title: Beyond short snippets: Deep networks for video classification

authors:
Michael Niemeyer
Lars M. Mescheder
Michael Oechsle
Andreas Geiger

title: Occupancy flow: 4d reconstruction by learning particle dynamics

authors:
Eshed Ohn-Bar
Mohan M. Trivedi

title: Joint angles similarities and HOG2 for action recognition

authors:
Omar Oreifej
Zicheng Liu

title: HON4D: histogram of oriented 4d normals for activity recognition from depth sequences

authors:
Lukas Prantl
Nuttapong Chentanez
Stefan Jeschke
Nils Thuerey

title: Tranquil clouds: Neural networks for learning temporally coherent features in point clouds

authors:
Charles Ruizhongtai Qi
Hao Su
Kaichun Mo
Leonidas J. Guibas

title: Pointnet: Deep learning on point sets for 3d classification and segmentation

authors:
Charles Ruizhongtai Qi
Li Yi
Hao Su
Leonidas J. Guibas

title: Pointnet++: Deep hierarchical feature learning on point sets in a metric space

authors:
Charles Ruizhongtai Qi
Or Litany
Kaiming He
Leonidas J. Guibas

title: Deep hough voting for 3d object detection in point clouds

authors:
Davis Rempe
Tolga Birdal
Yongheng Zhao
Zan Gojcic
Srinath Sridhar
Leonidas J. Guibas

title: Caspr: Learning canonical spatiotemporal point cloud representations

authors:
Germán Ros
Laura Sellart
Joanna Materzynska
David Vázquez
Antonio M. López

title: The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes

authors:
Amir Shahroudy
Jun Liu
Tian-Tsong Ng
Gang Wang

title: NTU RGB+D: A large scale dataset for 3d human activity analysis

authors:
Lei Shi
Yifan Zhang
Jian Cheng
Hanqing Lu

title: Skeleton-based action recognition with directed graph neural networks

title: Two-stream adaptive graph convolutional networks for skeleton-based action recognition

authors:
Chenyang Si
Wentao Chen
Wei Wang
Liang Wang
Tieniu Tan

title: An attention enhanced graph convolutional LSTM network for skeleton-based action recognition

authors:
Karen Simonyan
Andrew Zisserman

title: Two-stream convolutional networks for action recognition in videos

authors:
Hugues Thomas
Charles Ruizhongtai Qi
Jean-Emmanuel Deschaud
Beatriz Marcotegui
François Goulette
Leonidas J. Guibas

title: Kpconv: Flexible and deformable convolution for point clouds

authors:
Du Tran
Lubomir D. Bourdev
Rob Fergus
Lorenzo Torresani
Manohar Paluri

title: Learning spatiotemporal features with 3d convolutional networks

authors:
Du Tran
Heng Wang
Lorenzo Torresani
Jamie Ray
Yann LeCun
Manohar Paluri

title: A closer look at spatiotemporal convolutions for action recognition

authors:
Antônio Wilson Vieira
Erickson R. Nascimento
Gabriel L. Oliveira
Zicheng Liu
Mario Fernando Montenegro Campos

title: STOP: space-time occupancy patterns for 3d action recognition from depth map sequences. In Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications

authors:
Jiang Wang
Zicheng Liu
Ying Wu
Junsong Yuan

title: Mining actionlet ensemble for action recognition with depth cameras

authors:
Limin Wang
Yuanjun Xiong
Zhe Wang
Yu Qiao
Dahua Lin
Xiaoou Tang
Luc Van Gool

title: Temporal segment networks: Towards good practices for deep action recognition

authors:
Pichao Wang
Wanqing Li
Zhimin Gao
Chang Tang
Philip O. Ogunbona

title: Depth pooling based large-scale 3-d action recognition with convolutional neural networks

venue: IEEE Trans. Multimedia,

authors:
Yancheng Wang
Yang Xiao
Fu Xiong
Wenxiang Jiang
Zhiguo Cao
Joey Tianyi Zhou
Junsong Yuan

title: 3dv: 3d dynamic voxel for action recognition in depth video

authors:
Yue Wang
Yongbin Sun
Ziwei Liu
Sanjay E. Sarma
Michael M. Bronstein
Justin M. Solomon

title: Dynamic graph CNN for learning on point clouds

venue: ACM Trans. Graph.,

authors:
Wenxuan Wu
Zhongang Qi
Fuxin Li

title: Pointconv: Deep convolutional networks on 3d point clouds

authors:
Yang Xiao
Jun Chen
Yancheng Wang
Zhiguo Cao
Joey Tianyi Zhou
Xiang Bai

title: Action recognition for depth video using multi-view dynamic images

venue: Inf. Sci.,

authors:
Xiaodong Yang
Yingli Tian

title: Super normal vector for activity recognition using depth sequences

authors:
Lequan Yu
Xianzhi Li
Chi-Wing Fu
Daniel Cohen-Or
Pheng-Ann Heng

title: Pu-net: Point cloud upsampling network

authors:
Pengfei Zhang
Cuiling Lan
Junliang Xing
Wenjun Zeng
Jianru Xue
Nanning Zheng

title: View adaptive neural networks for high performance skeleton-based human action recognition

authors:
Linchao Zhu
Yi Yang

title: Label independent memory for semi-supervised few-shot video classification

authors:
Tao Zhuo
Zhiyong Cheng
Peng Zhang
Yongkang Wong
Mohan S. Kankanhalli

title: Explainable video action reasoning via prior knowledge and state transitions

venue: In ACM Multimedia,

heading: 1 INTRODUCTION

text:
Modern robotic and automatic driving systems usually employ real-time depth sensors, such as LiDAR, to capture the geometric information of scenes accurately while being robust to different lighting conditions. A scene geometry is thus represented by a 3D point cloud, i.e., a set of measured point coordinates {(x, y, z)}. Moreover, when RGB images are available, they are often used as additional features associated with the 3D points to enhance the discriminativeness of point clouds. However, unlike conventional grid based videos, dynamic point clouds are irregular and unordered in the spatial dimension while points are not consistent and even flow in and out over time. Therefore, existing 3D convolutions on grid based videos (Tran et al., 2015; Carreira & Zisserman, 2017; Hara et al., 2018) are not suitable to model raw point cloud sequences, as shown in Fig. 1.
To model the dynamics of point clouds, one solution is converting point clouds to a sequence of 3D voxels, and then applying 4D convolutions (Choy et al., 2019) to the voxel sequence. However, directly performing convolutions on voxel sequences require a large amount of computation. Furthermore, quantization errors are inevitable during voxelization, which may restrict applications that require precise measurement of scene geometry. Another solution MeteorNet (Liu et al., 2019e) is extending the static point cloud method PointNet++ (Qi et al., 2017b) to process raw point cloud sequences by appending 1D temporal dimension to 3D points. However, simply concatenating coordinates and time together and treating point cloud sequences as unordered 4D point sets neglect the temporal order of timestamps, which may not properly exploit the temporal information and lead to inferior performance. Moreover, the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible. Treating them equally is not conducive for network optimization. Besides, MeteorNet only considers spatial neighbors and neglects the local dependencies of neighboring frames. With the use of whole sequence length as its temporal receptive field, MeteorNet cannot construct temporal hierarchy. As points are not consistent and even flow in and out of the region, especially for long sequences and fast motion embedding points in a spatially local area along an entire sequence handicaps capturing accurate local dynamics of point clouds.
In this paper, we propose a point spatio-temporal (PST) convolution to directly process raw point cloud sequences. As dynamic point clouds are spatially irregular but ordered in the temporal dimension, we
decouple the spatial and temporal information to model point cloud sequences. Specifically, PST convolution consists of (i) a point based spatial convolution that models the spatial structure of 3D points and (ii) a temporal convolution that captures the temporal dynamics of point cloud sequences. In this fashion, PST convolution significantly facilitates the modeling of dynamic point clouds and reduces the impact of the spatial irregularity of points on temporal modeling. Because point cloud sequences emerge inconsistently across frames, it is challenging to perform convolution on them. To address this problem, we introduce a point tube to preserve the spatio-temporal local structure.
To enhance the feature extraction ability, we incorporate the proposed PST convolution into a spatiotemporally hierarchical network, namely PSTNet. Moreover, we extend our PST convolution to a transposed version to address point-level prediction tasks. Different from the convolutional version, the PST transposed convolution is designed to interpolate temporal dynamics and spatial features. Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of the proposed PST convolution and the superiority of PSTNet in modeling point clouds sequences. The contributions of this paper are fourfold:
• To the best of our knowledge, we are the first attempt to decompose spatial and temporal information in modeling raw point cloud sequences, and propose a generic point based convolutional operation, named PST convolution, to encode raw point cloud sequences.
• We propose a PST transposed convolution to decode raw point cloud sequences via interpolating the temporal dynamics and spatial feature for point-level prediction tasks.
• We construct convolutional neural networks based on the PST convolutions and transposed convolutions, dubbed PSTNet, to tackle sequence-level classification and point-level prediction tasks. To the best of our knowledge, our PSTNet is the first deep neural network to model raw point cloud sequences in a both spatially and temporally hierarchical manner.
• Extensive experiments on four datasets indicate that our method improves the accuracy of 3D action recognition and 4D semantic segmentation.

heading: 2 RELATED WORK

text:
Learning Representations on Grid based Videos. Impressive progress has been made on generating compact and discriminative representations for RGB/RGBD videos due to the success of deep neural networks. For example, two-stream convolutional neural networks (Simonyan & Zisserman, 2014; Wang et al., 2016) utilize a spatial stream and an optical flow stream for video modeling. To summarize the temporal dependencies of videos, recurrent neural networks (Ng et al., 2015; Fan et al., 2018) and pooling techniques (Fan et al., 2017) are employed. In addition, by stacking multiple 2D frames into a 3D tensor, 3D convolutional neural networks (Tran et al., 2015; Carreira & Zisserman, 2017; Tran et al., 2018; Hara et al., 2018) are widely used to learn spatio-temporal representations for videos, and achieve promising performance. Besides, interpretable video or action reasoning methods (Zhuo et al., 2019; Fan et al., 2021b) are proposed by explicitly parsing changes in videos.
Static Point Cloud Processing. Static point cloud analysis has been widely investigated for many problems, such as classification, object part segmentation, scene semantic segmentation (Qi et al., 2017a;b; Li et al., 2018b; Wu et al., 2019; Thomas et al., 2019; Wang et al., 2019), reconstruction (Dai et al., 2017; Yu et al., 2018) and object detection (Chen et al., 2017; Qi et al., 2019). Most recent works aim to directly manipulate point sets without transforming coordinates into regular voxel grids. Since a point cloud is essentially a set of unordered points and invariant to permutations of its points, static point cloud processing methods mainly focus on designing effective point based spatial correlation operations that do not rely on point orderings.
Dynamic Point Cloud Modeling. Compared with static point cloud processing, dynamic point cloud modeling is a fairly new task. Fast and Furious (FaF) (Luo et al., 2018) converts a point cloud frame into a bird’s view voxel and then extracts features via 3D convolutions. MinkowskiNet (Choy et al., 2019) converts a point cloud sequence into a 4D occupancy grid and then applies 4D Spatio-Temporal ConvNets. PointRNN (Fan & Yang, 2019) leverages point based recurrent neural networks for raw point cloud sequence forecasting. MeteorNet (Liu et al., 2019e) extends 3D points to 4D points and then appends a temporal dimension to PointNet++ to process these 4D points. 3DV (Wang et al., 2020) first integrates 3D motion information into a regular compact voxel set and then applies PointNet++ to extract representations from the set for 3D action recognition. P4Transformer (Fan et al., 2021a) employs transformer to avoid point tracking for raw point cloud sequence modeling. Niemeyer et al. (2019) learned a temporal and spatial vector field for 4D reconstruction. Prantl et al. (2020) learned stable and temporal coherent feature spaces for point based super-resolution. CaSPR (Rempe et al., 2020) learns to encode spatio-temporal changes in object shape from point clouds for reconstruction and camera pose estimation. In this paper, we propose a point based convolution to model raw point cloud sequences in a spatio-temporally hierarchical manner.

heading: 3 PROPOSED POINT SPATIO-TEMPORAL CONVOLUTIONAL NETWORK

text: In this section, we first briefly review the grid based 3D convolution operation as our point spatiotemporal (PST) convolution is motivated by it. Then, we introduce how the proposed PST convolution extracts features from dynamic point clouds in detail. In order to address dense point prediction tasks, i.e., semantic segmentation, we develop a PST transposed convolution. Finally, we incorporate our operations into deep hierarchical networks to address different dynamic point cloud tasks.

heading: 3.1 MOTIVATION FROM GRID BASED 3D CONVOLUTION

text:
The power of convolutional neural network (CNNs) comes from local structure modeling by convolutions and global representation learning via hierarchical architectures. Given an input feature map F ∈ RC×L×H×W , where C, L, H and W denote the input feature dimension, length, height and width, 3D convolution is designed to capture the spatio-temporal local structure of F, written as:
F′(x,y)t = bl/2c∑
k=−bl/2c bh/2c∑ i=−bh/2c bw/2c∑ j=−bw/2c W(i,j)k · F (x+i,y+j) t+k , (1)
where W ∈ RC′×C×l×h×w is the convolution kernel, (l, h, w) is the kernel size, C ′ represents the output feature dimension and · is matrix multiplication. The W(i,j)k ∈ RC
′×C denotes the weight at kernel position (k, i, j) and F(x+i,y+j)t+k ∈ RC×1 denotes the feature of pixel at input position (t+k, x+ i, y+ j). Usually, CNNs employ small kernel sizes, e.g., (3, 3, 3), which are much smaller than the input size, to effectively model the local structure. To construct hierarchical CNNs, stride (> 1) is used to subsample input feature maps during the convolution operation. In this fashion, following convolutional layers will have relatively larger receptive fields.

heading: 3.2 POINT SPATIO-TEMPORAL (PST) CONVOLUTION

text:
Let Pt ∈ R3×N and Ft ∈ RC×N denote the point coordinates and features of the t-th frame in a point cloud sequence, where N and C denote the number of points and feature channels, respectively. Given a point cloud sequence ( [P1;F1], [P2;F2], · · · , [PL;FL] ) , the proposed PST convolution will
encode the sequence to ( [P ′1;F ′ 1], [P ′ 2;F ′ 2], · · · , [P ′L′ ;F ′L′ ] ) , where L and L′ indicate the number of frames, and P ′t ∈ R3×N ′ and F ′t ∈ RC ′×N ′ represent the encoded coordinates and features.

heading: 3.2.1 DECOMPOSING SPACE AND TIME IN POINT CLOUD SEQUENCE MODELING

text:
Because point clouds are irregular and unordered, grid based 3D convolution cannot be directly applied to point cloud sequences. As point cloud sequences are spatially irregular and unordered but temporally ordered, this motivates us to decouple these two dimensions in order to reduce the impacts of the spatial irregularity of points on temporal modeling. Moreover, the scales of spatial displacements and temporal differences in point cloud sequences may not be compatible. Treating them equally is not conducive for network optimization. By decoupling the spatio-temporal information in point cloud sequences, we not only make the spatio-temporal modeling easier but also significantly improve the ability to capture the temporal information. Therefore, our PST convolution is formulated as:
F ′(x,y,z) t = bl/2c∑ k=−bl/2c ∑ ‖(δx,δy,δz)‖≤r W(δx,δy,δz)k · F (x+δx,y+δy,z+δz) t+k ,
= bl/2c∑ k=−bl/2c ∑ ‖(δx,δy,δz)‖≤r T(δx,δy,δz)k · ( S(δx,δy,δz)k · F (x+δx,y+δy,z+δz) t+k ) ,
(2)
where (x, y, z) ∈ Pt, (δx, δy, δz) represents displacement and r is spatial search radius. The convolution kernel W ∈ RC′×C×l × R is decomposed into a spatial convolution kernel S ∈ RCm×C×l × R and a temporal convolution kernel T ∈ RC′×Cm×l × R and Cm is the dimension of the intermediate feature. Moreover, because space and time are orthogonal and independent of each other, we further decompose spatial and temporal modeling as:
F ′(x,y,z) t = bl/2c∑ k=−bl/2c Tk · ∑ ‖(δx,δy,δz)‖≤r S(δx,δy,δz) · F (x+δx,y+δy,z+δz)t+k , (3)
where S ∈ RCm×C × R and T ∈ RC′×Cm×l. The above decomposition can also be expressed as applying temporal convolution first and then spatial convolution to input features. However, doing so requires point tracking to capture point motion. Because it is difficult to achieve accurate point trajectory and tracking points usually relies on point colors and may fail to handle colorless point clouds, we opt to model the spatial structure of irregular points first, and then capture the temporal information from the spatial regions. In this fashion, Eq. (3) can be rewritten as:
M (x,y,z) t = ∑ ‖(δx,δy,δz)‖≤r S(δx,δy,δz) · F (x+δx,y+δy,z+δz)t , F ′(x,y,z) t = bl/2c∑ k=−bl/2c Tk ·M (x,y,z)t+k . (4)
The spatial convolution captures spatial local structures of point clouds in 3D space, while the temporal convolution aims to describe the temporal local dynamics of point cloud sequences along the time dimension. In order to capture the distributions of neighboring points, it is required to learn a convolution kernel S for all displacements. However, this is impossible because point displacements are not discrete. To address this problem, we convert the kernel to a function of displacements, defined as:
M (x,y,z) t = ∑ ‖(δx,δy,δz)‖≤r f ( (δx, δy, δz);θ ) · F (x+δx,y+δy,z+δz)t , (5)
where f : R1×3 → RCm×C is a function of (δx, δy, δz) parametrized by θ to generate different RCm×C according to different displacements. There can be many ways to implement the f function. In principle, the f function should be both computation-efficient and memory-efficient so that PST convolution is able to encode long sequences. Existing static point cloud convolution methods (Li et al., 2018b; Wu et al., 2019; Wang et al., 2019) usually contains multilayer perceptrons (MLPs), which are computation and memory consuming. In this paper, we design a lightweight implementation for f , which contains only a few parameters. Specifically, we further decompose the f function as f ( (δx, δy, δz);θ ) = θd · (δx, δy, δz)T · 1 θs, where θ = [θd,θs],θd ∈ RCm×3 is a displacement transform kernel, θs ∈ RCm×C is a sharing kernel, 1 = (1, · · · , 1) ∈ R1×C is for broadcasting, and is element-wise product. The sharing kernel θs is to increase point feature dimension to improve the feature representation ability. The displacement kernel θd is to capture spatial local structure based on displacements. In this fashion, f generates a unique spatial kernel for each displacement so that the lightweight spatial convolution is able to increase feature dimension while capturing spatial local structure like conventional convolutions.

heading: 3.2.2 POINT TUBE

text:
Grid based 3D convolutions can be easily performed on regular conventional videos by sliding along the length, height and width. Because point cloud sequences are irregular and unordered in 3D space and emerge inconsistently across different frames, it is challenging to perform convolution on them. To address this problem, we introduce a point tube to preserve the spatio-temporal local structure. In contrast to pixel cubes in 3D convolution, in which pixels are distributed regularly, point tubes are dynamically generated based on the input sequences so that dense areas have more tubes than sparse ones. Specifically, the point tube is constructed as follows:
Temporal anchor frame selection. For an entire sequence of point clouds, we need to select some anchor frames to generate our tubes. Temporal anchor frames in a point cloud sequence are automatically selected based on temporal kernel size (l), temporal stride (st) and temporal padding (p), where l is set to an odd number so that an anchor frame is located in the middle of a point tube. Moreover, we set b l2c ≥ p to avoid selecting a padding frame as an anchor frame. Spatial anchor point sampling. Once a temporal anchor frame is selected, we need to choose spatial anchor points that can represent the distribution of all the points in the frame. Given a spatial subsampling rate ss, this operation aims to subsample N points to N ′ = bNss c points. We employ the farthest point sampling (FPS) (Qi et al., 2017b) to sample points in each anchor frame. Point tubes are generated according to sampled anchor points.
Transferring spatial anchor points. 3D convolutions can effectively capture local changes within a cube of a kernel size (l, h, w) rather than tracking a specific pixel. Following this idea, we propagate the positions of sampled anchor points to the neighboring frames without tracking, and they are regarded as the anchor points in those frames. Specifically, each anchor point is transferred to the b l2c nearest frames. The original and transferred anchor points form the central axis of a tube. Spatial neighbor. This step aims to find the spatial neighbors of every anchor point in each frame for performing spatial convolution. As indicated in Eq. (5), radius neighborhood r is used to search neighbors within the tube slice, where local structure of points is depicted. Note that, padding is usually used in grid based convolution to align feature maps. However, our point based spatial convolution is not conducted on grids and thus spatial padding is not employed.
By performing PST convolution on point tubes, our network is able to capture the dynamic changes in local areas. The temporal kernel size l and spatial search radius r allow our PST convolution to capture temporal and spatial local structure, respectively. Frame subsampling (according to st) and point subsampling (according to ss) make our PSTNet be both temporally and spatially hierarchical. Global movement can be summarized by merging the information from these tubes in a spatio-temporally hierarchical manner. An illustration of PST convolution is shown in Fig. 2.

heading: 3.3 POINT SPATIO-TEMPORAL TRANSPOSED CONVOLUTION

text:
After PST convolution, the original point cloud sequence is both spatially and temporally subsampled. However, for point-level prediction tasks, we need to provide point features for all the original points. Thus, we develop a PST transposed convolution. Suppose ( [P ′1;F ′ 1], [P ′ 2;F ′ 2], · · · , [P ′L′ ;F ′L′ ] ) is the
encoded sequence of the original one ( [P1;F1], [P2;F2], · · · , [PL;FL] ) . PST transposed convolution propagates features (F ′1,F ′ 2, · · · ,F ′L′) to the original point coordinates (P1,P2, · · · ,PL), thus outputting new features (F ′′1 ,F ′′ 2 , · · · ,F ′′L ), where F ′′t ∈ RC
′′×N and C ′′ denotes the new feature dimension. To this end, PST transposed convolution first recovers the temporal length by a temporal transposed convolution:
T′ · F ′(x,y,z)t = [ M ′(x,y,z) t−bl/2c, · · · ,M ′(x,y,z) t+bl/2c ] , (6)
where T′ ∈ Rl×C′m×C′ is the temporal transposed convolution kernel, and then increases the number of points by assigning temporal features to original points. Inspired by (Qi et al., 2017b), the interpolated features are weighted by inverse distance between an original point and neighboring anchor points:
F ′′(x,y,z) t = S ′ · ∑ ‖(δx,δy,δz)‖≤r w(δx, δy, δz)M ′(x+δx,y+δy,z+δz) t∑
‖(δx,δy,δz)‖≤r w(δx, δy, δz) , w(δx, δy, δz) =
1
‖(δx, δy, δz)‖2 , (7)
where S′ ∈ RC′′×C′m is a sharing kernel to enhance the interpolated features and change the feature dimension from C ′m to C ′′. An illustration of PST transposed convolution is shown in Appendix A.

heading: 3.4 PSTNET ARCHITECTURES

text:
PSTNet for 3D action recognition. We employ 6 PST convolution layers and a fully-connected (FC) layer for 3D action recognition. In the 1st, 2nd, 4th, 6th layers, the spatial subsampling rate is set to 2 to halve the spatial resolution. Because point clouds are irregular and unordered, spatial receptive fields cannot be relatively enlarged by spatial subsampling. To address this problem, we progressively increase the spatial search radius. In the 2nd and 4th layers, the temporal stride is set to 2 to halve the temporal resolution. In the 2-4 layers, the temporal kernel size is set to 3 to capture temporal correlation. Temporal paddings are added when necessary. After the convolution layers, average and max poolings are used for spatial pooling and temporal pooling, respectively. Finally, the FC layer maps the global feature to action predictions.
PSTNet for 4D semantic segmentation. We use 4 PST convolution layers and 4 PST transposed convolution layers for 4D semantic segmentation. The spatial subsampling rate is set to 4 to reduce the spatial resolution in the 1-3 PST convolutions and 2 in the fourth PST convolution. Similar to 3D action recognition, the spatial search radius progressively increases to grow spatial receptive fields along the network layers. In the 3rd PST convolution and 2nd transposed convolution, the temporal kernel size is set to 3 to reduce and increase temporal dimension, respectively. Skip connections are added between the corresponding convolution layers and transposed convolution layers. After the last PST transposed convolution layer, a 1D convolution layer is appended for semantic predictions.
In addition, batch normalization and ReLU activation are inserted between PST convolution and transposed convolution layers. Our PSTNet architectures are illustrated in Appendix B.

heading: 4 EXPERIMENTS

heading: 4.1 3D ACTION RECOGNITION

text:
To show the effectiveness in sequence-level classification, we apply PSTNet to 3D action recognition. Following (Liu et al., 2019e; Wang et al., 2020), we sample 2,048 points for each frame. Point
Table 1: Action recognition accuracy (%) on the MSR-Action3D dataset.
Method Input # Frames Accuracy
Vieira et al. (Vieira et al., 2012) depth 20 78.20 Kläser et al. (Kläser et al., 2008) depth 18 81.43 Actionlet (Wang et al., 2012) skeleton all 88.21
PointNet++ (Qi et al., 2017b) point 1 61.61
MeteorNet (Liu et al., 2019e) point
4 78.11 8 81.14
12 86.53 16 88.21 24 88.50
PSTNet (ours) point
4 81.14 8 83.50
12 87.88 16 89.90 24 91.20
Figure 3: Influence of temporal kernel size and (initial) spatial search radius on MSR-Action3D with 24 frames.
1 3 5 7 9 (a) temporal kernel size82
84
86
88
90
92
ac cu
ra cy
(% )
80.80
91.20 90.55 89.05
87.37
0.1 0.5 0.9 1.3 1.7 2.1 2.5 (b) spatial search radius
90.0 90.2 90.4 90.6 90.8 91.0 91.2 ac cu ra cy (% ) 90.94
91.20
90.86 90.77
90.59 90.41
90.07
Figure 4: Visualization of PST convolution’s output. Top: input point cloud sequences, where color encodes depth. Bottom: output of PST convolution, where brighter color indicates higher activation. Interestingly, PST convolution outputs high activation to salient motion. Best viewed in color.
cloud sequences are split into multiple clips (with a fixed number of frames) as inputs. For training, sequence-level labels are used as clip-level labels. For evaluation, the mean of the clip-level predicted probabilities is used as the sequence-level prediction. Point colors are not used.

heading: 4.1.1 MSR-ACTION3D

text:
The MSR-Action3D (Li et al., 2010) dataset consists of 567 Kinect depth videos, with 20 action categories and 23K frames in total. We use the same training/test split as previous works (Wang et al., 2012; Liu et al., 2019e). We conduct experiments with 10 times and report the mean.
Comparison with the state-of-the-art. We compare our PSTNet with skeleton-based, depth-based and point-based 3D action recognition methods on this dataset. As shown in Table 1, the proposed PSTNet significantly outperforms all the state-of-the-art methods, demonstrating the superiority of our PST convolution on feature extraction. Moreover, when encoding long point sequences, our spatio-temporally hierarchical PSTNet is more effective than the spatially hierarchical MeteorNet. Specifically, from 16 frames to 24 frames, MeteorNet only achieves a slight improvement of 0.29%, while our method increases the accuracy by 1.30%.
What does PST convolution learn? To investigate what PST convolution learns, we visualize the output of the middle layer (i.e., the 3rd layer) in Fig. 4. As ReLU is employed as the activation function, larger values indicate higher activation. As expected, PSTNet outputs higher activation on moving areas, demonstrating that our PSTNet captures the most informative clues in action reasoning.

heading: 4.1.2 NTU RGB+D 60 AND NTU RGB+D 120

text:
The NTU RGB+D 60 (Shahroudy et al., 2016) is the second largest dataset for 3D action recognition. It consists of 56K videos, with 60 action categories and 4M frames in total. The videos are captured
using Kinect v2, with 3 cameras and 40 subjects (performers). The dataset defines two types of evaluation, i.e., cross-subject and cross-view. The cross-subject evaluation splits the 40 performers into training and test groups. Each group consists of 20 performers. The cross-view evaluation uses all the samples from camera 1 for testing and samples from cameras 2 and 3 for training.
The NTU RGB+D 120 (Liu et al., 2019a) dataset, the largest dataset for 3D action recognition, is an extension of NTU RGB+D 60. It consists of 114K videos, with 120 action categories and 8M frames in total. The videos are captured with 106 performers and 32 collection setups (locations and backgrounds). Besides cross-subject evaluation, the dataset defines a new evaluation setting, i.e., cross-setup, where 16 setups are used for training, and the others are used for testing.
Comparison with state-of-the-art methods. As indicated in Table 2, PSTNet outperforms all the other approaches in all evaluation settings. Particularly, as indicated by the cross-setup evaluation on NTU RGB+D 120, PSTNet outperforms the second best 3DV-PointNet++ (Wang et al., 2020) by 4.6%. Moreover, compared to 3DV that extracts motion from voxels, PSTNet directly models the dynamic information of raw point cloud sequences and thus is efficient.
Computational efficiency. We provide a running time comparison with the second best 3DVPointNet++ (Wang et al., 2020). The average running time per video is shown in Fig. 5. Experiments are conducted using 1 Nvidia RTX 2080Ti GPU on NTU RGB+D 60. Impressively, compared to 3DV-PointNet++, PSTNet reduces running time by about 2s, showing that PSTNet is efficient.

heading: 4.2 4D SEMANTIC SEGMENTATION

text:
To demonstrate that our PST convolution can be generalized to point-level dense prediction tasks, we apply PSTNet to 4D semantic segmentation. Following the works (Choy et al., 2019; Liu et al., 2019e), we conduct experiments on video clips with length of 3 frames. Note that, although semantic segmentation can be achieved from a single frame, exploring temporal consistency would facilitate exploring the structure of scenes, thus improving segmentation accuracy and robustness to noise. The widely-used standard mean Intersection over Union (mIoU) is adopted as the evaluation metric.
Synthia 4D (Choy et al., 2019) uses the Synthia dataset (Ros et al., 2016) to create 3D video sequences. The Synthia 4D dataset includes 6 sequences of driving scenarios. Each sequence consists of 4 stereo
Table 3: Semantic segmentation results on the Synthia 4D dataset.
Method Input #Frms #Params (M) mIoU (%)
3D MinkNet14 (Choy et al., 2019) voxel 1 19.31 76.24 4D MinkNet14 (Choy et al., 2019) voxel 3 23.72 77.46
PointNet++ (Qi et al., 2017b) point 1 0.88 79.35 MeteorNet (Liu et al., 2019e) point 3 1.78 81.80
PSTNet (l = 1) point 3 1.42 80.79 PSTNet (l = 3) point 3 1.67 82.24
Figure 5: Comparison of recognition running time per video on NTU RGB+D 60.
3DV-PointNet++ PSTNet (ours)0
500
1000
1500
2000
2500
ru nn
in g
tim e
(m s) 2205
11
2678
696
CPU GPU
RGB-D images taken from the top of a moving car. Following (Liu et al., 2019e), we use the same training/validation/test split, with 19,888/815/1,886 frames, respectively.
As seen in Table 3, PSTNet (l = 3) exploits the temporal information and outperforms the state-ofthe-art. Moreover, our method saves 0.11M, relative 6% of parameters compared to the second best method MeteorNet (Liu et al., 2019e). We visualize a few segmentation examples in Appendix N.

heading: 4.3 ABLATION STUDY

text:
Clip length. Usually, information is not equally distributed in sequences along time. Short point cloud clips may miss key frames and thus confuse the models as noise. Therefore, as shown in Table 1, increasing clip length (i.e., the number of frames) benefits models for action recognition.
Temporal kernel size. The temporal kernel size l controls the temporal dynamics modeling of point cloud sequences. Fig. 3(a) shows the accuracy on MSR-Action3D with different l. (a) When l is set to 1, temporal correlation is not captured. However, PSTNet can still observe 24 frames and the pooling operation allows PSTNet to capture the pose information of an entire clip. Moreover, some actions (e.g., “golf swing”) can be easily recognized by a certain pose, and thus PSTNet with l = 1 can still achieve satisfied accuracy. (b) When l is greater than 1, PSTNet models temporal dynamics and therefore improves accuracy on actions that rely on motion or trajectory reasoning (e.g., “draw x”, “draw tick” and “draw circle”). (c) When l is greater than 3, the accuracy decreases. This mainly depends on motion in sequences. Because most actions in MSR-Action3D are fast (e.g., “high arm wave”), using smaller temporal kernel size facilitates capturing fast motion, and long-range temporal dependencies will be captured in high-level layers. Since we aim to present generic point based convolution operations, we do not tune the kernel size for each action but use the same size. When PSTNet is used in 4D semantic segmentation, we observe that PSTNet (l = 3) improves mIoU by 1.45% compared to PSTNet (l = 1) that neglects temporal structure (shown in Table 3).
Spatial search radius. The spatial search radius r controls the range of the spatial structure to be modeled. As shown in Fig. 3(b), using too small r cannot capture sufficient structure information while using large r will decrease the discriminativeness of spatial local structure for modeling.

heading: 5 CONCLUSION

text: In this paper, we propose an point spatio-temporal (PST) convolution to learn informative representations from raw point cloud sequences and a PST transposed convolution for point-level dense prediction tasks. We demonstrate that, by incorporating the proposed convolutions into deep networks, dubbed PSTNets, our method is competent to address various point-based tasks. Extensive experiments demonstrate that our PSTNet significantly improves the 3D action recognition and 4D semantic segmentation performance by effectively modeling point cloud sequences.

heading: ACKNOWLEDGMENTS

text: This research is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funding Scheme (#A18A2b0046).

heading: A PST TRANSPOSED CONVOLUTION

text: We illustrate an example of PST transposed convolution in Fig. 6. Given a convolved sequence( [P ′1;F ′ 1], [P ′ 2;F ′ 2], [P ′ 3′ ;F ′ 3′ ] ) , and its original coordinate sequence ( P1,P2,P3,P4,P5 ) , PST transposed convolution aims to generate new features (F ′′1 ,F ′′ 2 ,F ′′ 3 ,F ′′ 4 ,F ′′ 5 ) according to the original point coordinates. First, with the temporal kernel size l = 3, each input point feature is decoded to 3 features by a temporal transposed convolution. Second, 5 frames are interpolated in accordance with a temporal stride st = 2, and the input points with the decoded features are assigned to interpolated frames. In this way, the temporal cross-correlation is constructed. Third, given the original coordinates of a point cloud frame (e.g., P1) and the corresponding interpolated frame (i.e.,[ P ′1; [M ′a 1,M ′b 1] ] ), we generate features (i.e., F ′′1 ) for all the original point coordinates.

heading: B PSTNET ARCHITECTURES

text:
B.1 PSTNET FOR 3D ACTION RECOGNITION
As shown in Fig. 7, the architecture of PSTNet for 3D action recognition consists of 6 PST convolutions, i.e., PSTConv1, PSTConv2a, PSTConv2b, PSTConv3a, PSTConv3b, PSTConv4, and a fully-connected (FC) layer. In PSTConv1, PSTConv2a, PSTConv3a and PSTConv4, the spatial subsampling rate is set to 2 to halve the spatial resolution. In PSTConv2a and PSTConv3a, the temporal stride is set to 2 to halve the temporal resolution. The ro denotes the initial spatial search radius. The temporal padding p is split as [p1, p2], where p1 and p2 denote the left padding and the right padding, respectively. After these convolution layers, average and max poolings are used for spatial pooling and temporal pooling, respectively. The FC layer is then used as a classifier.
B.2 PSTNET FOR 4D SEMANTIC SEGMENTATION
As shown in Fig. 8, we employ 4 PST convolutional layers, i.e., PSTConv1, PSTConv2, PSTConv3, and PSTConv4, and 4 PST transposed convolutional layers, i.e., PSTConvTrans1, PSTConvTrans2, PSTConvTrans3, and PSTConvTrans4 for 4D Semantic Segmentation. The spatial subsampling rate is set to 4 to reduce the spatial resolution in each PST convolution layer. Following (Choy et al., 2019; Liu et al., 2019e), we conduct 4D semantic segmentation on clips with the length of 3 frames, i.e., L is fixed to 3. In PSTConv3 and PSTConvTrans2, the temporal kernel size is set to 3. Skip connections are added from input to PSTConvTrans1, PSTConv1 to PSTConvTrans2, PSTConv2 to PSTConvTrans3, and PSTConv3 to PSTConvTrans4. After PSTConvTrans4, a 1D convolution layer is appended for semantic predictions.
C IMPLEMENTATION DETAILS
To build a unified network and train the network with mini-batch, the number of spatial neighbors needs to be fixed. We follow existing point based works (Qi et al., 2017b; Liu et al., 2019e) to randomly sample a fixed number of neighbors. For 3D action recognition, the number is set to 9. For 4D semantic segmentation, we follow (Liu et al., 2019e) to set the number to 32. If the actual number of neighbors of a point is less than the set one (e.g., 9), we randomly repeat some neighbors.
We train our models for 35 epochs with the SGD optimizer. Learning rate is set to 0.01, and decays with a rate of 0.1 at the 10th epoch and the 20th epoch, respectively.
MSR-Action3D (Li et al., 2010): Following (Liu et al., 2019e), batch size is set to 16 and frame sampling stride are set to and 1, respectively. We set the initial spatial search radius ro to 0.5.
NTU RGB+D 60 /120 (Shahroudy et al., 2016; Liu et al., 2019a): Following (Wang et al., 2020), batch size is set to 32. We set the clip length, frame sampling stride and the initial spatial search radius ro to 23, 2 and 0.1, respectively.
Synthia 4D (Choy et al., 2019): Following (Liu et al., 2019e), batch size and frame sampling stride are set to 12 and 1, respectively. We set the spatial search radius ro to 0.9.
We have implemented our PSTNets using both PyTorch and PaddlePaddle, which have shown similar performance.
D INFLUENCE OF TEMPORAL KERNEL SIZE ON DIFFERENT ACTIONS
We show the influence of temporal kernel size on different actions in MSR-Action3D in Fig. 9. Clip length is 24.
When temporal kernel size l = 1, temporal correlation is not exploited. However, PSTNet can leverage pose information for recognition, especially for actions whose poses are unique in the dataset. For example, PSTNet with l = 1 can correctly recognize the “jogging” action because the action’s pose is discriminative in the MSR-Action3D dataset. However, PSTNet with motion modeling (l ≥ 3) can distinguish similar actions such as “draw x”, “draw tick” and “draw circle”. The results support our intuition that PSTNet effectively captures dynamics in point cloud sequences.

heading: E FEATURE VISUALIZATION

text: We qualitatively evaluate PSTNet’s ability to encode point cloud sequence by visualizing the learned features. We compare PSTNet with the sate-of-the-art MeteorNet (Liu et al., 2019e). Features are extracted from the layer before classifier (FC). These features are then projected to 2-dimensional space using t-SNE. As shown in Fig. 10, PSTNet feature is more compact and discriminative than MeteorNet.

heading: F SYNTHIA 4D SEMANTIC SEGMENTATION RESULT DETAILS

text: We list the segmentation result for each class in Table 4. The Synthia 4D dataset contains 12 categories. Our PSTNet achieves five best accuracies among them, demonstrating the effectiveness of PSTNet.

heading: G PST CONVOLUTION WITHOUT DISENTANGLING SPACE AND TIME

text:
To confirm the effectiveness of our disentangling structure for spatio-temporal modeling, we perform 3D action recognition on MSR-Action3D with non-decomposing convolution. To combine space and time, 3D displacements and time differences are encoded together to generate convolution kernel. We evaluate our PSTNet with different clip lengths. As shown in Fig. 11, the disentangling PST convolution achieves better accuracy than the non-decomposing structure, demonstrating the effectiveness of disentangling space and time for point cloud sequence modeling.
H IMPACT OF DISPLACEMENT KERNEL AND SHARING KERNEL
In this paper, we design a lightweight implementation for the f function in Eq. (5), i.e., f ( (δx, δy, δz);θ ) = θd · (δx, δy, δz)T · 1 θs, where θd is a displacement transform kernel, θs is a sharing kernel. In this section, we investigate the influence of these two kernels. We conduct 3D action recognition on MSR-Action3D. The experimental results are shown in Table 5.
Without θs, the spatial convolution degenerates intoM (x,y,z) t = ∑ ‖(δx,δy,δz)‖≤r θd · (δx, δy, δz)
T . In this way, only the points in the final layer of our PSTNet are used. Therefore, the accuracy decreases dramatically.
Without θd, the spatial convolution becomes M (x,y,z) t = ∑ ‖(δx,δy,δz)‖≤r θs · F (x+δx,y+δy,z+δz) t . In this fashion, the points in a neighborhood are treated equally. Point features are leveraged but their positions are ignored. Because spatial structure is not well captured, the accuracy decreases compared to the case where both θd and θs are used.

heading: I COMPUTATIONAL EFFICIENCY AND MEMORY USAGE

text:
In this section, we evaluate the computational efficiency and memory usage, i.e., the running time and number of parameters, of our method. We conduct 3D action recognition with a clip length of 16 on MSR-Action3D using 1 Nvidia Quadro RTX 6000 GPU.
As shown in Table 6, the proposed PSTNet is 41.5% faster than MeteorNet (Liu et al., 2019e). This is because that MeteorNet is based on MLPs, which are less efficient than convolutions.
Second, we study the impact of the number of PSTNet layers on running time and parameter. As shown in Fig. 12, when PSTNet becomes deep, the number of parameters significantly grows. This is because, like most deep neural networks, the point feature dimension exponentially increases to improve the feature representation ability in PSTNet, which needs more parameters. However, the running time does not increase significantly. This is because our PSTNet is spatio-temporally hierarchical, points are exponentially reduced along both spatial and temporal dimensions, thus saving running time.
Chart Title
Third, we investigate the running time of main operations in PSTNet, including convolution, farthest point sampling and spatial neighbor search. We show the running time proportion of these operations in Fig. 13.
Finally, because the f function in Eq. (5) can be implemented in different ways, we replace our lightweight spatial convolution with PointConv (Wu et al., 2019) to investigate the influence of the spatial modeling in our PSTNet. As shown in Table 6, using PointConv only improves accraucy slightly compared to our lightweight spatial convolutions, but significantly increases parameters and
running time. This is because PointConv utilizes MLP based operations to process points and their features, which are less efficient than our lightweight convolutions.
J IRREGULAR FRAME SAMPLING
Our PSTNet can also process irregularly sampled point cloud sequences by frame interpolation. In this section, we randomly remove some frames in sequences. Specifically, we conduct 3D action recognition on MSR-Action with 24 frames. We randomly remove 8 frames from each clip. Then, we use replication and Iterative Closest Point (ICP) (Besl & McKay, 1992) to interpolate missing frames, respectively. We compare our PSTNet with MeteorNet (Liu et al., 2019e), which explicitly encodes time and does not need interpolation.
As shown in Table 7, our PSTNet still achieves the best accuracy, indicating that PSTNet is able to model point cloud sequences with irregular frame sampling.

heading: K SCENE FLOW ESTIMATION

text:
To evaluate our method on the LiDAR data where points in a frame can be highly irregular, we apply our method to scene flow estimation. We follow the setting proposed by MeteorNet (Liu et al., 2019e), i.e., given a point cloud sequence, estimating a flow vector for every point in the last frame. However, because our point tube is constructed according to the middle frame, which is not applicable to the last-frame scene flow estimation, we adapt temporal anchor frame selection and spatial anchor point transferring. Specifically, we select the last frame in each tube as the anchor frame. Then, after spatial sampling, each anchor point is transferred to its previous nearest l frames. Following MeteorNet, we first train our model on a FlyingThings3D dataset according to the synthetic method in (Liu et al., 2019e), and then fine-tune the model on a KITTI scene flow dataset (Liu et al., 2019e). Point tracking is not used.
As shown in Table 8, our PSTNet achieves promising accuracy on scene flow estimation.

heading: L MOTION RECOGNITION ON THE SYNTHETIC MOVING MNIST POINT CLOUD DATASET

text:
In the real world, it is impossible to obtain point IDs. To evaluate our PSTNet in the scenario where points are correlated and can be accurately tracked across frames, we conduct motion recognition on a synthetic Moving MNIST Point Cloud dataset. To synthesize moving MNIST digit sequences, we use a generation process similar to that described in (Fan & Yang, 2019). Each synthetic sequence consists of 16 consecutive point cloud frames. Each frame contains one handwritten digit moving and bouncing inside a 64× 64 area. The digits are chosen randomly from the MNIST dataset. We
use the same training/test split as the original MNIST dataset. For each digit, we sample 128 points. Point order is maintained across frames so that tracking can be employed. We design 144 motions, including
• nine initial locations: (1, 1), (1, 3), (1, 5), (3, 1), (3, 3), (3, 5), (5, 1), (5, 3), (5, 5).
• eight velocities: (1, 1), (−1, 1), (−1,−1), (1,−1), (2, 2), (−2, 2), (−2,−2), (2,−2). • two kinds of digit distortion: scaling digits horizontally or vertically, i.e., (|0.4− 0.05t|+ 0.6)× (x− cx) + cx or (|0.4− 0.05t|+ 0.6)× (y − cy) + cy, where (cx, cy) is the digit center and t ∈ [1, 16].
A few motion examples are shown in Fig. 14. Because motion and appearance are independent of each other, it is challenging to recognize motion while avoiding interference from digit appearance.
To exploit point correlation, anchor points and their neighbors are selected in the first frame, which are then propagated to other frames. In this fashion, anchors and their neighbors are tracked across frames.
As shown in Table 9, both the original PSTNet and the tracking based PSTNet achieve promising accuracy on the Moving MNIST Point Cloud dataset. Our original PSTNet achieves similar accuracy with the tracking based PSTNet in this simulated case, demonstrating that our PSTNet does not heavily rely on point IDs or tracking.
M VISUALIZATION OF THE OUTPUT OF EACH PST CONVOLUTION LAYER IN PSTNET
We visualize the output of each layer in the PSTNet architecture trained on MSR-Action3D in Fig. 15. Because we use ReLU as the activation function, all outputs are greater than zero and large outputs indicate high activation. To visualize outputs, we squeeze output vectors to scalars via l1 norm. We observe that:
• For PSTConv1, because temporal kernel size l = 1, it does not capture the temporal correlation. In this case, PSTConv1 focuses on the appearance and therefore outputs high activation to performer contours.
• PSTConv2a to PSTConv4. When modeling the spatio-temporal structure of point cloud sequence, PST convolutions mainly focus on moving areas so as to achieve informative representations of actions.
The visualization results support our intuition that the proposed PST convolution effectively captures dynamics of point cloud sequences.
N VISUALIZATION OF 4D SEMANTIC SEGMENTATION
We visualize segmentation results from the Synthia 4D dataset in Fig. 16 and Fig. 17. Our PSTNet can accurately segment most objects.

heading: O LIMITATION

text:
Like most deep neural networks, the proposed PSTNet relies on large-scale labeled datasets for training. A potential improvement is to integrate some learning methods, e.g., few-shot learning (Liu et al., 2019c;f; Zhu & Yang, 2020; Liu et al., 2021), into point cloud sequences modeling to reduce the reliance on human-annotated data.
In p u t: 𝑁 = 2 0 4 8
P ST C o n v1
: 𝑁 = 1 0 2 4
w /o
t em
p o
ra l m
o d
el in
g
P ST C o n v2 a:
𝑁 = 5 1 2
P ST C o n v2 b
: 𝑁 = 5 1 2
P ST C o n v3 a:
𝑁 = 2 5 6
P ST C o n v3 b
: 𝑁 = 2 5 6
P ST C o n v4
: 𝑁 = 1 2 8
In p u t: 𝑁 = 2 0 4 8
P ST C o n v1
: 𝑁 = 1 0 2 4
w /o
t em
p o
ra l m
o d
el in
g
P ST C o n v2 a:
𝑁 = 5 1 2
P ST C o n v2 b
: 𝑁 = 5 1 2
P ST C o n v3 a:
𝑁 = 2 5 6
P ST C o n v3 b
: 𝑁 = 2 5 6
P ST C o n v4
: 𝑁 = 1 2 8
Fi gu
re 15
:V is
ua liz
at io
n of
th e
ou tp
ut of
ea ch
la ye
ri n
PS T
N et
.F or
th e
in pu
tp oi
nt cl
ou d
se qu
en ce
,c ol
or en
co de
s de
pt h.
Fo rt
he ou
tp ut
s, br
ig ht
er co
lo ri
nd ic
at es
hi gh
er ac
tiv at
io n.
In pu
ts eq
ue nc
es co
ns is
to f2
4 fr
am es
.D ue
to th
e sp
at ia
ls ub
sa m
pl in
g s s
an d
th e
te m
po ra
ls tri
de s t
,p oi
nt s
an d
fr am
es pr
og re
ss iv
el y
de cr
ea se
al on g th e ne tw or k de pt h.

title: PSTNET: POINT SPATIO-TEMPORAL CONVOLUTION ON POINT CLOUD SEQUENCES