summaries: This paper studies the problem of predicting the segmentations and poses (position + yaw orientation) of multiple objects, given the image of a scene. The paper introduces a method that is trained without supervision for the segmentations, similar to several other recent object-centric models. In contrast to these existing models, the method proposed in this paper additionally estimates the 3D location of each object by predicting a depth map and classifies the yaw angle by representing the pose domain as equally-spaced bins. To do so, during training the method operates on a short clip of the scene recorded by a moving camera, and uses self-supervision by predicting the scene’s image at the next time step. At test time, the model is able to infer a representation of each object in the scene and segment them given a single image of the scene.

summaries: This paper presents a method to learn how to parse 3-frame videos into object-centric representations, which include segmentation masks and 3D positions and yaws for those objects frame-by-frame, as well as an image representation of the background, and an overall depth map. This is accomplished with a depth network, an object network with an LSTM at the bottleneck to iteratively pick out objects and their positions and yaws, and a decoder to provide segmentations, a warping/re-compositing operation that pastes the inferred objects at their estimated positions for the NEXT frame (i.e., with a constant-velocity assumption), and finally an "imagination" network which refines the estimated image. The model learns with a combination of 4 losses, which include image reprojection/prediction, depth consistency, a spatial term that includes consistency and randomness (though I have complaints about this), and finally a penalty term that discourages object probabilities from being zero. The paper also introduces a new synthetic dataset where prior methods do badly, and the proposed method does slightly better. The learned depth maps look good, but this is perhaps expected because camera poses are known.

summaries: This paper presents an unsupervised object-centric scene representation technique that can decompose a scene into multiple objects (and segment the scene) and infer their 3D locations and pose. The overall setup is very similar to earlier models like MONet but this model works on sequences of images, more precisely on 3 consecutive images. It uses the first two images to infer the 3D position and pose of objects and combining this with known camera motion tries to predict the last (third) image. The main contribution here is an optical flow based method to warp the image at time t using the predicted object location/pose/depth to predict (some of) the pixels in image at time t+1.  In more detail, the object extraction network outputs the location and pose of each object. And a separate depth perception network outputs the depth for each pixel in the image. The location and pose of objects are used to estimate the velocity of each object (e.g., by subtracting the position at t-1 from position at t. note this requires matching each object at time t-1 to object in time t, which they do using a soft-matching approach). These along with the depth information are then used to warp the image at t to predict pixels in image at time t+1. This is possible only for a subset of the pixels so for the rest, they use a separate "imagination" network that takes in object information and predicts the color/depth and object masks at t+1. The predictions from warping and imagination network are then combined to form the final predicted color and depth images.  To train the model, they require images and camera motion, and use a combination of losses: reconstruction loss on predicted and ground truth image, self-supervised losses on object location, pose, and depth.

summaries: Inspired by ideas about how humans learn about objects, the authors detail a system to train a neural network to perceive generic objects using image triplets where objects move and the viewer also can move.  The viewer's motion is provided as an input. Object perception by parts of the network operating on the first two time points is rewarded by predicting what is seen at the third time point (the training signal). Having been trained, the object perception part of the system, which is a relatively basic neural network, can segment them from a single image.  This new setup requires different data that what has been used in this space, and the authors contribute a synthetic dataset as well.

abstractText: The representation of objects is the building block of higher-level concepts. Infants develop the notion of objects without supervision. The prediction error of future sensory input is likely the major teaching signal for infants. Inspired by this, we propose a new framework to extract object-centric representation from single 2D images by learning to predict future scenes in the presence of moving objects. We treat objects as latent causes whose function to an agent is to facilitate efficient prediction of the coherent motion of their parts in visual input. Distinct from previous object-centric models, our model learns to explicitly infer objects’ locations in 3D environment in addition to segmenting objects. Further, the network learns a latent code space where objects with the same geometric shape and texture/color frequently group together. The model requires no supervision or pre-training of any part of the network. We provide a new synthetic dataset with more complex textures on objects and background and found several previous models not based on predictive learning overly rely on clustering colors and lose specificity in object segmentation. Our work demonstrates a new approach for learning symbolic representation grounded in sensation and action.

authors:
Daniel M Bear
Chaofei Fan
Damian Mrowca
Yunzhu Li
Seth Alter
Aran Nayebi
Jeremy Schwartz
Li Fei-Fei
Jiajun Wu
Joshua B Tenenbaum

title: Learning physical graph representations from visual scenes

venue: arXiv preprint arXiv:2006.12373,

authors:
Christopher P Burgess
Loic Matthey
Nicholas Watters
Rishabh Kabra
Irina Higgins
Matt Botvinick
Alexander Lerchner. Monet

title: Unsupervised scene decomposition and representation

authors:
Liang-Chieh Chen
George Papandreou
Iasonas Kokkinos
Kevin Murphy
Alan L Yuille

title: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey Hinton

title: A simple framework for contrastive learning of visual representations

venue: Proceedings of the 37th International Conference on Machine Learning,

authors:
Coline Devin
Pieter Abbeel
Trevor Darrell
Sergey Levine

title: Deep object-centric representations for generalizable robot learning

venue: IEEE International Conference on Robotics and Automation (ICRA),

authors:
Martin Engelcke
Adam R Kosiorek
Oiwi Parker Jones
Ingmar Posner

title: Genesis: Generative scene inference and sampling with object-centric latent representations

authors:
Martin Engelcke
Oiwi Parker Jones
Ingmar Posner

title: Reconstruction bottlenecks in objectcentric generative models

venue: arXiv preprint arXiv:2007.06245,

title: Genesis-v2: Inferring unordered object representations without iterative refinement

venue: arXiv preprint arXiv:2104.09958,

title: Efference copy and corollary discharge: implications for thinking and its disorders

venue: Schizophrenia bulletin,

title: The developmental psychology of jean piaget

authors:
Klaus Greff
Raphaël Lopez Kaufman
Rishabh Kabra
Nick Watters
Christopher Burgess
Daniel Zoran
Loic Matthey
Matthew Botvinick
Alexander Lerchner

title: Multi-object representation learning with iterative variational inference

venue: Proceedings of the 36th International Conference on Machine Learning,

authors:
Paul Henderson
Christoph H Lampert

title: Unsupervised object-centric video generation and decomposition in 3d

venue: arXiv preprint arXiv:2007.06705,

authors:
Daniel Kersten
Pascal Mamassian
Alan Yuille

title: Object perception as bayesian inference

venue: Annu. Rev. Psychol.,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Thomas Kipf
Elise van der Pol
Max Welling

title: Contrastive learning of structured world models

venue: arXiv preprint arXiv:1911.12247,

authors:
Francesco Locatello
Dirk Weissenborn
Thomas Unterthiner
Aravindh Mahendran
Georg Heigold
Jakob Uszkoreit
Alexey Dosovitskiy
Thomas Kipf

title: Object-centric learning with slot attention

venue: arXiv preprint arXiv:2006.15055,

authors:
Li Nanbo
Cian Eastwood
Robert B Fisher

title: Learning object-centric representations of multiobject scenes from multiple views

venue: In 34th Conference on Neural Information Processing Systems,

authors:
Randall C O’Reilly
Jacob L Russin
Maryam Zolfaghar
John Rohrlich

title: Deep predictive learning in neocortex and pulvinar

venue: Journal of Cognitive Neuroscience,

authors:
Jean Piaget
Margaret Trans Cook

title: The origins of intelligence in children

authors:
Joseph Redmon
Santosh Divvala
Ross Girshick
Ali Farhadi

title: You only look once: Unified, real-time object detection

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Shaoqing Ren
Kaiming He
Ross Girshick
Jian Sun

title: Faster r-cnn: Towards real-time object detection with region proposal networks

venue: Advances in neural information processing systems,

authors:
Danilo Jimenez Rezende
Shakir Mohamed
Daan Wierstra

title: Stochastic backpropagation and approximate inference in deep generative models

venue: In International conference on machine learning,

authors:
Olaf Ronneberger
Philipp Fischer
Thomas Brox

title: U-net: Convolutional networks for biomedical image segmentation

venue: In International Conference on Medical image computing and computerassisted intervention,

authors:
Rishi Veerapaneni
John D Co-Reyes
Michael Chang
Michael Janner
Chelsea Finn
Jiajun Wu
Joshua Tenenbaum
Sergey Levine

title: Entity abstraction in visual model-based reinforcement learning

venue: In Conference on Robot Learning,

authors:
Dequan Wang
Coline Devin
Qi-Zhi Cai
Fisher Yu
Trevor Darrell

title: Deep object-centric policies for autonomous driving

venue: In 2019 International Conference on Robotics and Automation (ICRA),

authors:
Olivia Wiles
Georgia Gkioxari
Richard Szeliski
Justin Johnson

title: Synsin: End-to-end view synthesis from a single image

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

heading: 1 INTRODUCTION

text:
Visual scenes are composed of various objects in front of backgrounds. Discovering objects from 2D images and inferring their 3D locations is crucial for planning actions in robotics (Devin et al., 2018; Wang et al., 2019) and this can potentially provide better abstraction of the environment for reinforcement learning (RL), e.g. Veerapaneni et al. (2020). The appearance and spatial arrangement of objects, together with the lighting and the viewing angle, determine the 2D images formed on the retina or a camera. Therefore, objects are latent causes of 2D images, and discovering object is a process of inferring latent causes (Kersten et al., 2004). The predominant approach in computer vision for identifying and localizing objects rely on supervised learning to infer bounding boxes (Ren et al., 2015; Redmon et al., 2016) or pixel-level segmentation of objects (Chen et al., 2017). However, the supervised approach requires expensive human labeling. It is also difficult to label every possible category of objects. Therefore, an increasing interest has developed recently to build unsupervised or self-supervised models to infer objects from images, such as MONet (Burgess et al., 2019), IODINE (Greff et al., 2019) slot-attention (Locatello et al., 2020), GENESIS (Engelcke et al., 2019; 2021), C-SWM (Kipf et al., 2019) and mulMON (Nanbo et al., 2020). Our work also focuses on the same unsupervised object-centric representation learning (OCRL) problem, but offers a new learning objective and architecture to overcome the limitation of existing works in segmenting more complex scenes and explicitly represents objects’ 3D locations.
The majority of the existing OCRL works are demonstrated on relatively simple scenes with objects of pure colors and background lacking complex textures. As recently pointed out, the success of several recent models based on a variational auto-encoder (VAE) architecture (Kingma & Welling, 2013; Rezende et al., 2014) depends on a reconstruction bottleneck that needs to be intricately balanced (Engelcke et al., 2020). To evaluate how such models perform on scenes with more complex surface textures, we created a new dataset of indoor scenes with diverse texture patterns on the objects and background. We found that several existing unsupervised OCRL models overly rely on clustering pixels based on their colors. A challenge in our dataset and in real-world perception is that sharp boundaries between different colors exist both at contours of objects and within the surface of the same object. A model essentially has to implicitly learn a prior knowledge of which types of boundaries are more likely to be real object contours. The reconstruction loss in existing works appears to be insufficient for learning this prior.
To tackle this challenge, we draw our inspiration from development psychology and neuroscience. Infants understand the concept of object as early as 8 months, before they can associate objects with names (Piaget & Cook, 1952; Flavell, 1963). The fact that infants are surprised when object are hidden indicates that they have already learned to segment discrete objects from the scene and that their brains constantly make prediction and check for deviation from expected outcome. As the brain lacks direct external supervision for object segmentation, the most likely learning signal is from the error of this prediction. In the brain, a copy of the motor command (efference copy) is sent from the motor cortex simultaneously to the sensory cortex, which is hypothesized to facilitate the prediction of changes in sensory input due to self-generated motion (Feinberg, 1978). What remains to be predicted are changes in visual input due to the motion of external objects. Therefore, we believe that the functional purpose of grouping pixels into object is to allow the prediction of the motion of the constituting pixels in a coherent way by tracking very few parameters (e.g., the location, pose, and speed of an object). Driven by this hypothesis, our contribution in this paper is: (1) we combine predictive learning and explicitly 3D motion prediction to learn 3D aware objectcentric representation, which we call Object Perception by Predictive LEarning (OPPLE); (2) we provide a new dataset1 with complex surface texture and motion by both the camera and objects to evaluate object-centric representation models; (3) we found several previous models overly rely on clustering colors to segment objects; (4) although our model leverages image prediction as learning objective, the architecture generalize the ability of object segmentation and spatial localization to single-frame images.

heading: 2 METHOD

heading: 2.1 PROBLEM FORMULATION

text:
We denote a scene as a set of distinct objects and a background S = {O1, O2, . . . , OK , B}, whereK is the number of objects in scene. At any moment t, we denote two state variables, the location and pose of each object relative to the perspective of an observer (camera), as x(t)1:K andφ (t) 1:K , where x (t) k is the 3-d coordinate of the k-th object and φ(t)k is its yaw angle from a canonical pose, as viewed from the reference frame of the camera (for simplicity, we do not consider pitch and roll here and leave it for future work to extend to 3D pose). At time t, given the location of the camera o(t) ∈ R3 and its facing direction α(t), S renders a 2D image on the camera as I(t) ∈ Rw×h×3, where w × h is the size of the image. Our goal is to train a neural network that infers properties of objects given only a single image I(t) as the sole input without external supervision and with only the information of the intrinsics and egomotion of the camera without explicit knowledge of its extrinsic matrix:
{z(t)1:K ,π (t) 1:K+1, x̂ (t) 1:K ,p (t) φ1:K } = fobj(I(t)) (1)
Here, z(t)1:K is a set of view-invariant vectors representing the identity of each object k. “Viewinvariant” is loosely defined as |z(t)k − z (t+∆t) k | < |z (t) k − z (t) l | for k 6= l and ∆t > 0 in most cases, i.e., the vector codes are more similar for the same object across views than they are different across objects. π(t)1:K+1 ∈ R(K+1)×w×h are the probabilities that each pixel belongs to any of the objects or the background ( ∑ k πkij = 1 for any pixel at i, j), which achieve object segmentation. To localize objects, x̂(t)1:K are the estimated locations of each object relative to the observer and p (t) φ1:K are estimated probability distribution of the poses of each object. Each p(t)φk ∈ R b is a probability distribution over b equally-spaced bins of yaw angles in (0, 2π).

heading: 2.2 OVERALL PRINCIPLE: LEARNING OBJECT REPRESENTATION BY PREDICTING THE FUTURE

text:
Our hypothesis is that the notion of object emerges to meet the need of efficiently predicting the future fates of all parts of an object. With the (to be learned) ability to infer an object’s pose and location from each frame, the object’s speed of translation and rotation can be estimated from consecutive frames. If depth is further inferred for each pixel belonging to an object, then the optical
1We will release upon publication of the paper
flow of each pixel can be predicted based on the object’s speed and the position of each pixel relative to the object’s center. The pixel-segmentation of an object essentially prescribes which pixels should move together with the object. With the predicted optical flow, one can further predict part of the next image by warping the current image. The parts of the next image unpredictable by warping include surfaces of objects or the background that are currently occluded but will become visible, and the region of a scene newly entering the view due to self- or object-motion. These portions can only be predicted based on the learned statistics of the appearance of objects and background, which we call ”imagination”. In this work, we will show that with the information of self-motion, knowledge of geometry (rule of rigid-body movement) and the assumption of smooth object movement, the object representations captured by function fobj and depth perception can be learned without supervision.

heading: 2.3 NETWORK ARCHITECTURE

text:
To demonstrate the hypothesized principle above, we build our OPPLE networks as illustrated in Figure 1, which process two consecutive images individually and make prediction for the next image with the information extracted from them.
Object extraction network. We build an object extraction network fθobj by modificaiton of a U-Net (Ronneberger et al., 2015) to extract representation for each object, the pixels it occupies and its spatial location and pose. A basic U-Net is composed of a convolutional encoder and a transposed convolutional decoder, while each encoder layer sends a skip connection to the corresponding decoder layer, so that the decoder can combine both global and local information. Inside our fθobj , an image I(t+1) first passes through the encoder. Additional Atrous spatial pyramid pooling layer (Chen et al., 2017) is inserted between the middle two convolutional layers of the encoder to expand the receptive field. The top layer of the encoder outputs a feature vector et capturing the global information of the scene. A Long-Short Term Memory (LSTM) network further repeatedly reads in e(t)
and sequentially outputs one code for an object at a time. Each object code is then mapped through a one-layer fully connected network to predict object code z(t)k , object location x̂ (t) k and object pose probability p φ (t) k
, k = 1, 2, · · · ,K. The inferred location is restricted to the visible range of the camera with an upper limit of distance. The pose prediction is represented as log p
φ (t) k
for numerical
stability. Each object code vector z(t)k is then independently fed through the decoder with shared skip connection from the encoder. The decoder outputs one channel for each pixel, representing an un-normalized log likelihood that the pixel belongs to the object k. The unnormalized logit maps for all objects are concatenated with a map of all zero for the background, and compete through a softmax function to output the probabilistic segmentation map π(t)k .
Depth perception network. We use a standard U-Net for depth perception function hθdepth that processes images I(t) and output a single-channel depth mapD(t).
Object-based imagination network. We build the imagination network gθimag also with a modified U-Net. The input is concatenated image I(t) and log of depth log(D(t)) inferred by the depth perception network, both multiplied element-wise by one probabilistic mask π(t)k . The output of the encoder network is concatenated with a vector composed of the observer’s moving velocity v(t)obs and rotational speed ω(t)obs, and the estimated object location x̂ (t) k , velocity v̂ (t) k and rotational speed ω̂ (t) k before entering the decoder. The decoder outputs five channels for each pixel: three for predicting RGB colors, one for depth and one for the probability of the pixel belonging to any object k or background and is used to weight the predicted color and depth for the final ”imagination”.

heading: 2.4 LEARNING OBJECT REPRESENTATION BY PREDICTION

text: Below, we explain details of the approach. The pseudocode of the algorithm is provided in the appendix.

heading: 2.4.1 PREDICTION BY WARPING

text:
We first describe the prediction of part of the next image by warping the current image. Here we consider only rigid objects and the fates of all visible pixels belonging to an object. With depth D(t) ∈ Rw×h = hθ(I(t)) of all pixels in a view inferred by the Depth Perception network based on visual features in the image I(t) , the 3D location of a pixel m̂(t)(i,j) at any coordinate (i, j) in the image, can be determined given the focal length d of the camera. We use the inferred current and previous locations, x̂(t)k and x̂ (t−1) k to estimate the instantaneous velocity of the K
th object v̂
(t) k = x̂ (t) k − x̂ (t−1) k for all pixels (i, j) attributed to thatK
th object. Similarly, with the inferred the current and previous pose probabilities of the object, p(t)φk and p (t−1) φk , we can obtain the likelihood of its angular velocity p(φ(t)k , φ (t−1) k | ω (t) k = ω). We obtain the posterior distribution of the object’s next pose p(φ(t+1)k | φ (t) k , φ (t−1) k ) with a Von Mises prior. More details about warping and pose estimation are added in the Appendix.
Assuming a pixel (i, j) belongs to object k, we use the estimated motion information v̂(t)k and p(ω
(t) k | φ (t) k , φ (t−1) k ) of the object, together with the current location and pose of the object and the current 3D location m̂(t)(i,j) of the pixel, to predict the 3D location m ′(t+1) k,(i,j) of the pixel at the next moment asm′(t+1)k,(i,j) = M (t) −ωobs [M (t) ω̂k (m̂ (t) (i,j)− x̂ (t) k ) + x̂ (t) k + v̂ (t) k −v (t) obs], whereM (t) −ωobs andM (t) ω̂k are rotational matrices due to the rotation of the observer and the object, respectively, and v(t)obs is the velocity of the observer (relative to its own reference frame at t). In this way, assuming objects move smoothly most of the time, if the self motion information is known, the 3D location of each visible pixel can be predicted. If a pixel belongs to the background, ωK+1 = 0 and vK+1 = 0 (K+ 1 is the background’s index). Given the predicted 3D location, the target coordinate (i′, j′)(t+1)k of the pixel on the image and its new depth D′k(i, j)
(t+1) can be calculated. This prediction of pixel movement allows predicting the image I ′(t+1) and depth D′(t+1) by weighting the colors and depth of pixels predicted to land near each pixel at the discrete grid of the next frame.
Considering that object attribution of each pixel has to be inferred, we can write object attribution as a probability of belonging to each object, across all pixels, π(t)k , k = 1, 2, · · · ,K+1. The predicted motion of each pixel should be described as a probability distribution over K + 1 discrete target locations p(x′(t+1)(i,j) ) = ∑K+1 k=1 π (t) kij ·δ(x′ (t+1) k,(i,j)), i.e., pixel (i, j) has a probability of π (t) kij to move to location x′(t+1)k,(i,j). Using probabilistic prediction of pixel movements, we partially predict the color of next image at the pixel grids where some original pixels from the current will land by weighting their contribution I ′(t+1)Warp (p, q). This contribution term is used to calculate a weight for effect of a pixel (i,j) on position (p,q) on the grid, and is written as wk(i, j, p, q). More details in the appendix.

heading: 2.4.2 IMAGINATION

text:
For the regions not fully predictable by warping current image with appendix equation (7), i.e., for (p, q) where ∑ k,i,j wk(i, j, p, q) < 1, we learn a function g that “imagines” the appearance I ′ (t+1) kImag ∈ Rw×h×3 and the pixel-wise depth D′ (t+1) kImag ∈ Rw×h of the object or background k in the next frame, and the predicted probabilities that each pixel in the next frame belongs to each object or the background π′(t+1)kImag ∈ Rw×h. The function takes as input portion of the current image corresponding to each object I(t) π(t)k and the inferred depth D(t) π (t) k , both extracted by element-wise multiplying with the probabilistic segmentation mask π(t)k , the information of the camera’s self motion, and the location and motion of that object:
{I ′(t+1)kImag ,D′ (t+1) kImag ,π ′(t+1) kImag} = g(I (t) i π (t) k ,D (t) π(t)k , x̂ (t) k , v̂ (t) k , ω̂ (t) k , ω (t) obs) (2)
The “imagination” specific for each object and the background can then be merged using the weights prescribed by π′(t+1)1:KImag: I ′(t+1) Imag = ∑ k I ′(t+1) kImag π′ (t+1) kImag , andD ′(t+1) Imag = ∑ kD ′(t+1) kImag π′ (t+1) kImag .

heading: 2.4.3 COMBINING WARPING AND IMAGINATION

text:
The final predicted image or depth map are weighted average of the prediction made by warping the current image or predicted depth map and the corresponding predictions by imagination:
I ′ (t+1) = I ′ (t+1) Warp WWarp + I ′ (t+1) Imag (1−WWarp) (3)
Here, WWarp ∈ Rw×h with each element WWarp(p, q) = max{ ∑ k,i,j w(i, j, p, q), 1}. The intuition is that imagination is only needed when there is not sufficient contribution for predicting a pixel by warping. The same weighting applies for generating the final predicted depthD′(t+1).
In addition to the image, the states of the objects can also be predicted. The location of object k at t + 1 can be predicted as x′(t+1)k = M (t) −ωobs(x̂ (t) k + v̂ (t) k − v (t) obs). Its new pose probability can be
predicted by p′(t+1)(φk + ωobs = γ2) = ∑
γ1,ω,γ2−γ1∈ {ω−2π,ω,ω+2π}
p(ω̂ (t) k = ω)p(φ (t) k = γ1) for γ2 equal to
each fixed yaw angle bin. To obtain p′(t+1)(φk) instead of p′ (t+1)
(φk+ωobs) at the same set of bins, the vector p′(t+1)(φk+ωobs) can be shifted by multiplication with a pre-computed matrix composed of a bank of shifted Von Mises distributions to “move” the probability mass on the angle bins.
There is one important issue of object-centric representation when making prediction for future images: in order to predict the spatial state of each object at t + 1 based on the views at t and t− 1, the network needs to match the representation of an object at t from the representation of the same object at t − 1. As the dimensions of features (e.g., shape, surface texture, size, etc.) grows, the number of possible objects grows exponentially. Therefore, we cannot simply match object representations based on the order by which an LSTM extracts objects, as this requires learning a consistent order over enormous amount of objects. Instead, we take a soft-matching approach: we take a subset of the features in z(t)k extracted by f as an identity code for each object. For object k at time t, we calculate the distance between its identity code and those of all objects at t − 1, and pass the distances through a radial basis function to serve as a matching score rkl indicating how closely the object k matches each of the previous objects. The scores are used weight all the
estimated translational and rotational speeds for object k each assuming a different object l were the true object k at t− 1. We additionally introduce a fixed identity code zK+1 = 0 for the background and set the predicted motion of background to zero.

heading: 2.4.4 LEARNING OBJECTIVE

text:
Above, we have explained how the next image input I ′(t+1), the depth map D′(t+1) and the spatial states of each object, x′(t+1)k and p ′(t+1) φk can be predicted based on object-centric representation extracted by a function f from the current and previous images I(t)i and I (t−1) i , the depth D ′(t) extracted by a function h, combined with the prediction from object-based imagination function g that are all to be learned. Among the three prediction targets, only the ground truth of visual input I(t+1) is available, while the other can only be inferred by f and h from I(t+1). Therefore, for the prediction targets other than I(t+1), we use the self-consistent loss between the predicted value based on t and t− 1 and the inferred value based on t+ 1 as additional regularization terms to learn the functions f and g.
To learn the functions f , g and h, we approximate them with deep neural networks with parameters θ and optimize θ to minimize the following loss function:
L = Limage + λdepthLdepth + λspatialLspatial + λmapLmap (4)
Here, Limage = MSE(I ′ (t+1) , I(t+1) is the image prediction error. Ldepth = MSE(log(D′(t+1)), log(D̂(t+1))) is the error between the predicted and inferred depth. These provide major teaching signals. Lspatial = ∑K k=1 | ∑K+1 l=1 rklx ′(t+1) l − x̂ (t+1) k |2− ∑K k=1 min{|x̂ (t+1) rand − x̂ (t+1) k |, δ} + ∑K k=1 |x̂ (t+1) k − ∑ i,j m̂ (t+1) i,j πkij |2 + ∑K k=1DKL(p̂ (t+1) φk || ∑K+1 l=1 rklp ′(t+1) φl
) is the self-consistent loss on spatial information prediction. The first term is the error between inferred and predicted location of each object, while the calculation of the predicted location incorporates soft matching between objects in consecutive frames. The second term is the negative term of contrastive loss, which we found empirically prevents the network from reaching a local minimum where all objects are inferred at the same location relative to the camera (and covering minimal regions of the picture). x̂rand is the inferred object location from a random sample within the same batch. The third term penalizes the discrepancy between the inferred object location and the average location of pixels in its segmentation mask. The last term is the KL-divergence between the predicted and inferred pose for each object at t+ 1. Lmap = ReLu(10−4−π1:K) +πk ·πl, for k 6= l avoids loss of gradient due to zero probability of object belonging and discourages overlap between maps of different objects.

heading: 2.5 DATASET

text: We procedurally generated a dataset composed of 306445 triplets of images captured by a virtual camera with field of view of 90 degrees in a square room. The camera translates horizontally and pans with random small steps between consecutive frames to facilitate the learning of depth perception. 3 objects with random shape, size, surface color or textures are spawned at random locations in the room and each move with a randomly selected constant velocity and panning speed. The translation and panning of the the camera is known to the networks. No other ground truth information is provided. The first two frames serve as data and the last frame serve as the prediction target at t+ 1. An important difference between this dataset and other commonly used synthetic datasets for OCRL is that more complex and diverse textures are used on both the objects and the background. We further evaluated on a richer version of the Traffic dataset (Henderson & Lampert, 2020) in A.6.

heading: 2.6 COMPARISON WITH OTHER WORKS

text:
To compare our work with the states-of-the-art models of unsupervised object-centric representation learning, we trained MONet (Burgess et al., 2019), slot-attention (Locatello et al., 2020) and GENESIS2 (Engelcke et al., 2021) on the same dataset. Although these models are trained on single images, all images of each triplets are used for training.
2We failed to obtain reasonable result by training GENESIS V2 on our dataset, thus we adopted a GENESIS network pre-trained on GQN dataset and retrained on our dataset with K=7.
To address the concern that the original configurations of the models are not optimized for more difficult dataset, we trained variants of some of the models with large network size. For MONet, we tested channel numbers of [32, 64, 128, 128] (MONet-128) and [32, 64, 128, 256, 256] (MONet-128-bigger) for the hidden layers of encoder of the component VAE instead of [32, 32, 64, 64] and adjusted decoder layers sizes accordingly, and increased the base channel from 64 to 128 for the attention network. For slot attention, we tested a variant which increased the number of features in the attention component from 64 to 128 (slotattention-128). Slot numbers were chosen as 4 except for GENESIS.

heading: 3 RESULTS

text: After training the networks, we evaluate them on 4000 test images unused during training but generated randomly with the same procedure, thus coming from the same distribution. We compare the performance of different models mainly on their segmentation performance. Additionally, we demonstrate the ability unique to our model: inferring locations of objects in 3D space and the depth of the scene. The performance of depth perception is illustrated in the appendix.

heading: 3.1 OBJECT SEGMENTATION

text:
Following prior works (Greff et al., 2019; Engelcke et al., 2019; 2021), we evaluated segmentation with the Adjusted Rand Index of foreground objects (ARI). In addition, for each image, we matched ground-true objects and background with each of the segmented class by ranking their Intersection over Union (IoU) and quantified the average IoU over all foreground objects3. The performance is summarized in table 2.6 .
Our model outperforms all compared models on ARI and is second to a slot-attention-128 in IoU. As shown in Figure 2, MONet and GENESIS appear to heavily rely on color to group pixels into the same masks. Even though some of these models almost fully designate pixels of an object to a mask, the masks lacks specificity in that they often include pixels with similar colors from other objects or background. Patterns on the backgrounds are often treated as objects as well. The reason of such drawbacks awaits further investigation but we postu-
late there may be fundamental limitation in the approach that learns purely from static discrete images. Patches in the background with coherent color offer room to compress information similarly as objects with coherent colors do, and their shapes re-occur across images just as other objects. Our model is able to learn object-specific masks because these masks are used to predict optical flow specific to each object. A wrong segmentation would generate large prediction error even if the motion of an object is estimated correctly. Such prediction error forces the masks to be concentrated on object surface. They emerge first at object boundaries where the prediction error is the largest and gradually grow inwards during training. Figure 3A-D further compares the distribution
3Due to the artifact at the border of slot-attention model, IoU was calculated excluding those border pixels
of IoU across models. Most models have IoU < 0.01 for about 1/4 of objects. Only OPPLE and slot-attention-128 show bi-modal distributions while other models’ IoU are more skewed towards 0. Figure 3G plots each object’s distance and size on the picture with colors corresponding to their IoUs in our model. Objects with poor segmentation (blue dots) are mostly far away from the camera and occupy few pixels. This is reasonably because motion of farther objects causes less shift on the images and thus provide weaker teaching signal for the network to assign their pixels as separate from the background. For other models, blue dots are more spread even for near objects (not shown).

heading: 3.2 OBJECT LOCALIZATION

text: The Object Extraction Network infers object location relative to the camera. We convert the inferred locations to angles and distance in polar coordinate relative to the camera. Figure 3E-F plot the true and inferred angles and distance, color coded by objects’ IoUs. For objects well segmented (red dots), their angles are estimated high accurately (concentrated on the diagonal in E). Distance estimation is negatively biased for farther objects, potentially because the regularization term on the distance between the predicted and inferred object location at frame t + 1 favors shorter distance when estimation is noisy. Note that the ability to explicitly infer object’s location is not available in other models compared.

heading: 3.3 MEANINGFUL LATENT CODE

text: Because a subset of the latent code (10 dimensions) was used to calculate object matching scores between frames in order to soft-match objects, this should force the object embedding z to be similar for the same objects. We explored the geometry of the latent code by examining whether the nearest neighbours of each of the object in the test data with IoU > 0.5 are more likely to have the same property as themselves. 772 out of 3244 objects’ nearest neighbour had the same shape (out of 11 shapes) and 660 objects’ nearest neighbour had the same color or texture (out of 15). These numbers are 28 to 29 times the standard deviation away from the means of the distribution expected if the nearest neighbour were random (Figure 3H). This suggests the latent code reflects meaningful features of objects. However, texture and shape are not the only factors determining latent code, as we found the variance of code of all objects with the same shape and texture to still be big.

heading: 4 RELATED WORK

text:
Our work is on the same tracks as two recent trends in machine learning: object-centric representation (Locatello et al., 2020) and self-supervised learning (Chen et al., 2020). We take the same
logic as self-supervised learning that learning to predict part of the data based on another part forces a neural network to learn important structures in the data. However, most of the existing works in self-supervised learning do not focus on object-based representation, but instead encode the entire scene as one vector. Other works on object-centric representations overcome this by assigning one representation to each object, as we do. Although works such as MONet (Burgess et al., 2019), IODINE (Greff et al., 2019), slot-attention (Locatello et al., 2020), GENESIS (Engelcke et al., 2019) and PSGNet (Bear et al., 2020) can also segment objects and some of them can ”imagine” complete objects based on codes extracted from occluded objects or draw objects in the correct order consistent with occlusion, few works explicitly infer an object’s location in 3D space together with segmentation purely by self-supervised learning, with the exception of a closely related work O3V (Henderson & Lampert, 2020). Both our works learn from videos. One major distinction is that O3V interleaves spatial and temporal convolution, thus it still require video as input at test time. In contrast, our three major networks process each image independently. Therefore, once trained, our network can generalize to single images. Another distinction from Henderson & Lampert (2020) and many other works is that our model learns from prediction instead of reconstruction. Contrastivelearning of structured world model (Kipf et al., 2019) also learns object masks and predict their future states by linking each object mask with a node in a Graphic Neural Network (GNN). The order of mapping object slot to nodes of GNN is fixed through time, and the actions to objects are coded with specific associated nodes. This arrangement may become infeasible with combinatorial number of different possible objects as the order of assigning different objects to a limited number of nodes may not be consistent across scenes. We solve this by a soft matching of object representation between different time points, which does not require the RNN in the Object Extraction Network to learn a fixed order of extracting different types of objects. On the neuroscience side, our work is highly motivated by recent works on predictive learning (O’Reilly et al., 2021) which also yields view-invariance representation while self-motion signal is available. O’Reilly et al. (2021) used biologically plausible but less efficient learning and applied their model to an easier dataset with objects without background, and did not learn object localization. We should note that explicit spatial localization and depth perception were not pursued in previous works on self-supervised object-centric learning, and the images in our dataset have significantly richer texture information than those demonstrated in previous works (Burgess et al., 2019; Kipf et al., 2019), making the task more challenging. Although view synthesis is not our central goal, the principle illustrated here can be combined with recent advancement in 3D-aware image synthesis (Wiles et al., 2020).

heading: 5 DISCUSSION

text:
We provide a new approach to learn object-centric representation that includes explicit spatial localization of objects, object segmentation from image, automatic matching the same objects across scene based on a learned latent code and depth perception as a by-product. All of the information extracted by our networks are learned without supervision and no pre-training on other tasks is involved. The only additional information required is that of observer’s self-motion, which is available in the brain as efference copy. This demonstrate the possibility of learning rich embodied information of object, one step toward linking neural networks with symbolic representation in general. We expect future works to develop self-supervised learning model for natural categories beyond simple object identity, building on our work.
The work demonstrates that the notion of object can emerge as a necessary common latent cause of the pixels belonging to the object for the purpose of efficiently explaining away the pixels’ coherent movement across frames. In our experiment, object spatial location is inferred more easily than object pose (which we have not fully investigated), thus the predicted warping relies more on object translation than rotation. As a limitation, almost all existing object-centric representation works, including ours, focus on rigid bodies and simple environment. Future works need to explore how to learn object representation for deformable objects, objects with more complex shapes and lighting conditions, and more cluttered environment, towards more realistic application. There is important implication for learning 3D-aware object-based representation, because for such representation to be useful eventually for robotics and RL, agents need to understand an object’s spatial relation to itself based on vision. Learning object-centric representation with explicit 3D information is an important step toward embodied representation of the environment.

heading: A APPENDIX

text:
A.1 PSEUDO CODE OF THE OPPLE FRAMEWORK
Algorithm 1 Developing object-centric representation by predicting future scene InitInitialize: Network parameters θ Input: images I(t−1), I(t) ∈ Rw×h×3, self-motion v(t−1)obs , ω (t−1) obs ,v (t) obs, ω (t) obs Output: prediction I ′(t+1), segmentation π(t−1)1:K+1, π (t) 1:K+1, objects’ codes z (t−1) 1:K , z (t) 1:K , objects’
locations and poses x̂(t−1)1:K ,p (t−1) φ1:K , x̂ (t) 1:K ,p (t) φ1:K for τ = {t− 1, t} do scene code e(τ) ← U-NetEncoderfθ (I(τ)) object code z(τ)1:K , location x̂ (τ) 1:K , pose p (τ) φ1:K
← LSTMfθ (e(τ)) background code zK+1 = 0 depthD(τ) ← hθ(I(τ)) segmentation mask π(τ)1:K+1 ← Softmax (U-NetDecoderfθ (I (τ), z (τ) 1:K), 0)
end for object matching scores rkl ← RBF(z(t)k , z (t−1) l ), k, l ∈ 1 : K + 1 for k ← 1 to K do object motion v̂1:K ,ω1:K ← rk,l, x̂(t)k , x̂ (t−1) l ,p (t) φk ,p (t−1) φl , l = 1 : K + 1
onject-specific optical flowk ← v̂1:K ,ω1:K ,vobs, ωobs,D(t) end for I ′
(t+1) warp ←Warp(I(t), optical flow1:K+1)
I ′ (t+1) imagine ← gθ(I(t) π (t) 1:K+1, log(D (t)) π(t)1:K+1,vobs, ωobs, v̂1:K+1, x̂1:K) final image prediction: I ′(t+1) ← I ′(t+1)warp , I ′ (t+1) imagine,warping weights update parameters: θ ← θ − γ∇θ[|I ′(t+1) − I(t+1)|2 + regularization loss]
A.2 PERFORMANCE ON DEPTH PERCEPTION
We demonstrate a few example images and the inferred depth. Our network can capture the global 3D structure of the scene, although details on object surfaces are still missing. Because background occurs in every training sample, the network appears to bias the depth estimation on objects towards the depth of the walls behind, as is also shown in the scatter plot.
A.3 NETWORK TRAINING AND DATASET
We trained the three networks jointly using ADAM optimization Kingma & Ba (2014) with a learning rate of 3e−4, = 1e−6 and other default setting in PyTorch, with a batch size of 24. 40 epochs were trained on the dataset. We set λspatial = 1.0, λdepth = 0.1 and Lmap = 0.005 Images in datasets are rendered in Unity environment and downsampled to 128 × 128 resolution for training. Images are rendered in sequence of 7 steps each time in a room with newly selected texture and object. Camera moves with random steps and random rotation between consecutive frames. All possible triplets equally spaced by 1, 2, and 3 frames form training samples.
The model was implemented in PyTorch and trained on NVidia RTX 6000. We will release the code and dataset upon publication of the manuscript.
A.4 METHOD
A.4.1 PREDICTION BY WARPING
We first describe the prediction of part of the next image by warping the current image. Here we consider only rigid objects and the fates of all visible pixels belonging to an object. With depth D(t) ∈ Rw×h = hθ(I(t)) of all pixels in a view inferred by the Depth Perception network based on visual features in the image I(t) , the 3D location of a pixel at any coordinate (i, j) in the image, where |i| ≤ w−12 ,|j| ≤ h−1 2 , can be determined given the focal length d of the camera as m̂ (t) (i,j) = D(t)(i,j)√ i2+j2+d2 · [i, d, j]. Here, we take the coordinate of the center of an image as (0,0). On the other hand, with the inferred x̂(t)k and x̂ (t−1) k , the current and previous locations of the object k that the pixel (i, j) belongs to, from I(t) and I(t−1) respectively, we can estimate the instantaneous velocity of the object v̂(t)k = x̂ (t) k − x̂ (t−1) k . Similarly, with the inferred the current and previous pose probabilities of the object, p(t)φk and p (t−1) φk , we can obtain the likelihood of its angular velocity
p(φ (t) k , φ (t−1) k | ω (t) k = ω) ∝ ∑ γ1,γ2,γ1−γ2∈ {ω−2π,ω,ω+2π} p(φ (t) k = γ1) · p(φ (t−1) k = γ2) (5)
.
By additionally imposing a prior distribution (we use Von Mises distribution) over ω(t)k that favors slow rotation, we can obtain the posterior distribution of the object’s angular velocity p(ω
(t) k | φ (t) k , φ (t−1) k ), and eventually the posterior distribution of the object’s next pose p(φ (t+1) k |
φ (t) k , φ (t−1) k ).
Assuming a pixel (i, j) belongs to object k, using the estimated motion information v̂(t)k and p(ω (t) k | φ (t) k , φ (t−1) k ) of the object, together with the current location and pose of the object and the current 3D location m̂(t)(i,j) of the pixel, we can predict the 3D location m ′(t+1) k,(i,j) of the pixel at the next moment as
m′ (t+1) k,(i,j) = M (t) −ωobs [M (t) ω̂k (m̂ (t) (i,j) − x̂ (t) k ) + x̂ (t) k + v̂ (t) k − v (t) obs] (6)
where M (t)−ωobs and M (t) ω̂k are rotational matrices due to the rotation of the observer and the object, respectively, and v(t)obs is the velocity of the observer (relative to its own reference frame at t). In this way, assuming objects move smoothly most of the time, if the self motion information is known, the 3D location of each visible pixel can be predicted. If a pixel belongs to the background, ωK+1 = 0 and vK+1 = 0 (K + 1 is the background’s index). Given the predicted 3D location, the target coordinate (i′, j′)(t+1)k of the pixel on the image and its new depth D ′ k(i, j)
(t+1) can be calculated. This prediction of pixel movement allows predicting the image I ′(t+1) and depth D′(t+1) by weighting the colors and depth of pixels predicted to land near each pixel at the discrete grid of the next frame, as explained in Sec 2.4.2.
A.4.2 WARPING CONTRIBUTION WEIGHT
As the object attribution of each pixel is not known but is inferred by fobj(I(t)), it is represented for every pixel as a probability of belonging to each object and the background π(t)k , k = 1, 2, · · · ,K+ 1. Therefore, the predicted motion of each pixel should be described as a probability distribution over K + 1 discrete target locations p(x′(t+1)(i,j) ) = ∑K+1 k=1 π (t) kij · δ(x′ (t+1) k,(i,j)), i.e., pixel (i, j) has a probability of π(t)kij to move to location x ′(t+1) k,(i,j) at the next time point, for k = 1, 2, · · · ,K+1. With such probabilistic prediction of pixel movement for all visible pixel (i, j)(t), we can partially predict the colors of the next image at the pixel grids where some original pixels from the current view will land nearby by weighting their contribution:
I ′ (t+1) Warp (p, q) =
{∑ k,i,j wk(i,j,p,q)I
(t)(i,j)∑ k,i,j wk(i,j,p,q) , if ∑ k,i,j wk(i, j, p, q) > 0
0, otherwise (7)
We define the weight of the contribution from any source pixel (i, j) to a target pixel (p, q) as
wk(i, j, p, q) = π (t) kij · e −β·D′(t+1)k (i,j) ·max{1− |i′(t+1)k − p|, 0} ·max{1− |j′ (t+1) k − q|, 0} (8)
The first term incorporates the uncertainty of which object a pixel belongs to. The second term e−β·D ′(t+1) k (i,j) resolves the issue of occlusion when multiple pixels are predicted to move close to the same pixel grid by down-weighting the pixels predicted to land farther from the camera. These last two terms mean that only the source pixels predicted to land within a square of of 2 × 2 pixels centered at any target location (p, q) will contribute to the color I ′(t+1)Warp (p, q). The depth map D′(t+1)Warp can be predicted by the same weighting scheme after replacing I (t)(i, j) with each predicted depth D′(t+1)k (i, j) assuming the pixel belongs to object k.
A.5 DEPENDENCY OF SEGMENTATION PERFORMANCE ON OBJECT SIZE AND DISTANCE ACROSS MODELS
Our model (OPPLE) MONet-128
Slot-attention-128 Genesis *
IoU 0
1
A.6 ILLUSTRATION OF PREDICTION QUALITY
In the figure below, we display the first, second image, one of the masked objects, the predicted third image, and the ground truth of the third image, both for our dataset, and for a richer version of the Traffic dataset used by (Henderson & Lampert, 2020). We provide reference lines and some circles to aid the comparison between images and evaluate the warping quality. Imagination quality can be insprected usually at one side of the predicted images (the camera motion is typically larger in our dataset).


summaries: The paper proposes a VAE model for mobility forecasting. It focuses mainly on the disentanglement of spatial and temporal features by modelizing explicitly two groups of features in the VAE schema, a group of spatial features that are time-independent and a group of temporal features using a sequential prior. In order to make a prediction, the VAE is used to extract features from three groups of sequences describing the trend, the period, and the last few timesteps before the forecasting period.  An ablation study is provided concerning those three groups of sequences. Experiments show competitive results wrt SOTA.

summaries: This paper proposes learning a disentangled representation of spatio-temporal mobility data using a VAE-based architecture, which essentially tried to decompose spatial and temporal features and model them independently. Disentanglement in the learnt representation is encouraged through the introduction of total correction as a regularization term in the VAE loss. The authors empirically demonstrate the potential of the proposed approach in 3 different mobility datasets (BikeNYC, TaxiNYC and TaxiBJ). Overall, the results suggest competitive predictive performance with other state-of-the-art approaches.

summaries: This paper proposes a variation of the Variational Autoencoder (VAE) model to learn disentangled spatial and temporal representations from ST raster data. A separation module is designed in the proposed network. The proposed model is validated on three real-world datasets and achieves better performance compared with the baselines listed in the paper.

summaries: The authors propose a novel neural architecture to structurally learn disentangled spatio-temporal representations in the context of mobility forecasting. This work argues that predictive models can benefit from enforcing the independence of spatial and temporal dynamics and proposes a VAE-inspired architecture for doing so.  By conducting experiments on three mobility datasets, the authors empirically evaluate the prediction performance of the proposed approach against various baselines. Additionally, further experiments attempt to (i) quantify the effectiveness of different temporal features (short-term/daily/weekly correlation), and (ii) formalize and justify a strategy for feature selection.

abstractText: Spatio-temporal (ST) prediction task like mobility forecasting is of great significance to traffic management and public safety. There is an increasing number of works proposed for mobility forecasting problems recently, and they typically focus on better extraction of the features from the spatial and temporal domains. Although prior works show promising results on more accurate predictions, they still suffer in characterising and separating the dynamic and static components, making it difficult to make further improvements. Disentangled representation learning separates the learnt latent representation into independent variables associated with semantic factors (Duan et al., 2019). It offers a better separation of the spatial and temporal features, which could improve the performance of mobility forecasting models. In this work, we propose a VAE-based architecture for learning the disentangled representation from real spatio-temporal data for mobility forecasting. Our deep generative model learns a latent representation that (i) separates the temporal dynamics of the data from the spatially varying component and generates effective reconstructions; (ii) is able to achieve state-of-the-art performance across multiple spatio-temporal datasets. Moreover, we investigate the effectiveness of our method by eliminating the non-informative features from the learnt representations, and the results show that models can benefit from this operation.

authors:
Leonardo Alfonso
Arnold Lobbrecht
Roland Price

title: Optimization of water level monitoring network in polder systems using information theory

venue: Water Resources Research,

authors:
Mohammad Babaeizadeh
Chelsea Finn
Dumitru Erhan
Roy H Campbell
Sergey Levine

title: Stochastic variational video prediction

venue: arXiv preprint arXiv:1710.11252,

authors:
Justin Bayer
Christian Osendorfer

title: Learning stochastic recurrent networks

venue: arXiv preprint arXiv:1411.7610,

authors:
Yoshua Bengio
Aaron Courville
Pascal Vincent

title: Representation learning: A review and new perspectives

venue: IEEE transactions on pattern analysis and machine intelligence,

authors:
Ricky TQ Chen
Xuechen Li
Roger B Grosse
David K Duvenaud

title: Isolating sources of disentanglement in variational autoencoders

venue: In Advances in Neural Information Processing Systems,

authors:
Junyoung Chung
Kyle Kastner
Laurent Dinh
Kratarth Goel
Aaron Courville
Yoshua Bengio

title: A recurrent latent variable model for sequential data

venue: arXiv preprint arXiv:1506.02216,

authors:
Emily Denton
Vighnesh Birodkar

title: Unsupervised learning of disentangled representations from video

venue: arXiv preprint arXiv:1705.10915,

authors:
Emily Denton
Rob Fergus

title: Stochastic video generation with a learned prior

venue: In International Conference on Machine Learning,

authors:
Sunny Duan
Loic Matthey
Andre Saraiva
Nicholas Watters
Christopher P Burgess
Alexander Lerchner
Irina Higgins

title: Unsupervised model selection for variational disentangled representation learning

authors:
Otto Fabius
Joost R Van Amersfoort

title: Variational recurrent auto-encoders

venue: arXiv preprint arXiv:1412.6581,

authors:
Jonas Gehring
Michael Auli
David Grangier
Denis Yarats
Yann N Dauphin

title: Convolutional sequence to sequence learning

authors:
Irina Higgins
Loic Matthey
Arka Pal
Christopher Burgess
Xavier Glorot
Matthew Botvinick
Shakir Mohamed
Alexander Lerchner

title: beta-vae: Learning basic visual concepts with a constrained variational framework

authors:
Jun-Ting Hsieh
Bingbin Liu
De-An Huang
Li Fei-Fei
Juan Carlos Niebles

title: Learning to decompose and disentangle representations for video prediction

venue: arXiv preprint arXiv:1806.04166,

authors:
Wei-Ning Hsu
Yu Zhang
James Glass

title: Unsupervised learning of disentangled and interpretable representations from sequential data

venue: arXiv preprint arXiv:1709.07902,

authors:
Renhe Jiang
Zekun Cai
Zhaonan Wang
Chuang Yang
Zipei Fan
Xuan Song
Kota Tsubouchi
Ryosuke Shibasaki

title: Vluc: An empirical benchmark for video-like urban computing on citywide crowd and traffic prediction

venue: arXiv preprint arXiv:1911.06982,

authors:
Wenwei Jin
Youfang Lin
Zhihao Wu
Huaiyu Wan

title: Spatio-temporal recurrent convolutional networks for citywide short-term crowd flows prediction

venue: In Proceedings of the 2nd International Conference on Compute and Data Analysis,

authors:
Hyunjik Kim
Andriy Mnih

title: Disentangling by factorising

venue: arXiv preprint arXiv:1802.05983,

title: Auto-encoding variational bayes

authors:
Wei Li
Wei Tao
Junyang Qiu
Xin Liu
Xingyu Zhou
Zhisong Pan

title: Densely connected convolutional networks with attention lstm for crowd flows prediction

venue: IEEE Access,

authors:
Yanjun Li
Shujian Yu
Jose C Principe
Xiaolin Li
Dapeng Wu

title: Pri-vae: Principle-of-relevantinformation variational autoencoders

venue: arXiv preprint arXiv:2007.06503,

authors:
Yingzhen Li
Stephan Mandt

title: Disentangled sequential autoencoder

venue: arXiv preprint arXiv:1803.02991,

authors:
Ziqian Lin
Jie Feng
Ziyang Lu
Yong Li
Depeng Jin

title: Deepstn+: Context-aware spatial-temporal neural network for crowd flow prediction in metropolis

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Luis Moreira-Matias
Joao Gama
Michel Ferreira
Joao Mendes-Moreira
Luis Damas

title: Predicting taxi–passenger demand using streaming data

venue: IEEE Transactions on Intelligent Transportation Systems,

authors:
Chujie Tian
Xinning Zhu
Zheng Hu
Jian Ma

title: Deep spatial-temporal networks for crowd flows prediction by dilated convolutions and region-shifting attention mechanism

venue: Applied Intelligence,

authors:
Dmitry Ulyanov
Andrea Vedaldi
Victor Lempitsky

title: Instance normalization: The missing ingredient for fast stylization

venue: arXiv preprint arXiv:1607.08022,

authors:
Leye Wang
Xu Geng
Xiaojuan Ma
Feng Liu
Qiang Yang

title: Cross-city transfer learning for deep spatio-temporal prediction

venue: arXiv preprint arXiv:1802.00386,

authors:
Huaxiu Yao
Fei Wu
Jintao Ke
Xianfeng Tang
Yitian Jia
Siyu Lu
Pinghua Gong
Jieping Ye
Zhenhui Li

title: Deep multi-view spatial-temporal network for taxi demand prediction

authors:
Huaxiu Yao
Xianfeng Tang
Hua Wei
Guanjie Zheng
Zhenhui Li

title: Revisiting spatial-temporal similarity: A deep learning framework for traffic prediction

venue: In Proceedings of the AAAI conference on artificial intelligence,

authors:
Bing Yu
Haoteng Yin
Zhanxing Zhu

title: Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting

venue: arXiv preprint arXiv:1709.04875,

authors:
Junbo Zhang
Yu Zheng
Dekang Qi

title: Deep spatio-temporal residual networks for citywide crowd flows prediction

venue: arXiv preprint arXiv:1610.00081,

authors:
Yu Zheng
Licia Capra
Ouri Wolfson
Hai Yang

title: Urban computing: concepts, methodologies, and applications

venue: ACM Transactions on Intelligent Systems and Technology (TIST),

authors:
Yizhe Zhu
Martin Renqiang Min
Asim Kadav
Hans Peter Graf

title: S3vae: Self-supervised sequential vae for representation disentanglement and data generation

authors:
Ali Zonoozi
Jung-jae Kim
Xiao-Li Li
Gao Cong

title: Periodic-crn: A convolutional recurrent model for crowd density prediction with recurring periodic patterns

venue: In IJCAI,

heading: 1 INTRODUCTION

text:
Spatio-temporal prediction tasks like mobility forecasting are critically important for smart city applications (Zheng et al., 2014). With the help of the rapid deployment of IoT (Internet of Things) devices and sensors, massive amounts of saturated ST datasets are available, and many researchers have put their efforts to improve the performance of ST prediction (Zhang et al., 2016; Wang et al., 2018; Zonoozi et al., 2018; Jin et al., 2018; Yao et al., 2019b). A central problem in deep learning for crowd flow prediction tasks is the extraction of features from both spatial and temporal domains. Researchers first tried to extract the entangled spatio-temporal features directly from the data (Zhang et al., 2016). Then, they tried to extract features from spatial and temporal domains separately. Wang et al. (2018) and Zonoozi et al. (2018) proposed models that extract periodic representations or short-term temporal features directly from the ST data using recurrent-based convolution operations and find their effectiveness in producing more accurate predictions. Based on that, in STDN (Yao et al., 2019b), the explicit extraction of the long-term periodic information was shown to also improve the prediction.
Although the existing approaches appear to be powerful in terms of results and predictions, two major challenges hinder these models: 1) Difficulty in characterising dynamic and spatial components. Spatio-temporal correlation is more complex since it comprises dependencies from both the spatial and temporal regions. Models like LDRSN (Tian et al., 2020) and RegionTrans (Wang et al., 2018) are capturing temporal dependencies explicitly. However, their complicated structure makes it hard to characterise and validate the effectiveness of their dynamic and spatial components. 2) Difficulty in separating the extraction of spatial and temporal features. Except for the spatial and temporal features from the most recent data, many approaches try to extract long-term/periodical features to improve the performance. However, without an explicit separation mechanism, the extracted long-term temporal dependencies introduce irrelevant noises to the predictor (factors do not vary with time and are only relevant to the long-term sequence). The extent to how generative models can model, extract and disentangle the spatial and temporal features in ST-raster data is an open problem.
To address the above challenges, we introduce the disentangled representation learning to the mobility forecasting task. Disentangled representation learning, which separates the learnt representation into independent variables such that each variable relates to one semantic factor of sensory data (Bengio et al., 2013), offers a solution for the problem mentioned above. For a spatio-temporal task like mobility forecasting, an ideal disentangled representation should have the ability to separate time-relevant components from the factors that don’t vary with time, which can help improve the predictor’s performance.
Many prior works have explored disentangled representation learning for spatio-temporal data (Hsu et al., 2017; Li & Mandt, 2018; Denton & Birodkar, 2017; Zhu et al., 2020). Most of them assume that the features can be disentangled into two sets: a set of dynamic features that extract the temporal correlations and a set of static features that describe factors that are constant through the input sequence. However, since most of them focus on the movement of some predefined objects, their assumption is inaccurate when applying to datasets for mobility forecasting. For example, a sudden car accident might show no relationship to the temporal influence from the previous timestep and the static traffic network structure. Therefore, there are still gaps in how to model the spatio-temporal data using the disentangled representation learning method. To address this problem, we assume that each timestep has its own temporal features and features that do not vary with time. By doing so, our model can better formulate the spatio-temporal data, and we can achieve controlled data generation frame by frame.
In this work, we proposed a VAE-based model to learn disentangled spatio-temporal representation. Compared to the conventional methods for the ST prediction problem, our approach extracts the entangled features first and then explicitly separates them into temporal variables and spatial variables using disentangled representation learning method. This will force the model to keep the learnt spatial/temporal features as mutually exclusive as possible. For applying disentangled representation learning method to real ST data, we assume that each frame has its own spatial and temporal variables and separate these two groups with auxiliary regularisation. It helps the model to formulate the complicated spatio-temporal sequence. Our experimental results (see Section 4) show that the learnt representations have a similar level of performance with the current state-of-the-art methods. Our key contributions can be summarised as follows:
1. We propose a novel approach to learn disentangled spatio-temporal representations for mobility forecasting tasks. The learnt representation is separated into two independent groups: spatial and temporal factors.
2. We conducted several experiments on multiple spatio-temporal datasets and used the learnt representation for mobility forecasting. Results show that our methods achieve state-of-theart performance compared to other baseline mobility forecasting methods.
3. We investigate the effectiveness of our methods under the ”Closeness, Period, Trend” scheme and how to further improves the model’s performance by selecting the informative features from the learnt representations.

heading: 2 RELATED WORK

text:
Deep spatial-temporal networks for mobility forecasting: In order to make accurate traffic prediction, many researchers have paid attention to capturing the spatio-temporal dependencies hidden behind the traffic data. Besides the conventional methods like Seasonal ARIMA (Moreira-Matias et al., 2013), deep learning methods are increasingly used in more and more works for mobility forecasting.
ST-ResNet (Zhang et al., 2016) was proposed to capture the spatial dependencies through a stack of residual convolution layers. It also stacks the frame from a near and a distant time period to capture temporal dynamics. The goal of using the residual units is to overcome the gradient vanishing problem, and the results show that capturing distant spatial features can improve the performance of mobility forecasting. DeepSTN+ (Lin et al., 2019) proposed a ResPlus unit that is capable of capturing long-range spatial correlations. Although they succeed in extracting distant spatial features, they lack attention on capturing the temporal dependencies. To further explore the effectiveness of temporal features, recurrent-based approaches were introduced to capture the temporal correlations. PCRN (Zonoozi et al., 2018) was proposed, which first extract entangled spatio-temporal representation
using a convolutional recurrent network (CRN) and then updating the periodic representations by CRN’s hidden state. Attention-based LSTM methods are adopted by STDN (Yao et al., 2019b) and ST-DCCNAL (Li et al., 2019), which try to capture long-term temporal dependencies. In summary, our proposed method differs from other methods in the explicit separation of spatial and temporal features using a disentangled representation mechanism.
Disentangled Representation Learning for Spatio-temporal data: Most prior works on the disentangled representation learning problem are developed based on Variational Autoencoders (VAE) (Kingma & Welling, 2013), which is an unsupervised generative learning method. β-VAE, proposed by Higgins et al. (2016), forces the inference model to disentangle the latent representation by adding a new hyperparameter β to create an information bottleneck on the prior. FactorVAE (Kim & Mnih, 2018) further breaking down the objective function and try to enhance disentanglement by penalising the total correlation of the learnt representation. As for sequence modelling, a number of prior publications have extended VAE to video and speech data (Fabius & Van Amersfoort, 2014; Chung et al., 2015; Bayer & Osendorfer, 2014). These models, although being able to generate realistic sequences, do not explicitly disentangle the representation of time-invariant and time-dependent information. Thus it is inconvenient for these models to perform tasks such as controlled generation. S3VAE (Zhu et al., 2020) is proposed to separate static and dynamic factors of sequential data. Another approach proposed by (Li & Mandt, 2018) is also focusing on separating the dynamic factors from static factors. Although they share a similar idea which uses an RNN-based architecture to extract dynamic factors for each timestep, they use different prior setups. Each frame in (Li & Mandt, 2018) has its own content features while the time-invariant variables are shared by the whole sequence in S3VAE. As for the spatial-temporal type of data like video, SV2P (Babaeizadeh et al., 2017) uses the variational model to extract the time-invariant latent and make predictions for multiple frames. Models like (Denton & Birodkar, 2017; Hsieh et al., 2018) try to factorise each frame into a stationary part and a temporally dynamic component.

heading: 3 SPATIO-TEMPORAL VAE MODEL

heading: 3.1 SPATIO-TEMPORAL DATA

text: In this work, we will useD = {Xi}i=1:N to denote a spatio-temporal raster dataset that comprises N i.i.d. sequences. Each X ≡ x1:T = {x1, x2, ..., xT } in that dataset denotes a sequence of raster data with T frames. Since we focus on mobility forecasting tasks using grid-based spatial representations, which all have the similar dimensionsH×W . Hence each frame xt ∈ RH×W represents the mobility flow of a certain area at a given time interval t.

heading: 3.2 SEPARATING SPATIAL FEATURES FROM TEMPORAL FEATURES

text:
Many prior works have explored disentangled representation learning for spatio-temporal data. Models like FHVAE (Hsu et al., 2017) tried to separate global variables from segment (dynamic) variables for speech data. DSVAE (Li & Mandt, 2018), DRNET (Denton & Birodkar, 2017) and S3VAE (Zhu et al., 2020) aim at the disentangled representation for video, which also factorised latent variables into static and dynamic parts. However, current approaches for learning the disentangled representation on sequence or spatio-temporal data like video assume that there is a fixed content or object shared by all frames in the sequence since they are using video datasets like the Stochastic Moving MNIST (Denton & Fergus, 2018) and Sprite (Li & Mandt, 2018). Sequences in these datasets often comprise images describing the movement of a set of the same numbers or virtual avatars. Under this context, static features describing this object can be extracted, and its performance can be evaluated by the accuracy of a classification task. However, for most of the real spatio-temporal datasets, the static features are not enough. For example, in a traffic flow dataset, in addition to the fixed structure of the traffic network and the temporal influences, there might be some hotspot suddenly emerges. At that specific time, the rise of those events shows no correlation to the temporal features and the fixed content. Therefore, we assume that for each frame, data is generated based on two sets of features: a set of temporal features which comprise the influence from the sequence before it and a set of spatial features that describe the structure of the network and local events. In this work, we propose a novel architecture that extracts mixed (entangled) feature maps for each timestep in the input sequence and then separates them into temporal and spatial variables. It allows us to analyse and controlled generate each frame separately.
The architecture of our method for spatio-temporal disentangled representation learning is illustrated in Fig.1a. It consists of three major modules: (i) Image Encoder to yield initial feature maps from the input spatio-temporal sequence. (ii) A Feature Separation module to separate the spatial and temporal variables from the initially mixed feature maps. (iii) A Decoder to reconstruct the sequence frame by frame based on their corresponding spatial and temporal variables.
The Image Encoder comprises several layers of CNN, and the feature maps are extracted from each frame separately. The goal here is to yield some higher-level features for the next step. An Instance Normalisation layer is used after each convolutional layer to improve the reconstruction results since using batch normalisation might remove the instance-specific contrast information from the data, which is useful in the later step (Ulyanov et al., 2016).
Intuitively, without any disentanglement constraints, the feature maps that come out of the Image Encoder should be the entanglement of features from both spatial and temporal domain. Therefore, the goal for the Feature Separation module is to separate them explicitly. For temporal features, inspired by (Yu et al., 2017; Gehring et al., 2017), we use a Gated Convolutional structure to capture temporal dynamics from the mixed feature maps. This temporal gated convolution layer contains two 1-D causal convolution layers. As Fig.1b shows, those 1-D convolution operations will be applied on the time axis for each pixel of the feature maps, which means the capture of the temporal dynamics from each mixed high-level feature. It is much easier to understand the usage of the Image Encoder here since the 1-D convolutional operation on the high-level features map is definitely better for separating the temporal dynamics than applying them on the individual pixel of the original input.
The Decoder module is used to reconstruct each frame xt by using their corresponding spatial variables zSpt and temporal variables z Te 1:t as input.

heading: 3.3 SPATIO-TEMPORAL VAE MODEL

text:
Priors: In this work, we proposed a spatio-temporal variational autoencoder architecture to learn disentangled representations from ST raster datasets. Our assumption for variational autoencoder is that each input xt is generated from a corresponding latent representation zt which can be separated into two disentangled subgroup: variables zSpt which contains the spatial (time-irrelevant) features and the temporal (time-varying) features zTe1:t .
On the one hand, since the spatial variables zSpt is considered time-irrelevant, therefore its prior is defined as a standard Gaussian distribution zSpt ∼ N (0, 1) and the spatial variables for the whole sequence can be formed as zSp1:T = ∏T t=1 z Sp t . On the other hand, the prior of the temporal dynamic
variables follow a sequential prior zTe1:t = z Te t |zTe<t . The distribution of this sequential prior is defined as zTet |zTe<t ∼ N ( µt, diag ( σ2t )) where [ µt, σ 2 t ] = φTe ( zTe<t ) . The mean µt and variance σ2t are parameters that are conditioned on all of its previous temporal variables zTe<t and can be parameterised by the gated-convolution layer. Thelatent prior zt that combines both the spatial and temporal variables is formed as:
p(zt) = p(z Sp t )p(z Te 1:t ) = p(z Sp t ) T∏ t=1 p(zTet |zTe<t ) (1)
Generative model: For the generative model, we assume that the generation of each frame xt at a given time t depends on the combination of its corresponding spatial variables zSpt and temporal variables zTet . Therefore, the generation process for the single frame xt in the whole sequence x1:T can be formed as:
pθ(xt, zt) = pθ(xt|zt)p(zt) = pθ(xt|zt)p ( zSpt ) T∏ t=1 p ( zTet |zTe<t ) (2)
where θ are the parameters for the decoder.
Inference models: We use a deep structured model as an encoder to approximate the posterior distribution, which can factorise the latent z into disentangled spatial and temporal components. The amortised variational distribution is formed as:
qφ ( zTe1:T , z Sp T |x1:T ) = T∏ t=1 qφ ( zSpt |xt ) T∏ t=1 qφ ( zTet |x<t ) (3)

heading: 3.4 LOSS

text:
In this work, the objective of our proposed method is defined as the combination of the VAE loss and total correlation regularisation. It can be formulated as: L = LV AE + βLTC where β is the hyperparameter for the TC regularisation. Theoretically, applying higher value on β will emphasise the disentanglement of the learnt representation and lead to better separation of the spatial and temporal variables. We estimate the objective based on FactorVAE’s approach (Kim & Mnih, 2018).
VAE Objective Function: The objective function of the our method is derived from the variational lower bound (Evidence Lower Bound, ELBO) of the vanilla VAE (Kingma & Welling, 2013) and is formed as follow:
LV AE(θ, φ;x1:T ) = Eqφ(zSp1:T ,zTe1:T |x1:T )[ T∑ t=1 log pθ(xt|zSpt , zTet )]− T∑ t=1 DKL(qφ(z Sp t |xt)||pθ(z Sp t ))
− T∑ t=1 DKL(qφ(z Te t |x≤t)||pθ(zTet |zTe<t )) (4)
Note that S3VAE (Zhu et al., 2020) already propose a sequential VAE that consider the continuity of dynamic variables, but it only assumes that there are a whole set of statice features share by the whole sequence. In contrast, we model the temporal and spatial features for each time step independently, resulting in detailed information for emergencies in mobility forecasting data.
Total Correlation Regularization: To encourage the overall disentanglement of the learnt representation, we introduced the Total Correlation (TC) among variables as a regularization term. It quantifies the dependency among a set variables (Alfonso et al., 2010). Experimental results from β-TCVAE (Chen et al., 2018) and FactorVAE (Kim & Mnih, 2018) show that, by amplifying the penalty on this term, the dependence between the variables is reduced hence emphasising the disentanglement.
In this work, we estimate the total correlation using the same approach like FactorVAE (Kim & Mnih, 2018), i.e. by introducing a discriminator and using the independence testing trick and the density-ratio trick to approximate the KL term in the above equation.

heading: 4 EXPERIMENTS

heading: 4.1 DATASET AND METRICS

text:
In this work, we focus on learning disentangled representation of spatio-temporal raster data. Therefore, we choose to conduct the experiments on the following three real-world urban flow datasets:
BikeNYC (Lin et al., 2019) is a bike usage data collected from New York City’s Citi Bike bicycle sharing service, which records the trajectory of all shared bikes in the system. This work covers the time period from 2014-04-01 to 2014-09-30.
TaxiNYC (Yao et al., 2019a) is a dataset that contains taxi in-out flow data taxi New York City, created from the NYC-Taxi GPS data, which covers the period from 2015-01-01 to 2015-03-01.
TaxiBJ (Zhang et al., 2016) comprises the taxi in-out flow data that aggregate the taxi GPS position in Beijing from 2013 to the year 2016. Although it span across 4 consecutive years, the data is not continuous (covers the period: 2013-07-01 to 2013-10-30, 2014-05-01 to 2014-06-30, 2015-03-01 to 2015-06-30, and 2015-11-01 to 2016-04-10).
For disentangled representation learning, we use all of the data to train the feature separation module, and for the mobility forecasting tasks, we use the same setup described in (Xue & Salim, 2021), i.e., the first 80% data is used for training the prediction model, and the rest 20% data is used for testing. Besides, Min-Max normalization is adopted to transform the urban flow values into the range [0, 1] for better training purpose. We evaluate the effectiveness of our models with two commonly used metrics: the Mean Absolute Error (MAE) and the Root Mean Square Error (RMSE).
4.2 IMPLEMENTATION DETAILS
To compare the performance of our model in mobility forecasting with the current state-ofthe-art methods, we use the ”Closeness, Period, Trend” scheme to form the input of our prediction model. It is widely used in the urban flow prediction area, where all three of them comprise a sequence of raster data. It designs a set of unique input sequences, namely Closeness, Period, and Trend, which correspond to the recent time intervals, daily periodicity, and weekly trend, respectively (Zhang et al., 2016). Those three sequences are then fed as the input of their models.
As shown in Figure 2, in this work, we use the same setup used by VLUC-Net (Jiang et al., 2019), where the Closeness sequence contains the previous six steps before the prediction target; the Period is the sequence of the previous day, and the Trend comprises data from the pre-
vious week. We first train our model to learn disentangled representation on the closeness, period and trend sequences separately. Then the learnt representations are fused to train a Multi-Layer perceptron (MLP) regressor to predict the next frame of the sequence. It should be noted that the input of the MLP is not the actual features z, but the learnt µ and σ of the distribution. The logic of doing this is because the sampling part will introduce uncertainty which has a huge impact on the training.

heading: 4.3 MOBILITY FORECASTING: COMPARISON AGAINST OTHER METHODS

text:
In our experiments, we compare our method against the following mobility flow prediction methods: Historical Average (HA); Convolutional Neural Network(CNN); Convolutional LSTM (ConvLSTM); ST-ResNet(Zhang et al., 2016); DMVST(Yao et al., 2018); DeepSTN+(Lin et al., 2019); STDN(Yao et al., 2019a); VLUC-Net(Jiang et al., 2019).
The overall evaluation results on effectiveness are summarised in Table 1 for TaxiBJ, TaxiNYC and BikeNYC. The upper half of Table 1 shows the results of baseline methods on those datasets, and the lower part shows the results of our proposed approach. The best result for each column is given in bold, and methods except HA, CNN, and ConvLSTM are all the current state-of-the-art methods in crowd flow prediction. In general, we can find that our method shows the best results on the BikeNYC and TaxiNYC datasets and compatible performance on the TaxiBJ dataset.
Besides the crowd flow prediction with a full feature set, we also trained models with only spatial/temporal features to see which set contributes more to the mobility forecasting. We find that the prediction performance of the models trained on the temporal feature set leads those trained on the spatial features by a relatively large margin. This shows that for the crowd flow prediction task, the extraction of the temporal dependencies is important than the spatial dependencies, which also coincides with the research direction in this area. It should also be noted that the models trained on the temporal feature already show compatible results with the current state-of-the-art methods. The is still room for improvement since the input of our method is the raw data alone with no context information.

heading: 4.4 ABLATION STUDY

text:
To explore the effectiveness of each module in the ”Closeness, Period, Trend” scheme, we perform an ablation study consider different configurations of the input sequences. The detailed configuration and their corresponding results are summarised in Table 2. The left half of the table shows the sequences used for each setup. By discarding extracted features from some sequences in the scheme, we are able to evaluate their effectiveness in regrading the mobility forecasting task. Since the quality of the learnt disentangled representation is sensitive to the choice of hyperparameters (Duan et al., 2019), we trained multiple feature separation modules with different hyperparameters. The mean and standard deviation of each configuration on all three datasets are presented in the right half of Table 2.
We can first find that the performance of models using period or trend sequence alone is poor, and sometimes adding those extra information does not help the mobility forecasting task. On average, the configuration that achieves the best results for all three datasets is not configuration C6. Although the features from the trend sequence improve the results for the BikeNYC dataset by a tiny margin, introducing the middle/long-term data worsens the performance for both the TaxiNYC and TaxiBJ
datasets. One of the reasons might relate to the size of the representation. Although there might be some information that can contribute to the prediction, using all three sequences tripling the dimension of the input and let the noise offset the potential benefit. Besides, we can find out that the variance results are high for all three datasets, which agree with the assumption that the quality of the learnt representation regarding the mobility forecasting task is sensitive to the choice of hyperparameters.

heading: 4.5 INFORMATIVE FEATURES

text:
Given the thought that the dimension of MLP’s input might be too large and the learnt representations from the previous section, we want to investigate whether we can find the ”informative” features from the learnt representations. And for those ”normal” representations (contains valuable information but does not achieve the best results), will the models benefit when we exclude those ”non-informative” features from their input. In our work, we define the ”informative” features using the definition from Unsupervised Disentanglement Ranking (UDR)(Duan et al., 2019): A latent dimension is treated as an ”informative” feature if it learns a latent posterior that diverges from the prior.
IKL(a) = { 1 KL(qφ(za|x)||p(za)) > 0.001 0 otherwise
(5)
where a is the index of the variable. We use a smaller threshold here since the number of features is larger than the UDR (Duan et al., 2019). To avoid the influence from random noise, we test the performance of mobility forecasting on all representations that we get from the previous section, and the results are summarised in Figure 3a.
First, it should be noted that there exist representations that do not contain any informative features. As shown in the figure, those ”poorly-learnt” representations perform a lot worse compared to the ones with informative features. That proves that a ”well-disentangled, well-learnt” representation can contribute to the downstream task, at least for the mobility forecasting task. To prove the necessity of the filtering operation (remove the ”non-informative” features), we first decrease the representation size and train the feature separation module to extract representations. The results show that when the size is below a certain threshold, all representations with smaller sizes will not contain any informative features. And the ones containing informative features will always have some space. That indicates the necessity of the filtering operation since we still haven’t found a solution to learn disentangled representation that only contains informative features.
Another thing that we want to investigate is whether the models can benefit when removing those non-informative features from their input. Therefore, we train those MLPs with only the informative
features and calculate the difference in performance and summarised the results in Figure 3b. It is worth noted that the MLPs have the same structure except the first layer due to the reducing amount of input features.
In summary, for the BikeNYC and TaxiNYC datasets, we can find out that, although the MLP that trained with the best representation does not get performance improvement, the majority of the MLPs that trained with ”normal” representations show better results after removing the ”non-informative” features. As for the TaxiBJ dataset, there is a slight drop in performance but only with a tiny margin. The reason for that might be due to the different sizes and data distribution of TaxiBJ. The overall grid size for TaxiBJ is larger than the other two datasets, and the Cumulative Distribution Function (CDF) shows that it contains more large values in the dataset, which might lead to different behaviour when we are removing the ”non-informative” features. More details can be found in the Appendix.

heading: 5 CONCLUSION

text: Spatio-temporal (ST) prediction tasks like mobility forecasting have attracted significant attention since they greatly influence traffic management and public safety. We introduce the disentangled representation learning method and modify it to fit spatio-temporal data. The experimental evaluation results show that our method can achieve state-of-the-art performance and is able to extract desirable spatial/temporal features. Moreover, we investigate the effectiveness of recent/middle/long-term temporal features and find that sometimes our method can achieve state-of-the-art results without long-term temporal features. Finally, we also demonstrated that a well-learnt representation shows better results in mobility forecasting tasks and removing the non-informative features from the input of downstream models sometimes can also boost performance. Hence, we hope that our method can contribute to the mobility forecasting task by introducing the disentangled representation learning mechanism. One future direction of this work is forcing the model to learn a more compact representation that only contains ”informative” features. A better mechanism to separate the spatial/temporal features and link them to real semantics is also needed to be investigated.

heading: A APPENDIX

text:
A.1 THE CUMULATIVE DISTRIBUTION FUNCTION (CDF) OF THE DATASETS
The Cumulative Distribution Function (CDF) of all three datasets is shown in Figure 4. We can find that the TaxiBJ dataset contains more data with larger values, therefore shows a different data distribution comparing to the other two datasets.
A.2 VAE LOSS DECOMPOSITION AND TOTAL CORRELATION REGULARISER
Variational Autoencoder (VAE) is a generative model that tries to approximate a posterior distribution pθ(z|x) by a neural network component qφ(z|x). The generative part means that in this model, the data x is considered generated by some random process modelled as pθ(z)pθ(x|z). And we optimize θ to θ∗ by maximizing the marginal likelihood:
log pθ(x) = DKL(qφ(z|x)||pθ(z|x)) + LV AE(θ, φ;x) (6)
Due to the intractability of the posterior distribution, it is not likely to calculate the marginal likelihood directly. However, the author provided a variational lower bound (Evidence Lower Bound, ELBO), which can be optimized using stochastic gradient descent:
L(θ, φ;x) = −DKL(qφ(z|x)||pθ(x)) + Eqφ(z|x)[log pθ(x|z)] (7)
The second part in Eq. 7 represents the reconstruction loss, and by minimising this term, it forces the model to generate reliable reconstructions of the input and the synthesis data with better quality. The first term is a regularisation term which can be further tear down for the purpose of better disentanglement on the latent representation.
β-VAE (Higgins et al., 2016) is the first model for introducing the disentanglement. By adding a hyper-parameter β for penalising the Kullback-Liebler divergence term harder, the representation tends to become disentangled, and the reason behind it is quite straight forward if we decompose that term into the following form (Kim & Mnih, 2018):
DKL(qφ(z|x)||pθ(z|x)) = I(x, z) +DKL(qφ(z)||p(z)) (8)
The first term describes the mutual information between the input and its latent representation while the second term pushes the distribution of the latent towards the Gaussian prior thus emerging the
disentanglement. Although the hyperparameter β emphasises, it suppresses the model’s ability to produce a high-quality reconstruction. To address that limitation, the disentanglement term in Eq. 8 is further decomposed into the following form(Chen et al., 2018):
DKL(qφ(z)||p(z)) = DKL q(z)||∏ j=1 q(zj) +∑ d DKL(qφ(zj)||p(zj)) (9)
where z ∈ Rd and the original ELBO is structured as:
L(θ, φ;x) = Eqφ(z|x)[log pθ(x|z)]− I(x, z)
−DKL q(z)||∏ j=1 q(zj) −∑ d DKL(qφ(zj)||p(zj)) (10)
Not like β-VAE, FactorVAE(Kim & Mnih, 2018) and β-TCVAE(Chen et al., 2018) introduce the hyperparameter in front of the first term in Eq. 9 which is referred to as the total correlation (TC) (Chen et al., 2018). By amplifying the penalty on this term, the dependence between the variables is reduced hence emphasising the disentanglement. The second term in Eq. 9 is defined as dimensionwise KL divergence, which puts constraints on the generated latent code z and pushes them towards their predefined Gaussian prior (Li et al., 2020).
The total correlation is intractable since the prior contains mixtures with a large number of components. However, FactorVAE introduces a discriminator to approximate the the density ratio that arises in the TC term. In the training session, the model extract representations of two batch of input and then send them to the discriminator. The second batch of extracted representation will be randomly permuted and the goal for the discriminator is to identify whether its input is from q(z) or not. The discriminator and the VAE are trained jointly for getting accurate estimation.
A.3 THE IMPLEMENTATION DETAIL OF OUR PROPOSED METHOD
A.4 EXAMINING THE LEARNT REPRESENTATIONS
Quantitative evaluation of the learnt representations: We summarise the results in Table 4. As one can see, our approach consistently shows superior performance across all datasets in both the
disentanglement and reconstruction. This verifies the necessity of extracting time-irrelevant features for each time step, which can capture the unconventional changes in the ST raster data very well.
Table 4: Quantitative performance comparison on BikeNYC, TaxiNYC and TaxiBJ datasets.
BikeNYC TaxiNYC TaxiBJ MMD 0.858 0.918 0.354 KLD 0.108 0.131 8.305
Original add -1 add 1 0.0
0.8 z
add -1 add 1 -0.1
0.1 Difference
add -2
add -2 add 2
add 2
Spatial Features Temporal Features
Network Structure
Local Hotspot
Figure 5: Spatiotemporal data generation controlled by fixing the spatial/temporal variable and modifying the other variable set. The top row shows the synthesis image with changed features, while the images on the second row show the difference between the original image and their corresponding synthetic images.
Generating Attributed Spatio-temporal data: We visualise the spatio-temporal data generated by our method to validate what types of information are extracted by the spatial/temporal features. The generated images are shown in Figure 5, and the original input is placed on the top left corner. The remaining of the top row shows the reconstructed spatiotemporal data with one group of features fixed (spatial/temporal) while making changes to the other set of variables. The difference between images on the first row and the original input is calculated and visualised in the second row since those residual images show the impact on the reconstructed data when changing the latent representations.
The temporal features, as shown in the right half of Figure 5, do not focus on the spatial structure of the data. They capture the temporal influences that flow through the sequence. When we are adding negative/positive values to temporal features, the model tends to amplify/weaken the temporal dependencies as we can see that the areas with non-zero values on the previous timestep are enhanced/reduced on this timestep. On the contrary, the spatial features contain more than the static structure of the transportation network. As shown in the left half of Figure 5, when adding/subtracting values to the spatial features, not only the flow of the major roads are amplified/weakened, some hotspot events also emerge across the whole city. The results above demonstrate that verify the necessity of the disentangling design of our approach, which can capture and separate the spatial and temporal features very well.
A.5 REPRESENTATION SWAPPING
Besides the temporal, we also perform the representation swapping to show the ability of our proposed method to generate synthetic spatiotemporal data. Suppose two real sequences are given for spatial information and temporal information, denoted as Xs and Xt. The synthetic spatiotemporal data are expected to preserve the spatial structure in Xs and the temporal dependencies in Xt. The qualitative comparisons are shown in Figure 6