summaries: This paper has developed a framework for contrastive learning through time (CLTT). Instead of using augmentation operations to create positive pairs for contrastive learning, this paper creates several datasets with a near-photorealistic training environment by changing different view directions. The temporally close video frames of the new data are thus similar and should align in representation, which can naturally be considered as positive data.    The authors propose to perform self-supervised learning (SSL) by enforcing temporal consistency (as opposed to augmentation consistency in most recent efforts). They do this by adapting existing SSL approaches (SimCLR, RELIC and BYOL) to leverage video frames that are close in time as opposed to different augmentations of a given image (akin to a temporal augmentation). In addition, the authors introduce two new datasets which they use to train and study the learnt representations in detail, showing promising results. In this paper, author introduce an image-based contrastive learning framework which substitutes traditional data argumentation by using successive images along with the temporal dimension. A new dataset is also generated in this paper for better studying the topic. Extensive experiments demonstrate that the proposed method is able to learn as good feature representation as supervised learning  in some datasets. The paper proposes a new method to select positive pairs for contrastive learning: using adjacent frames in a sequence, referred to as Contrastive Learning Through Time (CLTT). This is relevant since CL approaches have been shown to depend a lot on what kind of augmentation is used. The method is integrated with a number of well-known CL approaches, and evaluated on three different datasets, two of which are novel --the method is shown to approach fully supervised results. The paper also provides a valuable discussion on how the notion of time can help representation learning. The method is also tested for latent representation similarity when one object or pattern is systematically shown after another, something which has been studied in the field of biological learning, and a similar trend is found here as well. 
abstractText: Human infants learn to recognize objects largely without supervision. In machine learning, contrastive learning has emerged as a powerful form of unsupervised representation learning. The utility of learned representations for downstream tasks depends strongly on the chosen augmentation operations. Taking inspiration from biology, we here study a framework for unsupervised learning of object representations we call Contrastive Learning Through Time (CLTT). CLTT simulates viewing sequences as they might be experienced by an infant while interacting with objects and avoids arbitrary augmentation operations. Instead, positive pairs are formed by successive views in such unsegmented viewing sequences. Generating viewing sequences procedurally, rather than using natural videos, gives us perfect control over the temporal structure of the input and allows us to ask the following two questions. First, can CLTT approach the performance of fully supervised learning? Second, if so, what are the required conditions on the temporal structure of the input? To answer these questions, we develop a new data set using a near-photorealistic training environment based on ThreeDWorld (TDW). We consider several state-of-the-art contrastive learning methods and demonstrate that CLTT allows linear classification performance that approaches that of the fully supervised setting if subsequent views are sufficiently likely to stem from the same object. We also consider the effect of one object being seen systematically before or after another object. We show that this leads to increased representational similarity between these objects, reminiscent of classic neurobiological findings. The data sets, code and pre-trained models for this paper can be downloaded at: (link will be added in the final version).
references.authors: Sven Bambach David Crandall Linda Smith Chen Yu
references.title: Toddler-inspired visual object learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A simple framework for contrastive learning of visual representations
references.venue: In International conference on machine learning,
references.authors: Xinlei Chen Haoqi Fan Ross Girshick Kaiming He
references.title: Improved baselines with momentum contrastive learning
references.venue: arXiv preprint arXiv:2003.04297,
references.authors: Jia Deng Wei Dong Richard Socher Li-Jia Li Kai Li Li Fei-Fei
references.title: Imagenet: A large-scale hierarchical image database
references.venue: IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Christoph Feichtenhofer Haoqi Fan Bo Xiong Ross Girshick Kaiming He
references.title: A large-scale study on unsupervised spatiotemporal representation learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Peter Földiák
references.title: Learning invariance from transformation sequences
references.venue: Neural computation,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Joshua Knights Ben Harwood Daniel Ward Anthony Vanderkop Olivia Mackenzie-Ross Peyman Moghadam
references.title: Temporally coherent embeddings for self-supervised video representation learning
references.venue: In 2020 25th International Conference on Pattern Recognition (ICPR),
references.authors: Alexander LaTourrette Sandra R Waxman
references.title: A little labeling goes a long way: Semi-supervised learning in infancy
references.venue: Developmental science,
references.authors: David Lipshutz Charles Windolf Siavash Golkar Dmitri Chklovskii
references.title: A biologically plausible neural network for slow feature analysis
references.venue: Advances in Neural Information Processing Systems,
references.authors: Ilya Loshchilov Frank Hutter
references.title: Decoupled weight decay regularization
references.venue: In International Conference on Learning Representations,
references.authors: Jovana Mitrovic Brian McWilliams Jacob C Walker Lars Holger Buesing Charles Blundell
references.title: Representation learning via invariant causal mechanisms
references.venue: In International Conference on Learning Representations,
references.authors: Yasushi Miyashita
references.title: Neuronal correlate of visual associative long-term memory in the primate temporal cortex
references.venue: Nature, 335(6193):817–820,
references.authors: Yasushi Miyashita Sei-Ichi Higuchi Kuniyoshi Sakai Naohiko Masui
references.title: Generation of fractal patterns for probing the visual memory
references.venue: Neuroscience Research,
references.authors: Hossein Mobahi Ronan Collobert Jason Weston
references.title: Deep learning from temporal coherence in video
references.venue: In Proceedings of the 26th Annual International Conference on Machine Learning,
references.authors: Sameer A Nene Shree K Nayar Hiroshi Murase
references.title: Columbia object image library (coil-100)
references.venue: Technical report,
references.authors: Emin Orhan Vaibhav Gupta Brenden M Lake
references.title: Self-supervised learning through the eyes of a child
references.venue: Advances in Neural Information Processing Systems,
references.authors: Tian Pan Yibing Song Tianyu Yang Wenhao Jiang Wei Liu
references.title: Videomoco: Contrastive video representation learning with temporally adversarial examples
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Edmund T Rolls T Milward
references.title: A model of invariant object recognition in the visual system: learning rules, activation functions, lateral inhibition, and information-based performance measures
references.venue: Neural computation,
references.authors: Stefan Stojanov Samarth Mishra Ngoc Anh Thai Nikhil Dhanda Ahmad Humayun Chen Yu Linda B Smith James M Rehg
references.title: Incremental object learning from contiguous views
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Jessica Sullivan Michelle Mei Andrew Perfors Erica Wojcik Michael C Frank
references.title: Saycam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective
references.venue: Open Mind, pp
references.authors: Xiaolong Wang Abhinav Gupta
references.title: Unsupervised learning of visual representations using videos
references.venue: In Proceedings of the IEEE International Conference on Computer Vision (ICCV),
references.authors: Yingfan Wang Haiyang Huang Cynthia Rudin Yaron Shaposhnik
references.title: Understanding how dimension reduction tools work: An empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization
references.venue: Journal of Machine Learning Research,
references.authors: Laurenz Wiskott Terrence J Sejnowski
references.title: Slow feature analysis: Unsupervised learning of invariances
references.venue: Neural computation,
sections.heading: 1 INTRODUCTION
sections.text: A hallmark of biological organisms is their ability to learn to understand the world around them in a largely autonomous fashion. Consider learning about visual objects. A human infant is not exposed to object views sampled i.i.d. from some fixed distribution and conveniently labeled image by image, but forms representations of objects and categories during extended interactions with individual objects (Bambach et al., 2018) and requires hardly any (verbal) labels for this (LaTourrette & Waxman, 2019). Mimicking such learning abilities in artificial systems would represent a giant leap forward for artificial intelligence.
  Self-supervised learning has emerged as a promising alternative to fully supervised approaches. In the domain of visual object recognition, recent contrastive learning approaches have obtained strong results on standard object recognition benchmarks (Chen et al., 2020a;b; Mitrovic et al., 2021; Grill et al., 2020). These approaches rely on a range of so-called augmentation operations. The basic idea is that an image is transformed through a number of operations (e.g., scaling, flipping, cropping, rotating, blurring, color distortions, pixel noise, ...) that change its appearance but not its meaning (e.g., “cat”). The key mechanism of contrastive learning is to form a representation where such augmented versions of an image are mapped on to close-by latent representations, while at the same time avoiding a “collapse” of the representation, i.e., making sure the network does not simply map all inputs to the same point in the latent space. Not surprisingly, the quality of the learned representations for downstream tasks strongly depends on the chosen augmentation operations (Grill et al., 2020).
  Theories of biological learning have also addressed the requirement to learn object representations without (or with only few) labels. The classic theory of how biological organisms learn invariant
  representations uses the notion of time to substitute for explicit labeling (Földiák, 1991; Rolls & Milward, 2000; Wiskott & Sejnowski, 2002). Biological organisms (including the human infant mentioned above) experience objects across time, typically seeing a sequence of different views of the same object before directing their attention elsewhere (Fig. 1A). Thus, by learning a representation such that subsequent views are mapped onto close-by latent codes (Fig. 1B), a representation should emerge that maps different views of the same object onto similar latent codes, thereby establishing (partial) invariance. While this idea has a long history in biological theorizing, it has only recently been explored in a contrastive learning context. The basic idea is to replace the augmentation operations in contrastive learning with natural appearance variation occurring during object interactions. To systematically study this approach, we propose a new Contrastive Learning Through Time (CLTT) framework that permits perfect control over the generated viewing sequences. For our experiments, we utilize the ThreeDWorld (TDW) virtual environment (Gan et al., 2021), which allows near-photorealistic rendering. We also simulate classic biological experiments by Miyashita (1988), demonstrating that objects form similar latent representations in the brain when they are systematically seen one after the other, even if they are visually dissimilar (Miyashita, 1988). We summarize our contributions as follows:
  • We develop the CLTT framework using state-of-the-art contrastive learning methods.
  • We introduce novel data sets to study CLTT under controlled conditions.
  • We systematically analyze the conditions for CLTT to be successful and demonstrate that it approaches fully supervised learning.
  • We show that CLTT maps objects that are systematically seen in temporal succession on similar latent representations, reminiscent of classic neurobiological findings.
sections.heading: 2 RELATED WORK
sections.text: An early demonstration that the temporal structure of visual inputs shapes object representations in primate visual cortex was given by Miyashita (1988). He showed 97 images of fractal-like objects to monkeys in always the same order. As the monkeys learned to represent these images in their visual cortices, Miyashita’s observations suggested that the representations of objects which were neighbors in the sequence became aligned — even if these objects were visually dissimilar. This
  effect extended over a few objects, i.e., the representations of objects six steps apart in the sequence were still more similar than the average similarity.
  In part motivated by such findings, there is a long history of neural network and machine learning models exploiting temporal structure for unsupervised representation learning. Földiák (1991) introduced so-called trace learning rules to explain how neurons in the mammalian visual system learn invariance properties, setting a starting point for later models considering multi-layered network architectures (Rolls & Milward, 2000). Another line of research introduced by Wiskott & Sejnowski (2002) explicitly considers the objective of extracting components from an input stream that are slowly changing. A recent variant attempts to do so in a biologically plausible fashion (Lipshutz et al., 2020).
  The use of temporal learning objectives in contrastive learning has received increasing attention recently. Among the first, Mobahi et al. (2009) have proposed a method for learning object representations that combines supervised learning and unsupervised learning based on temporal coherence using a siamese neural network architecture. Subsequently, Wang & Gupta (2015) used tracking of patches in videos for unsupervised pre-training, by learning an embedding that keeps patches from the same track close in the embedding space.
  An approach more closely related to ours has recently been proposed by Orhan et al. (2020). They consider learning on a longitudinal headcam video set from three developing children (Sullivan et al., 2020). They focus on a temporal classification approach, where they divide the videos into a finite number of contiguous segments of the same length that they call temporal classes. The learning objective is to predict from which of the classes a particular video frame originates. They also consider a temporal contrastive learning objective with the MoCo contrastive learning implementation of Chen et al. (2020b). This objective also aims to make the latent representations of adjacent video frames similar. However, the use of uncontrolled headcam video does not permit determining the required the temporal statistics of the visual input for the approach to work. Another related approach is that of Knights et al. (2021), who learn embeddings of video clips. Their learning objective makes latent codes of adjacent frames within a video clip similar, while making them distinct from latent codes of frames from other video clips. Note that this setup requires the video clips to be segmented, i.e., the system has access to the information where each video starts and ends rather than being exposed to an unlabeled continuous video stream as in Orhan et al. (2020) and CLTT. The same holds true for the recent approach of Feichtenhofer et al. (2021) and also Pan et al. (2021). Yet, they don’t make the connection between continuous time frames and biologically-inspired vision. Finally, the interesting work of Stojanov et al. (2019) has modeled continual infant-like learning of object representations from unsegmented input streams, but they did not consider a contrastive learning approach.
sections.heading: 3 METHODS
sections.heading: 3.1 SAMPLING SEQUENCES OF OBJECT VIEWS IN CLTT
sections.text: CLTT aims to mimic the stream of object views that an infant may experience during natural interactions with objects, while giving precise control over the statistical properties of this view sequence. Specifically, an object is always viewed for Nfix fixations before another object comes into view. Importantly, the learner does not have access to the information when a new object comes into view, making the approach fully unsupervised. Generating these view sequences involves two sampling procedures. The first describes how the next view of the same object is sampled during the Nfix fixations on the same object. We refer to this as view sampling. The second describes how the identity of the next object is determined at the end of the Nfix fixations on the same object, which we refer to as object sampling.
  For the view sampling, we distinguish two sampling methods. Our default method is the random walk view sampling. Here, the next view of an object is a “neighbor” of the previous view. For example, in the TDW data set (see below) this corresponds to changing azimuth or elevation by 10◦ or viewing distance by 10 cm. This procedure mimics the infant gradually turning an object or moving around an object while fixating it. The second method is the uniform view sampling. Here, the next view is picked uniformly at random. While this does not mimic infants’ viewing sequences, it provides more diversity among successive views, which is expected to aid learning.
  For the object sampling we also distinguish two methods. Our default method is the random order method. Here, the order of the objects is a new random permutation during each training cycle. This corresponds to the case that the infant encounters objects randomly. The second method uses a fixed order of objects, i.e., object A is always followed by object B, etc. in all training cycles. This situation matches the neurobiological experiments by Miyashita (1988) and is expected to lead to an alignment of the latent codes of objects that are consistently seen in succession.
  Training occurs in cycles. In every cycle each of the Nobj objects will be chosen exactly once — the order depending on the chosen object sampling method. Thus, a cycle consists of a total of Nobj × Nfix object views. The Nfix views for each object are determined by the chosen view sampling procedure. Several cycles of Nobj×Nfix views are stored in a buffer. For our experiments we use a batch size of 256. During learning, batches of pairs of subsequent views are sampled uniformly from the buffer. This leads to a probability 1/Nfix of sampling inter-class transitions and a probability of (Nfix−1)/Nfix of sampling intra-class transitions (compare Fig. 1). We consider an epoch to be a full run through the buffer, which is chosen to be of equal size as the underlying data set. That means that on average, every image of the data set is presented once during each epoch.
sections.heading: 3.1.1 CONTRASTIVE LEARNING ALGORITHMS FOR CLTT
sections.text: The sampled sequences can be fed into a wide range of contrastive learning methods. Here we consider SimCLR (Chen et al., 2020a) and RELIC (Mitrovic et al., 2021) in which for every positive pair, all the other pairs in the batch are considered negative pairs. We also experiment with BYOL (Grill et al., 2020), that uses only positive pairs. We refer to the CLTT versions of these algorithms as SimCLR-TT, RELIC-TT, and BYOL-TT, respectively. In all cases, we use a ResNet-18 architecture (He et al., 2016) to transform the input images into 128-dimensional latent representations, which is followed by a single layer as the projection head for RELIC-TT and SimCLR-TT. For BYOL-TT we use a two layer projection head with batch normalization after the hidden layer.
  SimCLR-TT. SimCLR (Chen et al., 2020a) is an effective contrastive learning approach for visual representation learning. By defining a wide range of augmentation operations and treating differently augmented images as positive samples, SimCLR has achieved state-of-the-art performance. Its groundbreaking success has triggered a surge of interest in augmentation-based self-supervised learning. For SimCLR-TT we replace the traditional augmentations with successive views as pairs defined in the previous subsection. For each positive pair, all remaining pairs in the batch are considered negative samples. The loss of SimCLR-TT is then defined as follows:
  L (zi) = − log exp (sim (zi, z ′ i) /τ)∑NB
  k=1,k ̸=i [exp (sim (zi, zk)) + exp (sim (zi, z ′ k))] /τ
  , (1)
  where zi and z′i are the latent codes of a sampled view pair, and NB is the number of pairs in a batch. Specifically, we use the cosine similarity as the similarity function sim (u, v) = u
  ⊤v ∥u∥∥v∥ .
  For simplicity, we set the temperature parameter τ in the original SimCLR loss to be 1. In all our experiments, we use the AdamW optimizer (Loshchilov & Hutter, 2019) with a starting learning rate of 10−3, which decays by a factor of 0.3 after every 10 epochs. The batch size for training is 256 and the buffer size is determined by the size of the data set.
  RELIC-TT. RELIC (Mitrovic et al., 2021) is an approach that uses an additional penalty loss compared to SimCLR. It obtains state-of-the-art results by keeping the similarity distribution of one sample invariant against differently augmented views of other samples. Incorporating this notion into the CLTT approach, we derive RELIC-TT by adding another loss term that. The loss can be described as follows:
  Lp (zi) = KL (p (Y | zi) , p (Y | z′i)) , (2)
  where KL is the Kullback-Leibler divergence and
  p (Y = j | zi) = exp
  ( sim ( zi, z ′ j ))∑NB k=1 exp (sim (zi, z ′ k)) . (3)
  In the definition above, zi and z′i are the latent codes of the first view and the second view of the sampled view pair. The similarities between zi and all the second views in every pair are calculated and passed to a Softmax function as shown in equation 3. The same computation is done for the second view z′i for all the other first views, resulting in totally two probability distributions denoted as p (Y | zi) and p (Y | z′i) that resembles a classification task of predicting the label Y out of totally NB classes. And then those two distributions are pulled together by minimizing the KL divergence between them. The training configuration is the same as used in SimCLR-TT.
  BYOL-TT. BYOL-TT builds on the Bootstrap Your Own Latent (BYOL) architecture by (Grill et al., 2020). BYOL has been shown to outperform other contrastive learning architectures like SimCLR (Chen et al., 2020a) or MoCov2 (Chen et al., 2020b) on the ImageNet data set (Deng et al., 2009). A key advantage of the BYOL architecture is that it works without negative pairs, which sets it apart from other contrastive learning algorithms. Instead, it uses a second so-called target network. The target network receives an augmented version of the input data like the online network. It will produce a target projection and the online network tries to predict this target projection. The loss function minimizes the similarity between the target projection and the prediction of the online network. In our BYOL-TT implementation, we use the AdamW optimizer with a learning rate of 2× 10−4 which decays by a factor of 0.3 after every 30 epochs and a batch size of 256. The loss is given by:
  Lθ,ξ = 2− 2 · ⟨qθ(zθ), z′ξ⟩
  ||qθ(zθ)||2 · ||z′ξ||2 , (4)
  where qθ(zθ) is the prediction of the online network of z′ξ. θ corresponds to the weights of the online network and ξ represent the weights of the target network. The optimization will be performed with respect to θ, the weights ξ of the target network will be updated using an exponential moving average of the online network. Here we choose τ = 0.99 as target decay rate. In the original BYOL architecture, an input image x produces two augmented views v and v′ which are then shown to the target and online network. In our case no augmentations are applied to the input image, positive pairs are formed by images next to each other in the presented data timeline. Two consecutive images in this timeline will then be used as views v and v′.
sections.heading: 3.2 DATA SETS
sections.text: ThreeDWorld data set. This data set was created with the ThreeDWorld (TDW) software (Gan et al., 2021). TDW was chosen, because it enables near-photorealistic rendering including depthof-field effects and the possibility of simulating physical interaction with objects (which we plan to utilize in the future). Our data set comprises twelve high quality 3D models of common household objects such as a hair brush, a hammer or scissors (compare Fig. 2A). They are arranged in a 4× 3 grid on a white floor. To simulate viewing sequences that an infant may experience while interacting with an object such as turning an object in hand or moving relative to an object while fixating it, we
  render object views from different directions and distances. The individual images of the data set can then be arranged into videos simulating different viewing sequences, where an object is seen from different directions and distances. Specifically, to create the different views, we define a spherical coordinate system around the center of each object. Different views are created by changing viewing direction in terms of azimuth (from 0◦ to 350◦ in 10◦ steps), elevation (from 10◦ to 70◦ in 10◦ steps), and distance (0.3 cm to 0.6 cm in 10 cm steps). This gives rise to 1,008 views per object and a total of 12,096 images, which are down-sampled to the size of 64 by 64 pixels. We also created a test set with distinct viewing directions by shifting the azimuth and elevation by 5◦ (from 15◦ to 75◦ and from 5◦ to 355◦ in 10◦ steps). This procedure resulted in a second data set with 12,096 images. Note that due to the varying distances from the objects, sometimes not all parts of an object are visible inside the image and sometimes (parts of) other objects may be present. These appear blurry due to the simulated depth-of-field effect. By sampling different paths through this object, azimuth, elevation and distance space, we generate different viewing sequences that form the input to the CLTT algorithms presented above.
  Miyashita-style data set. The Miyashita-style data set draws inspiration from the neuroscience experiments from Miyashita (1988). We adhered to the generation procedure described in (Miyashita et al., 1991) and wrote python code to generate 100 different fractals of size 64 × 64. Each of those fractals is unique and highly distinguishable from all others. For our experiments, we give each fractal its own class label and allow for a certain degree of variability to simulate fixation inaccuracies during viewing of the images. Each fixation applies a transformation with a random rotation between −10 and +10 degrees, a random rescaling between 90 and 100 % and a random translation of up to 15 % in x- and y-direction (compare Fig. 2B). The fractals are used exclusively with the fixed order procedure, i.e., there is a predefined order of fractals that is repeatedly shown to the network.
  COIL-100 data set. To validate the CLTT approach on real images, we employ the Columbia Object Image Library (COIL-100) database (Nene et al., 1996). It is composed of color images of 100 objects, each viewed from 72 different directions by placing the objects on a motorized turntable making a total of 7,200 images. The objects are seen against a homogeneous black background (compare Fig. 2C).
sections.heading: 4 RESULTS
sections.heading: 4.1 CLTT APPROACHES REPRESENTATION QUALITY OF SUPERVISED LEARNING
sections.text: Results with ThreeDWorld data set. We evaluate the proposed family of CLTT methods, namely SimCLR-TT, RELIC-TT, BYOL-TT on the novel ThreeDWorld data set (Fig. 3). We also add a supervised method as baseline, which has access to the true label of each image. We focus on the random walk sampling of views and the random sequence procedure for sampling objects. We train the network for 100 epochs and vary Nfix to study its effect. In order to evaluate the quality of learned representations, we use a Linear Least Squares (LLS) classifier to test linear separability. In general, a good representation, like that resulting from supervised learning, should be linearly separable and have high LLS classification accuracy. In Fig. 3A we show the LLS classification accuracy for Nfix = 5 as a function of training epoch. Both SimCLR-TT and RELIC-TT achieve the same or even slightly better performance than the supervised approach. Figure 3B shows the final LLS classification accuracy after 100 training epochs for different values of Nfix. Noticeably, SimCLR-TT and RELIC-TT perform on par with the supervised approach with Nfix ≥ 5, while BYOL-TT achieves the best score at Nfix = 5 and larger Nfix seems to have no positive effect on BYOL-TT. Figure 3C depicts a PacMAP (Wang et al., 2021) visualization of the latent space resulting from training with RELIC-TT for 100 epochs using Nfix = 30. The objects form well separated clusters.
  Results with COIL-100 data set. To evaluate CLTT on real (rather than computer rendered) images and a data set with a larger number of objects, we use the classic COIL-100 data set. Note that the COIL-100 data set does not have separate validation and test sets, thus we evaluated performance on the training set in the following experiments. This approach may have led to some over-fitting in the supervised setting, but this effect would only make it harder for our unsupervised method to
  match the performance of the supervised counterpart. We compare performance of SimCLR-TT, RELIC-TT, and supervised methods for different values of Nfix. We also compare the two sampling strategies: random walk sampling where successive views are generated via a random walk in the space of viewing directions and uniform sampling where successive views are picked uniformly at random across all possible viewing directions. Figure 4A shows the LLS accuracy of the different algorithms and sampling strategies as a function of the number of training epochs for Nfix = 30. The SimCLR-TT method outperforms RELIC-TT. Furthermore, we observe that the uniform sampling leads to better results than the random walk sampling. This is not surprising, as it creates more diversity in the training data by allowing for very different views of an object to be grouped as a positive pair during contrastive learning. Also, although learning is somewhat slower, the performance of SimCLR-TT and RELIC-TT with uniform sampling approaches that of fully supervised learning. Figure 4B compares the final LLS classification accuracy of the algorithms after 50 training epochs for different values of Nfix. The performance improves monotonically with Nfix as expected and approaches that of the fully supervised setting. Figure 4C visualizes the clustering of latent representations with PacMAP for RELIC-TT with Nfix = 30 and the random walk sampling procedure. The 100 different objects have formed distinct clusters in the latent space. Finally, in Appendix A.1, we compare uniform sampling and random walk sampling for the three methods with Nfix = 2 and Nfix = 10. As expected, the results show that uniform sampling achieves better performance.
sections.heading: 4.2 CLTT ALIGNS LATENT CODES OF SUCCESSIVELY VIEWED OBJECTS
sections.text: Results with Miyashita-style data set. To relate our framework to biological findings we use the Miyashita-style data set combined with our SimCLR-TT approach. We train the network for 106 stimuli presentations (100 epochs, buffer size = 10,000) and vary Nfix. Here, the buffer-size is larger than the number of fractals, because the data set is dynamically generated using the aforementioned fixations. In line with results from neuroscience, fractals that were presented in succession evoke more similar activations in the latent space than fractals that are far apart in the predefined sequence. Fig. 5A depicts the mean cosine similarity between a fractal’s latent representation and that of its two n-th nearest neighbors along the sequence (in the positive and negative time direction). Note that neighbor zero has a similarity of one, as it is de-facto the same activation pattern. The curves for all values of Nfix display the property of a downward slope that is clearly separated from the baseline (dashed lines). This indicates that over time not only immediate neighbors, but also more distant fractals become associated. In fact for Nfix = 2 we see significant deviations (p < .01, twosided Kolmogorov-Smirnov two-sample test) for the nine next neighbors before merging with the baseline. The number of deviating neighbors shrinks with increasing Nfix, but significant effects still can be observed. This experiment illustrates a fundamental property of CLTT that can be directly
  related to biological findings. We also compare our results to supervised learning, which can solve the classification of 100 fractals, but the resulting latent representations do not display any structure of temporal associations and thus do not differ significantly from its baseline (not shown).
  Results with ThreeDWorld data set. We tested if the alignment of latent representations of objects that are consistently shown in succession could also be observed with the TDW data set. For this we sampled objects using the fixed order procedure, i.e., the objects were always seen in the same sequence order. We focused on SimCLR-TT for this experiment. Figure 5B replicates the effect from the Fractals data set (compare Fig. 5A). Figure 5C shows the equivalent experiment for the COIL-100 data set. The fixed order aids classification and SimCLR-TT achieves excellent classification performance for different values of Nfix. Thus, the network is able to learn good representations of the input data even with low values of Nfix when the order in which the objects are shown to the network is fixed.
sections.heading: 5 DISCUSSION
sections.text: We have developed a general framework for contrastive learning through time (CLTT). CLTT emulates viewing sequences as may be experienced by infants and uses a temporal contrastive loss that maps subsequent inputs occurring during such interactions onto close-by latent representations (“close in time, will align”). To systematically investigate this approach, we have created a new data set using the ThreeDWorld environment (Gan et al., 2021), allowing us to flexibly simulate different kinds of viewing scenarios in a near photo-realistic fashion. We also validated our approach using a new fractal-like data set inspired by biological experiments and the classic COIL-100 computer vision data set. We have demonstrated that CLTT can approach the quality of representations learned using full supervision. For this it is important that intra-object transitions (successive fixations fall on the same object) dominate over inter-object transitions (successive fixations fall on different objects). We have also shown that CLTT produces effects reminiscent of classic biological findings showing that inputs that occur close in time are mapped onto close-by latent representations by the brain.
  Our work has a number of limitations. First, while the use of TDW gives us perfect control over all parameters of viewing sequences including scene geometry, sequence of views, lighting conditions, etc., the approach needs to be validated in the real world. Using the COIL-100 data set has been a first step in this direction. To address this more thoroughly, future work could consider first person videos from infants wearing head-mounted cameras (Bambach et al., 2018; Orhan et al., 2020). Second, we have assumed well-separated objects without major occlusions in front of uniform backgrounds. Learning in a cluttered environment is expected to be harder, but it may benefit from foveated vision and additional attentional and figure-ground separation mechanisms, which we plan to incorporate in future work. Third, our eye movement model, which kept gaze on the same object for a certain number of fixations before redirecting it elsewhere, is overly simplistic. In the future, it will be interesting to use more refined models that better reflect (measured) gaze sequences of children and adults. Indeed, it is an interesting question what gaze sequences are particularly beneficial for learning and if and how infants and artificial vision systems can optimize viewing sequences to maximize their own learning progress. This links CLTT to research on intrinsic motivation and (artificial) curiosity. Exploring these issues will bring us closer to building artificial vision systems that can learn truly autonomously.
sections.heading: ACKNOWLEDGMENTS
sections.text: Removed for double blind review.
sections.heading: A APPENDIX
sections.text: A.1 COIL-100: ADDITIONAL EXPERIMENTAL RESULTS
  In this section, we supply additional results on the COIL-100 data set. Figure A.1. compares the two view sampling methods for the different algorithms for Nfix = 2 and Nfix = 10. It shows improved LSS accuracy for the uniform sampling strategy. This is not suprising, since it creates more diverse views than the random walk sampling strategy. It also shows that Nfix has a larger influence on the random walk sampling.
title: CONTRASTIVE LEARNING THROUGH TIME