Time is an important feature in many applications involving events that occur synchronously and/or asynchronously. To effectively consume time information, recent studies have focused on designing new architectures. In this paper, we take an orthogonal but complementary approach by providing a model-agnostic vector representation for time, called Time2Vec, that can be easily imported into many existing and future architectures and improve their performances. We show on a range of models and problems that replacing the notion of time with its Time2Vec representation improves the performance of the final model.
This paper introduces a particular learnable vector representation of time which is applicable across problems without the use of a hand-crafted time representation. Their representation makes use of a feed-forward layer with sine activations which operates on time data. As it is a vector representation, it combines well with other deep neural network methods. They motivate their problem well, explaining why time data is important to a variety of problems and situate their solution as an orthogonal approach to many current solutions in the literature. They make reference to fourier analysis as motivation for their representation. Finally, they provide experimental results to support their claims using fabricated and real-world time series datasets, as well as ablation studies to support their design decisions.
While I think this work has the potential to be a significant contribution, I rate this a weak reject because the theoretical motivation and analysis of the experimental results are lacking the depth of evidence I would expect for an ICLR paper. If you provide a deeper discussion of the provable claims about the power of your model via Fourier analysis and provide a table of test accuracy/recall@K with/without your representation for more than one other state of the art algorithm for these datasets, I would be convinced to strong accept.
Specific comments:
* p.3 third paragraph: you repeat yourself in math notation a few times here. Repeated equations usually indicate that there is something new happening, but all of these are just restatements of your theta sin(omega tau + phi) term. I would introduce the notation for t2v(tau) upfront and use that to define a(tau, k)[j] and f_j
* p.3 A clearer explanation of the theory here would help, as I think Fourier's theorem nicely supports your claims.
* p.4 first paragraph you claim that this method responds well to data which exhibits seasonality, but none of your datasets deal with data that would exhibit seasonality. There are plenty of simple real-world datasets available which show multi-scale periodic phenomena (activity or location data, weather data, travel data, etc.). In fact, segmentation and recognition of wearable device activity would be a great application for this method.
* p.4 third paragraph: Your claim of invariance to time rescaling is technically correct, but I am not convinced that a model can learn the correct omega values for an arbitrary rescaling (e.g. if the period is smaller than the time unit). You show that this works for a rescaling from 2pi/7 to 2pi/14, but it would be nice if there was experimental confirmation of this property with frequency > 1.
* p.6 Showing accuracy/recall across training epochs is not sufficient evidence to show that this is a useful representation. There should be some kind of comparison with test set results from other state-of-the-art work on these datasets. If adding your representation to the SOTA model improved test set performance (or at least sped up training without hurting test set performance), then that would be better evidence. If LSTM+T is the SOTA, say so and restate the author's test performance compared to yours. If this is what these graphs show, consider using a different visualization to make it clearer that you're improving the final performance, not just the training process.
* p.8 I think sine functions make optimization harder because they make the gradient function periodic with respect to the weights, creating infinitely many local extrema. Historically this may have been an issue, but deep neural networks have so many local minima it might not matter. Still, it would be good to show that trained performance doesn't depend on the initialization values more than a standard LSTM+T model.
* You have an interesting corner case where your neural network parameters are interpretable: you can interpret the omega values from your model as frequencies and investigate their values to see which kinds of periodicity your model uses. You do something like this on p.7, but it would be neat to see a histogram like the one you have for EventMNIST for one of the real-world datasets to see if it learns the domain-relevant time knowledge you claim that it should learn.