Inverted Transformers Are Effective for Time Series Forecasting
Gah, this paper is hard to read, but here's my understanding:<p>Let's say you have 100 intersections, and you want to predict the traffic on each in cars/sec. You sample every hour, and you keep 24 hours of context, and try to predict the next 4.<p>First, you'd make 100 "tokens" (really stretching the meaning of token here), one for each stoplight, and loading 24 samples (the history of that stoplight) into each token, and normalize.<p>Next, you run each token through a Multi-Layer Perceptron (vanilla, old-school neural network) to make a vector of dim D.<p>Next, for each layer of the transformer, you:
1. Perform "cross-attention," i.e. the query/key/value dance. This is how the different time series (erm, tokens) get to share information.
2. Normalize across all.
3. Run another bog-standard MLP independently on each token. This is the opportunity to examine the history of each time series.
4. Normalize again across all.<p>Then, you map each "token" (ugh) from being D-dimensional to 4-dimensional, so for each stoplight it predicts the traffic ahead for the next 4 hours. This is also a regular MLP.<p>So specifically, if you're only predicting a single time series (one stoplight), this method is equivalent to running a regular neural network.<p>It also, interestingly enough, skips the cool sinusoidal position embedding that transformers use to embed token position. Fair enough, since here the time dimension is fixed and the index of the feed-forward neurons in each MLP layer corresponds (roughly) to the time index of the sample.<p>The architecture looks weird to me, but apparently it works so that's cool! But I'm not sure how well it works, and my unscientific gut feel is that there's a better and simpler architecture crying out to be found, because this looks a bit tortured. Like, nothing in it explicitly models the time dimension - that task is left to the MLPs - and that seems weird.