A NEURAL STOCHASTIC VOLATILITY MODEL
In this paper, we show that the recent integration of statistical models with recurrent neural networks provides a new way of formulating volatility models that have been popular in time series analysis and prediction. The model comprises a pair of complementary stochastic recurrent neural networks: the generative network models the joint distribution of the stochastic volatility process; the inference network approximates the conditional distribution of the latent variables given the observable ones. Our focus in this paper is on the formulation of temporal dynamics of volatility over time under a stochastic recurrent neural network framework. Our derivations show that some popular volatility models are a special case of our proposed neural stochastic volatility model. Experiments demonstrate that the proposed model generates a smoother volatility estimation, and outperforms standard econometric models GARCH, EGARCH, GJR-GARCH and some other GARCH variants as well as MCMC-based model stochvol and a recent Gaussian processes based volatility model GPVOL on several metrics about the fitness of the volatility modelling and the accuracy of the prediction.
1 INTRODUCTION
The volatility of the price movements reflects the ubiquitous uncertainty within financial markets. It is critical that the level of risk, indicated by volatility, is taken into consideration before investment decisions are made and portfolio are optimised (Hull, 2006); volatility is substantially a key variable in the pricing of derivative securities. Hence, estimating and forecasting volatility is of great importance in branches of financial studies, including investment, risk management, security valuation and monetary policy making (Poon & Granger, 2003).
Volatility is measured typically by using the standard deviation of price change in a fixed time interval, such as a day, a month or a year. The higher the volatility, the riskier the asset. One of the primary challenges in designing volatility models is to identify the existence of latent (stochastic) variables or processes and to characterise the underlying dependences or interactions between variables within a certain time span. A classic approach has been to handcraft the characteristic features of volatility models by imposing assumptions and constraints, given prior knowledge and observations. Notable examples include autoregressive conditional heteroskedasticity (ARCH) model (Engle, 1982) and its generalisation GARCH (Bollerslev, 1986), which makes use of autoregression to capture the properties of time-variant volatility within many time series. Heston (1993) assumed that the volatility follows a Cox-Ingersoll-Ross (CIR) process (Cox et al., 1985) and derived a closed-form solution for options pricing. While theoretically sound, those approaches require strong assumptions which might involve complex probability distributions and non-linear dynamics that drive the process, and in practice, one may have to impose less prior knowledge and rectify a solution under the worst-case volatility case (Avellaneda & Paras, 1996).
In this paper, we take a fully data driven approach and determine the configurations with as few exogenous input as possible, or even purely from the historical data. We propose a neural network re-formulation of stochastic volatility by leveraging stochastic models and recurrent neural networks (RNNs). We are inspired by the recent development on variational approaches of stochastic (deep) neural networks (Kingma & Welling, 2013; Rezende et al., 2014) to a recurrent case (Chung et al., 2015; Fabius & van Amersfoort, 2014; Bayer & Osendorfer, 2014), and our formulation shows that existing volatility models such as the GARCH (Bollerslev, 1986) and the Heston model (Heston, 1993) are the special cases of our neural stochastic volatility formulation. With the hidden latent
variables in the neural networks we naturally uncover the underlying stochastic process formulated from the models.
Experiments with synthetic data and real-world financial data are performed, showing that the proposed model outperforms the widely-used GARCH model on several metrics of the fitness and the accuracy of time series modelling and prediction: it verifies our model’s high flexibility and rich expressive power.
2 RELATED WORK
A notable volatility method is autoregressive conditional heteroskedasticity (ARCH) model (Engle, 1982): it can accurately capture the properties of time-variant volatility within many types of time series. Inspired by ARCH model, a large body of diverse work based on stochastic process for volatility modelling has emerged. Bollerslev (1986) generalised ARCH model to the generalised autoregressive conditional heteroskedasticity (GARCH) model in a manner analogous to the extension from autoregressive (AR) model to autoregressive moving average (ARMA) model by introducing the past conditional variances in the current conditional variance estimation. Engle & Kroner (1995) presented theoretical results on the formulation and estimation of multivariate GARCH model within simultaneous equations systems. The extension to multivariate model allows the covariances to present and depend on the historical information, which are particularly useful in multivariate financial models. Heston (1993) derived a closed-form solution for option pricing with stochastic volatility where the volatility process is a CIR process driven by a latent Wiener process such that the current volatility is no longer a deterministic function even if the historical information is provided. Notably, empirical evidences have confirmed that volatility models provide accurate forecasts (Andersen & Bollerslev, 1998) and models such as ARCH and its descendants/variants have become indispensable tools in asset pricing and risk evaluation.
On the other hand, deep learning (LeCun et al., 2015; Schmidhuber, 2015) that utilises nonlinear structures known as deep neural networks, powers various applications. It has triumph over pattern recognition challenges, such as image recognition (Krizhevsky et al., 2012; He et al., 2015; van den Oord et al., 2016), speech recognition (Hinton et al., 2012; Graves et al., 2013; Chorowski et al., 2015), machine translation (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014; Luong et al., 2015) to name a few.
Time-dependent neural networks models include RNNs with advanced neuron structure such as long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997), gated recurrent unit (GRU) (Cho et al., 2014), and bidirectional RNN (BRNN) (Schuster & Paliwal, 1997). Recent results show that RNNs excel for sequence modelling and generation in various applications (Graves, 2013; Gregor et al., 2015). However, despite its capability as non-linear universal approximator, one of the drawbacks of neural networks is its deterministic nature. Adding latent variables and their processes into neural networks would easily make the posterori computationally intractable. Recent work shows that efficient inference can be found by variational inference when hidden continuous variables are embedded into the neural networks structure (Kingma & Welling, 2013; Rezende et al., 2014). Some early work has started to explore the use of variational inference to make RNNs stochastic (Chung et al., 2015; Bayer & Osendorfer, 2014; Fabius & van Amersfoort, 2014). Bayer & Osendorfer (2014) and Fabius & van Amersfoort (2014) considered the hidden variables are independent between times, whereas (Fraccaro et al., 2016) utilised a backward propagating inference network according to its Markovian properties. Our work in this paper extends the work (Chung et al., 2015) with a focus on volatility modelling for time series. We assume that the hidden stochastic variables follow a Gaussian autoregression process, which is then used to model both the variance and the mean. We show that the neural network formulation is a general one, which covers two major financial stochastic volatility models as the special cases by defining the specific hidden variables and non-linear transforms.
3 PRELIMINARY: VOLATILITY MODELS
Stochastic processes are often defined by stochastic differential equations (SDEs), e.g. a (univariate) generalised Wiener process is dxt = µd t + σ dwt, where µ and σ denote the time-invariant rates of drift and standard deviation (square root of variance) while dwt ∼ N (0,d t) is the increment of
standard Wiener process at time t. In a small time interval between t and t+∆t, the change in the variable is ∆xt = µ∆t+ σ∆wt. Let ∆t = 1, we obtain the discrete-time version of basic volatility model:
xt = xt−1 + µ+ σ t, (1)
where t ∼ N (0, 1) is a sample drawn from standard normal distribution. In the multivariate case,Σ represents the covariance matrix in place of σ2. As presumed that the variables are multidimensional, we will useΣ to represent variance in general case except explicitly noted.
3.1 DETERMINISTIC VOLATILITY
The time-invariant variance Σ can be extended to be a function Σt = Σ(x<t) relying on history of the (observable) underlying stochastic process {x<t}. The current variance Σt is therefore determined given the history {x<t} up to time t. An example of such extensions is the univariate GARCH(1,1) model (Bollerslev, 1986):
σ2t = α0 + α1(xt−1 − µt−1)2 + β1σ2t−1, (2)
where xt−1 is the observation from N (µt−1, σ2t−1) at time t − 1. Note that the determinism is in a conditional sense, which means that it only holds under the condition that the complete history {x<t} is presented, such as the case of 1-step-ahead forecast. otherwise the current volatility would still be stochastic as it is built on stochastic process {xt}. However, for multi-step-ahead forecast, we usually exploit the relation Et−1[(xt−µt)2] = σ2t to substitute the corresponding terms and calculate the forecasts with longer horizon in a recursive fashion, for example, σ2t+1 = α0 + α1Et−1[(xt − µt) 2] + β1σ 2 t = α0 + (α1 + β1)σ 2 t . For n-step-ahead forecast, there will be n iterations and the procedure is hence also deterministic.
3.2 STOCHASTIC VOLATILITY
Another extension is applicable for Σt from being conditionally deterministic (i.e. deterministic given the complete history {x<t}) to fully stochastic: Σt = Σ(z≤t) is driven by another latent stochastic process {zt} instead of the observable process {xt}. Heston (1993) model instantiates a continuous-time stochastic volatility model for univariate processes:
dxt = (µ− 0.5σ2t ) d t+ σt dw 〈1〉 t , (3) dσt = aσt d t+ bdw 〈2〉 t , (4)
where the correlation between dw〈1〉t and dw 〈2〉 t applies: E[dw 〈1〉 t ·dw 〈2〉 t ] = ρd t. We apply Euler’s scheme of quantisation (Stoer & Bulirsch, 2013) to obtain the discrete analogue to the continuoustime Heston model (Eqs. (3) and (4)):
xt = (xt−1 + µ− 0.5σ2t ) + σ t σt = (1 + a)σt−1 + bzt where [ t zt ] = N (0, [ 1 ρ ρ 1 ] ). (5)
3.3 VOLATILITY MODEL IN GENERAL
As discussed above, the observable variable xt follows Gaussian distribution of which the mean and variance depend on the history of observable process {xt} and latent {zt}. We presume in addition that the latent process {zt} is an autoregressive model such that zt is (conditionally) Gaussian distributed. Therefore, we formulate the volatility model in general as:
zt ∼ N (µz(z<t),Σz(z<t)), (6) xt ∼ N (µx(x<t, z≤t),Σx(x<t, z≤t)), (7)
where µz(x<t, z≤t) and Σz(x<t, z≤t) denote the autoregressive time-varying mean and variance of the latent variable zt while µx(x<t, z≤t) and Σx(x<t, z≤t) represent the mean and variance of observable variable xt, which depend on not only history of the observable process {x<t} but that of the latent process {z≤t}. These two formulas (Eqs. (6) and (7)) abstract the generalised formulation of volatility models. Together, they represents a broad family of volatility models with latent variables, where the Heston
model for stochastic volatility is merely a special case of the family. Furthermore, it will degenerate to deterministic volatility models such as the well-studied GARCH model if we disable the latent process.
4 NEURAL STOCHASTIC VOLATILITY MODELS
In this section, we establish the neural stochastic volatility model (NSVM) for stochastic volatility estimation and forecast.
4.1 GENERATING OBSERVABLE SEQUENCE
Recall that the latent variable zt (Eq. (6)) and the observable xt (Eq. (7)) are described by autoregressive models (xt has the exogenous input {z≤t}.) For the distributions of {zt} and {xt}, the following factorisation applies:
pΦ(Z) = ∏ t pΦ(zt|z<t) = ∏ t N (zt;µzΦ(z<t),ΣzΦ(z<t)), (8)
pΦ(X|Z) = ∏ t pΦ(xt|x<t, z≤t) = ∏ t N (xt;µxΦ(x<t, z≤t),ΣxΦ(x<t, z≤t)), (9)
where X = {xt} and Z = {zt} are the sequences of observable and latent variables, respectively, while Φ represents the parameter set of the model. The full generative model is defined as the joint distribution:
pΦ(X,Z) = ∏ t pΦ(xt|x<t, z≤t)pΦ(zt|z<t)
= ∏ t N (zt;µzΦ(z<t),ΣzΦ(z<t))N (xt;µxΦ(x<t, z≤t),ΣxΦ(x<t, z≤t)). (10)
It is observed that the means and variances are conditionally deterministic: given the historical information {z<t}, the current mean µzt = µzΦ(z<t) and varianceΣzt = ΣzΦ(z<t) of zt is obtained and hence the distribution N (zt;µzt ,Σzt ) of zt is specified; after sampling zt from the specified distribution, we incorporate {x<t} and calculate the current meanµxt = µxΦ(x<t, z≤t) and variance Σxt = Σ x Φ(x<t, z≤t) of xt and determine its distribution N (xt;µxt ,Σxt ) of xt. It is natural and convenient to present such a procedure in a recurrent fashion because of its autoregressive nature. As is known that RNNs can essentially approximate arbitrary function of recurrent form (Hammer, 2000), the means and variances, which may be driven by complex non-linear dynamics, can be efficiently computed using RNNs.
It is always a good practice to reparameterise the random variables before we go into RNN architecture. As the covariance matrix Σ is symmetric and positive definite, it can be factorised as Σ = UΛU>, where Λ is a full-rank diagonal matrix with positive diagonal elements. Let A = UΛ 1 2 , we have Σ = AA>. Hence we can reparameterise the latent variable zt (Eq. (6)) and observable xt (Eq. (7)):
zt = µ z t +A z t z t , (11) xt = µ x t +A x t x t , (12)
where Azt (A z t ) > = Σzt ,A x t (A x t ) > = Σxt and x t ∼ N (0, Ix), zt ∼ N (0, Iz) are auxiliary variables. Note that the randomness within the variables of interest (e.g. zt) is extracted by the auxiliary variables (e.g. t) which follow the standard distributions. Hence, the reparameterisation guarantees that gradient-based methods can be applied in learning phase (Kingma & Welling, 2013).
In this paper, the joint generative model is comprised of two sets of RNN and multilayer perceptron (MLP): RNNzg/MLP z g for the latent variable, while RNN x g /MLP z g for the observables. We stack these two RNN/MLP together according to the causal dependency between those variables. The
joint generative model is implemented as the generative network:
{µzt ,Azt } = MLP z g(h z t ;Φ), (13)
hzt = RNN z g(h z t−1, zt−1;Φ), (14)
zt = µ z t +A z t z t , (15)
{µxt ,Axt } = MLP x g(h x t ;Φ), (16)
hxt = RNN x g(h x t−1,xt−1, zt;Φ), (17) xt = µ x t +A x t x t , (18)
where hzt and h x t denote the hidden states of the corresponding RNNs. The MLPs map the hidden states of RNNs into the means and deviations of variables of interest. The parameter set Φ is comprised of the weights of RNNs and MLPs.
One should notice that when the latent variable z is obtained, e.g. by inference (details in the next subsection), the conditional distribution pΦ(X|Z) (Eq. (9)) will involve in generating the observable xt instead of the joint distribution pΦ(X,Z) (Eq. (10)). This is essentially the scenario of predicting future values of the observable variable given its history. We will use the term “generative model” and will not discriminate the joint generative model or the conditional one as it can be inferred in context.
4.2 INFERENCING THE LATENT PROCESS
As the generative model involves latent variable zt, of which the true valus are unaccessible even we have observed xt. Hence, the marginal likelihood pΦ(X) becomes the key that bridges the model and the data. The calculation of marginal likelihood involves the posterior distribution pΦ(Z|X), which is often intractable as complex integrals are involved. We are unable to learn the paramters or to infer the latent variables. Therefore, we consider instead a restricted family of tractable distributions qΨ(Z|X), referred to as the approximate posterior family, as approximations to the true posterior pΦ(Z|X) such that the family is sufficiently rich and flexible to provide good approximations (Bishop, 2006; Kingma & Welling, 2013; Rezende et al., 2014).
We define the inference model in accordance with the approximate posterior family we have presumed, in a similar fashion as (Chung et al., 2015), where the factorised distribution is formulated as follows:
qΨ(Z|X) = ∏ t qΨ(zt|z<t,x<t) = ∏ t N (zt; µ̃zΨ(z<t,x<t), Σ̃zΨ(z<t,x<t)), (19)
where µ̃zΨ(z<t,x<t) and Σ̃ z Ψ(z<t,x<t) are functions of the historical information {z<t}, {x<t}, representing the approximated mean and variance of the latent variable zt, respectively. Note that Ψ represents the parameter set of inference model.
The inference model essentially describes an autoregressive model on zt with exogenous input xt. Hence, in a similar fashion as the generative model, we implement the inference model as the inference network using RNN/MLP:
{µ̃zt , Ãzt } = MLP z i (h̃ z t ), (20)
h̃zt = RNN z i (h̃ z t−1, zt−1,xt−1), (21)
zt = µ̃ z t + Ã z t ̃ z t , (22)
where Ãzt (Ã z t ) > = Σ̃zt = Σ̃ z Ψ(z<t,x<t) while h̃ z t represents the hidden state of RNN and ̃ z t ∼ N (0, Iz) is an auxiliary variable to extract randomness. The inference mean µ̃zt and deviation Ãzt is computed by an MLP from the hidden state h̃zt . We use the subscript i instead of g to distinguish the architecture used in inference model in contrast to generative model.
4.3 FORECASTING OBSERVATIONS IN FUTURE
In the realm of time series analysis, we usually pay more attention on forecasting over generating (Box et al., 2015). It means that we are essentially more interested in the generation procedure
conditioning on the historical information rather than generation purely based on a priori belief since the observations in the past of x<t influences our belief of the latent variable zt. Therefore, we apply the approximate posterior distribution of the latent variable zt (Eq. (19)) as discussed in previous subsection, in place of the prior distribution (Eq. (8)) to build our predictive model.
Given the historical observations x<t, the predictive model infers the current value of latent variable zt using inference network and then generates the prediction of the current observation xt using generative network. The procedure of forecasting is shown in Fig. 1.
NSVM is learned using Stochastic Gradient Variational Bayes following (Kingma & Welling, 2013; Rezende et al., 2014). For readability, we provide the detailed derivation in Appendix A.
4.4 LINKS TO GARCH(1,1) AND HESTON MODEL
Although we refer to GARCH and Heston as volatility models, the purposes of them are quite different: GARCH is a predictive model used for volatility forecasting whereas Heston is more of a generative model of the underlying dynamics which facilitate closed-form solutions to SDEs in option pricing. The proposed NSVM has close relations to GARCH(1,1) and Heston model: both of them can be regarded as a special case of the neural network formulation. Recall Eq. (2), GARCH(1,1) is formulated as σ2t = α0 + α1(xt−1 − µt−1)2 + β1σ2t−1, where µt−1 is the trend estimate of {xt} at time step t calculated by some mean models. A common practice is to assume that µt follows the ARMA family (Box et al., 2015), or even simpler, as a constant that µt ≡ µ. We adopt the constant trend for simplicity as our focus is on volatility estimation.
We define the hidden state as hxt = [µ, σt] >, and disable the latent variable zt ≡ 0 as the volatility modelled by GARCH(1,1) is conditionally deterministic. Hence, we instantiate the generative network (Eqs. (16), (17) and (18)) as follows:
{µ, σt} = MLPxg(hxt ;Φ) = {[1, 0]hxt , [0, 1]hxt }, (23) hxt = RNN x g(h x t−1, xt−1;Φ)
= √[ 0 α0 ] + [ 0 α1 ] (xt−1 − [1, 0]hxt−1)2 + [ 1 0 0 β1 ] (hxt−1) 2, (24)
xt = µ+ σt t where t ∼ N (0, 1). (25)
The set of generative parameters is Φ = {µ, α0, α1, β1}. Next, we show the link between NSVM and (discrete-time) Heston model (Eq. (5)). Let hxt = [xt−1, µ, σt] > be the hidden state and zt be i.i.d. standard Gaussian instead of autoregressive vari-
able, we represent the Heston model in the framework of NSVM as:[ t zt ] = N (0, [ 1 ρ ρ 1 ] ), (26)
{µt, σt} = MLPxg(hxt ;Φ) = {[1, 1, 0]hxt − [0, 0, 0.5](hxt )2, [0, 0, 1]hxt }, (27) hxt = RNN x g(h x t−1, xt−1, zt;Φ)
= [ 0 0 0 0 1 0 0 0 1 + a ] hxt−1 + [ 1 0 0 ] xt−1 + [ 0 0 b ] zt, (28)
xt = µt + σt t. (29)
The set of generative parameters is Φ = {µ, a, b}. One should notice that, in practice, the formulation may change in accordance with the specific architecture of neural networks involved in building the model, and hence a closed-form representation may be absent.
5 EXPERIMENTS
In this section, we present our experiments1 both on the synthetic and real-world datasets to validate the effectiveness of NSVM.
5.1 BASELINES AND EVALUATION METRICS
To evaluate the performance of volatility modelling, we adopt the standard econometric model GARCH(1,1) Bollerslev (1986) as well as its variants EGARCH(1,1) Nelson (1991), GJR-GARCH(1,1,1) Glosten et al. (1993), ARCH(5), TARCH(1,1,1), APARCH(1,1,1), AGARCH(1,1,1), NAGARCH(1,1,1), IGARCH(1,1), IAVGARCH(1,1), FIGARCH(1,d,1) as baselines, which incorporate with the corresponding mean model AR(20). We would also compare our NSVM against a MCMC-based model “stochvol” and the recent Gaussian-processes-based model “GPVOL” Wu et al. (2014), which is a non-parametric model jointly learning the dynamics and hidden states via online inference algorithm. In addition, we setup a naive forecasting model as an alternative baseline referred to as NAIVE, which maintains a sliding window of size 20 on the most recent historical observations and forecasts the current values of mean and volatility by the average mean and variance of the window.
For synthetic data experiments, we take four metrics into consideration for performance evaluation: 1) the negative log-likelihood (NLL) of observing the test sequence with respect to the generative model parameters; 2) the mean-squared error (MSE) between the predicted mean and the ground truth (µ-MSE), 3) MSE of the predicted variance against the true variance (σ-MSE); 4) smoothness of fit, which is the standard deviation of the differences of succesive variance estimates. As for the real-world scenarios, the trend and volatility are implicit such that no ground truth is accessible to compare with, we consider only NLL and smoothness as the metrics for evaluation on real-world data experiment.
5.2 MODEL IMPLEMENTATION
The implementation of NSVM in experiments is in accordance with the architecture illustrated in Fig. 1: it consists of two neural networks, namely inference network and generative network. Each network comprises a set of RNN/MLP as we have discussed above: the RNN is instantiated by stacked LSTM layers whereas the MLP is essentially a 1-layer fully-connected feedforward network which splits into two equal-sized sublayers with different activation functions – one sublayer applies exponential function to impose the non-negativity and prevents overshooting of variance estimates while the other uses linear function to calculate mean estimates. During experiment, the model is structured by cascading the inference network and generative network as depicted in Fig. 1. The input layer is of size 20, which is the same as the embedding dimension DE ; the layer on the
1Repeatable experiment code: https://github.com/xxj96/nsvm
interface of inference network and generative network – we call it latent variable layer – represents the latent variable z, where its dimension is 2. The output layer has the same structure as the input one, therefore the latent variable layer acts as a bottleneck of the entire architecture which helps to extract the key factor. The stacked layers between input layer, latent variable layer and output layer are the hidden layers of either inference network or generative network, it consists of 1 or 2 LSTM layers with size 10, which contains recurrent connection for temporal dependencies modelling.
State-of-the-art learning techniques have been applied: we introduce Dropout (Zaremba et al., 2014) into each LSTM recurrent layer and impose L2-norm on the weights of each fully-connected feedforward layer as regularistion; NADAM optimiser (Dozat, 2015) is exploited for fast convergence, which is a variant of ADAM optimiser (Kingma & Ba, 2014) incorporated with Nesterov momentum; stepwise exponential learning rate decay is adopted to anneal the variations of convergence as time goes.
For econometric models, we utilise several widely-used packages for time series analysis: statsmodels (http://statsmodels.sourceforge.net/), arch (https://pypi.python. org/pypi/arch/3.2), Oxford-MFE-toolbox (https://www.kevinsheppard. com/MFE_Toolbox), stochvol (https://cran.r-project.org/web/packages/ stochvol) and fGarch (https://cran.r-project.org/web/packages/fGarch). The implementation of GPVOL is retrived from http://jmhl.org and we adopt the same hyperparameter setting as in Wu et al. (2014).
5.3 SYNTHETIC DATA EXPERIMENT
We build up the synthetic dataset by generating 256 heteroskedastic univariate time series, each with 2000 data points i.e. 2000 time steps. At each time step, the observation is drawn from a Gaussian distribution with pre-determined mean and variance, where the tendency of mean and variance is synthesised as linear combinations of sine functions. Specifically, for the trend and variance, we synthesis each using 3 sine functions with randomly chosen amplitudes and frequencies; then the value of the synthesised signal at each timestep is drawn from a Gaussian distribution with the corresponding value of trend and variance at that timestep. A sampled sequence is shown in Fig. 2a. We expect that this limited dataset could well simulate the real-world scenarios: one usually has very limited chances to observe and collect a large amount of data from time-invariant distributions. In addition, it seems that every observable or latent quantity within time series varies from time to time and seldom repeats the old patterns. Hence, we presume that the tendency shows long-term patterns and the period of tendency is longer than observation. In the experiment, we take the former 1500 time steps as the training set whereas the latter 500 as the test set.
For the synthetic data experiment, we simplify the recurrent layers in both inference net and generative net as single LSTM layer of size 10. The actual input {~xt} fed to NSVM is DEdimensional time-delay embedding (Kennel et al., 1992) of raw univariate observation {xt} such that ~xt = [xt+1−DE , . . . , xt]. 2-dimensional latent variable zt is adopted to capture the latent process, and enforces an orthogonal representation of the process by using diagonal covariance matrix. At each time step, 30 samples of latent variable zt are generated via reparameterisation (Eq. (22)).
5.4 REAL-WORLD DATA EXPERIMENT
We select 162 out of more than 1500 stocks from Chinese stock market and collect the time series of their daily closing prices from 3 institutions in China. We favour those with earlier listing date of trading (from 2006 or earlier) and fewer suspension days (at most 50 suspension days in total during the period of observation) so as to reduce the noise introduced by insufficient observation or missing values, which has significant influences on the performance but is essentially irrelevant to the purpose of volatility forecasting. More specifically, the dataset obtained contains 162 time series, each with 2552 data points (7 years). A sampled sequence is shown in Fig. 2b. We divide the whole dataset into two subsets: the training subset consists of the first 2000 data points while the test subset contains the rest 552 data points.
Similar model configuration is applied to the real-world data experiment: time-delay embedding of dimension DE on the raw univariate time series; 2-dimensional latent variable with diagonal
0 500 1000 1500 2000 timestep
3
2
1
0
1
2
3
x
data
µ
µ+ σ µ− σ
0 500 1000 1500 2000 timestep
0.0
0.1
0.2
0.3
0.4
0.5
v a ri
a n ce
ground truth variance garch's prediction nsvm's prediction
(a) Synthetic time series prediction. (up) The data and the predicted µx and bounds µx ± σx. (down) The groundtruth data variance and the corresponding prediction from GARCH(1,1) and NSVM.
0 500 1000 1500 2000 2500 timestep
0.0
0.5
1.0
1.5
2.0
2.5
3.0
x
data
µ
µ+ σ µ− σ
0 500 1000 1500 2000 2500 timestep
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
v a ri
a n ce
garch's prediction nsvm's prediction
(b) Real-world stock price prediction. (up) The data and the predicted µx and bounds µx ± σx. (down) The variance prediction from GARCH(1,1) and NSVM. The prediction of NSVM is more smooth and stable than that of GARCH(1,1), also yielding smaller NLL.
Figure 2: A case study of time series prediction.
covariance matrix; 30 sampling for the latent variable at each time step. Instead of single LSTM layers, here we adopt stacked LSTM layers composed of 2× 10 LSTM cells.
5.5 RESULT AND DISCUSSION
The overall performance of NSVM and baselines is listed in details in Table 1 and case studies on synthetic data and real-world financial data are illustrated in Fig. 2. The results show that NSVM has higher accuracies for modelling heteroskedastic time series on various metrics: NLL shows the fitness of the model under likelihood measure; the smoothness indicates that NSVM obtains more robust representation of the latent volatility; µ-MSE and σ-MSE in synthetic data experiment imply the ability of recognising the underlying patterns of both trend and volatility, which in fact verifies our claim of NSVM’s high flexibility and rich expressive power for volatility (as well as trend) modelling and forecasting compared with the baselines. Although the improvement comes at the cost of longer training time before convergence, it can be mitigated by applying parallel computing techniques as well as more advanced network architecture or training procedure.
The newly proposed NSVM outperforms standard econometric models GARCH(1,1), EGARCH(1,1), GJR-GARCH(1,1,1) and some other variants as well as the MCMC-based model “stochvol” and the recent GP-based model “GPVOL”. Apart from the higher accuracy NSVM obtained, it provides us with the ability to simply generalise univariate time series analysis to multivariate cases by extending network dimensions and manipulating the covariance matrices. Furthermore, it allows us to implement and deploy a similar framework on other applications, for example signal processing and denoising. The shortcoming of NSVM comparing to GPVOL is that the training procedure is offline: for short-term prediction, the experiments have shown the accuracy, but for long-term forecasting, the parameters need retraining, which will be rather time consuming. The online algorithm for inference will be one of the work in the future.
Specifically, our NSVM outperforms GARCH(1,1) on 142 out of 162 stocks on the metric of NLL. In particular, NSVM obtains −2.111, −2.044, −2.609 and −1.939 on the stocks corresponding to Fig2(b), Fig 4(a), (b) and (c) respectively, each of which is better than the that of GARCH (0.3433, 0.589, 0.109 and 0.207 lower on NLL).
6 CONCLUSION
In this paper, a novel volatility model NSVM has been proposed for stochastic volatility estimation and forecast. We integrated statistical models and RNNs, leveraged the characteristics of each model, organised the dependences between random variables in the form of graphical models, implemented the mappings among variables and parameters through RNNs, and finally established a powerful stochastic recurrent model with universal approximation capability. The proposed architecture comprises a pair of complementary stochastic neural networks: the generative network and inference network. The former models the joint distribution of the stochastic volatility process with both observable and latent variables of interest; the latter provides with the approximate posterior i.e. an analytical approximation to the (intractable) conditional distribution of the latent variables given the observable ones. The parameters (and consequently the underlying distributions) are learned (and inferred) via variational inference, which maximises the lower bound for the marginal log-likelihood of the observable variables. Our NSVM has presented higher accuracy compared to GARCH(1,1), EGARCH(1,1) and GJR-GARCH(1,1,1) as well as GPVOL for volatility modelling and forecasting on synthetic data and real-world financial data. Future work on NSVM would be to incorporate well-established models such as ARMA/ARIMA and to investigate the modelling of seasonal time series and correlated sequences.
As we have known, for models that evolve explicitly in terms of the squares of the residuals (e2t = (xt − µt)2), e.g. GARCH, the multi-step-ahead forecasts have closed-form solutions, which means
that those forecasts can be efficiently computed in a recursive fashion due to the linear formulation of the model and the exploitation of relation Et−1[e2t ] = σ 2 t .
On the other hand, for models that are not linear or do not explicitly evolve in terms of e2, e.g. EGARCH (linear but not evolve in terms of e2), our NSVM (nonlinear and not evolve in terms of e2), the closed-form solutions are absent and thus the analytical forecast is not available. We will instead use simulation-based forecast, which uses random number generator to simulate draws from the predicted distribution and build up a pre-specified number of paths of the variances at 1 step ahead. The draws are then averaged to produce the forecast of the next step. For n-step-ahead forecast, it requires n iterations of 1-step-ahead forecast to get there.
NSVM is designed as an end-to-end model for volatility estimation and forecast. It takes the price of stocks as input and outputs the distribution of the price at next step. It learns the dynamics using RNN, leading to an implicit, highly nonlinear formulation, where only simulation-based forecast is available. In order to obtain reasonably accurate forecasts, the number of draws should be relatively large, which will be very expensive for computation. Moreover, the number of draws will increase exponentially as the forecast horizon grows, so it will be infeasible to forecast several time steps ahead. We have planned to investigate the characteristics of NSVM’s long-horizontal forecasts and try to design a model specific sampling method for efficient evaluation in the future.
A COMPLEMENTARY DISCUSSIONS OF NSVM
In this appendix section we present detailed derivations of NSVM, specifically, the parameters learning and calibration, and covariance reparameterisation.
A.1 LEARNING PARAMETERS / CALIBRATION
Given the observationsX , the objective of learning is to maximise the marginal log-likelihood ofX given Φ, where the posterior is involved. However, as we have discussed in the previous subsection, the true posterior is usually intractable, which means exact inference is difficult. Hence, approximate inference is applied instead of rather than exact inference by following (Kingma & Welling, 2013; Rezende et al., 2014). We represent the marginal log-likelihood ofX in the following form:
ln pΦ(X) = EqΨ(Z|X) [ ln pΦ(X,Z)
pΦ(Z|X)
] = EqΨ(Z|X) [ ln pΦ(X,Z)
qΨ(Z|X) qΨ(Z|X) pΦ(Z|X) ] = EqΨ(Z|X)[ln pΦ(X,Z)− ln qΨ(Z|X)] +KL[qΨ(Z|X)‖pΦ(Z|X)] ≥ EqΨ(Z|X)[ln pΦ(X,Z)− ln qΨ(Z|X)] (as KL ≥ 0), (30)
where the expectation term EqΨ(Z|X)[ln pΦ(X,Z) − ln qΨ(Z|X)] is referred to as the variational lower bound L[q;X,Φ,Ψ] of the approximate posterior qΨ(Z|X,Ψ). The lower bound is essentially a functional with respect to distribution q and parameterised by observationsX and parameter sets Φ,Ψ of both generative and inference model. In theory, the marginal log-likelihood is maximised by optimisation on the lower bound L[q;X,Φ,Ψ] with respect to Φ and Ψ. We apply the factorisations in Eqs. (10) and (19) to the integrand within expectation of Eq. (30):
ln pΦ(X,Z)− ln qΨ(Z|X) = ∑ t [ lnN (xt;µxΦ(x<t, z≤t),ΣxΦ(x<t, z≤t))
+ lnN (zt;µzΦ(z<t),ΣzΦ(z<t))− lnN (zt; µ̃zΨ(z<t,x<t), Σ̃zΨ(z<t,x<t)) ] . (31)
As there is usually no closed-form solution for the expecation (Eq. (30)), we have to estimate the expectation by applying sampling methods to latent variable zt through time in accordance with the causal dependences. We utilise the reparameterisation of zt as shown in Eq. (22) such that we sample the corresponding auxiliary standard variable ̃t rather than zt itself and compute the value of zt on the fly. This ensures that the gradient-based optimisation techniques are applicable as the reparameterisation isolates the model parameters of interest from the sampling procedure. By sampling N sample paths, the estimator of the lower bound is defined as the average of paths:
L̂ =− 1 2N ∑ t [ ln detΣzt + (µ̃ z t + Ã z t ̃ z t − µzt )>(Σzt )−1(µ̃zt + Ãzt ̃zt − µzt )
+ ln detΣxt + (xt − µxt )>(Σxt )−1(xt − µxt )− ln det Σ̃t ] + const, (32)
where Ãzt (Ã z t ) > = Σ̃zt and ̃ z t ∼ N (0, Iz) is parameter-independent and considered as constant when calculating derivatives.
A.2 COVARIANCE PARAMETERISATION
As is known, it entails a computational complexity of O(M3) to maintain and update the full-size covariance Σ with M dimensions (Rezende et al., 2014). In the case of very high dimensions, the full-size covariance matrix would be too computationally expensive to afford. Hence, we use instead the covariance matrices with much fewer parameters for efficiency. The simplest setting is to use diagonal precision matrix (i.e. the inverse of covariance matrix) Σ−1 = D. However, it draws very strong restrictions on representation of the random variable of interest as the diagonal precision matrix (and thus diagonal covariance matrix) indicates independence among the dimensions. Therefore, the tradeoff becomes low-rank perturbation on diagonal matrix: Σ−1 = D + V V >, where V = {v1, . . . ,vK} denotes the perturbation while each vk is a M -dimensional column vector.
The corresponding covariance matrix and its determinant is obtained using Woodbury identity and matrix determinant lemma:
Σ =D−1 −D−1V (I + V >D−1V )−1V >D−1 (33) ln detΣ = − ln det (D + V V >) = − ln detD − ln det (I + V >D−1V ) (34)
To calculate the deviation A for the factorisation of covariance matrix Σ = AA>, we first consider the rank-1 perturbation where K = 1. It follows that V = v is a column vector, and I + V >D−1V = 1 + v>D−1v is a real number. A particular solution ofA is obtain:
A =D− 1 2 − [γ−1(1−√η)]D−1vv>D− 12 (35)
where γ = v>D−1v, η = (1 + γ)−1. The computational complexity involved here is merely O(M).
Observe that V V > = ∑K
k=1 vkv > k , the perturbation of rank K is essentially the superposition of
K perturbations of rank 1. Therefore, we can calculate the deviation A iteratively, an algorithm is provided to demonstrate the procedure of calculation. The computational complexity for rank-K perturbation remains to be O(M) given K M . Algorithm 1 gives the detailed calculation scheme.
Algorithm 1 Calculation of rank-K perturbation of precision matrices Input: The original diagonal matrixD; The rank-K perturbation V = {v1, . . . ,vK} Output: A such that the factorisationAA> = Σ = (D + V V >)−1 holds
1: A(0) =D − 12 2: i = 0 3: while i < K do 4: γ(i) = v > (i)A(i)A > (i)v(i) 5: η(i) = (1 + γ(i)) −1 6: A(i+1) = A(i) − [γ−1(i) (1− √ η(i))]A(i)A > (i)v(i)v > (i)A(i)
7: A = A(K)
B MORE CASE STUDIES
In this appendix section we add more case studies of NVSM performance on both synthetic data and real-world stock data.
NSVM obtains −2.044, −2.609 and −1.939 on the stocks corresponding to Fig 4(a), (b) and (c) respectively, each of which is better than the that of GARCH (0.589, 0.109 and 0.207 lower on NLL).
The reason of the drops in Fig 4(b) and (c) seems to be that NSVM has captured the jumps and drops of the stock price using its nonlinear dynamics and modelled the sudden changes as part of the trend: the estimated trend “mu” goes very close to the real observed price even around the jumps and drops (see the upper figure of Fig 4(b) and (c) around step 1300 and 1600). The residual (i.e. difference between the real value of observation and the trend of prediction) therefore becomes quite small, which lead to a lower volatility estimation.
On the other hand, for the baselines, we adopt AR as the trend model, which is a relatively simple linear model compared with the nonlinear NSVM. AR would not capture the sudden changes and leave those spikes in the residual; GARCH then took the residuals as input for volatility modelling, resulting in the spikes in volatility estimation.
0 500 1000 1500 2000 2500 timestep
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
x
data
µ
µ+ σ µ− σ
0 500 1000 1500 2000 2500 timestep
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
0.18
v a ri
a n ce
garch's prediction nsvm's prediction
(a) Real-world stock price prediction II. (up) The data and the predicted µx and bounds µx ± σx. (down) The variance prediction from GARCH(1,1) and NSVM. The prediction of NSVM is more smooth and stable than that of GARCH(1,1), also yielding smaller NLL.
0 500 1000 1500 2000 2500 timestep
0.0
0.5
1.0
1.5
2.0
2.5
3.0
x
data
µ
µ+ σ µ− σ
0 500 1000 1500 2000 2500 timestep
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
v a ri
a n ce
garch's prediction nsvm's prediction
(b) Real-world stock price prediction III. (up) The data and the predicted µx and bounds µx ± σx. (down) The variance prediction from GARCH(1,1) and NSVM. The prediction of NSVM is more smooth and stable than that of GARCH(1,1), also yielding smaller NLL.
0 500 1000 1500 2000 2500 timestep
0.0
0.5
1.0
1.5
2.0
2.5
x
data
µ
µ+ σ µ− σ
0 500 1000 1500 2000 2500 timestep
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
v a ri
a n ce
garch's prediction nsvm's prediction
(c) Real-world stock price prediction IV. (up) The data and the predicted µx and bounds µx ± σx. (down) The variance prediction from GARCH(1,1) and NSVM. The prediction of NSVM is more smooth and stable than that of GARCH(1,1), also yielding smaller NLL.
Figure 4: A case study of real-world stock time series prediction.

The authors propose a recurrent variational neural network approach to modelling volatility in financial time series. This model consists of an application of Chung et al.’s (2015) VRNN model to volatility forecasting, wherein a Variational Autoencoder (VAE) structure is repeated at each time step of the series.
The paper is well written and easy to follow (although this reviewer suggests applying a spelling checking, since the paper contains a number of harmless typos).
The paper’s main technical contribution is to stack two levels of recurrence, one for the latent process and one for the observables. This appears to be a novel, if minor contribution. The larger contribution is methodological, in areas of time series modelling that are both of great practical importance and have hitherto been dominated by rigid functional forms. The demonstration of the applicability and usefulness of general-purpose non-linear models for volatility forecasting would be extremely impactful.
I have a few comments and reservations with the paper:
1) Although not  mentioned explicitly, the authors’ framework are couched in terms of carrying out one-timestep-ahead forecasts of volatility. However, many applications of volatility models, for instance for derivative pricing, require longer-horizon forecasts. It would be interesting to discuss how this model could be extended to forecast at longer horizons.

2) In Section 4.4, there’s a mention that a GARCH(1,1) is conditionally deterministic. This is true only when forecasting 1 time-step in the future. At longer horizons, the GARCH(1,1) volatility forecast is not deterministic.
3) I was initially unhappy with the limitations of the experimental validation, limited to comparison with a baseline GARCH model. However, the authors provided more comparisons in the revision, which adds to the quality of the results, although the models compared against cannot be considered state of the art. It would be well advised to look into R packages such as `stochvol’ and ‘fGarch’ to get implementations of a variety of models that can serve as useful baselines, and provide convincing evidence that the modelled volatility is indeed substantially better than approaches currently entertained by the finance literature.
4) In Section 5.2, more details should be given on the network, e.g. number of hidden units, as well as the embedding dimension D_E (section 5.4)
5) In Section 5.3, more details should be given on the data generating process for the synthetic data experiments.
6) Some results in the appendix are very puzzling: around jumps in the price series, which are places where the volatility should spike, the model reacts instead by huge drops in the volatility (Figure 4(b) and (c), respectively around time steps 1300 and 1600). This should be explained and discussed.
All in all, I think that the paper provides a nice contribution to the art of volatility modelling. In spite of some flaws, it provides a starting point for the broader impact of neural time series processing in the financial community.


We build a large-scale visual search system which finds similar product images given a fashion item. Defining similarity among arbitrary fashion-products is still remains a challenging problem, even there is no exact ground-truth. To resolve this problem, we define more than 90 fashion-related attributes, and combination of these attributes can represent thousands of unique fashion-styles. We then introduce to use the recurrent neural networks (RNNs) recognising multiple fashion-attributes with the end-to-end manner. To build our system at scale, these fashion-attributes are again used to build an inverted indexing scheme. In addition to these fashion-attributes for semantic similarity, we extract colour and appearance features in a region-of-interest (ROI) of a fashion item for visual similarity. By sharing our approach, we expect active discussion on that how to apply current deep learning researches into the e-commerce industry.
1 INTRODUCTION
Online commerce has been a great impact on our life over the past decade. We focus on an online market for fashion related items1. Finding similar fashion-product images for a given image query is a classical problem in an application to computer vision, however, still challenging due to the absence of an absolute definition of the similarity between arbitrary fashion items.
Deep learning technology has given great success in computer vision tasks such as efficient feature representation (Razavian et al., 2014; Babenko et al., 2014), classification (He et al., 2016a; Szegedy et al., 2016b), detection (Ren et al., 2015; Zhang et al., 2016), and segmentation (Long et al., 2015). Furthermore, image to caption generation (Vinyals et al., 2015; Xu et al., 2015) and visual question answering (VQA) (Antol et al., 2015) are emerging research fields combining vision, language (Mikolov et al., 2010), sequence to sequence (Sutskever et al., 2014), long-term memory (Xiong et al., 2016) based modelling technologies.
These computer vision researches mainly concern about general object recognition. However, in our fashion-product search domain, we need to build a very specialised model which can mimic human's perception of fashion-product similarity. To this end, we start by brainstorming about what makes two fashion items are similar or dissimilar. Fashion-specialist and merchandisers are also involved. We then compose fashion-attribute dataset for our fashion-product images. Table 1 explains a part of our fashion-attributes. Conventionally, each of the columns in Table 1 can be modelled as a multi-class classification. Therefore, our fashion-attributes naturally is modelled as a multi-label classification.
∗This work was done by the author at SK Planet. 1In our e-commerce platform, 11st (http://english.11st.co.kr/html/en/main.html), al-
most a half of user-queries are related to the fashion styles, and clothes.
Multi-label classification has a long history in the machine learning field. To address this problem, a straightforward idea is to split such multi-labels into a set of multi-class classification problems. In our fashion-attributes, there are more than 90 attributes. Consequently, we need to build more than 90 classifiers for each attribute. It is worth noting that, for example, collar attribute can represent the upper-garments, but it is absent to represent bottom-garments such as skirts or pants, which means some attributes are conditioned on other attributes. This is the reason that the learning tree structure of the attributes dependency can be more efficient (Zhang & Zhang, 2010; Fu et al., 2012; Gibaja & Ventura, 2015).
Recently, recurrent neural networks (RNN) are very commonly used in automatic speech recognition (ASR) (Graves et al., 2013; Graves & Jaitly, 2014), language modelling (Mikolov et al., 2010), word dependency parsing (Mirowski & Vlachos, 2015), machine translation (Cho et al., 2014), and dialog modelling (Henderson et al., 2014; Serban et al., 2016). To preserve long-term dependency in hidden context, Long-Short Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) and its variants (Zaremba et al., 2014; Cooijmans et al., 2016) are breakthroughs in such fields. We use this LSTM to learn fashion-attribute dependency structure implicitly. By using the LSTM, our attribute recognition problem is regarded to as a sequence classification. There is a similar work in Wang et al. (2016), however, we do not use the VGG16 network (Simonyan & Zisserman, 2014) as an image encoder but use our own encoder. To the best of our knowledge, it is the first work applying LSTM into a multi-label classification task in the commercial fashion-product search domain.
The remaining of this paper is organized as follows. In Sec. 2, We describe details about our fashion-attribute dataset. Sec. 3 describes the proposed fashion-product search system in detail. Sec. 4 explains empirical results given image queries. Finally, we draw our conclusion in Sec. 5.
2 BUILDING THE FASHION-ATTRIBUTE DATASET
We start by building large-scale fashion-attribute dataset in the last year. We employ maximum 100 man-months and take almost one year for completion. There are 19 fashion-categories and more than 90 attributes for representing a specific fashion-style. For example, top garments have the Tshirts, blouse, bag etc. The T-shirts category has the collar, sleeve-length, gender, etc. The gender attribute has binary classes (i.e. female and male). Sleeve-length attribute has multiple classes (i.e. long, a half, sleeveless etc.). Theoretically, the combination of our attributes can represent thousands of unique fashion-styles. A part of our attributes are in Table 1. ROIs for each fashion item in an image are also included in this dataset. Finally, we collect 1 million images in total. This internal dataset is to be used for training our fashion-attribute recognition model and fashion-product ROI detector respectively.
3 FASHION-PRODUCT SEARCH SYSTEM
In this section, we describe the details of our system. The whole pipeline is illustrated in Fig. 3. As a conventional information retrieval system, our system has offline and online phase. In offline process, we take both an image and its textual meta-information as the inputs. The reason we take additional textual meta-information is that, for example, in Fig. 1a dominant fashion item in the image is a white dress however, our merchandiser enrolled it to sell the brown cardigan as described
in its meta-information. In Fig. 1b, there is no way of finding which fashion item is to be sold without referring the textual meta-information seller typed manually. Therefore, knowing intension (i.e. what to sell) for our merchandisers is very important in practice. To catch up with these intension, we extract fashion-category information from the textual meta. The extracted fashion-category information is fed to the fashion-attribute recognition model. The fashion-attribute recognition model predicts a set of fashion-attributes for the given image. (see Fig. 2) These fashion-attributes are used as keys in the inverted indexing scheme. On the next stage, our fashion-product ROI detector finds where the fashion-category item is in the image. (see Fig. 8) We extract colour and appearance features for the detected ROI. These visual features are stored in a postings list. In these processes, it is worth noting that, as shown in Fig. 8, our system can generate different results in the fashion-attribute recognition and the ROI detection for the same image by guiding the fashioncategory information. In online process, there is two options for processing a user-query. We can
take a guided information, what the user wants to find, or the fashion-attribute recognition model automatically finds what fashion-category item is the most likely to be queried. This is up to the user's choice. For the given image by the user, the fashion-attribute recognition model generates fashion-attributes, and the results are fed into the fashion-product ROI detector. We extract colour and appearance features in the ROI resulting from the detector. We access to the inverted index addressed by the generated a set of fashion-attributes, and then get a postings list for each fashionattribute. We perform nearest-neighbor retrieval in the postings lists so that the search complexity is reduced drastically while preserving the semantic similarity. To reduce memory capacity and speed up this nearest-neighbor retrieval process once more, our features are binarized and CPU depen-
dent intrinsic instruction (i.e. assembly popcnt instruction2) is used for computing the hamming distance.
3.1 VISION ENCODER NETWORK
We build our own vision encoder network (ResCeption) which is based on inception-v3 architecture (Szegedy et al., 2016b). To improve both speed of convergence and generalization, we introduce a shortcut path (He et al., 2016a;b) for each data-flow stream (except streams containing one convolutional layer at most) in all inception-v3 modules. Denote input of l-th layer , xl ∈ R , output of the l-th layer, xl+1, a l-th layer is a function, H : xl 7→ xl+1 and a loss function, L(θ;xL). Then forward and back(ward)propagation is derived such that
xl+1 = H(xl) + xl (1) ∂xl+1
∂xl = ∂H(xl) ∂xl + 1 (2)
Imposing gradients from the loss function to l-th layer to Eq. (2),
∂L ∂xl := ∂L ∂xL . . . ∂xl+2 ∂xl+1 ∂xl+1 ∂xl
= ∂L ∂xL
( 1+ · · ·+ ∂H(x L−2)
∂xl + ∂H(xL−1) ∂xl ) =
∂L ∂xL
( 1+ l∑ i=L−1 ∂H(xi) ∂xl ) . (3)
As in the Eq. (3), the error signal, ∂L ∂xL
, goes down to the l-th layer directly through the shortcut path, and then the gradient signals from (L − 1)-th layer to l-th layer are added consecutively (i.e.∑l i=L−1 ∂H(xi) ∂xl
). Consequently, all of terms in Eq. (3) are aggregated by the additive operation instead of the multiplicative operation except initial error from the loss (i.e. ∂L
∂xL ). It prevents
from vanishing or exploding gradient problem. Fig. 4 depicts network architecture for shortcut
2http://www.gregbugaj.com/?tag=assembly (accessed at Aug. 2016)
paths in an inception-v3 module. We use projection shortcuts throughout the original inception-v3 modules due to the dimension constraint.3 To demonstrate the effectiveness of the shortcut paths in the inception modules, we reproduce ILSVRC2012 classification benchmark (Russakovsky et al., 2015) for inception-v3 and our ResCeption network. As in Fig. 5a, we verify that residual shortcut paths are beneficial for fast training and slight better generalization.4 The whole of the training curve is shown in Fig. 5b. The best validation error is reached at 23.37% and 6.17% at top-1 and top-5, respectively. That is a competitive result.5 To demonstrate the representation power of our ResCeption, we employ the transfer learning strategy for applying the pre-trained ResCeption as an image encoder to generate captions. In this experiment, we verify our ResCeption encoder outperforms the existing VGG16 network6 on MS-COCO challenge benchmark (Chen et al., 2015). The best validation CIDEr-D score (Vedantam et al., 2015) for c5 is 0.923 (see Fig. 5c) and test CIDEr-D score for c40 is 0.937.7
3.2 MULTI-LABEL LEARNING AS SEQUENCE PREDICTION BY USING THE RNN
The traditional multi-class classification associates an instance x with a single label a from previously defined a finite set of labels A. The multi-label classification task associates several finite sets of labels An ⊂ A. The most well known method in the multi-label literature are the binary relevance method (BM) and the label combination method (CM). There are drawbacks in both BM
3If the input and output dimension of the main-branch is not the same, projection shortcut should be used instead of identity shortcut.
4This is almost the same finding from Szegedy et al. (2016a) but our work was done independently. 5http://image-net.org/challenges/LSVRC/2015/results 6https://github.com/torch/torch7/wiki/ModelZoo 7We submitted our final result with beam search on MS-COCO evaluation server and found out the beam
search improves final CIDEr-D for c40 score by 0.02.
and CM. The BM ignores label correlations that exist in the training data. The CM directly takes into account label correlations, however, a disadvantage is its worst-case time complexity (Read et al., 2009). To tackle these drawbacks, we introduce to use the RNN. Suppose we have random variables a ∈ An, An ⊂ A. The objective of the RNN is to maximise the joint probability, p(at, at−1, at−2, . . . a0), where t is a sequence (time) index. This joint probability is factorized as a product of conditional probabilities recursively,
p(at, at−1, . . . a0) = p(a0)p(a1|a0)︸ ︷︷ ︸ p(a0,a1) p(a2|a1, a0)
︸ ︷︷ ︸ p(a0,a1,a2)
· · ·
︸ ︷︷ ︸ p(a0,a1,a2,... )
(4)
= p(a0) ∏T t=1 p(at|at−1, . . . , a0).
Following the Eq. 4, we can handle multi-label classification as sequence classification which is illustrated in Fig. 6. There are many label dependencies among our fashion-attributes. Direct modelling of such label dependencies in the training data using the RNN is our key idea. We use the ResCeption as a vision encoder θI , LSTM and softmax regression as our sequence classifier θseq, and negative log-likelihood (NLL) as the loss function. We backpropagage gradient signal from the sequence classifier to vision encoder.8 Empirical results of our ResCeption-LSTM based attribute recognition are in Fig. 2. Many fashion-category dependent attributes such as sweetpants, fading, zipper-lock, mini, and tailored-collar are recognized quite well. Fashion-category independent attributes (e.g., male, female) are also recognizable. It is worth noting we do not model the fashionattribute dependance tree at all. We demonstrate the RNN learns attribute dependency structure implicitly. We evaluate our attribute recognition model on the fashion-attribute dataset. We split this dataset into 721544, 40000, and 40000 images for training, validating, and testing. We employ the early-stopping strategy to preventing over-fitting using the validation set. We measure precision and recall for a set of ground-truth attributes and a set of predicted attributes for each image. The quantitative results are in Table 2.
8Our attribute recognition model is parameterized as θ = [θI ; θseq]. In our case, updating θI as well as θseq in the gradient descent step helps for much better performance.
3.3 Guided ATTRIBUTE-SEQUENCE GENERATION
Our prediction model of the fashion-attribute recognition is based on the sequence generation process in the RNN (Graves, 2013). The attribute-sequence generation process is illustrated in Fig. 7. First, we predict a probability of the first attribute for a given internal representation of the image i.e. pθseq(a0|gθI (I)), and then sample from the estimated probability of the attribute, a0 ∼ pθseq(a0|gθI (I)). The sampled symbol is fed to as the next input to compute pθseq(a1|a0, gθI (I)). This sequential process is repeated recursively until a sampled result is reached at the special endof-sequence (EOS) symbol. In case that we generate a set of attributes for a guided fashion-category, we do not sample from the previously estimated probability, but select the guided fashion-category, and then we feed into it as the next input deterministically. It is the key to considering for each seller's intention. Results for the guided attribute-sequence generation is shown in Fig. 8.
3.4 Guided ROI DETECTION
Our fashion-product ROI detection is based on the Faster R-CNN (Ren et al., 2015). In the conventional multi-class Faster R-CNN detection pipeline, one takes an image and outputs a tuple of (ROI coordinate, object-class, class-score). In our ROI detection pipeline, we take additional information, guided fashion-category from the ResCeption-LSTM based attribute-sequence generator. Our fashion-product ROI detector finds where the guided fashion-category item is in a given image. Jing et al. (2015) also uses a similar idea, but they train several detectors for each category independently so that their works do not scale well. We train a detector for all fashion-categories jointly. Our detector produces ROIs for all of the fashion-categories at once. In post-processing, we reject ROIs that their object-classes are not matched to the guided fashion-category. We demonstrate that the guided fashion-category information contributes to higher performance in terms of mean average precision (mAP) on the fashion-attribute dataset. We measure the mAP for the intersection-of-union (IoU) between ground-truth ROIs and predicted ROIs. (see Table 3) That is due to the fact that our guided fashion-category information reduces the false positive rate. In our fashion-product search pipeline, the colour and appearance features are extracted in the detected ROIs.
3.5 VISUAL FEATURE EXTRACTION
To extract appearance feature for a given ROI, we use pre-trained GoogleNet (Szegedy et al., 2015). In this network, both inception4 and inception5 layer's activation maps are used. We evaluate this feature on two similar image retrieval benchmarks, i.e. Holidays (Jegou et al., 2008) and UK-benchmark (UKB) (Nistér & Stewénius, 2006). In this experiment, we do not use any postprocessing method or fine-tuning at all. The mAP on Holidays is 0.783, and the precision@4 and recall@4 on UKB is 0.907 and 0.908 respectively. These scores are competitive against several deep feature representation methods (Razavian et al., 2014; Babenko et al., 2014). Examples of queries and resulting nearest-neighbors are in Fig. 9. On the next step, we binarize this appearance feature by simply thresholding at 0. The reason we take this simple thresholding to generate the hash code is twofold. The neural activation feature map at a higher layer is a sparse and distributed code in nature. Furthermore, the bias term in a linear layer (e.g., convolutional layer) compensates for
aligning zero-centering of the output feature space weakly. Therefore, we believe that a code from a well-trained neural model, itself, can be a good feature even to be binarized. In our experiment, such simple thresholding degrades mAP by 0.02 on the Holidays dataset, but this method makes it possible to scaling up in the retrieval. In addition to the appearance feature, we extract colour feature using the simple (bins) colour histogram in HSV space, and distance between a query and a reference image is computed by using the weighted combination of the two distances from the colour and the appearance feature.
4 EMPIRICAL RESULTS
To evaluate empirical results of the proposed fashion-product search system, we select 3 million fashion-product images in our e-commerce platform at random. These images are mutually exclusive to the fashion-attribute dataset. We have again selected images from the web used for the queries. All of the reference images pass through the offline process as described in Sec. 3, and resulting inverted indexing database is loaded into main-memory (RAM) by our daemon system. We send the pre-selected queries to the daemon system with the RESTful API. The daemon system then performs the online process and returns nearest-neighbor images correspond to the queries. In this scenario, there are three options to get similar fashion-product images. Option 1 is that the fashion-attribute recognition model automatically selects fashion-category, the most likely to be queried in the given image. Option 2 is that a user manually selects a fashion-category given a query image. (see Fig. 10) Option 3 is that a user draw a rectangle to be queried by hand like Jing et al. (2015). (see Fig. 11) By the recognized fashion-attributes, the retrieved results reflect the user's main needs, e.g. gender, season, utility as well as the fashion-style, that could be lacking when using visual feature representation only.
5 CONCLUSIONS
Today's deep learning technology has given great impact on various research fields. Such a success story is about to be applied to many industries. Following this trend, we traced the start-of-the art computer vision and language modelling research and then, used these technologies to create value for our customers especially in the e-commerce platform. We expect active discussion on that how to apply many existing research works into the e-commerce industry.

The paper presents a large-scale visual search system for finding product images given a fashion item. The exploration is interesting and the paper does a nice job of discussing the challenges of operating in this domain. The proposed approach addresses several of the challenges.
However, there are several concerns.
1) The main concern is that there are no comparisons or even mentions of the work done by Tamara Berg’s group on fashion recognition and fashion attributes, e.g.,
-  “Automatic Attribute Discovery and Characterization from Noisy Web Data” ECCV 2010
- “Where to Buy It: Matching Street Clothing Photos in Online Shops” ICCV 2015,
- “Retrieving Similar Styles to Parse Clothing, TPAMI 2014,
etc
It is difficult to show the contribution and novelty of this work without discussing and comparing with this extensive prior art.
2) There are not enough details about the attribute dataset and the collection process. What is the source of the images? Are these clean product images or real-world images? How is the annotation done? What instructions are the annotators given? What annotations are being collected? I understand data statistics for example may be proprietary, but these kinds of qualitative details are important to understand the contributions of the paper. How can others compare to this work?
3) There are some missing baselines. How do the results in Table 2 compare to simpler methods, e.g., the BM or CM methods described in the text?
While the paper presents an interesting exploration, all these concerns would need to be addressed before the paper can be ready for publication.


DYNAMIC PARTITION MODELS
We present a new approach for learning compact and intuitive distributed representations with binary encoding. Rather than summing up expert votes as in products of experts, we employ for each variable the opinion of the most reliable expert. Data points are hence explained through a partitioning of the variables into expert supports. The partitions are dynamically adapted based on which experts are active. During the learning phase we adopt a smoothed version of this model that uses separate mixtures for each data dimension. In our experiments we achieve accurate reconstructions of high-dimensional data points with at most a dozen experts.
1 INTRODUCTION
We consider the task of learning a compact binary representation (e.g. Goessling & Amit, 2015). That means we are seeking a parsimonious set of experts, which can explain a given collection of multivariate data points. In contrast to most existing approaches the emphasis here is on finding experts that are individually meaningful and that have disjoint responsibilities. Ideally, each expert explains only one factor of variation in the data and for each factor of variation there is exactly one expert that focuses on it.
Formally, the experts Pk, k = 1, . . . ,K, are probability distributions that depend on binary latent variables h(k). The latent state h specifies which experts are active and has to be inferred for each D-dimensional data point x. The active experts then define a probability distribution P. The goal of representation learning is to train experts such that the conditional likelihood P(x |h) of the data given the latent activations is maximized.
We start by describing a simple model family, which forms the basis of our work. A partition model (Hartigan, 1990) makes use of a manually specified partitioning of the D variables into subsets
{1, . . . , D} = L∪
ℓ=1
Sℓ.
For each subset of variablesx(Sℓ) = (x(d))d∈Sℓ there exists a separate model Pℓ. It is then typically assumed that variables in different subsets are conditionally independent, i.e.,
P(x |h) = L∏
ℓ=1
Pℓ(x(Sℓ) |h(ℓ)). (1)
The model is completed by specifying a prior distribution P(h) for the latent state h. One advantage of partition models is that estimating Pℓ from observations is straightforward, while learning expert models in general requires computationally involved procedures (Bengio et al., 2013). However, in order to be able to define a satisfactory partitioning of the variables some prior knowledge about the dependence structure is needed. For image data a common choice is to use a regular grid that divides the image into patches (e.g. Pal et al., 2002). In general, a good partitioning is characterized by providing weakly dependent subsets of variables so that the conditional independence assumption (1) is reasonable and the distribution of the latent variables is easy to model. Unfortunately, often there simply is no single fixed partitioning that works well for the whole dataset because the set
of variables, which are affected by different factors of variation, might overlap. This restricts the scenarios in which partition models are useful.
In this paper we extend partition models to allow for dynamically adapting partitionings. In Section 2 we introduce the model and present an appropriate learning procedure. Related work is discussed in Section 3. Special emphasis is given to the comparison with products of experts (Hinton, 2002). Experiments on binary and real-valued data are performed in Section 4. While it is important to explain high-dimensional data points through multiple experts, our work shows that it is possible to assign the responsibility for individual variables to a single expert (rather than having all active experts speak for every variable).
2 DYNAMIC PARTITION MODELS
Our main proposal is to define for each expert Pk its level of expertise ek ∈ RD+ for all variables. We can then dynamically partition the variables based on the active experts. Specifically, for each variable we employ the most reliable (active) expert
P(x |h) = D∏
d=1
Pk⋆(d)(x(d)), k⋆(d) = argmax k:h(k)=1 ek(d). (2)
That means, each variable x(d) is explained by only a single expert k⋆(d). The partitioning into expert supports Sk(h) = {d ∈ {1, . . . , D} : k⋆(d) = k} is determined dynamically based on the latent configuration h. We hence call our model a dynamic partition model.
2.1 INFERENCE
In the inference step we try to find for each data point xn the subset of experts {k : hn(k) = 1} that maximizes P (xn |hn). To do this, we suggest to sequentially activate the expert that most improves the likelihood, until the likelihood cannot be improved anymore. This approach is called likelihood matching pursuit (Goessling & Amit, 2015). The greedy search works well for our model because we are working with a small set of experts and each expert focuses on a rather different structure in the data. Consequently, the posterior distribution on the latent variables given xn is often highly peaked at a state hn (note that for high-dimensional data the effect of the prior P(h) is typically negligible).
2.2 LEARNING
In contrast to traditional approaches, which combine multiple experts for individual variables, training the experts in a dynamic partition model is trivial. Indeed, the maximum-likelihood estimates are simply the empirical averages over all observations for which the expert was responsible. For example, the expert means can be estimated from training data xn, n = 1, . . . , N , as
◦ µk(d) =
N∑ n=1 1{k⋆n(d)=k}xn(d)
N∑ n=1 1{k⋆n(d)=k} . (3)
Here, k⋆n(d) denotes the expert with the highest level of expertise ek(d) among all experts k with hn(k) = 1.
2.2.1 EXPERTISE-WEIGHTED COMPOSITION
In order to compute the estimator in (3) the levels of expertise ek have to be known. Since in this paper we are trying to train the experts as well as the associated levels of expertise we consider a smoothing of the maximum-expertise composition (2) to motivate our learning procedure. Rather than using the expert with the highest level of expertise, we form a mixture of the active experts, where the mixture weight is proportional to the level of expertise. Thus, the smoothed composition
rule is P̃(x |h) = D∏
d=1 K∑ k=1 rk(d)Pk(x(d)), rk(d) =
{ ek(d)∑
k′:h(k′)=1 ek′ (d) if h(k) = 1
0 if h(k) = 0 . (4)
In contrast to classical mixture models (e.g. McLachlan & Peel, 2004) we use different mixture weights for each dimension d ∈ {1, . . . , D}. The mixture weight rk(d) is the degree of responsibility of k-th expert for the d-th dimension and depends on the latent state h. An expert with a medium level of expertise assumes full responsibility if no other reliable expert is present and takes on a low degree of responsibility if experts with a higher level of expertise are present.
According to the total variance formula V[P̃] = Erk [V[Pk]] + Vrk [E[Pk]] the variance of a mixture is always larger than the smallest variance of its components. In other words, the precision of the smoothed model is maximized when all the mixture weight (individually for each dimension) is concentrated on the most precise expert. We can thus learn a dynamic partition model in an EM manner (Dempster et al., 1977) by interleaving inference steps with updates of the experts and levels of expertise in the smoothed model.
2.2.2 EXPERT UPDATE
The sequential inference procedure (from Section 2.1) provides for each data point xn the latent representation hn. We denote the corresponding expert responsibilities (using the current estimates for the level of expertise) by rnk. The smooth analog to the hard update equation (3) is a responsibilityweighted average of the training samples
µk(d) =
N∑ n=1 rnk(d)xn(d) + ϵµ0
N∑ n=1 rnk(d) + ϵ
. (5)
For stability we added a term that shrinks the updated templates towards some target µ0 if the total responsibility of the expert is small. In our experiments we set µ0 to the average of all training examples. The update rule implies that the experts have local supports, in the sense that they are uninformative about variables for which they are not responsible.
For binary data the mean templates µk are all we need. Continuous data x ∈ RD is modeled through Gaussians and hence we also have to specify the variance vk of the experts. We again use a responsibility-weighted average
vk(d) =
N∑ n=1
rnk(d)(xn(d)− µk(d))2 + ϵv0 N∑
n=1 rnk(d) + ϵ
, (6)
where v0 is the empirical variance of all training samples.
2.2.3 EXPERTISE UPDATE
We now turn to the updates of the levels of expertise. The log-likelihood of the smoothed model (4) as a function of ek is rather complex. Using gradient descent is thus problematic because the derivatives with respect to ek can have very different scales, which makes it difficult to choose an appropriate learning rate and hence the convergence could be slow. However, exact optimization is not necessary because in the end only the order of the levels of expertise matters. Consequently, we propose to adjust ek(d) only based on the sign of the gradient. We simply multiply or divide the current value by a constant C. If the gradient is very close to 0 we leave ek(d) unchanged. For all our experiments we used C = 2. Larger values can speed up the convergence but sometimes lead to a worse solution. Using an exponential decay is common practice when learning levels of expertise (e.g. Herbster & Warmuth, 1998).
In the learning procedure we perform the expertise update first. We then recompute the responsibilities using these new levels of expertise and update the experts. Our algorithm typically converges after about 10 iterations.
3 RELATED WORK
Herbster & Warmuth (1998) proposed an algorithm for tracking the best expert in a sequential prediction task. In their work it is assumed that a linear ordering of the variables is known such that the expert with the highest level of expertise is constant on certain segments. In contrast to that, our approach can be applied to an arbitrary permutation of the variables. Moreover, they consider a single sequence of variables with a fixed partitioning into experts supports. In our setup the partitioning changes dynamically depending on the observed sample. However, the greatest difference to our work is that Herbster & Warmuth (1998) do not learn the individual experts but only focus on training the levels of expertise.
Lücke & Sahani (2008) studied a composition rule that also partitions the variables into expert supports. In their model the composed template is simply the maximum of the experts templates µk. This rule is only useful in special cases. A generalization, in which the composition depends on the maximum and the minimum of the expert templates µk(d), was considered by Goessling & Amit (2015). While the motivation for that rule was similar, the maximum-expertise rule in this paper is more principled and can be applied to continuous data.
In the work by Amit & Trouvé (2007) a simple average (i.e., an equal mixture) of the individual templates was used. With such a composition rule, all experts are equally responsible for each of the variables and hence specialization on local structures is not possible. To circumvent this problem, in their work ek(d) was manually set to 1 for some subset of the dimensions (depending on a latent shift variable) and to 0 elsewhere.
A popular model family with latent binary representation are products of experts (Hinton, 2002). In such a model the individual distributions Pk are multiplied together and renormalized. Computation of the normalizing constant is in general intractable though. A special case, in which an explicit normalization is possible, are restricted Boltzmann machines (Hinton, 2002). In these models the experts are product Bernoulli distributions with templates µk ∈ [0, 1]D. The composed distribution is then also a product Bernoulli distribution with composed template
µRBM(d) = σ (∑ k:h(k)=1 wk(d) ) ,
where the weights wk(d) = log(µk(d)/(1 − µk(d)) ∈ R are the log-odds of the experts and σ(t) = (1 + exp(−t))−1 is the logistic function. This sum-of-log-odds composition rule arises naturally from generalized linear models for binary data because the log-odds are the canonical parameter of the Bernoulli family. In a product of experts, the variance of the composition is usually smaller than the smallest variance of the experts. As a consequence, products of experts tend to employ many experts for each dimension (for more details on this issue see Goessling & Amit (2015)). Even with an L1-penalty on the votes wk(d) the responsibility for individual variables x(d) is typically still shared among many experts. The reason for this is that under the constraint∑
k wk(d) = w(d) the quantity ∑
k |wk(d)| is minimized whenever wk(d) has the same sign for all k. The usual inference procedure for products of experts independently activates experts based on their inner product with the data point. In particular, not just the most probable expert configuration is determined but the whole posterior distribution on latent states given the data is explored through Monte Carlo methods. For learning in products of experts, simple update rules like (5) and (6) cannot be used because for each expert the effects of all other experts have to be factored out. Dynamic partition models essentially decompose the expert votes wk into expert opinions µk and levels of expertise ek. Apart from the computational advantages for learning, this introduces an additional degree of flexibility because the expert supports are adjusted depending on which other experts are present (cf. Figure 5). Moreover, the decomposition into opinions and levels of expertise avoids ambiguities. For example, a votewk(d) ≈ 0 could mean that µk(d) ≈ 1/2 or that ek(d) ≈ 0. Another common model for representation learning are autoencoders (Vincent et al., 2008), which can be considered as mean-field approximations of restricted Boltzmann machines that use latent variables h(k) with values in [0, 1]. To obtain a sparse representation a penalty on the number of active experts can be added (Ng, 2011). Such approaches are also known as sparse dictionaries (e.g., Elad, 2010) and are based on opinion pools of the form ∑ k h(k)wk(d). The strength of the sparsity penalty is an additional tuning parameter which has to be tuned. In dynamic partition models sparse activations are inherent. In the next section, we experimentally compare products of experts, autoencoders and sparse dictionaries to our proposed model.
4 EXPERIMENTS
4.1 SYNTHETIC DATA
We consider a synthetic example and try to learn the underlying factors of variation. The dataset consists of the 32-element subset {(0, 1), (1, 0)}5 ⊂ {0, 1}10. Note that there are 5 factors of variation corresponding to the state of the pairs (x(2ℓ−1),x(2ℓ)) for ℓ = 1, . . . , 5 with the two factor levels (0, 1) and (1, 0). Indeed, the distribution can be easily expressed through a partition model with partitioning
{1, 2} ∪ {3, 4} ∪ {5, 6} ∪ {7, 8} ∪ {9, 10} and corresponding models
Pℓ(x(2ℓ−1),x(2ℓ)) = 12 · 1{x(2ℓ−1)=0, x(2ℓ)=1}+ 1 2 · 1{x(2ℓ−1)=1, x(2ℓ)=0}.
We show that our dynamic partition model is able to learn these factors of variation without requiring a manual specification of the partitioning. Here, the total number of experts we need to accurately reconstruct all data points happens to be equal to the number of dimensions. However, in other cases the number of required experts could be smaller or larger than D. We ran our learning algorithm for 15 iterations starting from a random initialization of the experts. The resulting templates after 3, 5 and 15 iterations are shown in Figure 1. We see that each of the final experts specializes in exactly two dimensions d and d+1. Its opinion for these variables are close to 0 and 1, respectively, while the opinions for the remaining variables are about 1/2. Every data point can now be (almost) perfectly reconstructed by using exactly 5 of these experts.
For comparison we trained various other models with 10 experts, which use a sum-of-log-odds composition. We first tried an autoencoder (Vincent et al., 2008), which in principle could adopt the identity map because it uses (in contrast to our model) a bias term for the observable and latent variables. However, the gradient descent learning algorithm with tuned step size yielded a different representation (Figure 2, 1st panel). While the reconstruction errors are rather low, they are clearly nonzero and the factors of variations have not been disentangled. Next, we considered a dictionary with a sparse representation (e.g., Elad, 2010). The sparsity penalty was adjusted so that the average number of active dictionary elements was around 5. The learning algorithm again yielded highly dependent experts (Figure 2, 2nd panel). Finally, we trained a restricted Boltzmann machine through batch persistent contrastive divergence (Tieleman, 2008) using a tuned learning rate. Note that a
restricted Boltzmann machine in principle only requires 5 experts to model the data appropriately because it uses bias terms. However, we again learned 10 experts (Figure 2, 3rd panel). While the results look better than for the previous two models they are still far from optimal. In earlier work Goessling & Amit (2015) we performed a quantitative comparison for a similar dataset, which showed that the reconstruction performance of models with sum-of-log-odds composition is indeed suboptimal.
4.2 MNIST DIGITS
We now consider the MNIST digits dataset (LeCun et al., 1998), which consists of 60,000 training samples and 10,000 test samples of dimension 28× 28 = 784. We ran our learning algorithm for 10
iterations and trained 100 experts (Figure 3). We see that some experts specialize on local structures while others focus on more global ones. In Figure 4 we visualize the inference procedure for some test samples using these 100 learned experts. On average 12 experts were activated for each data point. For easier visualization we show at most 10 iterations of the likelihood matching pursuit algorithm. The reconstructions are overall accurate and peculiarities of the samples are smoothed out. In Figure 5 we illustrate how the expert supports change based on the latent representation. Depending on which other experts are present the supports can vary quite a bit.
4.3 WEIZMANN HORSES
The following experiment shows that our model is able to cope with very high-dimensional data. The Weizmann horse dataset (Borenstein & Ullman, 2008) consists of 328 binary images of size 200 × 240. We used the first 300 images to train 20 experts (Figure 6) and used the remaining 28 images for testing. Some of the experts are responsible for the background and the central region of the horse while other experts focus on local structures like head posture, legs and tail. In Figure 7 we illustrate the partitioning of the test examples into expert opinions. For simplicity we used exactly 4 experts to reconstruct each sample. Not all characteristics of the samples are perfectly reconstructed but the general pose is correctly recovered. The same dataset was used to evaluate the shape Boltzmann machine (Eslami et al., 2014), where 2,000 experts were learned. For those experiments the images were downsampled to 32 × 32 pixels. This is a factor 50 smaller than the full resolution of 48,000 dimensions that we use.
4.4 CALTECH MOTORCYCLES
We also experimented with real-valued data using the Caltech-101 motorcycle dataset (Fei-Fei et al., 2007), which consists of 798 images of size 100× 180. The first 750 images were used for training and the remaining 48 images for testing. We trained 50 experts by running our learning procedure for 10 iterations. In Figure 8 we visualize the reconstructed test examples. The reconstructions are a bit blurry since we use a fairly sparse binary representation. Indeed, for each data point on average only 7 experts were employed. Note that the shapes of the motorcycles are reconstructed quite accurately.
5 DISCUSSION
In order to improve the reconstructions for continuous image data we could use real-valued latent variables in addition to binary ones (as in Hinton et al. (1998)). This would allow us to model intensities and contrasts more accurately. The inference procedure would have to be adapted accordingly such that continuous activations can be returned.
Our work focused on product distributions. In order to apply the proposed approach to models with dependence structure one can make use of an autoregressive decomposition (e.g., Goessling & Amit, 2016). If the joint distribution is written as a product of conditional distributions then we can employ the same composition rule as before. Indeed, we can model composed the conditionals as
P(x(d) |x(1:d−1),h) = Pk⋆(d)(x(d) |x(1:d−1)),
where Pk are autoregressive expert models and k⋆(d) is the active expert with the highest level of expertise for dimension d.
6 DERIVATIVES
We provide here the derivatives of the log-likelihood in the expertise-weighted compositional model (4) with respect to the expert parameters.
6.1 BERNOULLI MODEL
The Bernoulli log-likelihood is
f(µ) = x logµ+ (1− x) log(1− µ), where the composition rule for the probability is
µ = ∑ k rkµk, rk = ek∑ k′ ek′ .
6.1.1 DERIVATIVES WITH RESPECT TO THE COMPOSED PROBABILITY
The first and second derivative of the log-likelihood with respect to the composed probability are df
dµ =
x µ − 1− x 1− µ = x− µ µ(1− µ) ,
d2f
dµ2 = − x µ2 − 1− x (1− µ)2 = − (x− µ) 2 µ2(1− µ)2 .
6.1.2 DERIVATIVES WITH RESPECT TO THE EXPERT PROBABILITIES
The first and second derivative of the composed probability with respect to the expert probabilities are
dµ
dµk = rk,
d2µ dµ2k = 0.
Consequently, the derivatives of the log-likelihood with respect to the expert probabilities are df
dµk =
df dµ · dµ dµk = rk x− µ µ(1− µ) ,
d2f dµ2k = d2f dµ2 · ( dµ dµk )2 + df dµ · d 2µ dµ2k = −r2k (x− µ)2 µ2(1− µ)2 .
We see that d2f/dµ2k < 0 for µ ∈ (0, 1), i.e., the log-likelihood is a strictly concave function of µk.
6.1.3 DERIVATIVE WITH RESPECT TO THE LEVELS OF EXPERTISE
The derivative of the composed probability with respect to the levels of expertise is
dµ dek =
µkE − ∑ ek′µk′
E2 = µk − µ E ,
where E = ∑
k′ ek′ . The derivative of the log-likelihood with respect to the levels of expertise can be computed as
df
dek =
df dµ · dµ dek .
6.2 GAUSSIAN MODEL
The Gaussian log-likelihood is
f(µ, v) = − (x− µ) 2 2v − 1 2 log(v)− 1 2 log(2π),
where the composition rules for the mean and variance are µ = ∑ k rkµk, v = ∑ k rk(vk + µ 2 k)− µ2, rk = ek∑ k′ ek′ .
6.2.1 DERIVATIVE WITH RESPECT TO THE COMPOSED MEAN AND VARIANCE
The derivative of the log-likelihood with respect to the composed mean and variance are
df dµ = x− µ v , df dv = (x− µ)2 2v2 − 1 2v = (x− µ)2 − v 2v2 .
6.2.2 DERIVATIVE WITH RESPECT TO THE LEVELS OF EXPERTISE
The derivative of the composed mean and variance with respect to the levels of expertise are
dµ dek =
µkE − ∑ ek′µk′
E2 = µk − µ E ,
dv
dek =
qkE − ∑ ek′qk′
E2 − 2µ dµ dek = qk − q E − 2µµk − µ E = vk − v + (µk − µ)2 E ,
whereE = ∑
k′ ek′ and qk = vk+µ 2 k, q = v+µ 2. The derivative of the log-likelihood with respect to the levels of expertise can be computed as
df
dek =
df dµ · dµ dek + df dv · dv dek .
7 NUMERICAL OPTIMIZATION
For binary data, the log-likelihood of the smoothed model is a concave function of µk(d), see Section 6.1.2. We could therefore in principal perform an optimization for the experts opinions using Newton’s method. There are a few complications though. One problem is that the second derivative is proportional to the squared responsibility and hence close to 0 if the level of expertise is small. Consequently, template updates in regions with low expertise would be unstable. To deal with that we could add a penalty on the squared log-odds for example. Another problem is that the Newton steps may lead to probability estimates outside of [0, 1]. This can be dealt with by pulling the estimates back into the unit interval. Note that working on the log-odds scale is not possible because the log-likelihood of our model is not concave in the expert log-odds. Because of these complications we use the simple, fast and robust heuristic (5) instead of Netwon’s method.

The goal of this paper is to learn “ a collection of experts that are individually
meaningful and that have disjoint responsibilities.” Unlike a standard mixture model, they “use a different mixture for each dimension d.” While the results seem promising, the paper exposition needs significant improvement.
Comments:
The paper jumps in with no motivation at all. What is the application, or even the algorithm, or architecture that this is used for? This should be addressed at the beginning.
The subsequent exposition is not very clear. There are assertions made with no justification, e.g. “the experts only have a small variance for some subset of the variables while the variance of the other variables is large.”
Since you’re learning both the experts and the weights, can this be rephrased in terms of dictionary learning? Please discuss the relevant related literature.
The horse data set is quite small with respect to the feature dimension, and so the conclusions may not necessarily generalize.