text:
\subsection{Wavenet}

Wavenet \cite{VanDenOord2016} is an autoregressive sequential model originally used to model speech. The core of Wavenet is the Temporal Convolutional Network (TCN) \cite{Bai2018} which uses ResNet \cite{He2016} to enable training of deep networks. Wavenet is built up from a series of so-called Wavenet blocks, where each block consists of a residual connection and an identity connection, respectively.  The residual connection in turn consists of three parts: a dilated convolution, a gate-like activation function, and a final $1{\times}1$ convolution. This can be summarized as
\begin{equation}
\text{WavenetBlock}(x) =  \text{Conv}_{1{\times}1}(\tanh(\text{DilatedConv}(x)) \odot \sigma(\text{DilatedConv}(x))) + x,
\end{equation}
where $\odot$ denotes element-wise multiplication and $\sigma$ denotes the sigmoid activation function. The dilated convolution ensure that the receptive field of  Wavenet can increase exponentially with only a linear increase in the number of parameters \cite{Yu2016}. This makes them viable as the core element in autoregressive models.

\subsection{Hierarchical Variational Autoencoders}

A hierarchical variational autoencoder models the data $y$ (in general not necessarily of sequential nature) by introducing hierarchical stacked latent variables, $\{z^{(l)} \}_{l=1}^L$, in $L$ layers. Using stacked latent variables was proposed along with the original variational autoencoders by \citet{Rezende2014} and \citet{Kingma2014}.  However, the full extent of these hierarchical models was better utilized by \citet{Sonderby2016}, where the approximate posterior is linked to the prior in the so called ladder variational autoencoders (see \Cref{fig:ladder_network}). This idea has since then proved to be fertile ground for numerous other architectures, see e.g. ResNet VAE \cite{Kingma2016} and BIVA \cite{Maaloe2019}.

The generative distribution of a hierarchical VAE can be expressed as
\begin{equation}
\label{eq:gen_ladder}
p(y) = \expect_{p} \bigg  [ p(y \given z^{(1)},h^{(1)}) \prod_{l=1}^{L} p(z^{(l)} \given  h^{(l)}) \bigg ],
\end{equation}
where $p(z^{(l)} \given h^{(l)})$ is modeled with a parametric distribution and $h^{(l)}$ are intermediate features extracted from latent variables higher up in the hierarchy (see right hand side of \Cref{fig:ladder_network}). The parametric distribution is typically a diagonal Gaussian distribution due to its compatibility with the reparametrization trick \cite{Rezende2014,Kingma2014}.

The expectation in \Cref{eq:gen_ladder} is typically intractable and a common approximation is to use Monte Carlo sampling. As for the ordinary VAE, we can decrease the Monte Carlo variance of the log-likelihood for the generative model by using a parameterized approximate posterior distribution,~$q$. Using this approximate posterior and Jensen's inequality, we can bound the log-likelihood from below at the cost of introducing some bias. This yields the following evidence lower bound (ELBO)
\begin{equation}
\mathcal{L}_\text{ELBO}(y;\theta) =  \expect_{q} \bigg [ \log p(y \given z^{(1)}, h^{(1)}) -  \sum_{l=1}^{L} D_\text{KL} \left(q(z^{(l)}) || p(z^{(l)} \given h^{(l)}) \right ) \bigg ],
\label{eq:elbo}
\end{equation}
where $D_\text{KL}$ denotes the Kullback-Lieber divergence. The expectation is evaluated with respect to $q$ and is most commonly approximated with a single Monte Carlo sample. However, it is possible to use more samples as is done in importance weighted autoencoders \cite{Burda2015}.

The approximate posterior's dependence on $y$ is implemented by extracting a bottom-up hierarchy of features, $d^{(l)}$, where each successive layer of features depends the previous layer, $d^{(l-1)}$. As already hinted at, the link between the prior and the approximate posterior has proven to be essential for efficient training of these models. The implementation of this link varies between different versions of hierarchical VAEs, but can be represented by letting the posterior also depend on the extracted features $h^{(l)}$ from the generative network (see \Cref{fig:ladder_network}). The parameters of the approximate posterior are optimized jointly with the parameters of the generative model.

Without any additional tricks, the hierarchical VAE suffers from \emph{posterior collapse}, a state during the optimization where the model gets stuck in a local minimum and the KL units do not transfer any information  \cite{Sonderby2016, Kingma2016, Goyal2017, Bowman2016}. This state can be circumvented through a number of different strategies, e.g. \emph{KL-annealing} \cite{Sonderby2016, Bowman2016}, \emph{free bits} \cite{Kingma2016} or auxiliary loss functions as in Z-forcing \cite{Goyal2017}.

\begin{figure}[t]
	\centering
	\begin{subfigure}[t]{0.46\textwidth}
		\centering
		\scalebox{0.5}{\input{proposed_network_ladder.tikz}}
		\caption{Hierarchical VAE. The vertical dashed line was not present in the original ladder VAE \cite{Sonderby2016} but it was used in the ResNet VAE \cite{Kingma2016}. }
		\label{fig:ladder_network}
	\end{subfigure}
	\hspace{0.05\textwidth}
	\begin{subfigure}[t]{0.46\textwidth}
		\centering
		\scalebox{0.5}{\input{proposed_network.tikz}}
		\caption{Multiscale VAE. The horizontal dashed line enables the useful autoregressive relationship. }
		\label{fig:our_ladder_network}
	\end{subfigure}
	\caption{\textbf{\textcolor{red}{Approximative posterior model}}, \textbf{\textcolor{blue}{generative model}} and \textbf{common features} for hierarchical VAEs (\subref{fig:ladder_network}) and our proposed model (\subref{fig:our_ladder_network}). The dotted line encodes the KL divergence between the prior distribution and the approximate posterior. Note that during likelihood evaluation $z^{(l)}$ is sampled from the posterior and no sampling is done from the prior.}

\end{figure}

\section{Model}

The idea underlying our proposed model is to combine convolutional neural networks and striding with sequential modeling and to achieve depth in time. The core of our model is a hierarchical VAE where we condition the latent states on the previously observed data in an autoregressive fashion, as visualized in \Cref{fig:our_ladder_network}. We refer to the proposed model as the multiscale VAE. Analogously to the hierarchical VAE, the model can roughly be split into to a bottom-up and a top-down network. The  bottom-up network serve both the autoregressive conditioning network and the approximate posterior inspired by the work of \citet{Aksan2019}.

The bottom-up network produce a set of features that is divided into hierarchical layers, $d^{(l)}$. Each hierarchical layer (except for the first layer) consists of a downsampling layer followed by a Wavenet,
\begin{subequations}
	\begin{align}
	\label{eq:top_down_in}
	d^{(1)}&= \text{Wavenet}(y), \\
	d^{(l)} &= \text{Wavenet}(\text{Downsample}(d^{(l-1)})), \qquad \forall l = 2,\dots,L,
	\end{align}
\end{subequations}
where Downsample is implemented using an ordinary convolutional layer with strides.

The top-down part of the model follows the top-down network of the hierarchical VAE with the additional conditioning and upsampling. The conditioning is performed by concatenating the output from the previous hierarchical layer, $g^{(l)}$, with the delayed features of the bottom-up network, followed by a $1{\times}1$ convolution
\begin{subequations}
	\begin{align}
	\label{eq:top_down_cond}
	h^{(l)} &= \text{Wavenet}_{1{\times}1}([g^{(l)}, \text{Delay}(d^{(l)})]), \qquad \forall l = 1,\dots,L-1,\\
	h^{(L)} &= \text{Wavenet}_{1{\times}1}(\text{Delay}(d^{(L)})),
	\end{align}
\end{subequations}
where Delay delays $d^{(l)}$ one step and adds a zero in the beginning to keep the length, $[ \cdot ]$ denotes concatenation and $\text{Wavenet}_{1{\times}1}$ denotes a Wavenet where the dilated convolutions are replaced with $1{\times}1$ convolutions. The prior distribution for the latent variable is given by
\begin{equation}
\label{eq:top_down_dist}
p(z^{(l)} \given h^{(l)}) = \mathcal{N}(z^{(l)} \given \text{Linear}( h^{(l)}),  \text{softmax}(\text{Linear}(h^{(l)}))^2 ),
\end{equation}
where Linear denotes a single $1{\times}1$ convolution without an activation function. Finally, the output of the top-down layer is calculated as,
\begin{subequations}
	\begin{align}
	\label{eq:top_down_out}
	g^{(0)} &= \text{Wavenet}_{1{\times}1}([z^{(1)}, h^{(1)}]), \\
	g^{(l-1)} &= \text{Upsample}(\text{Wavenet}_{1{\times}1}([z^{(l)}, h^{(l)}])), \qquad \forall l = 2,\dots,L,
	\end{align}
\end{subequations}
where the Upsample layer is implemented using transposed convolution \cite{Dumoulin2018}.

The predictive distribution is chosen as $p(y \given z^{(1)}, h^{(l)}) = p(y \given \text{Linear}(g^{(0)}))$. The shape of the predictive distribution is chosen with respect to the problem at hand --- Gaussian for real valued data and Bernoulli for binary data. We also consider a Gaussian mixture model (GMM) and a Bernoulli mixture model (BMM). See \Cref{sec:hyperparameters} for details of this parametrization.

The approximate posterior is expressed using the previously defined features $d^{(l)}$ and $h^{(l)}$ as
\begin{equation}
q(z^{(l)}) = \mathcal{N}(z^{(l)} \given  \text{Linear}([h^{(l)},d^{(l)}]),  \text{softmax}( \text{Linear}([h^{(l)},d^{(l)}])^2 ), \qquad \forall l = 1,\dots,L.
\end{equation}
Note that $d^{(l)}$ here do not include the delay used for the generative network. Also note that we do not calculate $q$ as a combination of $p$ and another normal distribution, as advocated by \citet{Sonderby2016} and \citet{Aksan2019}. Instead $q$ is parameterized independently of the common features, similar to \citet{Kingma2016}.

\Cref{fig:gen_struct,fig:latent_levels} visualize the model in an alternative fashion compared to \Cref{fig:our_ladder_network}. \Cref{fig:gen_struct} resolves the temporal dimension and \Cref{fig:latent_levels} visualizes a hierarchical layer locally in time. The parameters of this model (both the generative model and the approximate posterior) are the parameters for the Wavenets, the Linear layers, the Upsample layers and the Downsample layers.

\begin{figure}[ht]
	\centering

	\begin{subfigure}[t]{0.46\textwidth}
		\resizebox{0.9\textwidth}{!}{\input{deep_seq_ar.tikz}}
		\caption{The hierarchical multiscale structure of the generative model for a sample with four timesteps. This structure repeats to the right for longer sequences.}
		\label{fig:gen_struct}
	\end{subfigure}
	\hspace{0.05\textwidth}
	\begin{subfigure}[t]{0.46\textwidth}
		\resizebox{0.9\textwidth}{!}{\input{latent_level.tikz}}
		\caption{A close-up of the dashed square in \Cref{fig:gen_struct} for the generative model. Note that $\tau$ iterates at halfspeed compared to $t$ due to the downsampling.}
		\label{fig:latent_levels}
	\end{subfigure}

	\caption{An alternative description of the generative part of the model in \Cref{fig:our_ladder_network} where the temporal dimension is explicitly shown. }
	\label{fig:gen_struct_full}
\end{figure}

\paragraph{Linear free bits}  The objective we use to train the model is the ELBO in \Cref{eq:elbo}. To avoid posterior collapse we use an alternative to free bits \cite{Kingma2016}. The ordinary free bits sets a minimum value of how low the KL units can be minimized by altering the ELBO objective \Cref{eq:elbo} to
\begin{equation}
\mathcal{L}_\text{FB} = \expect_q \log p(y \given z) - \sum_i \text{maximum} \left ( D_\text{KL} \left ( q(z_i) || p(z_i) \right ), \lambda \right ),
\end{equation}
where $\lambda$ denotes the free bits threshold and $i$ iterates through all latent variables, $z$. To smoothen the transition from where the threshold is active to where it is not, we propose a linear smoothing of the free bits objective. Hence, we define linear free bits
\begin{equation}
\mathcal{L}_\text{LFB} = \expect_q \log p(y \given z) - \sum_i  \text{StopGradient} \left ( \text{minimum} \left ( \frac{D_\text{KL} \left ( q(z_i) || p(z_i) \right ) }{\lambda}  , 1  \right )   \right )  D_\text{KL} \left ( q(z_i) || p(z_i) \right ),
\end{equation}
where $\text{StopGradient}(\cdot)$ ensures that no gradient information is passed through the function in the backward pass while acting as the identity function in the forward pass.

\section{Experiments}

We experiment on three different kinds of hierarchically structured sequential data: speech, handwritten text and polyphonic music, briefly introduced below. For all experiments we use linear free bits and the Adam optimizer with a fixed learning rate of $10^{-4}$ for speech and handwritten text and $10^{-3}$ for the polyphonic music data. For free bits we used a scheme to smoothly decay the threshold of free bits exponentially over the course of training at a rate of halving the threshold every $300\thinspace000$ iterations. We train using early stopping as stopping criteria on a validation split. For exact hyperparameter settings and additional implementation details we refer to \Cref{sec:hyperparameters}.

\paragraph{Speech} Blizzard contains $300$ hours of raw waveform speech from a single speaker and originates from the 2013 Blizzard challenge \cite{Karaiskos2013}. The raw signal is split into $0.5$ second ($8\thinspace000$ samples) long sequences. We divide these sequences into frames of $25$ samples each to exploit our model's longer memory. Previous research on this dataset used frames of $200$ samples. Additionally, we remove some artifacts from the mp3 decoding that appear before and after every sequence (we did not observe any effect because of this but it could be a potential problem). The remaining preprocessing is performed according to \citet{Aksan2019}. The smaller framesize increases the computational complexity of the model compared to that of the STCN. We compensate for this by using fewer filters for the Wavenet. This way the computational complexity is approximately the same for the proposed model as for the (moderate) STCN, while at the same time using slightly fewer parameters. For this dataset we only consider a Gaussian distribution for the final prediction.

\paragraph{Handwritten text} IAM-OnDB\cite{Liwicki2005_iamondb} and Deepwriting \cite{Aksan2018_DeepWriting} are handwritten text datasets where every sample consists of the pen position ($\mathbb{R}^2$) and a binary variable indicating whether the pen was lifted after this position or not. The IAM-OnDB consists of $13\thinspace000$ lines of handwritten text by more than $200$ writers. The Deepwriting dataset is an extension to IAM-OnDB, of roughly the same size, where some of the noisy samples have been replaced with new writers. We use the same preprocessing as  \citet{Chung2015} and \citet{Aksan2019}. For this dataset we only consider GMM for the final prediction of the pen position as done in previous work.

\paragraph{Polyphonic music} Piano-midi.de, Nottingham, MuseData  and JSB Chorales are four midi piano datasets presented in \citet{BoulangerLewandowski2012}. The sequences contain information about which keys, out of the 88 keys on the keyboard, are pressed during each time step. The temporal resolution corresponds to eight notes for all dataset except for JSB Chorales that use quarter notes. Each of the datasets express different levels of complexity and they are all quite small. To somewhat alleviate this we make use of data augmentation, shifting the key and starting timestep randomly. For this dataset we consider both Bernoulli and BMM as the final prediction distribution.

\subsection{Results}

The ELBO evaluated on test data for the speech and handwritten text is presented in \Cref{tab:results_speech_text}. Note that for the Blizzard dataset our model makes use of significantly fewer parameters than the large STCN model and should rather be compared with the moderately sized STCN. Also for the handwritten text datasets we use fewer parameters than STCN. \Cref{fig:samples} illustrate samples generated from the model for the IAM-OnDB dataset.

The results on the polyphonic music datasets are presented in \Cref{tab:results_piano}, where both the ELBO and an importance weighted estimate of the log-likelihood are included. The model performs very poorly on the polyphonic music dataset MuseData. We pinpoint this bad performance to a single sequence in the test data. This sequence contains pressed keys in a completely different register (up to 2 octaves higher) than all other sequences (in training, validation and test splits). The inability to model this sequence is not necessarily bad, but can be seen as the model's ability to be used as an outlier detector. If this, according to the model, unlikely sample is removed from the dataset, the likelihood estimated on training and validation data is in line with what is calculated on the test data.

For the Blizzard, IAM-OnDB and deepwriting datasets the model was stopped before the validation error started to increase after ${\sim}400$, ${\sim}2000$ and ${\sim}1000$ epochs respectively. The polyphonic music datasets converged in ${\sim}100$ to ${\sim}2000$ epochs for the different datasets. Note that the model performed well with ordinary free bits also but it was easier to tune the threshold parameter with linear free bits.

\begin{wrapfigure}{R}{0.5\textwidth}
	\centering
	\begin{subfigure}[]{0.24\textwidth}
		\includegraphics[width=\linewidth, height=\linewidth]{img/ground_truth.png}
		\caption{Dataset samples}
	\end{subfigure}

	\begin{subfigure}[]{0.24\textwidth}
		\includegraphics[width=\linewidth, height=\linewidth]{img/samples.png}
		\caption{Multiscale VAE}
	\end{subfigure}
	\caption{An illustration of samples from the dataset and generated samples from our model for the IAM-OnDB dataset.}
	\label{fig:samples}
\end{wrapfigure}

\begin{table}[ht]
	\sisetup{table-format = <5, table-number-alignment = center, mode=text}
	\begin{center}
		\caption{Average log-likelihood per sequence comparison for speech and handwritten text. Entries marked with $^*$ are acquired from \citet{Aksan2019}.  }
		\label{tab:results_speech_text}
		{ \small
			\begin{tabular}{l S[table-format = <5] S[table-format = <4] S[table-format = <4]}
				\toprule
				Model & \multicolumn{1}{c}{Blizzard}  & \multicolumn{1}{c}{IAM-OnDB} & \multicolumn{1}{c}{Deepwriting}\\
				\midrule
				HM-LSTM \cite{Chung2016_HMRNN} & \multicolumn{1}{c}{-}  & 1167 & \multicolumn{1}{c}{-} \\
				RNN (GMM)  \cite{Chung2015} & 7413 & 1358 & 528 $^*$ \\
				VRNN (GMM) \cite{Chung2015} & \approx 9392  & \approx 1384 &
				\geq 673 $^*$\\
				SRNN \cite{Fraccaro2016} & \geq 11991 & \multicolumn{1}{c}{-}  & \multicolumn{1}{c}{-}  \\
				Z-Forcing \cite{Goyal2017} & \geq 15430 &   \multicolumn{1}{c}{-} &   \multicolumn{1}{c}{-} \\
				STCN (Moderate) \cite{Aksan2019} & \geq 16288 & \geq 1796  & \geq 797 \\
				STCN (Large) \cite{Aksan2019} & \B \geq 17670 & \multicolumn{1}{c}{-} & \multicolumn{1}{c}{-} \\
				\midrule
				Our model & \geq 17094 & \B \geq 2925 & \B \geq 1094\\
				\bottomrule
			\end{tabular}
		}
	\end{center}
\end{table}

\begin{table}[ht]
	\sisetup{table-format = <-1.2, table-number-alignment = center, mode=text}
	\begin{center}
		\caption{Log-likelihood per timestep comparison for polyphonic music datasets. Models with BMM are using a Bernnolli mixture model with 10 components. The log-likelihood (LL) is estimated using 50 importance weighted samples. Results in parentheses are calculated without the unlikely sample in the test dataset. }
		\label{tab:results_piano}
		{ \small
			\begin{tabular}{l SSSS}
				\toprule
				Model & \multicolumn{1}{c}{Piano-midi.de} & \multicolumn{1}{c}{Nottingham} & \multicolumn{1}{c}{MuseData} & \multicolumn{1}{c}{JSB Chorales} \\
				\midrule
				SRNN \cite{Fraccaro2016} & \geq -8.20 & \geq -2.94  & \geq - 6.28 & \geq -4.74 \\
				STORN \cite{Bayer2014} & \approx -7.13 & \approx -2.85   & \approx - 6.16 & \approx -6.91 \\
				NASMC \cite{Gu2015} &  \approx -7.61 &  \approx  -2.72 &  \approx  6.89 &  \B \approx -3.99 \\
				RNN-RBM \cite{BoulangerLewandowski2012} &  \approx -7.09 &  \approx -2.39 &  \approx -6.01 &  \approx -6.27 \\
				RNN-DBN \cite{Goel2014} &  \approx -7.15 &  \approx -2.54 &  \approx -6.28 &  \approx -5.68 \\
				RNN-NADE \cite{BoulangerLewandowski2012} &  -7.05 & \B -2.31 & \B -5.60 &  -5.56 \\
				\midrule

				Our model (ELBO) & \geq -6.59 & \geq -2.53 & \multicolumn{1}{r}{ $\approx -10161 (-5.64)$ } & \geq -6.97\\
				Our model (LL) & \approx -6.53 & \approx -2.42 &\multicolumn{1}{r}{ $\approx -6877 (-5.56)$ }& \approx -6.70 \\
				Our model (BMM, ELBO) & \geq -6.53 & \geq -2.54& \multicolumn{1}{r}{ $\approx -1764 (-5.62)$ }& \geq -7.01 \\
				Our model (BMM, LL) & \B \approx -6.47 & \approx -2.44& \multicolumn{1}{r}{ $\approx -184 (-5.54)$ } & \approx -6.73 \\
				\bottomrule
			\end{tabular}
		}
	\end{center}
\end{table}

\section{Related work}
\label{sec:related_work}
The stochastic TCN \cite{Aksan2019} is closely related to and used as inspiration for our proposed model. Our model generalize the STCN to the hierarchical VAE in favor of the Ladder VAE and extend it by also incorporating multiple timescales. \citet{Aksan2019} argues that their model also express multiple timescales but we disagree since it is missing the downsampling and upsampling steps. Indeed, the deeper features of the STCN gathers information from a larger receptive field. However, these features do only affect the current time step and are not shared between different timesteps.

Other models that use multiple timescales are HM-RNN \cite{Chung2016_HMRNN}, HS-LSTM \cite{Zhang2018} and the clockwork RNN \cite{Koutnik2014}. Implementation-wise they resemble each other by stacking multiple RNNs on top of each other running at different update rates. Specifically, RNNs higher in the hierarchy are updating their state using a slower update rate than the RNNs closer to the data. However, these structures do not make use of stochasticity.

Stochastic state space models are commonly implemented using VAEs, e.g. SRNN \cite{Fraccaro2016}, VRNN \cite{Chung2015} and STORN \cite{Bayer2014} and they share several similar traits. The core sequential model is an RNN which is extended with Gaussian latent variables. In addition to the forward pass of the state space model, a backward flow is needed for the approximate posterior, which is implemented by an RNN running backwards in time. None of these models adopt deep features or multiple timescale structure.

An alternative to stochastic state space models is to avoid the recurrence of the latent variable. This has the advantage that the approximate posterior can be calculated without a backward flow. Thus, the latent variables are only used to produce a flexible generative distribution at every time step individually (if no upsampling is used). Examples from this class of models are Stochastic TCN \cite{Aksan2019}, RNN-RBM \cite{BoulangerLewandowski2012}, RNN-DBN \cite{Goel2014} and Stochastic Wavenet \cite{Lai2018}. The PixelVAE \cite{Gulrajani2016} is closely related to these models even though it is not used for sequential data. It employs an autoregressive model with a flexible generative distribution.

The generative distribution can be chosen from a large class of models. Stochastic TCN use hierarchical VAEs inspired by Ladder VAE \cite{Sonderby2016}. Another choice is Gaussian mixture models, which can be used either alone or in combination with other models. RNN-RBM and RNN-DBN use Restricted Boltzmann Machines (RBM) \cite{Rumelhart1987} as the core for the predictive distribution, which is suitable for binary data but can be harder to train. In principle, any kind of flexible distribution model can be used --- e.g. other variations of VAEs, normalizing flows or GANs \cite{Ziegler2019,Im2016}.

\section{Conclusion and future work}

The results indicate that our proposed model has a strong inductive bias towards hierarchically structured data as the model fits the data very well while at the same time being more parameter efficient compared to related models. We hypothesis that the proposed model exploit the hierarchical nature of the sequential data similar to how the CNN exploits the hierarchical structure in image data, where the downsampling/pooling layers are essential for the inductive bias of the CNNs. In the future we hope to be able to use the model for other problems in machine learning, e.g. semi-supervised learning or model-based reinforcement learning.

The number of epochs required for the model to converge is quite large, which was also observed in \citet{Sonderby2016}. This might be something that is inherent to the model structure but future research have to answer that. We did however observe that increasing the learning rate significantly sped up the rate of convergence for the polyphonic music dataset while still achieving good results. This indicates that the convergence rate can be improved with properly adjusted hyperparameters, something we have not yet fully explored.

An ablation study could straighten out some of the question marks regarding the effect different parts of the model have on model performance, e.g. the Wavenet structure, free bits, upsampling/downsampling, and other hyperparameters and how these properties interact. This could hopefully also give hints on how to speed up training.

The proposed model is in principle not limited to one dimensional temporal data. Similar to PixelVAE \cite{Gulrajani2016}, our model can be applied to images or even video. Investigating whether the promising results of this model transfers into other regimes of hierarchically structured data, with or without causality, is a most interesting avenue for future work. For example image data where one could investigate the model performance both with and without the autoregressive connection, similar to PixelVAE \cite{Gulrajani2016}.

\section*{Broader Impact}

As argued in this paper we believe that models that use the desired properties possess a stronger inductive bias towards learning from hierarchically structured sequential data, e.g. natural language and speech. We this said we believe that the direct consequences from this paper are, both an ethical and societal perspective, very limited. We hope, however, that the work can contribute to future understanding of sequential data and inspire new research that benefit the larger society.

\section*{References}

\newpage

\section{Introduction}

System identification and sequence modeling with deep learning are two different research areas that essentially solve the same problem, to model a sequence, $y_{1:T}$, for predictive purposes. Even though the methods are very similar there are differences when it comes to what kind of data the method typically is applied to. Whereas system identification traditionally has focused on systems with relatively short memory and small datasets (e.g. systems on the nonlinear benchmark website \citep{NonLinear}), deep learning has focused on the opposite, i.e. systems with long memory and larger datasets (e.g. text modeling). This dichotomy is not a product of any fundamental difference in methodology, but rather an effect of the rapid advancement in computational power that has accompanied the advancements in deep learning.

Neural networks are by no means new to the system identification community, they have in fact a rather rich history \citep{Sjoberg1995}. What deep learning brought with it was instead a new paradigm of model design and implicit regularization. Whereas the system identification community typically view neural networks as black box function approximators, the deep learning community instead augment how these black box models are built up and adapt them to the data, e.g. long-short term memory (LSTM) models \citep{Hochreiter1997}, convolutional neural networks (CNN) \citep{Krizhevsky2012_AlexNet} and temporal convolutional networks (TCN). A very prominent example of the connection between model and data structure is indeed deep CNNs applied to images, where the model mirrors the hierarchical composition of natural images and  locality of low-level features \citep{LeCun2015} while high-level features are represented in a downsampled image. Such hierarchical composition can also be seen in many sequential datasets, e.g. language modeling, handwritten text modeling (see \Cref{fig:data_samples}), human motion tracking or other systems of (hierarchical) switching nature where the high-level features evolve at a slower pace than low-level features \citep{Chung2016_HMRNN, Koutnik2014}.

\begin{figure}[t]
	\centering
	\begin{subfigure}[]{0.24\textwidth}
		\includegraphics[width=\linewidth, height=\linewidth]{img/ground_truth_v3.png}
		\caption{Dataset samples}
		\label{fig:data_samples}
	\end{subfigure}\hfill
	\begin{subfigure}[]{0.24\textwidth}
		\includegraphics[width=\linewidth, height=\linewidth]{img/samples_v3.png}
		\caption{Generated samples}
	\end{subfigure}
	\caption{An illustration of samples from the dataset and generated samples from our model for the IAM-OnDB dataset \citep{Liwicki2005_iamondb}.}
	\label{fig:samples}
\end{figure}

In the light of this we propose a sequential hierarchical latent variable model, represented in \Cref{fig:deep_seq}, that focus on modeling multiscale hierarchically structured sequential data. This can seen as an extension the Stochastic TCN \citep{Aksan2019} which also uses a sequential hierarchical latent variable model although without the multiscale property. The latent variables $z= \{z^{(l)}_{1:T}\}$ are structured sequentially with hierarchical layers, where layer $l$ is downsampled with a factor $S^{(l)}$. To facilitate training we propose to relax the state space formulation similarily to \citet{Aksan2019}, see \Cref{fig:deep_seq_ar_v1}. With this we get a deep sequential model that exploits the hierarchical nature of the data similarly to how a deep convolutional network exploits the nature of images. We realize the model using a combination of a Wavenet model \citep{VanDenOord2016} for the sequential component with a hierarchical variational autoencoder \citep{Rezende2014,Kingma2014} for the latent variables.  We employ the proposed model on two different datasets and show that by including the prior information of the hierarchical nature of the data we can improve the model in terms of both parameter effectiveness and computational complexity.

\begin{figure}[t]
	\centering
	\begin{subfigure}[t]{0.46\textwidth}
		\resizebox{!}{0.6\linewidth}{\input{deep_seq.tikz}}
		\caption{State space form}
		\label{fig:deep_seq}
	\end{subfigure}
	\hspace{0.02\textwidth}
	\begin{subfigure}[t]{0.46\textwidth}
		\resizebox{!}{0.6\linewidth}{\input{deep_seq_ar_v1.tikz}} \hfill
		\caption{Autoregressive form}
		\label{fig:deep_seq_ar_v1}
	\end{subfigure}
	\caption{Two versions of a hierarchical generative model for sequential data, $y$, on state space form (top) and autoregressive form (bottom), with latent variables, $z$. $z^{(2)}$ updates only every other timestep and $z^{(3)}$ updates only every forth. }
\end{figure}

\section{Background}

In this section we will introduce and give some background on the components needed to realize the proposed model.

\subsection{Temporal convolutional network}
\vspace{-0.5em}

The temporal Convolutional Network (TCN) is a type of non-linear autoregressive (NARX) model that makes use  of a deep autoregressive neural network model. \cite{Bai2018} observe that these autoregressive models are easier to train, less sensitive to hyperparameter selection and in many circumstances achieves superior results, compared to deterministic state space models, i.e. LSTMs and RNNs. This can partly be explained by the efficient implementation, both computationally and parameterwise, of the  autoregressive neural networks here realized with convolutions.

The depth of the TCN is achieved by stacking multiple affine transformations intertwined with activation functions. Each affine transformation/activation function pair can be expressed with
\begin{align}
	h_t = \phi(x_{t-k:t-1}),
\end{align}
where $\phi$ combines an affine transformation and a nonlinear activation function of the regression vector, $x_{t-k:t-1}$. The notation $x_{i:j}$ denotes the sequence of $x$ from $i$ to $j$. An alternative formulation of this is to view the entire $h_{1:T}$ as a convolution between $\phi$ and the input $x_{1:T}$. The width of this convolution filter, $k$, corresponds to the so called the \emph{receptive field} of the convolution. We use the notation
\begin{align}
	h_{1:T} = f(\text{Conv}(x_{1:T})),
\end{align}
where Conv stands for the affine transformation and $f$ is an activation function.

The dilated convolution \citep{VanDenOord2016} is an option to the convolution that sets some of the parameters in the affine transformation to zero. With a dilation rate $\mu$ only every $\mu$:th element in the affine transformation is nonzero. By convention $k$ denotes the number of effective parameters in this transformation. Thus, this increase the receptive field of the filter to $k \mu$. By stacking several dilated convolutions with exponentially increasing dilation rate it is possible to produce a model with exponentially long memory (exponentially large respective field) with a linear increase in the number of parameters \citep{Yu2016}. The parameters of the affine transformation are the parameters of the convolutional layer, while both $k$ and $\mu$ are hyperparameters.

\subsection{Wavenet}
\vspace{-0.5em}

One of the most prominent TCNs is Wavenet \citep{VanDenOord2016}. Wavenet is built up of stacked so-called Wavenet blocks, where each block consists of a residual connection and an identity connection (inspired by  ResNet \citep{He2016}).  Each residual connection in turn consists of a sequence of: a dilated convolution, a gate-like activation function, and a final $1{\times}1$ convolution. An $1{\times}1$ convolution is a type of convolution that only operates locally (the filter has size 1). The Wavenet block can be summarized as
\begin{align}
\text{WavenetBlock}(x_{1:T}) &= x_{1:T} +  \nonumber \\
\text{Conv}_{1{\times}1} (
\tanh(&\text{Conv}(x_{1:T})) \odot
\sigma(\text{Conv}(x_{1:T}))),
\end{align}
where $\odot$ denotes element-wise multiplication and $\sigma$ denotes the sigmoid activation function.

When merging two data sequences with the Wavenet model \cite{VanDenOord2016} proposed to consider one of the sequences as the main sequence ($x_{1:T}$) and the other as a conditional sequence ($c_{1:T}$). The two sequences are concatenated (in feature space) forming a new stream and this is used as input to the residual network. The identity connection only passes $x_{1:T}$ forward. Thus, a conditional Wavenet block can be written as,
\begin{align}
	&\text{CondWavenetBlock}(x_{1:T},c_{1:T}) = x_{1:T} +  \nonumber \\ &\text{Conv}_{1{\times}1}(\tanh(\text{Conv}([x_{1:T},c_{1:T}])) \odot \sigma(\text{Conv}([x_{1:T},c_{1:T}]))) ,
\end{align}
where $[\cdot]$ denotes concatenation in the feature dimension. When stacking several such blocks together, all blocks share the same conditioning sequence. \citet{Lai2018} with Stochastic Wavenet and later \citet{Aksan2019} extended these models to also include latent variables similar to the proposed model.

\subsection{Variational autoencoder}
\vspace{-0.5em}

\label{sec:VAE}
A common approach for latent variables in deep learning is to use variational autoencoders (VAE) \citep{Rezende2014,Kingma2014}. This section aims to introduce the notation we use in this paper and for a more in depth and pedagogical description of the VAE see \citet{Kingma2019}. The idea of a VAE is to model the distribution of $y$ with a generative model as
\begin{equation}
\label{eq:vae_gen}
p(y) = \expect \big  [ p(y \given z) p(z) \big ],
\end{equation}
with
\begin{align*}
	p(y \given z) &= \mathcal{N}(y \given \mu_y(z), \Sigma_y(z)), \\
	p(z) &= \mathcal{N}(z\given 0, I),
\end{align*}
where $\mu_y(z)$ and $\Sigma_y(z)$ are modeled as neural networks.

In the VAE setting the expectation in \eqref{eq:vae_gen} is estimated with Monte Carlo samples and (amortized) variational inference. The variational inference is done by introducing a parameterized approximate posterior distribution,~$q$,
\begin{align*}
q(z) = \mathcal{N}(z \given \mu_q(y), \Sigma_q(y)),
\end{align*}
where $\mu_q(y)$ and $\Sigma_q(y)$ are also modeled as neural networks. It is also possible to condition the VAE on some arbitrary data, $c$, in which case the prior is similarly expressed with neural networks,
\begin{align}
p(z \given c) &= \mathcal{N}(z \given \mu_p(c), \Sigma_p(c)).
\end{align}
The approximate posterior $q(z)$ and the generative distribution will then also depend on $c$.

Using this approximate posterior and Jensen's inequality, we can bound the log-likelihood from below. This gives us the evidence lower bound (ELBO)
\begin{equation}
\mathcal{L}_\text{ELBO}(y;\theta) =  \mathbb{E}_{q} \big [ \log p(y \given z) \big ] -  D_\text{KL} \left(q(z) || p(z) \right ) ,
\label{eq:elbo}
\end{equation}
where $D_\text{KL}$ denotes the Kullback-Lieber divergence and $\theta$ are the parameters of both the approximative posterior and the generative model. The expectation here is evaluated with respect to $q$ and it is most commonly approximated with a single Monte Carlo sample. The ELBO objective is used to update both the parameters of the generative model and the approximate posterior.

The variational RNN (VRNN) \citep{Chung2015}, stochastic RNN (SRNN) \citep{Fraccaro2016} and STORN \citep{Bayer2014} are all examples of VAEs that have been adapted to sequential data. The core of these models is an RNN which is extended with a latent variable. These models require a backward flow for inference which is implemented by an RNN running backwards in time.

\subsection{Hierarchical variational autoencoders}
\vspace{-0.5em}

A hierarchical variational autoencoder extends the VAE by introducing hierarchically stacked latent variables, $\{z^{(l)} \}_{l=1}^L$, in $L$ layers. The use of stacked latent variables was proposed along with the original variational autoencoder, though with small improvements compared to the single layer VAE. The hierarchical models were later improved by \citet{Sonderby2016} with the so called ladder VAE, where the approximate posterior is linked to the generative distribution (see \Cref{fig:ladder_network}). This idea has since then proved to be fertile ground for numerous other architectures, e.g. ResNet VAE \citep{Kingma2016}, BIVA \citep{Maaloe2019} and NVAE \citep{vahdat2020}.

The approximate posterior's dependence on $y$ is implemented by extracting a bottom-up hierarchy of features, $d^{(l)}$, where each successive layer of features depends on the previous layer, $d^{(l-1)}$. The link between the generative and the approximate posterior distribution has proven to be essential for efficient training of these models. Although, the implementation details for this link varies it can represented as \Cref{fig:ladder_network} in that the posterior also depends on the extracted features $h^{(l)}$ from the generative model. The parameters of the approximate posterior are optimized jointly with the parameters of the generative model.

\begin{figure}[t]
	\centering
	\begin{subfigure}[t]{0.46\textwidth}
		\centering
		\hspace{0.4cm}\scalebox{0.7}{\input{proposed_network_ladder.tikz}}
		\caption{Hierarchical VAE. The vertical dashed line was not present in the original ladder VAE \citep{Sonderby2016} but it was used in the ResNet VAE \citep{Kingma2016}. }
		\label{fig:ladder_network}
	\end{subfigure}

	\begin{subfigure}[t]{0.46\textwidth}
		\centering
		\scalebox{0.7}{\input{proposed_network.tikz}}
		\caption{The proposed model, the horizontal dashed line enables the useful autoregressive relationship. }
		\label{fig:our_ladder_network}
	\end{subfigure}
	\caption{\textbf{\textcolor{red}{Approximative posterior model}}, \textbf{\textcolor{blue}{generative model}} and \textbf{common features} for hierarchical VAEs (\subref{fig:ladder_network}) and our model (\subref{fig:our_ladder_network}). The dotted line encodes the KL divergence between the prior distribution and the approximate posterior. Note that during likelihood evaluation $z^{(l)}$ is sampled from the posterior.}
\end{figure}

\vspace{-0.5em}
\section{Model}
\vspace{-0.5em}

In our model we combine temporal convolutional networks and striding as building blocks to instantiate \Cref{fig:deep_seq_ar_v1}. We combine this with a hierarchical VAE where we condition the latent states on the previously observed data in an autoregressive fashion, as visualized in \Cref{fig:gen_struct}. Analogously to the hierarchical VAE, the model can roughly be split into to a bottom-up (from data to more abstract features) and a top-down (from abstract features to predictions) network \citep{Maaloe2019}. The autoregressive features will also be used as features for the approximate posterior, inspired by the work of \citet{Aksan2019} which this work extends.

The bottom-up network produces a set of features that is divided into hierarchical layers. This network also downsamples the signal in between every layer to capture the multiscale property of the model. To avoid excessive cluttering we use the notation $d^{(l)} = d^{(l)}_{1:S^{(l)}:T}$ to denote the bottom-up features which have been downsampled by a total factor of $S^{(l)}$. The downsample is implemented inside the first WavenetBlock function where the residual connection simply uses strided convolutions and the identity connection use the average. We use superscript $D$ to denote Wavenet with downsampling.
\begin{subequations}
\begin{align}
	\label{eq:top_down_in}
	d^{(1)}&= \text{Wavenet}(y), \\
	d^{(l)} &= \text{Wavenet}^\text{D}(d^{(l-1)}), \qquad l \neq 1.
\end{align}
\end{subequations}

The features of the top-down network, $h^{(l)}~=~h^{(l)}_{1:S^{(l)}:T}$, follows the hierarchical VAE with an additional conditioning and upsampling. The conditioning is performed by concatenating the output from the previous hierarchical layer, which we denote $g^{(l)} = g^{(l)}_{1:S^{(l)}:T}$, with the delayed features of the bottom-up network, followed by a $1{\times}1$ convolution
\begin{subequations}
\begin{align}
	\label{eq:top_down_cond}
	h^{(l)} &= \text{Wavenet}_{1{\times}1}([g^{(l)}, \text{Delay}(d^{(l)})]), \qquad l \neq L,\\
	h^{(L)} &= \text{Wavenet}_{1{\times}1}(\text{Delay}(d^{(L)})),
\end{align}
\end{subequations}
where $[\cdot]$ denotes concatenation in feature domain. The Delay function delays $d^{(L)}$ one step at the current downsample level and prepends zeros, i.e.
\begin{align}
	\text{Delay}(d_{1:S:T}) = [0, d_{1:S:T-S}],
\end{align}
here the concatenation is done in time domain. Finally $\text{Wavenet}_{1{\times}1}$ denotes a Wavenet where all the dilated convolutions are replaced with $1{\times}1$ convolutions.

The prior distribution for the latent variable is given by
\begin{equation}
	\label{eq:top_down_dist}
	\begin{split}
	p(z^{(l)} \given h^{(l)}) =&\\
	 \mathcal{N} \bigg (z^{(l)}& \bigg \vert
	\begin{split}
	&\mu = \text{Conv}_{1{\times}1}( h^{(l)}),\\
	&\Sigma = \text{diag}(\text{softplus}(\text{Conv}_{1{\times}1}(h^{(l)}))^2)
	\end{split} \bigg ),
	\end{split}
\end{equation}
Finally, the output of the top-down layer is calculated as,
\begin{subequations}
\begin{align}
	\label{eq:top_down_out}
	g^{(0)} &= \text{CondWavenet}_{1{\times}1}( h^{(1)},z^{(1)}),\\
	g^{(l-1)} &= \text{CondWavenet}^\text{U}_{1{\times}1}(h^{(l)},z^{(l)}), \,\,\, l \neq 1,
\end{align}
\end{subequations}
thus, the latent variables are used as a conditioning sequence in the Wavenet. The Upsampling here, $\text{CondWavenet}^\text{U}$, is similar to the downsample made as a part of the Wavenet. The identity connection is hear upsampled with nearest neighbor and the residual with strided transpose convolutions. The predictive distribution is chosen as $p(y \given \text{Conv}_{1{\times}1}(g^{(0)}))$. The shape of the predictive distribution is chosen with respect to the problem -- Gaussian for real valued data and Bernoulli for binary data or mixtures thereof.

The approximate posterior is expressed using the previously defined features $d^{(l)}$ and $h^{(l)}$ as
\begin{equation}
	\begin{split}
	q(z^{(l)}) = &\\
	\mathcal{N} \bigg ( &z^{(l)} \bigg \vert
	\begin{split}& \mu = \text{Conv}_{1{\times}1}([h^{(l)},d^{(l)}]),  \\
	& \Sigma = \text{diag}(\text{softplus}( \text{Conv}_{1{\times}1}([h^{(l)},d^{(l)}])^2 )
	\end{split}
	\bigg ).
	\end{split}
\end{equation}
We do not calculate $q$ as a combination of $p$ and another normal distribution, as advocated by \citet{Sonderby2016} and \citet{Aksan2019}. Instead $q$ is parameterized independently of the common features, similar to \citet{Kingma2016}.

\Cref{fig:gen_struct,fig:latent_levels} visualize the model in an alternative fashion compared to \Cref{fig:our_ladder_network}. \Cref{fig:gen_struct} resolves the temporal dimension and \Cref{fig:latent_levels} visualizes a hierarchical layer locally in time. The parameters of this model (both the generative model and the approximate posterior) are the parameters for the Wavenets, the Convolutional layers, the Upsample layers and the Downsample layers.

Although the stochastic TCN (STCN) \citep{Aksan2019} is very related to our model, there are some distinctions. We generalize the STCN with the hierarchical VAE in favor of the Ladder VAE and also extend it by incorporating multiple timescales. \citet{Aksan2019} argue that their model expresses multiple timescales but we disagree since it is missing the downsampling and upsampling steps. Indeed, the deeper features of the STCN gathers information from a larger receptive field. However, these features do only affect the current time step and are not shared between different timesteps.

\begin{figure}[ht]
	\centering

	\begin{subfigure}[t]{0.46\textwidth}
		\resizebox{0.9\textwidth}{!}{\input{deep_seq_ar.tikz}}
		\caption{The hierarchical structure of the generative model for a sample with four timesteps. This structure repeats to the right for longer sequences. The gray arrows corresponds to the bottom-up paths. Note that the $d$s are delayed with $S^{(l)}$. }
		\label{fig:gen_struct}
	\end{subfigure}
	\hspace{0.05\textwidth}
	\begin{subfigure}[t]{0.46\textwidth}
		\resizebox{0.9\textwidth}{!}{\input{latent_level.tikz}}
		\caption{A close-up of the dashed square in \Cref{fig:gen_struct} for the generative model. Here $r$ is the receptive field of the Wavenet model. In this instance $S^{(l)} = 2$. }
		\label{fig:latent_levels}
	\end{subfigure}

	\caption{An alternative description of the generative part of the model in \Cref{fig:our_ladder_network} where the temporal dimension is explicitly shown. }
	\label{fig:gen_struct_full}
\end{figure}

\section{Experiments}

We conduct experiments on two hierarchically structured sequential datasets, raw waveform speech (Blizzard) and handwritten text (IAM-OnDB). For all experiments we use linear free bits (see \Cref{sec:hyperparameters}) and the Adam optimizer with an exponentially decaying learning rate of $5\cdot10^{-4}$. For free bits we used a scheme to smoothly decay the threshold of free bits exponentially over the course of training at a rate of halving the threshold every $300\thinspace000$ iterations. We train for $100$ epochs and do not see any tendencies of overfitting on the training data. Finally, each bottom-up Wavenet consists of four Wavenet blocks and each top-down constsits of two Wavenet blocks. The number of Wavenet blocks are thus the same in the bottom-up and the top-down networks. For exact hyperparameter settings and additional implementation details we refer to the appendix and code\footnote{\url{https://github.com/carl-andersson/MS-STCN}}.

{\bf Blizzard} is a raw waveform speech dataset that originates from the 2013 Blizzard challenge \citep{Karaiskos2013}. The raw signal is split into $0.5$ second ($8\thinspace000$ samples) long sequences. We experiment by dividing these sequences into frames of $200$ samples down to $2$ to really enable the long memory requirements and hierarchical decomposition of the data. Previous research on this dataset used frames of $200$ samples. Additionally, we remove some artifacts from the mp3 decoding that appear before and after every sequence. The remaining preprocessing is performed according to \citet{Aksan2019}.

The smaller framesize increases the computational complexity of the model compared to that of the STCN if we were to use the same number of filters. We compensate for this by decreasing the number of filters in our model for layers with small $l$ compared to those of the STCN. This way the computational complexity is approximately the same for the all of proposed models as for the (moderate) STCN, while at the same time using significantly fewer parameters. In the case of frame width of 200 we keep all the filters the same size which to be comparable with STCN.

The ELBO evaluated on test data for the Blizzard dataset is presented in \Cref{tab:results_speech_text}. Our model without striding is essentially a reimplementation of STCN using the model setup described in this paper. As a control experiment we also report the result with our model without the stochastic latent variables (i.e. only deterministic autoregressive). However, this control showed instabilities and eventually diverged so we report the best fit we got before it diverged.  This divergence might be an indication that the latent variables are somehow regularizing the model. In \Cref{tab:results_speech_text_kl} we also report the amount of information that is used in the KL-term for the different layers. Results show that KL units on all levels are being used.

{\bf IAM-OnDB} \citep{Liwicki2005_iamondb} is a handwritten text dataset where every sample consists of the pen position ($\mathbb{R}^2$) and a binary variable indicating whether the pen was lifted after this position or not. We use the same preprocessing as \citet{Chung2015} and \citet{Aksan2019}. However, we noted that the discretization of the pen position in some cases led to models that greatly overestimated the likelihood fit to the data. To cope with these discretization artifacts  we added artificial uniform measurement noise to the pen position.

Samples from the model trained on IAM-OmDB (\Cref{fig:samples}) shows complex sequences were it is even possible to recognize a few words. However, the added noise means that the likelihood estimates of the model are not easily comparable with previous results on this dataset.

\begin{table}[ht]
	\sisetup{table-format = <5, table-number-alignment = center, mode=text}
	\begin{center}
		\captionsetup{width=.9\linewidth}
		\caption{Average ELBO per sequence comparison for speech with varying frame width (FW) and with and without down-/upsampling (multiscale). }
		\label{tab:results_speech_text}
		{ \small
			\begin{tabular}{l S[table-format = <5] S[table-format = <4] S[table-format = <4]}
				\toprule
				Model & \multicolumn{1}{c}{Blizzard} \\
				\midrule
				RNN (GMM)  \citep{Chung2015} & 7413 \\
				Wavenet (GMM) \cite{Aksan2019} & 8190 \\
				VRNN (GMM) \citep{Chung2015} & \approx 9392 \\
				SRNN \citep{Fraccaro2016} & \geq 11991 \\
				Z-Forcing \citep{Goyal2017} & \geq 15430 \\
				STCN (Moderate) \citep{Aksan2019} & \geq 16288 \\
				STCN (Large) \citep{Aksan2019} &  \geq 17670 \\
				\midrule
				Our model, FW=200 & \geq 13645 \\
				Our model, FW=200 w/o multiscale  & \geq 15746 \\
				Our model, FW=25 & \geq 17060 \\
				Our model, FW=25 w/o multiscale   & \geq 16562  \\
				Our model, FW=2 &  \B \geq 18809 \\
				Our model, FW=2 w/o multiscale & \geq 18372 \\
				Our model, FW=2 w/o latent variables & 18523 \\
				\bottomrule
			\end{tabular}
		}
	\end{center}
\end{table}

\vspace{-0.5em}
\section{Conclusion and future work}
\vspace{-0.5em}

In this paper we argue that a deep learning model is not that different from a structured autoregressive model. However, the philosophy behind the model and the problems it is applied to, are significantly different from a typical system identification model. The model we proposed has shown to model hierarchically structured data very well while at the same time being more parameter efficient compared to existing models. We hypothesize that the proposed model exploit the hierarchical nature of the sequential data similar to how the CNN exploits the hierarchical structure in image data.

The proposed model is in principle not limited to one-dimensional temporal data. Similar to PixelVAE \citep{Gulrajani2016}, our model can be applied to images or even video. Investigating whether the promising results of this model transfers into other regimes of hierarchically structured data is an interesting avenue for future work. Another area for further experiments is to apply the model to other hierarchically structured data discussed in the introduction, one such example is pose and orientation estimation from inertial sensors \citep{Manon2017}.

\begin{table}[ht]
	\sisetup{table-format = 3, table-number-alignment = center, mode=text, round-precision=0, round-mode = places}
	\begin{center}
		\captionsetup{width=.9\linewidth}
		\caption{Average KL-divergence per layer on the Blizzard dataset where KL1 corresponds to the layer closest from the data. }
		\label{tab:results_speech_text_kl}
		{ \small
			\begin{tabular}{l S[table-format = 3] S[table-format = 3] S[table-format = 3] S[table-format = 3] S[table-format = 3]}
				\toprule
				Model & \multicolumn{1}{c}{KL1} & \multicolumn{1}{c}{KL2} & \multicolumn{1}{c}{KL3} & \multicolumn{1}{c}{KL4} & \multicolumn{1}{c}{KL5}\\
				\midrule
				FW=200 & 0.5849201800005052& 7.99086228375918& 4.451699510602553& 3.931846768133616& 74.69217626576035 \\
				FW=200 w/o multiscale & 48.63927638095835 & 33.65645178274352 & 47.96300490885383 & 146.00738400771718 & 142.97979292863562 \\
				FW=25 & 235.4 & 45.28940935 & 44.33512293 & 66.62584468 & 101.70034407 \\
				FW=25 w/o multiscale  &  320.93469148379603 & 126.59908707128865 & 488.13101236620463 & 542.9513418013241 & 180.42520826180387 \\
				FW=2 &  629.1813384740739 & 48.9450172994962 & 45.18734085857684 & 35.25978321085142 & 57.97964441838387 \\
				FW=2 w/o multiscale & 203.5077593894518 & 210.2916540831487 & 158.27784056474766 & 209.4174622465976 & 50.33677824322617 \\

				\bottomrule
			\end{tabular}
		}
	\end{center}
\end{table}

\section*{Acknowledgment}
\vspace{-0.5em}

We thank Joakim Jaldén's group at KTH, Royal Institute of Technology, Stockholm, for lending computational power.

redpajama_set_name: RedPajamaArXiv


Ah, the mundane tasks of everyday life, the chores and domestic duties that never seem to end but always repeat: housecleaning. The experience of cleaning our homes should fill us with pride and gratitude for the many material blessings that we have the privilege of cleaning. You have dishes to wash, clothes to iron, carpets to vacuum and floors to sweep – aren't you lucky? During the course of a life, most things change: jobs, homes, friends, lovers, careers, political persuasions… but what will always be there? Dirty dishes.
However, trying to appreciate the positives of housework can be difficult when you're on your knees unclogging the bathtub drain or wiping the gunk out from under your fridge. All too often, the endless cycle of repetitive cleaning tasks brings on a feeling of wasted hours – meaningless work to finish as quickly as possible. If you can't tap into the higher philosophical reasons to appreciate cleaning, there are ways to make it more fun and enjoyable. Really. Try the following tips to cruise through your chores and liven up your cleaning time a little.
1. Buy pretty and colorful cleaning gear. Spending extra money on bright pink plastic gloves and a fancy, frilly purple apron with green ruffles may seem like an unnecessary expense – until you find yourself giggling as you put them on in anticipation of doing the dishes. Treat yourself to a cheerful cleaning outfit to give your mood a boost.
2. Try out a new look. You know those makeup samples you've wanted to try out – in private? Break out the new eye shadow and lipstick before you clean. Put your hair up in braids or in Princess Leia buns. There's no commitment, because you'll probably hop in the shower once you've finished cleaning, and every time you glimpse yourself in the mirror with purple eyelids and orange lips, you'll smile – maybe even laugh.
3. Crank the tunes. Before you start scrubbing, turn on some upbeat, funky music that makes you want to shake it. Whether its some throbbing dance music, '90s gangsta rap or throwback disco favorites, blast some fun booty-shaking beats and you may just find yourself dancing as you sweep and mop.
4. Break it up. If you are cleaning your whole house, plan for a short break after you finish each room – spend 10 minutes looking at silly cat videos online, read a few pages of a juicy novel or watch a bit of Comedy Central. Don't make the break too long, or you might not get back up. Set a timer and relish your little breaks.
5. Make it a game. If you aren't in for long haul cleaning and just need to pick up a small area, set a timer for 15 minutes and see just how much you can get done in the short amount of time. You might be surprised!
6. Sticker chart. Remember how much you loved getting stickers for achievement as a child? Create a cleaning sticker chart to keep track of your rotating tasks, and watch yourself go above and beyond to get the extra stickers for housework. It sounds silly and it is – but you probably need a laugh anyway, right?
7. Call a friend. Some housework, such as folding laundry and dusting, are quiet activities that don't take much brain or body power. Use this time to chat up an old friend on the phone. Invest in a headset, and you can make a call even when both hands are busy.
8. Plan for a treat. The finished product of a clean, sparkling house is a fabulous feeling in itself, but if you plan to indulge in an additional treat once you're done for the day, you'll train your brain to better appreciate the cleaning process for the rewards that come after. Enjoy a pistachio ice cream cone, sit back with a glass of wine and your favorite magazine, or order dinner in and dine in your fresh clean abode.


Embracing our fifth year anniversary, we wanted to take the opportunity to look at how tremendously successful The Arena at Gwinnett Center has been. The Arena has hosted countless concerts, community events and sporting events, and is also home to both the Gwinnett Gladiators ECHL Hockey Team and the Georgia Force Arena Football Team. The Arena opened with an inaugural event featuring George Strait on Feb. 21, 2003 and has experienced incredible growth in these past five years.
Constructed with the patrons in mind, there is truly not a bad seat in the house. Events can be held to accommodate groups from 3,500 in an intimate theater-style setting, up to 13,000 for a major concert or sporting event. Over the span of the past five years, The Arena has hosted more than 500 events.
In the past five years, The Arena at Gwinnett Center has held 157 Concerts.
Sixty-six of those concerts have played to a SOLD OUT house!
The Arena was nominated for the 2005 Pollstar Arena of The Year Award.
Gwinnett Center does not have any plans to slow in the future; new construction will be breaking ground this year to include an Arena parking deck and additional meeting space. The Master Plan for the campus was unveiled last year that calls for a four-star hotel, expanded meeting room and exhibit space, and architectural enhancements. We look forward to the development of Gwinnett Center and making it an even better place for our community and patrons.
The Arena at Gwinnett Center would like to thank everyone who has made the facility such a success over the past five years.


Food safety is EVERYONE'S responsibility – we are all in this together. As a distributor, we take that job very seriously, and we know you do too. With the numerous recalls we have been experiencing lately there is a continuous need to be more diligent in regards to food safety.
Our role at Nelson-Jameson is to help provide you with the items you need to ensure your processing facilities are compliant, providing safe, quality food to all. Nelson-Jameson knows food safety, and offers a wealth of products and solutions that can help your facility reduce the risk of food-borne illness and contamination. See our product solutions below for more information.
Sanitation Basics – COMING SOON!
This entry was posted in Food Safety, Laboratory & QA/QC, Production & Material Handling, Safety & Personnel, Sanitation & Janitorial. Bookmark the permalink. Follow any comments here with the RSS feed for this post. Post a comment or leave a trackback: Trackback URL.


text:
Government Of Assam Handloom Textiles & Sericulture	Directorate of Sericulture
Tasar
Sericulture Technological Center
We have tried to link all Information & Services together to help you locate them faster.
Schemes & Projects
North Eastern Region Textile Promotion Scheme(NERTPS)
Find information about Projects. Details of the completed and ongoing projects can be obtained.
Office Memorandum
A document repository where all types of the documents of the organization can be searched and located in the shortest possible time.
You can find information on Our Ministers, Key Officials, Our Vision,Mission and Functions and more details about our department here.
Silk Reeling and Spinning
The larvae after maturation crawl down the tree at dusk, which are then handpicked and placed in "Jali" (cocoonage) for spinning of cocoons. For continuation of generations 5% seeds cocoons are selected and kept in grainage hall for emergence of moth and production of eggs. Rest 95% good cocoons are stifled for reeling purpose. Normally 1kg of raw silk can be obtained from 4500-6000 cocoons depending upon the quality compactness and weight of shell.
Extraction of silk filament from cocoons by employing a set of processes is known as silk reeling. Muga silk are generally reeled by traditional 'Bhir' reeling process. In recent years, pedal driven reeling machines (RMRS type, Choudhury type) and motor driven machines (CSTRI reeling cum twisting machine) are also practiced in some areas.
The bulk of the cocoon used for reeling is obtained from the "katia" (autumn) crop. The leaves at this time are quite suitable, the seasonal temperature wasps and flies etc. are much less and cocoons produced are comparatively richer in silk. The "jura" (winter) crop is raised for producing seed cocoons only. During the winter the worms take more than one month to spin cocoons, which are also very poor in silk content. The "jethua" (spring) crop is also important crop. The golden colour of the silk from cocoons of this crop is higher than the "aherua" and "bhodia" crop. The "aherua" (summer) and "bhodia" (late summer) crop are raised mainly for seed cocoons.
Generally, the muga reeling is done by the rearers themselves. Boiling and reeling is carried out by all classes of people in a corner of the household. But there is a large section of people, for whom muga reeling is a good trade. In some places of Assam, muga reeling is done extensively as a profession. All the cocoons produced are converted into silk-yarn and fabric within the state of Assam.
In case of Muga reeling about 56% production is done on "Bhir", the traditional age-old device and remaining 44% is coming from CSTRI motorized cum peddle operated machine. The productivity of "Bhir" is very low (80 gm per day) and the quality of yarn is also not uniform. However, the weavers prefer to use the untwisted yarn produced on "Bhir" for weft, which accounts for 60% of the total requirement. Therefore, there is need for an alternative technology to fulfill the need of weft yarn. Central Silk Board has developed a 'Bani' reeling machine particularly for production of better quality untwisted weft yarn. The productivity of "Bani" is much higher (150 gm/day) as compared to "Bhir". The technology may be popularized to enhance the production of quality yarn and also remuneration of the reelers. It may help in replacing "Bhir" in future. Besides, efforts are also required to reduce the high cost of production by utilizing reeling wastes and production of value added products.
The cocoons are processed within the state to produce Muga raw silk & Eri Spun silk. The Mulberry reeling is done in Govt as well private sector by using Multi-end reeling machines located in Mangaldoi, Sibsagar and Jorhat districts. Muga reeling activities are mainly concentrated in Kamrup, Kokrajhar, Goalpara, Sibsagar, Tinsukia, Lakhimpur & Dhemaji districts. Sualkuchi & Bamundi are the main reeling and weaving clusters of Muga silk. Before reeling, the Muga cocoons are cooked in an alkaline solution of soda ash for an hour. This helps to soften the natural gum, sericin, which holds the filaments together. After deflossing the true end of the continuos filament is found and a number of cocoons are transferred to the reeling basin containing tepid water. Two methods of reeling are prevalent - the traditional on Bhir, which involves two persons, and a recent one that employs a fast operating machine with the operator using both hands for reeling i.e. CSTRI Reeling cum Twisting machine. About half of the silk in each cocoon could be reeled and the remainder, used as silk waste, noil, is further processed to spun silk. After the reeling, the Muga threads are dried in the shade for three-four days, following which they are wound into skeins on a 'Sereki'. The sizing of the skeins involves the application of a mixture of powdered rice and water.
Hyperlinking Policy
Assam State Portal
CM Portal
Website Governance
Website Information Manager
Standardised Website Framework of Govt. of Assam
Last Reviewed & Updated: 06 Sep 2021
Content Ownership Directorate of Sericulture, Govt. of Assam.
National Informatics Centre (NIC)
Nodal Department
Secretariat Administration Department,Govt. of Assam

redpajama_set_name: RedPajamaCommonCrawl


May 6, 2015 By This post may contain affiliate links.
Wearing ladies hats may have gone out of fashion, but these festively decorated Ladies' Hat Piñata Cookies will still be fun treats to serve on Mother's Day.
I wish I would have gotten this recipe posted last week before the Kentucky Derby. It is still quite fashionable for women to wear elaborate hats to the horse race and these brightly colored candy filled hat cookies would have made perfect party favors or desserts for the event.
But, alas, my crazy schedule kept me from writing the post even though I made these cookies 2 weeks ago.
When I hit the road for an 8 day trip to Chicago, I thought I'd find plenty of time to work, but that just didn't happen. I traveled with a friend and after attending the Wilton Sweet Up event for two days, my friend and I took a few classes at the Wilton School of Cake Decorating. In between we did lots of shopping, sightseeing, eating, and talking. It felt like we never stopped talking and I was having such a good time, that I didn't even want to pick up my computer.
I'm home now, unpacked, and back to work. I can't wait to share all the fun things I learned at the Wilton school, but that will have to wait until I get some Mother's Day treats posted.
These candy filled hat cookies are very much like the Baseball Hat Piñata Cookies I shared with you a few weeks ago. Like those cookies, I once again used a cake ball pan to shape the rounded top of the hats.
These hat cookies can be colored any way you'd like and decorated using fondant, modeling chocolate, candies, or sprinkles. I decorated mine with modeling chocolate daisies. They were easy to make and added just enough flair to the hats to dress them up.
Note: I love using Rodelle's Vanilla Bean Paste in my cookies. The flavor is so intense and it doesn't bake out as it does when using liquid extracts, so the cookies have an amazing vanilla flavor.
Color the dough. I divided my dough into 4 equal parts and used pink, violet, blue, and yellow. Just note that being the dough is a light tan color, you wont get true colors. My blue was really more teal, which I thought was quite pretty.
Lightly dust your work surface, roll out the dough to between 1/8th of an inch and 1/16th of an inch thickness. Cut out 4 hat brims using a round scalloped cutter.
Place on a parchment paper lined baking sheet and refrigerate for 15 minutes.
Meanwhile preheat oven to 350 degrees Fahrenheit.
Bake these cookies for 12-14 minutes.
Re-roll each colored cookie dough out to a 1/16 inch thickness. Cut 4 circles of each color using round cutters, so that when you drape the dough over your cake ball pan, the dough drapes over but doesn't touch the base of the pan. The cookies will spread slightly while baking so you want to give them a bit of room to spread. You don't want the dough forming a lip at the base of the cookie. I used a 2 1/2 inch cutter, but your pan may be smaller and require a smaller cutter.
Bake the cookies on the bottom rack of your oven for 8-11 minutes.
Note: You will have some excess dough and can make a few more hat cookies. Being my cake ball pan only allowed me to make 13 cookies at a time, I made 4 of each then just rolled out the excess dough and baked up a few cookies to snack on.
You will notice a few of my cookies browned quite a bit on top which was caused by a few hot spots in my oven. If you have those too, it's best to rotate the cookies half way through baking. I got distracted and forgot to do that.
To assemble your hat cookies, fill the rounded cookies with candies, then pipe a thin line of royal icing around the top edge of the cookie. Turn one of the scalloped cookies upside down, set it on top of the iced cookie, and press them together to seal. Turn the cookie right side up and allow to dry for at least 30 minutes.
To decorate your hat cookies, roll out some modeling chocolate or fondant and cut into 1/8th inch wide by 8 inch long strips and attach to the hats using royal icing, cutting off any excess.
Color small pieces of white modeling chocolate or fondant to match your hats. Color some green as well.
Roll out the modeling chocolate and cut out flowers using a small daisy plunger cutter. Cut the center of the flowers out using a small round plunger cutter.
Attach the centers to the daisies using a small dab of water.Note: I used Wilton's new Water Brush, which I love. I don't have to mess around with bowls of water, which I've been known to spill all over my modeling chocolate. The water is contained in the brush. It was perfect for attaching these tiny pieces together.
Once all of your cookies are assembled and decorated they are ready to dress up your dessert table.
LOL, you'd have to have a mighty small head to make these fit, but I could see you doing it!!!
I love these for mothers day! ( I really love the baseball ones too) – you are becoming the expert at pinata cookies!
These are so amazing! I am always so in awe of your creativity when I am here!
LOVE these Mom's Day..Derby Day..or Just for Fun Day cookies! Can't wait to hear more about all your "Sugar Trips"..plus I'm gonna have to try using your vanilla bean paste!


text:
ATI Header Menu
Consultative Groups
Login-Logout-Menu
CIAT presents the Digital Economy Compliance Tool
The tool can be customised according to the necessities of each tax administration and implemented by other ATI members when required.
The digitalisation of the economy has raised many challenges for taxation. While the global trend of online commerce and international trade of services and intangibles consolidates, national governments in developing countries struggle to mobilise resources to finance their development goals. Tax administrations face the challenge of adapting their operative systems to collect revenues from cross-border transactions and new business models. To address this challenge and harness the revenue potential of digital transactions, the Inter-American Center of Tax Administrations (CIAT), with the support from the Norwegian Agency for Development Cooperation (Norad), has developed the Digital Economy Compliance Tool (DEC). In an insightful webinar, CIAT presented the DEC tool and explained how other ATI members can profit from it.
The DEC tool offers a simplified voluntary mechanism for registration, declaration, payment and communication of companies operating without a physical presence in a country. In his introductory remarks, the Director of Cooperation and International Taxation at CIAT, Mr. Isaac Gonzalo Arias Esteban, stated that the tool has a global potential because it is open source and completely parametrisable. The tool can contribute to expedite the collection of VAT, allowing countries to apply the destination principle in cross-border transactions, and ultimately fostering fair competition in international trade.
Against this background, Mr. Eduardo Jiménez, advisor in the Consumption Taxes Unit, Centre for Tax Policy and Administration at the OECD, presented the OECD recommendations on indirect taxation of digital economy transactions. The recommendations have four core pillars (i) the creation of an effective legal basis for a right to tax inbound international online sales; (ii) the implementation of an efficient and effective collection mechanism; (iii) the leverage of the potentials of existing digital platforms to enhance VAT collection and; (iv) the facilitation of compliance and strengthening of enforcement through modern, data-driven risk management and administrative cooperation. Mr. Jiménez brought attention to the publication of the first VAT digital toolkit, which has a regional focus on Latin America and the Caribbean.
To showcase the revenue potential that e-commerce has, Mr. Santiago Díaz de Sarralde, director of Tax Studies and Research at CIAT, presented empirical evidence on the levels of VAT collection that could be levied through effective taxation of digital services. Taking a sample of eight Latin American countries, it is estimated that yearly additional revenues could amount to USD 255 million in total, which represents between 0.03% and 0.06% percentage points of the national GDPs in the selected countries. The areas with the highest revenue potentials are the digital advertising and the audio-visual production, respectively.
Next, Mr. Raúl Zambrano, Technical Assistance and Information Technology and Communication director at CIAT, presented the DEC Tool, a software-based solution that aims to facilitate the collection of VAT for transactions taking place virtually. The tool has two interfaces: one for the tax administration and one for the taxpayer, and it can be adapted to any language. Mr. Zambrano explained how to utilise the multiple functions of the DEC tool, such as registering new taxpayers, filling return requests, uploading documents and paying obligations. Since the tool can be customised and adapted to the necessities of any tax administration, there are countless additional options to collect information and optimise processes. In terms of security, the DEC Tool fulfils the highest security standards and can be operated on-prem (within the tax administration's internal system and hardware) or as cloud-based solution (via internet).
To conclude, Mr. Simen Bjørnerud, Policy Advisor at Norad and representative of ATI Consultative Group 1, pointed out the importance of improving fair and effective VAT collection, as it represents the most important revenue source for developing countries. Furthermore, he celebrated the increasing cooperation among CIAT and the OECD in the last years to address relevant challenges for the taxation of the digitalised economy. Mr. Bjørnerud emphasised the potential of the DEC tool to foster domestic revenue mobilisation, particularly in the Least Developing Countries (LDCs), and emphasised that the ATI will continue fostering the exchange of experiences and peer learning among members on the topic.
To know more about the DEC implementation process, interested tax administrations can access the DEC handbook and contact CIAT's project manager, Mr. Décio Carreta (dcarreta@ciat.org).
ITC Secretariat
at GIZ Offices
Friedrich-Ebert-Allee 32 - 36
secretariat@taxcompact.net
ATI Footer Menu
DRM Database
Legal and Data Protection
ATI Icon Footer Menu
© Addis Tax Initiative, 2021


ACE (Access to Construction and Engineering) will give you a range of opportunities to develop your skills in the Construction and Engineering industries. It will also help build your confidence through working in teams with your fellow students.
This course has helped many students to gain employment and apprenticeships both locally and nationally.
Prospective learners must demonstrate an interest in a career in construction or engineering.
Consideration will also be given to learners who have completed the Personal Empowerment Programme.
Woodwork - Using hand and power tools to create traditional woodworking joints. Find out how to repair damaged items in your own home.
Plumbing - Learn how to fit out a bathroom suite and take the skill away with you to save money.
Painting & Decorating - Prepare and hang wall paper to surfaces you have prepared. Upgrade your home decor with your own hands.
Electrical Installation - Learn how to fit and test lighting circuits.
Welding & Fabrication - Build your very own mini steam engine and learn how to Weld using different techniques.
Employability - We will help you become more attractive to employers by encouraging you achieve the best possible goal.
Core Skills - You will be given the chance to enhance your core skills through enterprise.


text:
Vila Trocnov, původně evangelický hospic, stojí ve čtvrti Westend v Karlových Varech v ulici Krále Jiřího 3, č. p. 1053. Stavba vznikla z podnětu evangelického pastora Camillo Fellera, podle projektu architekta Julia Zeissiga a probíhala v letech 1897–1899.

Vila byla prohlášena kulturní památkou, památkově chráněna je od 18. srpna 1995, rejstř. č. ÚSKP 10846/4-5037.

Historie

V období výstavby městské části Westend v souvislosti se stavbou dvou sakrálních staveb – anglikánského kostela sv. Lukáše a pravoslavného kostela sv. Petra a Pavla – přesvědčil evangelický pastor Camillo Feller městskou radu o potřebě postavit v těchto místech též hospic.

V roce 1897 zakoupilo Evangelické společenství Karlovy Vary pozemek v tehdejší Eduard Knoll-Strasse, dnes ulici Krále Jiřího. Projekt vypracoval lipský architekt Julius Zeissig. Výstavba probíhala v letech 1897–1899. Objekt byl postaven k příležitosti oslav jubilea nastoupení na trůn císaře Františka Josefa I., k slavnostnímu otevření došlo 16. května 1899.

Po roce 1948 byla budova znárodněna a v této souvislosti pak i přejmenována.

V roce 1995 došlo k její kompletní renovaci. Od téhož roku je chráněna jako nemovitá kulturní památka České republiky.

V současnosti (únor 2021) je evidována jako objekt občanské vybavenosti ve vlastnictví Statutárního města Karlovy Vary.

Popis

Budova stojí v ulici Krále Jiřího 3, č. p. 1053. Jedná se o typickou reprezentativní lázeňskou vilu se zahradou s oplocením. Má německý charakter hrázděných středověkých hradů a tvrzí. Poslední, hrázděné patro věže má na spodních výplních motiv slunce. Použití ondřejských křížů na zábradlí balkonů dodává budově saský výraz.

Odkazy

Reference

Literatura

Související články
 Seznam kulturních památek v Karlových Varech

Externí odkazy

 Karlovy Vary znovu odmítly prodej sanatoria Trocnov – článek na stránkách Seznam Zprávy, autor ČTK, 2020-08-30 [cit. 2020-10-10]

Trocnov
Hotely v Karlových Varech
Kulturní památky v Karlových Varech
Postaveno v Česku 1899
Lázeňské hotely v Česku

redpajama_set_name: RedPajamaWikipedia


text:
Share ↗
A day with the Harleston Parker jury
by Maaha Rafique
The jury at the Harvard Science and Engineering Complex.
Mirror, mirror on the wall, who's the most beautiful of them all? Well, it depends on what you mean by "beautiful."
Every year, a jury selected to award the Harleston Parker Medal wrestles with the task of choosing "the most beautiful piece of architecture, building, monument or structure within the City or Metropolitan Parks District limits" built within the previous 10 years.
It isn't easy to do—the definition of "beautiful" changes based on context, personal perspective, and more. This year, the eclectic jury, made up of photographers, writers, sustainability experts, architects, and more, came together to narrow the list of hopefuls down to five finalists.
Established in 1921 in memory of Boston architect J. Harleston Parker, the Harleston Parker is the highest honor awarded by the BSA. To get a better sense of each finalist's story and meet the teams behind the projects, the jury embarked on site visits, visiting all five over the course of one blustery and eventful day in December. By the time that day was over, they had heard from designers and stakeholders, engaged in passionate discussions, broken bread together, and finally chosen a winner.
Their day began early in the morning with the story of a structure that no longer exists. As If It Were Already Here was a floating art piece located above the Rose Kennedy Greenway, designed by artist Janet Echelman.
Photo by Melissa Henry
A Harleston Parker finalist in both 2018 and 2021, the sculpture may only have been visible for a few months in 2015, but its dominant presence as a public art installation profoundly affected those who saw it, even inspiring a poem by Boston University professor Robert Pinsky.
The jury gathered in a conference room in a building situated on the Greenway to hear about the piece and its history. As If… was offered to the city as a "civic gift"— Echelman's contract was for $1. Representatives from the Greenway and the Smith Family Foundation, which helped fund the piece, were present and answered some of the jury's questions.
As If… was built specifically to evoke Boston's history, even using the same kind of knots traditionally used by New England fishermen in the cables that kept it in the air. That was one of the reasons Echelman turned down an offer from a city in China to purchase it. If placed elsewhere, the piece "would have to be a different work," Echelman said. She compared the piece to the cherry blossoms you can see if you happen to visit Japan at just the right time in the spring—leaving an impression that's all the sweeter for how brief it is.
The jury then headed to their next destination, Four51 Marlborough, a residential building in the Back Bay. The area surrounding the building screams "Boston," with a view of the Citgo sign and a nearby riverfront area designed by Frederick Olmstead.
Hacin + Associates designed the luxury five-story building. The project team's challenge was to create something that "subtly [added] a new layer for the neighborhood" while integrating naturally into the environment of the historic street. The team worked with the Back Bay Architectural Commission to make sure the project was in accordance with Boston's exhaustive zoning regulations.
Photo by Trent Bell
Featuring a façade of Roman bricks, a unique tile mural in the lobby, and a focus on light and shadow in the interior, the building was constructed with the utmost attention to detail. By filling in empty space on a historic street with a contemporary building, the project marked "the last opportunity to say something about our time in a district that has marked time over a century," said David Hacin FAIA.
Next on the list was the major redesign of the Roxbury branch of the Boston Public Library. The renovation was completed in 2020 and headed by Utile.
Once a very unpopular, closed-off building that residents wanted torn down, the library has completely transformed, replacing sullen concrete blocks and hard-to-find entrances with glass walls and a welcoming entry plaza.
Photo by Anton Grassl
After a tour of the outside, the jury was welcomed into the building and briefed on the history of the project before being shown some of the public art the City of Boston commissioned specifically for the library. The presence of the art exemplified the library's new focus on community, as did some new areas like a technology learning lab, and even a nutrition lab with a kitchen on the second floor. There, cooking classes and showcases are held by nutrition experts and professional chefs, and both children and adults have the opportunity to learn about healthy eating.
"Connecting people to information is the mission" of including such spaces in the library, the library staff told the jury.
After they'd had their fill of admiring the building and chatting with the head librarian, the jury moved on to another place of learning. Their destination was the King Open/Cambridge Street Upper Schools & Community Complex in Cambridge. William Rawn Associates in association with Arrowstreet designed the campus, which combines King Open Elementary and Cambridge Street Upper School.
The two public schools needed to be separated while still maintaining a sense of connection. Another point of emphasis in the design plan was the need to create a space that could be enjoyed and utilized by the wider community. This was accomplished through the inclusion of spaces like a branch of the Cambridge Public Library, a public pool, and lots of open greenery.
Photo by Robert Benson
Principal Darrell Williams took time out of his busy schedule to speak to the jury as they toured the campus, referencing the effectiveness of project-based learning and how classrooms were designed to encourage students to work together. He emphasized that teachers and staff seek to ensure a quality educational experience for each child, incorporating a progressive approach to teaching and learning. "The work is founded in social justice," Williams said.
The students, too, are fans of the school's innovative design—when questioned, school staff told the project team that their favorite places are the shared breakout rooms where much of the project-based learning is done.
The last stop was the Harvard Science and Engineering Complex. The 554,000 square foot building in Harvard's Allston complex was completed in 2020 and designed by Behnisch Architekten.
As the jury climbed out of the van that had been carting them around to their destinations all day, the building's remarkable facade rose up above them. The steel screen is one of the most distinguishing features of the structure.
Photo by Brad Feinknopf
The building has a long history complicated by the abrupt pausing of construction on the site three years after breaking ground in 2006. By the time construction began again in 2014, what was originally planned as a life science building was being reconceptualized as a state-of-the-art science and engineering facility.
Venturing further in, the jury had the opportunity to see students using the space for study and work, relaxation, socializing, and more. The World Cup was in full swing at the time, and a few students were avidly watching a game of soccer on a large projection screen in the central lobby. Up the stairs, others were making use of foosball tables and Peloton machines. The jury also had the opportunity to tour one of the labs, getting a glimpse of the space used by doctorate students to work on microbiotics.
After seeing the whole Harvard building, the tours were over, and the jury gathered to celebrate with drinks and snacks. But their work wasn't done—they needed to decide which project they ranked above the others. The deliberations had begun.
Unsurprisingly, there were a variety of opinions around the table. Those who attended the BSA Awards Gala or have been staying up to date with announcements will know the jury ultimately selected As If it Were Already Here as their winner, an unprecedented choice for an award that has almost always honored buildings.
The significance of the selection was not lost on the jury. Their discussion, which lasted nearly two hours at the end of a long day, was marked by consciousness of the Medal's importance and invigorating debate between As If…'s staunchest advocates and those who initially favored other finalists.
A recurring topic of discussion was the piece's ephemeral nature. How valuable is a structure that can only be seen by a limited number of people, for a limited amount of time? For the jury, this impermanence spoke to an important truth about the nature of reality. They debated amongst themselves, challenged each other with probing questions about aesthetics and scale, and shared personal impressions.
Everything ends eventually—some things just disappear faster than others. It's a slightly uncomfortable notion that inspires contemplation of what matters most to us. In this way, As If… evokes particularly important questions about art and life.
Of course, no matter how existentially interesting a piece is, to win the Harleston Parker Medal it must also be worthy of being called the "most beautiful." The fruitful discussions about the nature of the piece aside, the jury were unanimous in their appreciation of the unique and enthralling beauty As If… represents. The complexity of its design and the feats of engineering that allowed it to stand contrasted with its free-flowing and effervescent appearance, creating an unbelievable sight that had visitors to the Greenway stopping in their tracks and craning their necks to look up at the sky.
As the complexity of the jury's discussion indicates, each finalist impressed jurors over the course of the day, bringing together a diverse group of individuals who brought varying perspectives on art and architecture to the table. Ultimately, As If It Were Already Here won for its distinctive, breathtaking beauty—the essence of what the Harleston Parker rewards.
Sara Kudra AIA
An interview with Sara Kudra AIA, Affordable Housing Director at DREAM Collaborative.


text:
Mumbai Preview: Documentary Film Screening at Alliance Francais
It's auteur's Tuesday hère in Mumbai and if your movie coffer's been running a little low these days, then Alliance Francais' special film screening tonight might help flesh it out a bit. Tonight's a documentation of muscle, both of the physical and emotional kind, in the form of a documentary called 'Ramin.'
Directed by the renowned Audrius Stonys, the narrative follows a veteran Georgian wrestler and his journey from a once unbeatable wrestler to a present version of himself with no one but a cat for companionship. The filmmaker delves deep into the wrestler's introspection ultimately getting him to admit that even during his heydays, he never gave love a real chance and so he wakes up at a later stage of his life to seek out what he truly desires. Reviews rave about it being a truly heart-warming story but there's only one way to find out. You can watch the trailer here.
The screening will be followed by a stimulating discussion with filmmaker Tangella Madhavi, best known for her moving documentation of migrant daily wage labourers in the city and their utterly helpless feeling of 'homelessness,' a sentiment that was easily identifiable for her. You can follow up the movie watching with a little groove at The Little Door, where Groove Gully will be exploring different genres of music.
Alliance Francais will screen Ramin at 6.30pm tonight at the institution in Churgate. Entry is free. You can call +91 22 22036187 for more details.
Words: Mandovi Menon


text:
China's wet dream: the underwater railway
Pyramid schemes: a look into Egypt's megaprojects
According to reports, China has plans for a 13,000km high-speed railway line that would travel from mainland China, through Siberia and Eastern Russia, and then under the sea to Alaska. Although the project is currently on hold, Frankie Youd investigates whether this line could be a future possibility.
echnological developments have seen the transport industry introduce many new impressive fuel types, touchless technology, robot helpers, and autonomous vehicles. However, China is wanting to push advancements one step further by constructing a 13,000km high-speed underwater railway line.
Starting in mainland China, the line is planned to run through Siberia, pass under the sea for 200km through the Bering Strait into Alaska, and then continue to Canada, and finally, the US.
Although this may sound farfetched, underwater railway lines are more reality than science fiction, with Japan having its own underwater railway line currently running today: the Seikan railway.
Japan's 53.85km railway tunnel connects Honshu Island and Hokkaido Island via a track located 140m below the seabed. At present, the Seikan Tunnel is the world's deepest and longest railway tunnel with approximately 23.3km of the tunnel being located under the seabed making it the world's longest undersea tunnel.
If Japan can achieve this impressive underwater train experience, what is standing in the way for China to achieve similarly?
Brightline senior vice president, corporate affairs, Ben Porritt
A big splash now treading water
With flight times from Russia to the US lasting over ten hours on average, the introduction of an underwater bullet train connecting both countries could enable travel between the two in as little as 20 minutes.
Plans for this design were first brought forward in 2014, with reports claiming that China was in advanced talks with Russia – who had been discussing the construction of a railway line under the Bering Strait for years. The ambitious plan received a great deal of media coverage at the time with many reporting that it would boost trade links between China, Russia, Canada, and the US.
However, after the initial announcement and excitement, not much more was heard about the project – with no plans being put in place for construction. Recent reports from South China Morning Post have seen the plans being heavily criticised, due to the proposed budget of $200bn.
China has proposed to build a 13,000km long underwater railway line to the US. Do you think this is a realistic project? #china #usa #railway #project
— Future Rail Magazine (@FutureRail_Mag) July 30, 2021
Credit: SignLab Amsterdam
That being said, there are still indications that China's underwater rail project may still go ahead. The Chinese Government approved plans for the world's first underwater bullet train in 2018, with aims to demonstrate that high-speed railways are feasible under the sea.
The planned bullet train project would stretch 77km from Ningbo, a port city south of Shanghai, to the archipelago islands of Zhousan, including a 16.2km underwater section of the line. Although this is considerably smaller than the proposed China–Russia–Canada–US route, if this project is successful it could see China moving forwards with its larger goal.
However, since the announcement of this project in 2018, media coverage and further information have ceased being released, so the progress of the project is hard to gauge.
An ocean of technical problems
It goes without saying that the construction of this project would be one of the greatest engineering challenges, paired with the sky-high costs. However, arguably most importantly, it could create dynamic political relations between the US, Russia, and China.
Experts have noted that the most challenging element of the project would be constructing the crossover at the Bering Strait, due to this section containing the longest underwater tunnel in the world – extending for over 103km. This construction would be expected to take between 12-15 years, at an estimated cost of $35bn just for the underwater link.
To connect the land to the sea, new infrastructure will need to be laid out due to the closest terminus being located 3,000km away in Russia, while in Alaska the project would require over 1,200km of new railway line. To develop the missing infrastructure to connect the project, it has been estimated that over $200bn would be needed, which according to critics is out of proportion.
It seems like it could be more a pipedream than an actual project.
Willis Rooney, an economist at GlobaData, comments on the economic feasibility of the project: "I can't see much evidence of intention to develop this project from any official sources, only an announcement by someone at the Chinese Academy of Engineering, who may have now passed away. It seems like it could be more a pipedream than an actual project."
"If it was to proceed – though I believe that would be incredibly unlikely – project costs would be a significant hindrance to its feasibility, particularly given the growing public debt levels at Chinese state-owned enterprises, which are likely to discourage investment in more fanciful projects such as this.
"Geopolitical tensions are another likely stumbling block. These ambitious multinational projects tend to struggle to get off the ground, with differing national priorities and demands hampering the progress of development."
As impressive as it sounds, the project appears to be on hold for now with critics continuing to slam the proposal for being 'economically redundant', saying that flying and cargo ships are cheaper and more efficient options for trade due to the proposed project being too complex.
However, as technology advances and infrastructure develops, the China–Russia–Canada–US line may just one day be more than a pipe dream.
The HydroFLEX team talks about the train's development and testing.
Main image: Alstom's Coradia iLint was the world's first passenger train powered by hydrogen fuel cell. Credit: Alstom
Home | The camera doesn't lie
Texat Decor Engineering Company Insight
Texat Decor Engineering
SAB Bröckskes
Industry briefing
Covid-19 executive briefing by GlobalData
Wayside Inspection Devices
CRRC Ziyang
South Western Rail's new assistance scheme
Lumo's low-cost rail launch to challenge domestic flights
Can vaccine passports be truly inclusive?
London's new tube stations
MA Exhibitions
U-Lift
All aboard the Stockholm–Berlin night train
The technology that will improve post-pandemic UK rail journeys
Travelling through the mountains: Tibet's first bullet train
Should train guards wear body cameras?
Assisting deaf passengers with train travel: AI sign language
Siemens Mobility joins a growing list of complainants against HS2
Global markets and indices
Macro-economic indicators
Macro-economic indicators (page 2)
Satek
General Plastics