text:
\section{Introduction}

Unsupervised data generation has been a time-hallowed problem in machine learning theory. Though not its strongest suit, Variational Autoencoders (VAE) \cite{Kingma2014} have been implemented for generative purposes to a great extent. Once emerged as a crucial unsupervised method for representation learning, its poor performances in this particular area somewhat depreciates its appeal. On the contrary, generative adversarial networks (GANs) \cite{10.5555/2969033.2969125} have proven to be superior when it comes to the quality of the generated image samples. The theoretical elegance of the adversarial training method can be duly credited for this achievement. In an attempt to adopt this merit, such 'adversarial loss' functions were incorporated into VAE objectives \cite{makhzani2016adversarial, mescheder2018adversarial}. The Wasserstein metric, also being used in GANs \cite{pmlr-v70-arjovsky17a} previously, was an inevitable choice, since it brings along the beckoning of  Transportation Theory. Using the same to measure the reconstruction loss in a VAE, Wasserstein Autoencoder (WAE) \cite{tolstikhin2019wasserstein} was conceived. By approaching generative modeling from an Optimal Transport (OT) point of view, it not only achieved remarkable reconstruction standards but also strengthened the theoretical backbone.

An appropriate WAE framework consists of two components, namely an \textit{encoder} $(E)$ and a \textit{decoder} $(D)$, which are represented by neural networks in practice. Samples from an input distribution $(P_X)$ are fed into the encoder, which aims at resulting in a target law $(P_Z)$ in the latent space $(\mathcal{Z})$. The decoder deals with the reconstruction of the original distribution from this 'encoded' signal. Mathematically, the objective of WAEs is to minimize $\mathcal{L} = \mathcal{R}(D,E) + \Omega(E)$ over the set of probabilistic encoders $E$. Here, $\mathcal{R}()$ ensures the reconstruction of samples put in whereas, $\Omega()$ ensures the attainment of the latent distribution. Needless to say, $\mathcal{R}()$ is represented by the Wasserstein discrepancy.

As the name suggests, our primary interest lies in investigating whether a suitably characterized WAE can \textit{learn} a distribution, which represents its latent behaviour, without losing the \textit{memory} of the initial signal it was fed. When it comes to memorizing the input distribution in the form of reconstruction, theoretical guarantees for adversarial training, in the context of GANs, have been provided rigorously. In the absence of a flexible discriminator, WAEs need more care and closer observation. Moreover, the concurrent task of sculpting a desired latent distribution adds nuances to its mechanism. Most of the existing literature on deep generative models, under adversarial loss, misses out on this very point. Against such a backdrop, this first of its kind study substantiates the simultaneous performances of WAE, from a theoretical viewpoint.

Our analysis starts with a special family of WAEs, namely \textit{f}-Wasserstein Autoencoders (\textit{f}-WAE) \cite{NEURIPS2019_eae27d77}, which arises due to the specific choice of $\Omega()$ as \textit{f}-divergences (\ref{Definition1}). We follow a non-parametric approach in defining both the networks involved and the distributions under consideration. The transformations induced by the encoder and the decoder are conceived as measurable functions between spaces. The nature of an `ideal' encoder in our language, is a network that preserves information. Whereas, the decoder is characterized by a map that propagates the information retained, precisely. Our statistical assumptions on the same reflect these very ideas. As a result, the two concurrent objectives of \textit{f}-WAE boil down to non-parametric estimation of laws, under vastly different circumstances.

Our major findings can be summarized in the following way,
\begin{itemize}
    \item Given a non-negative margin of error, \textit{f}-WAEs can learn latent space distributions with a convergence rate of $\mathcal{O}(n^{-\frac{1}{2}})$, under fairly gentle assumptions on the input and target law (\ref{Corollary1}). The proof involves a realistic characterization of the transformation, induced by the encoder network.

    \item Under a geometry preserving representation of the decoder map, H\"{o}lder densities can be successfully reconstructed by \textit{f}-WAEs up to a constant margin (\ref{Theorem2}). Our work also produces deterministic concentration bounds on the regeneration error and hence, the rate of convergence of the empirical laws.

    \item A particular instance of our analysis goes on to show that the input distribution converges strongly to its latent counterpart under encoding, in the form of empirical measures. Also, \textit{f}-WAEs can exactly reproduce samples from smooth densities in the sense of weak convergence (\ref{Remark}).

    \item Inferences, identical to that in case of \textit{f}-WAE, can be drawn for the original WAE \cite{tolstikhin2019wasserstein}, under additional assumptions on the decoder (\ref{Corollary2}).
\end{itemize}

\section{Related Work}

In recent times, scrutiny of the empirical achievements of deep generative models from a theoretical point of view has gained great momentum. GANs \cite{10.5555/2969033.2969125}, being at the helm, have received most of the attention \cite{pmlr-v70-arora17a, 10.1214/19-AOS1858}. Possibly, the cornerstone of success for a GAN lies in its adversarial training and the accompanying loss functions. A plethora of GAN variants have been devised by incorporating different losses, such as Wasserstein distance (WD) \cite{pmlr-v70-arjovsky17a}, \textit{f}-divergence \cite{NIPS2016_cedebb6e}, Maximum Mean Discrepancy \cite{li2017mmd} and Sobolev distances \cite{mroueh2017sobolev}, to name a few. The generative aspect, in particular, has attracted a lot of research in the form of distributional estimation studies \cite{liang2018generative,chen2020statistical}. Additionally, \cite{liu2017approximation} has proved weak convergence of distributions under several of the aforementioned adversarial losses. In a recent study, \cite{biau2020theoretical} has also discussed the convergence of the empirical measures in the case of WGAN, under parametrized network assumptions. VAEs, despite being used vastly as generative tools, have not experienced such a level of independent theoretical investigation, which comprehends its latent space mechanism.

In a novel approach to unify the deep generative techniques, Adversarial AEs \cite{makhzani2016adversarial} were introduced. Subsequent work to establish resemblance \cite{hu2018unifying,mescheder2018adversarial}, has only strengthened that foundation. Eventually, the formulation of WAE \cite{tolstikhin2019wasserstein} bridged the gap significantly by matching regeneration standards. The equivalence in performance is a direct reflection of the similarity of its objective to that of GANs' \cite{NEURIPS2019_eae27d77}. This 'primal-dual relationship' prepares a basis to explore the generative facets of WAE, in particular, now with the collective experience provided by GANs.

Typically, WD serves as the measure of discrepancy used to represent the reconstruction loss in WAEs. Non-parametric density estimation under this metric has seen the establishment of minimax bounds based on covering numbers, under fairly general assumptions \cite{weed2017sharp,Lei_2020}. The same has also been achieved by \cite{singh2019minimax}, for unbounded metric spaces, assuming that some moments are finite. Singh \textit{et al.} \cite{10.5555/3327546.3327686} established minimax convergence rates in the same context, under smoothness assumptions on the networks employed in the generation. Despite being influential in their own ways, none of the above aim at discussing the coinciding objectives of WAEs. In a recent empirical study, Diffusion VAE \cite{ijcai2020-375} was introduced, which attempts at capturing arbitrary latent manifolds using kernels of Brownian motion. However, without precise conceptual insight. As such, the theoretical questions regarding the parallel machinery of WAEs remain largely unanswered. Our analysis is a humble attempt at filling this void of statistical understanding.

\section{Preliminaries}

\subsection{Notations}

We start with denoting the input space by $\mathcal{X}$, which can be assumed to be Polish (i.e., separable and completely metrizable). We will use $\mathcal{Z}$ for the latent space. Both may be thought of as Euclidean spaces, since it is a typical characterization. The space of probability measures (distributions) defined on $\mathcal{X}$ is denoted by $\mathscr{P}(\mathcal{X})$. The set of all measurable functions from $\mathcal{X}$ to $\mathcal{Z}$ is refered to as $\mathscr{F}(\mathcal{X},\mathcal{Z})$. For example, observe that a probabilistic encoder $E$ belongs to $\mathscr{F}(\mathcal{X},\mathscr{P}(\mathcal{Z}))$. For any function $T:\mathcal{X} \rightarrow \mathcal{Z}$, we denote the \textit{pushforward measure} of $\mu$ as $T_{\#}\mu \in \mathscr{P}(\mathcal{Z})$, i.e. for any measurable set $\omega \subset \mathcal{Z}$ we have $T_{\#}\mu(\omega) = \mu(\{x \in \mathcal{X}:T(x) \in \mathcal{Z}\}) = \mu(T^{-1}(\omega))$. To denote \textit{absolute continuity} of a measure $\mu$ with respect to $\lambda$, we use the notation $\mu \ll \lambda$. The \textit{Support} of $\mu$ is denoted by $Supp(\mu)$. To indicate the \textit{Closure} of $\mathcal{X}$ we use $\bar{\mathcal{X}}$ and $Int(\mathcal{X})$ denotes the set of its \textit{Interior Points}. Given $\mathcal{S} \subseteq \mathcal{X}$, the $\mathcal{S}$-\textit{distance} between a pair of functions $f$ and $g$ is given by, $\;\norm{f - g}_{\mathcal{S}} = \sup_{x \in \mathcal{S}}\abs{f(x) - g(x)}$. When the $\sup$ is taken over the whole space, we call it the \textit{uniform norm}. The \textit{total variation distance} between probability measures $\mu$ and $\gamma \in \mathscr{P}(\mathcal{X})$ is denoted by $\norm{\mu - \gamma}_{TV}$. Now we introduce some useful concepts that are made use of later.

\begin{definition}[\textit{f}-Divergence] \label{Definition1}
Let $P,Q \in \mathscr{P}(\mathcal{X})$ such that $P \ll Q$. For a convex function $f$ with $f(1) = 0$, the \textit{f}-divergence of $P$ from $Q$ is defined as
\[D_f(P,Q) = \int_{\mathcal{X}}f\Big(\frac{dP}{dQ}\Big)dQ.\]
\end{definition}

\begin{definition}[Yatracos Family \cite{10.2307/2241209}] \label{Definition2}
For a class of real-valued functions $\mathcal{F}$, defined on $\mathcal{Z}$, the Yatracos family associated to $\mathcal{F}$ is a collection of subsets of $\mathcal{Z}$ defined as
\[\mathcal{Y}(\mathcal{F}) = \{x\in\mathcal{Z} : f(x) \geq g(x); \;f,g \in \mathcal{F}\}.\]
\end{definition}

\begin{definition}[Empirical Yatracos Minimizer \cite{Devroye2001CombinatorialMI}] \label{Definition3}
The Empirical Yatracos Minimizer is a function $\mathcal{L}^{\mathscr{P}(\mathcal{Z})} : \bigcup\limits^{\infty}_{n=1}\mathcal{Z}^{n} \longrightarrow \mathscr{P}(\mathcal{Z})$ defind as
\[\mathcal{L}^{\mathscr{P}(\mathcal{Z})}_n = \argmin_{q \in \mathscr{P}(\mathcal{Z})}\:{\norm{q - \hat{p}_n}}_{\mathcal{Y}(\mathscr{P}(\mathcal{Z}))} ,\]
where $\hat{p}_n$ is an empirical distribution based on a sequence of observations $\{Z_i\}^{n}_{i=1}$ from $\mathcal{Z}$.
\end{definition}

\begin{definition}[$\epsilon$-Closeness] \label{Definition4}
A pair of distributions $f, g \in \mathscr{P}(\mathcal{X})$ are said to be $\epsilon$-close to each other with respect to the metric c if $c(f,g) \leq \epsilon$.
\end{definition}

\begin{definition}[Quasi-Isometric Embedding] \label{Definition5}
A map $f:(\mathcal{Z},d_1) \longrightarrow (\mathcal{X},d_2)$ is called a quasi-isometric embedding if there exists a constant $A \geq 1$ such that,
\[\frac{1}{A}d_{1}(x,y) \leq d_{2}(f(x),f(y)) \leq Ad_{1}(x,y) \;\:\forall \:x,y \in \mathcal{Z}.\]
\end{definition}

\begin{definition}[Covering Number \cite{weed2017sharp}] \label{Definition6}
Let $S \subseteq \mathcal{X}$. Then, the minimum integer $m$ such that there exists $m$ closed balls $\{B_{i}\}^{m}_{i=1}$ of diameter $\epsilon$ each $\ni S \subseteq \bigcup_{1 \leq i \leq m}B_i$, is called the $\epsilon$-covering number of $S$. We denote it by $\mathcal{N}_{\epsilon}(S)$.
\end{definition}

\subsection{Wasserstein Duality: A Brief Exposition}

The equivalence relation between Wasserstein Distance and Integral Probability Metrics (IPMs) \cite{10.2307/1428011} is a fairly well-studied topic in optimal transport theory. We exploit this fact to serve our purpose. For a detailed elucidation on this subject, one may take refuge in \cite{OptimalTransport,peyre2020computational}.

Let $P,Q \in \mathscr{P}(\mathcal{X})$ and $c:\mathcal{X}\times\mathcal{X} \rightarrow \mathbb{R}_{\geq 0}$ forms a metric. Then,
\begin{align*}
    W_c(P,Q) =& \inf_{\gamma \in \Gamma(P,Q)}\Big\{\int_{\mathcal{X}\times\mathcal{X}}c(x,y)d\gamma(x,y)\Big\} \\
    =& \sup_{l \in \mathscr{L}^1_c}\Big\{\int_{\mathcal{X}}l(x)dP(x) - \int_{\mathcal{X}}l(x)dQ(x)\Big\} = \textrm{IPM}_{\mathscr{L}^1_c}(P,Q),
\end{align*}
where $\Gamma(P,Q) = \big\{\gamma \in \mathscr{P}({\mathcal{X}\times\mathcal{X}}) : \int_{\mathcal{X}}\gamma(x,y)dy = P, \;\int_{\mathcal{X}}\gamma(x,y)dx = Q\big\}$ denotes the coupling between $P$ and $Q$. The class of 1-Lipschitz functions with respect to the metric $c$ is represented by $\mathscr{L}^1_c$. Later in the study however, we will use the notation $d_{\mathscr{L}^1_c}$ for Wasserstein metric, for clarity.

\subsection{Problem Setup}

A typical WAE architecture intends to solve the minimization problem,
\begin{equation} \label{Equation1}
    \textrm{WAE}_{c,\lambda}(P_X, D) = \inf_{E \in \mathscr{F}(\mathcal{X},\mathscr{P}(\mathcal{Z}))} \Big\{ \int_{\mathcal{X}} \mathbb{E}_{z \sim E(x)}[c(x,D(z))] dP_{X}(x) + \lambda.\Omega(E_{\#}P_X, P_Z)\Big\} ,
\end{equation}
where $c:\mathcal{X}\times\mathcal{X} \rightarrow \mathbb{R}_{\geq 0}$ and $\lambda >0$ serves as a hyperparameter. Arbitrary divergence metrics between measures, belonging to $\mathscr{P}(\mathcal{Z})$, are deployed in the form of $\Omega$. In this paper however, we modify this formulation to facilitate theoretical analyses. The reshaped objective that our study is based on is given by,
\begin{equation} \label{Equation2}
    f\textrm{-WAE}_{c,\lambda}(P_X, D) = \inf_{E \in \mathscr{F}(\mathcal{X},\mathscr{P}(\mathcal{Z}))} \Big\{ W_{c}(P_X, (D \circ E)_{\#}P_X) + \lambda.D_f(E_{\#}P_X, P_Z)\Big\} .
\end{equation}
This category of AEs are called \textit{f}-Wasserstein Autoencoders \cite{NEURIPS2019_eae27d77}. The key difference between this family and the ordinary WAE arises due to the choice of the reconstruction loss. This alteration, however, does not impact the characterization significantly. Equivalence between these two systems can be attained by imposing mild restrictions on the decoder transformation, which we elaborate on at a later stage. As such, this can be marked as a valid foundation to work on.

Observe that, the regularizer metric used in (\ref{Equation2}), to measure the discrepancy between the target latent distribution and the encoded signal is taken to be $D_f$. In this analysis, our specific choice of \textit{f}-Divergence is Total Variation Distance (TV). Its rich theoretical appeal motivates us to do so. TV as a measure of deviation is symmetric with respect to the distributions under consideration. Moreover, it can pose both as a \textit{f}-Divergence and an IPM \cite{sriperumbudur2009integral}. The original WAE, in one of its characterizations, uses Jensen–Shannon divergence (JS) to penalize the loss instead \cite{tolstikhin2019wasserstein}. TV, on the other hand, acts as an upper bound to JS \cite{Sason_2016}, which also justifies our choice.

\begin{remark}
WAEs act as a generalization to AAEs. We stress on the fact that, a specific choice of $c$ as $L_{2}^{2}$ in (\ref{Equation1}), establishes an equivalence between the two \cite{tolstikhin2019wasserstein}. Keeping that in mind, our analysis does not restrict the choice of $c$ and hence can also cater to the theory of AAEs.
\end{remark}

\section{Theoretical Analysis}

Before moving forward, let us rewrite the minimization problem (\ref{Equation2}) in the so-called constrained form
\begin{equation} \label{Equation3}
    \inf_{E \in \mathscr{F}(\mathcal{X},\mathscr{P}(\mathcal{Z}))} \big\{ W_{c}(P_X, (D \circ E)_{\#}P_X) \big\} \,\,\:\textrm{subject to} \,\:D_f(E_{\#}P_X, P_Z) \leq t,
\end{equation}
for some $t\geq0$. By Lagrangian duality \cite{10.5555/993483}, for every value of $t$ where the constraint $D_f(E_{\#}P_X, P_Z) \leq t$ holds, there exists a value of $\lambda$ that produces the same solution from (\ref{Equation2}). Conversely, the solution to the problem (\ref{Equation2}), say $\hat{E}$ also solves the bound version with $t = D_f(\hat{E}_{\#}P_X, P_Z)$. As such, there is an one-to-one correspondence between the constrained problem (\ref{Equation3}) and the Lagrangian one (\ref{Equation2}). This formulation gives us a direction to move ahead to establish regeneration guarantees and consistency of measures in the latent space.

Our analysis consists of two stages, going forward. Firstly we show that probabilistic encoders can successfully transform input densities into realistically characterized latent space laws, under TV distance, up to a constant margin of error. The mild assumptions imposed in the process are fairly practical. Once the constraint of the minimization problem is met in the form of latent space consistency, we provide deterministic upper bounds to the empirical reconstruction loss.

Throughout our discussion, we denote with $\mu \in \mathscr{P}(\mathcal{X})$ the input data distribution. The target distribution which characterizes the latent space is denoted by $\rho$. For simplicity, we assume that both $\mu$ and $\rho$ happen to have corresponding densities, namely $p_{\mu}$ and $p_{\rho}$ with respect to the Lebesgue measure $\lambda$, in their respective spaces. Both densities need explicit characterization, which we state in the form of certain assumptions.

\begin{assumption} \label{Assumption1}
$p_{\mu}$, (i) based on a compact and convex support is (ii) bounded away from 0 by a constant factor $c>0$, i.e. $\inf_{x}p_{\mu}(x) \geq c$.
\end{assumption}

\begin{assumption} \label{Assumption2}
$p_{\rho}$, (i) also defined on a compact support, is easy-to-sample-from and (ii) infinitely smooth, i.e. $p_{\rho} \in C^{\infty}(\mathcal{Z})$.
\end{assumption}

The input density being far from nullity, passes on the same nature to the reconstructed law, even under near ideal situations. This poses as a measure to prevent the regenerated law from being a degenerated one, at zero. We point out that, in the non-parametric approach of this work, we often use empirical distributions corresponding to $\rho$, which are based on observations from the same. In high-dimensional statistics, there are distributions from which it is practically not feasible to draw samples. For example, the drawing of uniform samples from a high dimensional unit sphere. Although in our analysis, the need for such samples is strictly theoretical and the practical feasibility is somewhat immaterial, we describe $\rho$ this way to avoid complication. Given these assumptions, suppose the `encoded' pushforward measure $E_{\#}\mu \in \mathscr{P}(\mathcal{Z})$ has density $p_{E_{\#}\mu}$. Then, every such $p_{E_{\#}\mu}$ is a potential candidate to represent $p_{\rho}$. However, our analysis does not assume the existence of a density. We denote $Supp(p_{\rho})$ by $\mathcal{C}$ such that,

\begin{assumption} \label{Assumption3}
$\textrm{VC-dim}[\mathcal{Y}(\mathscr{P}(\mathcal{C}))]$ is finite.
\end{assumption}

This assumption too is very much plausible in its own right. For example, let $\mathcal{G}_d$ and $\mathcal{A}_d$ denote the class of \textit{d}-dimensional Gaussian distributions and the class of \textit{d}-dimensional axis-aligned Gaussian distributions respectively. It can be shown that, $\textrm{VC-dim}[\mathcal{Y}(\mathscr{P}(\mathcal{G}_d))] = \mathcal{O}(d^2)$ and $\textrm{VC-dim}[\mathcal{Y}(\mathscr{P}(\mathcal{A}_d))] = \mathcal{O}(d)$ \cite{ashtiani2018techniques}. The introduction of Yatracos class of distributions, in turn, facilitates the search for the minimum distance estimate of a measure. For a detailed exposition on the same, one may turn to \cite{Devroye2001CombinatorialMI}.

Perhaps the most challenging of tasks in studying the theoretical aspects of deep neural networks lies in their basic characterization. Several depictions of networks as a collection of functions, indexed by parametric classes have produced notable results in the past \cite{10.1214/19-AOS1858,liang2018generative}. This paper rather explores the mathematical properties a network has to embody to function in a WAE system. The following representation of an encoder map reflects our motivation. We consider an $E \in \mathscr{F}(\mathcal{X},\mathscr{P}(\mathcal{Z}))$ such that,

\begin{assumption} \label{Assumption4}
(i) $E_{\#}\mu$ is supported on $\mathcal{C}$ and is also easy to draw samples from.

(ii) Given any arbitrary empirical measure $\hat{\mu}_n$ (based on $n$ samples) corresponding to $\mu$, there exists an empirical counterpart $\widehat{(E_{\#}\mu)}_n$ of $E_{\#}\mu$ which is $\epsilon$-close to $\hat{\mu}_n$ under E, with probability at least $1-k\exp\{-n^{r}\epsilon^2\}$, where $r \geq 1$ and $k \geq 0$. In other words, $\mathbb{P}\Big(\norm{E_{\#}{\hat{\mu}_n} - \widehat{(E_{\#}\mu)}_n}_{TV} \leq \epsilon\Big) \geq 1-k\exp\{-n^{r}\epsilon^2\}$.
\end{assumption}

Part (ii) of Assumption (\ref{Assumption4}) is provided only to give coherence to the definition of $E$. It suggests that the probability of the discrepancy between the two quantities involved decreases exponentially with the \textit{r}-th power of $n$. As such, if one has a superlinear rate of increment in information in the form of samples, the portion lost in transition decays sharply. This embodies our idea of an encoder, that preserves information. It is often seen in practice that AEs, with ReLU encoders, are able to reconstruct images accurately, despite having the encoded observations to be almost degenerate. In such a case, the two measures under consideration can always be said to be arbitrarily close. Part (ii) is a gentler approximation of the same idea. We now present the main result that enables one to prove consistency in estimating the latent space distribution.

\begin{theorem} \label{Theorem1}
Let, $\widehat{(E_{\#}\mu)}_{n}$ stands for an empirical measure based on $n \geq 1$ i.i.d. samples from $E_{\#}\mu$. Then, under assumptions [\ref{Assumption1}(i), \ref{Assumption2}(i), \ref{Assumption4}(i)], for positive constants $c_1, c_2, c_3$ and $\delta \in (0,1)$ there exists $\lambda^* \geq 0$ such that,
\[\norm{\rho - \widehat{(E_{\#}\mu)}_{n}}_{TV} - \lambda^* \leq \frac{1}{\sqrt{n}}\Big\{c_1\sqrt{v} + c_2\sqrt{\ln{\big(\frac{c_3}{\delta}\big)}}\Big\}\]
holds with probability $\geq 1-\delta$, where $v = \textrm{VC-dim}[\mathcal{Y}(\mathscr{P}(\mathcal{C}))]$.
\end{theorem}

To prove this theorem, we introduce some crucial results from the estimation theory of measures. Proofs of all the lemmas stated henceforth, can be found in the supplement.

\begin{lemma} \label{Lemma1}
For $f,g \in \mathscr{P}(\mathcal{C})$, $\norm{f - g}_{TV} = \:\norm{f - g}_{\mathcal{Y}(\mathscr{P}(\mathcal{C}))}.$
\end{lemma}

\begin{lemma} \label{Lemma2}
Let $\gamma \in \mathscr{P}(\mathcal{Z})$. Also, let $\hat{\gamma}_{n}$ denote the empirical distribution based on $n \geq 0$ i.i.d. samples drawn from $\gamma$. Then there exist positive constants $k_1,k_2$ such that
\[\mathbb{P}\Big(\norm{\gamma - \hat{\gamma}_{n}}_{\mathcal{Y}(\mathscr{P}(\mathcal{Z}))} \leq k_{1}\sqrt{\frac{v}{n}} + t\Big) \geq 1 - \exp{(-k_{2}nt^{2})},\]
where $v$ stands for $\textrm{VC-dim}[\mathcal{Y}(\mathscr{P}(\mathcal{Z}))]$.
\end{lemma}

\begin{proof}[Proof of Theorem \ref{Theorem1}]
Given $n$ observations from $\rho$, let $\hat{\rho}_n$ denote the corresponding empirical measure. Now, using the triangle inequality we have,
\begin{align}
    \norm{\rho - \widehat{(E_{\#}\mu)}_{n}}_{TV} \leq& \;\norm{\rho - E_{\#}\mu}_{TV} + \;\norm{E_{\#}\mu - \widehat{(E_{\#}\mu)}_{n}}_{TV} \nonumber \\
    \leq &\;\norm{\rho - \hat{\rho}_n}_{TV} +  \;\norm{E_{\#}\mu - \widehat{(E_{\#}\mu)}_{n}}_{TV} + \;\norm{\hat{\rho}_n - E_{\#}\mu}_{TV} \label{4}.
\end{align}

Define,
\begin{equation*}
    \mathcal{L}^{*}_n = \argmin_{q \in \mathscr{P}(\mathcal{C})}\:{\norm{q - \hat{\rho}_n}}_{\mathcal{Y}(\mathscr{P}(\mathcal{C}))}.
\end{equation*}
Observe that,
\begin{align}
    \norm{\hat{\rho}_n - E_{\#}\mu}_{TV} \leq &\;\norm{\hat{\rho}_n - \mathcal{L}^{*}_n}_{TV} + \;\norm{\mathcal{L}^{*}_n - E_{\#}\mu}_{TV} \nonumber \\
    = &\;\norm{\hat{\rho}_n - \mathcal{L}^{*}_n}_{\mathcal{Y}(\mathscr{P}(\mathcal{C}))} + \;\norm{\mathcal{L}^{*}_n - E_{\#}\mu}_{TV} \label{5} \\
    \leq &\;\norm{\rho - \hat{\rho}_n}_{\mathcal{Y}(\mathscr{P}(\mathcal{C}))} + \;\norm{\mathcal{L}^{*}_n - E_{\#}\mu}_{TV} \label{6}.
\end{align}

Lemma (\ref{Lemma1}) ensures the equality in (\ref{5}). (\ref{6}) is due to the definition of $\mathcal{L}^{*}_n$.

Now,
\begin{align}
    \norm{\mathcal{L}^{*}_n - E_{\#}\mu}_{TV} =& \;\inf_{\tau \in \mathcal{T}(\mathcal{L}^{*}_n, E_{\#}\mu)}\int c(x,y)d\tau(x,y) \label{7} \\
    =& \;\inf_{\tau \in \mathcal{T}^{'}(\mathcal{L}^{*}_n, \mu)}\int c(x,E(y))d\tau(x,y), \label{8}
\end{align}
where $\mathcal{T}$ and $\mathcal{T}^{'}$ are couplings of measures. The metric $c: \mathcal{Z}\times\mathcal{Z} \rightarrow \mathbb{R}_{\geq 0}$ in (\ref{7}) is the trivial metric $c(x,y) = 1_{x \neq y}$ \cite{OptimalTransport}. This implies, the transformation induced by the encoder should be such that, it minimizes the adversarial loss (\ref{8}). In case $c$ is lower semi-continuous, there exists a $\tau \in \mathcal{T}^{'}$ which minimizes (\ref{8}), and hence an $E$ \cite{alma991004453279705596}. For example, $c(x,y) = 1_{C}(x,y)$, where $1_{C}(x,y)$ is the indicator on an open set $C$. In our situation however, we can always choose a sub-optimal $E$ such that (\ref{8}) $\leq \lambda^*$, for a non-negative $\lambda^*$. Villani \textit{et al.}\:\cite{OptimalTransport} describes various ways to get hold of such a $\tau$.

Going back to (\ref{4}),
\begin{equation}
    \norm{\rho - \widehat{(E_{\#}\mu)}_{n}}_{TV} - \lambda^{*} \leq 2\norm{\rho - \hat{\rho}_n}_{\mathcal{Y}(\mathscr{P}(\mathcal{C}))} +  \;\norm{E_{\#}\mu - \widehat{(E_{\#}\mu)}_{n}}_{\mathcal{Y}(\mathscr{P}(\mathcal{C}))}. \label{9}
\end{equation}

Thus, (\ref{9}) along with lemma (\ref{Lemma2}) proves the theorem.
\end{proof}

It is worth mentioning that in reality however, we only have a sample of size $n \in \mathbb{N}^+$ from the input distribution, say $\{X_i\}^{n}_{i=1}$. Based on the same, an empirical distribution $\hat{\mu}_n = \frac{1}{n}\sum^{n}_{i=1}\delta_{X_i}$ is what we have at hand.

\begin{corollary} \label{Corollary1}
Under the additional assumptions [\ref{Assumption3}, \ref{Assumption4}(ii)],
\[\norm{E_{\#}{\hat{\mu}_{n}} - \rho}_{TV} - \lambda^{*} = \mathcal{O}_{\mathbb{P}}{(n^{-\frac{1}{2}})}.\]
\end{corollary}

\begin{remark}
Despite the fact that the optimal value of (\ref{8}) cannot be made arbitrarily small directly, in case we obtain $\lambda_{n}, n \in \mathbb{N}$, such that, $\limsup_{n \rightarrow \infty}\lambda_{n} = \lambda^{*}$; the same conclusion can be drawn. We call Corollary (\ref{Corollary1}) 'consistency up to $\lambda^{*}$' in this light. Note that, for a given margin of error $\lambda$ in the latent space, $E$ needs to be chosen such that, $\lambda^{*} \leq \lambda$. On the other way round, starting with an unspecified $\lambda$, one can learn about the feasible margin of error by observing $\lambda^{*}$.
\end{remark}

Theorem (\ref{Theorem1}) shows that the discrepancy between the 'encoded' and the 'target' law can be made arbitrarily small around a constant with high probability. Corollary (\ref{Corollary1}), consequently establishes our idea of consistency in latent space. Proof for the same can be found in the supplement. As such, for any suitably chosen $t^*$, the constraint in problem (\ref{Equation3}) is satisfied. We now investigate whether the decoder network is capable of regenerating a distribution 'close' to $\mu$, from $E_{\#}\mu$. A probabilistic decoder, in this context, is formally defined as a function $D:\mathcal{Z} \longrightarrow \mathcal{X}$ such that $D_{\#}\rho \in \mathscr{P}(\mathcal{X})$.

The residual task at hand can be formally written as the minimization of
\[d_{\mathscr{L}^1_c}((D \circ E)_{\#}\mu, \mu) = \sup_{l \in \mathscr{L}^1_c}\Big\{\mathbb{E}_{x \sim (D \circ E)_{\#}\mu}\big[l(x)\big] - \mathbb{E}_{x \sim \mu}\big[l(x)\big]\Big\}\] over $E \in \mathscr{F}(\mathcal{X},\mathscr{P}(\mathcal{Z}))$ which follows assumption (\ref{Assumption4}). Which, however under an observed empirical measure $\hat{\mu}_n$ boils down to its sample counterpart, given by
\begin{equation} \label{Equation4}
    d_{\mathscr{L}^1_c}((D \circ E)_{\#}{\hat{\mu}_n}, {\mu}) = \sup_{l \in \mathscr{L}^1_c}\Big\{\mathbb{E}_{x \sim (D \circ E)_{\#}{\hat{\mu}_n}}\big[l(x)\big] - \mathbb{E}_{x \sim \mu}\big[l(x)\big]\Big\}.
\end{equation}

We remark that the role of the class of functions $\mathscr{L}^1_c$ while minimizing (\ref{Equation4}) is to behave as `critics'. GANs, during their training, gradually learn to be a better judge by casting this functional class through a discriminator \cite{pmlr-v70-arjovsky17a}. This freedom of choice in turn gives one more flexibility to interpret its regenerative capabilities \cite{liang2018generative,10.1214/19-AOS1858,chen2020statistical}. In the case of AEs however, the choice depends solely on the preassigned discrepancy metric. Hence, the theoretical justification of reconstruction for AEs needs additional maneuvering. Our first step in doing so is an assumption on the input distribution.

\begin{assumption} \label{Assumption5}
(i) $p_{\mu} \in \mathcal{H}^{\alpha}(\mathcal{X})$ with $\alpha > 0$, where $\mathcal{H}^{\alpha}(\mathcal{X})$ denotes the H\"{o}lder class based on $\mathcal{X}$; such that, the $i^{\textrm{th}}$ order derivative $p^{(i)}_{\mu}$ exists $(i \leq \lfloor{\alpha}\rfloor)$ and is bounded on $\bar{\mathcal{X}}$. Also,\\ $D^{\lfloor{\alpha}\rfloor}_{p_{\mu}} = \sup_{x,y}\frac{\abs{p^{(\lfloor{\alpha}\rfloor)}_{\mu}(x)-p^{(\lfloor{\alpha}\rfloor)}_{\mu}(y)}}{{\norm{x-y}_2}^{\alpha - \lfloor{\alpha}\rfloor}} < \infty$, for $x\neq y \in \textrm{Int}(\mathcal{X})$.
\end{assumption}

\begin{remark}
Assumption (\ref{Assumption5}) enables one to assign the H\"{o}lder norm, defined as
\[\norm{p_\mu}_{\mathcal{H}^{\alpha}} = \max_{i \leq \lfloor{\alpha}\rfloor}\norm{p^{(i)}_{\mu}}_{\infty} + D^{\lfloor{\alpha}\rfloor}_{p_{\mu}}.\]
Moreover, $\norm{p_\mu}_{\mathcal{H}^{\alpha}}$ turns out to be bounded.
\end{remark}

Smoothness assumptions on input data distribution, in the form of H\"{o}lder or Sobolev densities are quite familiar in the theory of density estimation for generative models \cite{liang2018generative,chen2020statistical,10.5555/3327546.3327686}. Observe that, earlier results still hold under this additional assumption. The smoothness of both the input and latent densities (\ref{Assumption2}(ii)) together, ensures the existence of certain regenerative maps, which are crucial to the forthcoming discussion. Lemma (\ref{Lemma3}) formalizes the same.

\begin{lemma}[\cite{Caffarelli1992TheRO}] \label{Lemma3}
Under assumptions \big[\ref{Assumption1}, \ref{Assumption2}, \ref{Assumption5}\big], there exists a map $T \in \mathcal{H}^{\alpha + 1}$ from $\mathcal{Z}$ to $\mathcal{X}$ such that, $T_{\#}{\rho} = \mu$.
\end{lemma}

In our discussion, similar to the encoder transformation, we do not impose specific structural restrictions on the decoder network. However, the mathematical assumptions that are made, are inspired and supported by real architectures.

\begin{assumption} \label{Assumption6}
The quasi-isometric embedding $D:(\mathcal{Z}, \;\norm{\;}_{TV}) \longrightarrow (\mathcal{X},c)$ induced by the decoder network is  $\epsilon$-close to $T$ in uniform norm, where $T$ is as defined in lemma (\ref{Lemma3}). Also, $(D \circ E)_{\#}\mu$ shares the same support with $\mu$.
\end{assumption}

Classes of functions, learned by generator networks in GANs, are often taken to be smooth \cite{liang2018generative,10.5555/3327546.3327686} to achieve minimax convergence rate. Our assumption of quasi-isometric embedding is gentler in principle. We mention that an alternative weaker assumption of Lipschitz continuity would have sufficed in our analysis. Assumption (\ref{Assumption6}) on the other hand, takes one step forward to comment on the broader geometry of the two spaces under consideration. Moreover, precise specification of parameters of a ReLU network can always result in a map, which is capable of approximating the optimal Monge solution \cite{NEURIPS2019_fd95ec8d,YAROTSKY2017103}, which in our case turns out to be smooth. Note that, the metric on $\mathcal{Z}$ can be equivalently taken as
$\:\norm{\;}_{1}$.

The following lemma provides an oracle inequality by fragmenting the empirical loss as in (\ref{Equation4}), into disjoint parts.

\begin{lemma} \label{Lemma4}
Given an encoder transformation $E^*$,
\[d_{\mathscr{L}^1_c}((D \circ E^*)_{\#}{\hat{\mu}_n}, {\mu}) \leq \mathcal{E}_1 + \mathcal{E}_2 + 2\mathcal{E}_3,\]
where $\mathcal{E}_1 = d_{\mathscr{L}^1_c}((D \circ E^*)_{\#}{\hat{\mu}_n}, D_{\#}{\rho})$ is the 'Encoder Approximation Error'; \\
$\mathcal{E}_2 = d_{\mathscr{L}^1_c}(D_{\#}{\rho}, T_{\#}{\rho})$ stands for the 'Decoder Approximation Error' and \\
$\mathcal{E}_3 = d_{\mathscr{L}^1_c}(\hat{\mu}_n, \mu)$, the 'Statistical Estimation Error'.
\end{lemma}

We remark that successful regeneration by the \textit{f}-WAE can be ensured by showing: these individual errors are arbitrarily small or they approach zero in any probabilistic sense. To demonstrate the same in the case of $\mathcal{E}_3$, we introduce some concepts from \cite{10.1214/aoms/1177697802,weed2017sharp}.

\begin{definition}[1-Upper Wasserstein Dimension] \label{Definition7}
For $\gamma \in \mathscr{P}(\mathcal{X})$, the $(\epsilon, \tau)$-covering number is given by \[\mathcal{N}_{\epsilon}(\gamma, \tau) = \inf_{S \subseteq \mathcal{X}}\{\mathcal{N}_{\epsilon}(S): \gamma(S) \geq 1-\tau\}.\]
Consequently, we define the $(\epsilon, \tau)$-dimension as
$\delta_{\epsilon}(\gamma, \tau) = \frac{\mathcal{N}_{\epsilon}(\gamma, \tau)}{-\log(\epsilon)}$.
Under this setup, the Upper Wasserstein Dimension for $p=1$ is
\[\delta^*_{1}(\gamma) = \inf\{s \in (2,\infty): \limsup_{\epsilon \rightarrow 0}\delta_{\epsilon}(\gamma,\epsilon^{\frac{s}{s-2}}) \leq s\}.\]
\end{definition}

\begin{lemma}[\cite{weed2017sharp}] \label{Lemma5}
Let $diam_{c}(Supp(\mu)) = B$. Then, for $n \geq 0$ and $s > \delta^*_{1}(\mu)$,
\[\mathbb{P}\Big(d_{\mathscr{L}^1_c}(\hat{\mu}_n, \mu) \leq \mathcal{O}(n^{-\frac{1}{s}}) + t\Big) \geq 1 - \exp\Big\{-\frac{2nt^2}{B^2}\Big\}.\]
\end{lemma}

Now, we propose the main theorem which provides a guarantee regarding regeneration.

\begin{theorem} \label{Theorem2}
Given a margin of error $\lambda^{*}$ in latent space, let $\hat{\mu}_{n}$ denote an empirical input measure, consistent up to $\lambda^{*}$, under encoder map $E^*$. Also, let $diam_{c}(Supp(\mu)) = B$. Then, there exists a constant $\zeta^* \geq 0$ and $\delta \in (0,1)$ such that
\[d_{\mathscr{L}^1_c}((D \circ E^*)_{\#}{\hat{\mu}_n}, {\mu}) - \zeta^* \leq \mathcal{O}(n^{-\frac{1}{s}}) + \mathcal{O}(n^{-\frac{1}{2}})\]
with probability at least $1-\delta$, where $s > \delta^*_{1}(\mu)$.
\end{theorem}

\begin{proof}[Proof of Theorem \ref{Theorem2}]
Observe that, the error committed by the encoder
\begin{align}
    \mathcal{E}_1 =& \;d_{\mathscr{L}^1_c}((D \circ E^*)_{\#}{\hat{\mu}_n}, D_{\#}{\rho})
    = \inf_{\upsilon \in \Upsilon(E^{*}_{\#}\hat{\mu}_{n},\rho)} \int c(D(x),D(y))d\upsilon(x,y) \label{12} \\
    \leq & \;A\inf_{\upsilon \in \Upsilon(E^{*}_{\#}\hat{\mu}_{n},\rho)} \int \norm{x-y}_{TV}d\upsilon(x,y) \nonumber \\
    \leq & \;\lambda^{*}AB + \mathcal{O}(n^{-\frac{1}{2}}) \label{13},
\end{align}
where (\ref{13}) holds with probability $\geq 1-\delta$ (using Theorem (\ref{Theorem1})). $\Upsilon$ in (\ref{12}) denotes the associated coupling and $A \geq 1$ is the constant related to the embedding $D$. Say, $\zeta^{*} = \lambda^{*}AB$.

Similarly,
\begin{align}
    \mathcal{E}_2 = \;d_{\mathscr{L}^1_c}(D_{\#}{\rho}, T_{\#}{\rho}) =& \sup_{l \in \mathscr{L}^1_c}\Big\{\mathbb{E}_{x \sim \rho}\big[l(D(x))\big] - \mathbb{E}_{x \sim \rho}\big[l(T(x))\big]\Big\} \nonumber \\
    \leq& \;\mathbb{E}_{x \sim \rho}\big[\norm{D - T}_{\infty}\big] \leq \epsilon^*  \label{14},
\end{align}
which is a consequence of the fact that, $\norm{D - T}_{\infty} \leq \epsilon$ (Assumption (\ref{Assumption6})). Here, $\epsilon^*$ is an arbitrarily small quantity corresponding to $\epsilon$.

Also, lemma (\ref{Lemma5}) suggests that for $s > \delta^*_{1}(\mu)$,
\begin{equation}
    \mathcal{E}_3 = \;d_{\mathscr{L}^1_c}(\hat{\mu}_n, \mu) \leq \mathcal{O}(n^{-\frac{1}{s}}) + \frac{1}{\sqrt{n}}B\sqrt{\frac{1}{2}\ln\big({\frac{1}{\delta}}\big)} \label{15}
\end{equation}
with probability at least $1 - \delta$.

As such, (\ref{13}), (\ref{14}) and (\ref{15}) together prove the theorem.
\end{proof}

\begin{remark} \label{Remark}
Let us weigh up on a special case, which carries notable significance. Suppose, one obtains $\lambda_n$ as an upper bound to (\ref{8}), corresponding to $\mathcal{L}^{*}_n, \;n \in \mathbb{N}^+$, such that $\lambda_n = o(1)$. We do not prove the existence of such a sequence, which might be taken up as a future work. However, under the assumption of existence, corollary (\ref{Corollary1}) implies that,
\[\sup_{\mathcal{A} \in \mathcal{F}}\norm{E_{\#}{\hat{\mu}_{n}}(\mathcal{A}) - \rho(\mathcal{A})} \longrightarrow 0,\]
where $\mathcal{F}$ is the Borel $\sigma$-algebra on $\mathcal{Z}$. As such, $E_{\#}\hat{\mu}_{n}$ converges strongly to $\rho$. This theoretically ensures the achievement of a target latent law.

Similarly, such an $\{\lambda_n\}$ will have a remarkable effect on the regeneration process. In that case, Theorem (\ref{Theorem2}) will imply,
\[d_{\mathscr{L}^1_c}((D \circ E^*)_{\#}{\hat{\mu}_n}, {\mu}) \longrightarrow 0,\]
i.e. $(D \circ E^*)_{\#}{\hat{\mu}_n}$ converges weakly in $\mathscr{P}(\mathcal{X})$ to $\mu$ \cite{OptimalTransport}. This result serves as a reconstruction guarantee.
\end{remark}

Now, let us look back at the original WAE objective in (\ref{Equation1}). Denote by
\[d^{*}_{c}(P_X,(E,D)) = \int_{\mathcal{X}} \mathbb{E}_{z \sim E(x)}[c(x,D(z))] dP_{X}(x)\]
the reconstruction loss. In general, this acts as an upper bound to the \textit{f}-WAE counterpart \cite{NEURIPS2019_eae27d77}. However, under additional restrictions on D, they become equivalent \cite{NEURIPS2019_eae27d77}. In such a case, if $\Omega$ in ($\ref{Equation1}$) is taken to be $D_f$, the two reconstructions also follow the same trait. In our study, the measure of discrepancy of distributions in latent space was throughout taken to be TV. JS divergence, being upper bounded by TV, is obliged to follow the results established exactly. As such, the only task remaining is the transferral of regeneration guarantees to WAEs \cite{tolstikhin2019wasserstein}.

\begin{corollary} \label{Corollary2}
There exists a specific margin of error $\kappa \geq 0$ in the latent space such that, Theorem (\ref{Theorem2}) holds exactly for $d^{*}_{c}(\hat{\mu}_{n},(E^*,D))$, where $D$ is also invertible.
\end{corollary}

\begin{proof}[Proof of Corollary (\ref{Corollary2})]
Husain \textit{et al.}\:\cite{NEURIPS2019_eae27d77} had suggested the existence of a $\kappa \geq 0$, such that for all margin or error $\lambda > \kappa$, (\ref{Equation1}) and (\ref{Equation2}) are exactly equal. The proof however, demands the decoder transform to be invertible. In such a case, for an encoder $E^*$ and $\Omega = D_f$ imply that
\[d^{*}_{c}(\hat{\mu}_n,(E^*,D)) = \int_{\mathcal{X}} \mathbb{E}_{z \sim E^{*
}(x)}[c(x,D(z))] d\hat{\mu}_{n}(x) \leq d_{\mathscr{L}^1_c}((D \circ E^*)_{\#}{\hat{\mu}_n}, {\mu}) + d_{\mathscr{L}^1_c}(\hat{\mu}_n, {\mu}).\]

Hence, using lemma (\ref{Lemma2}) and Theorem (\ref{Theorem2}) we say that
\[d^{*}_{c}(\hat{\mu}_n,(E^*,D)) - \zeta^* \leq \mathcal{O}(n^{-\frac{1}{s}}) + \mathcal{O}(n^{-\frac{1}{2}})\]
holds with probability at least $1-\delta$, for $s > \delta^*_{1}(\mu)$.
\end{proof}

We emphasize the fact that the argument of weak convergence, however, as given in (\ref{Remark}), does not apply directly to this case. The Wasserstein distance metrizes $\mathscr{P}(\mathcal{X})$, i.e. weak convergence holds if and only if $d_{\mathscr{L}^1_c} \longrightarrow 0$. On the other hand, $d^{*}_{c}$ does not satisfy the axioms of a distance. As such, WD produces stronger inferences, even though under carefully taken assumptions they tend to be equal.

\section{Conclusion}
Our analysis, besides serving as the first of its kind theoretical promises regarding the simultaneous tasks a WAE has to perform, also points toward untraveled avenues. An immediate extension might look into broader classes of discrepancy measures in latent space. The latent space and hence the distribution that characterizes it might also be in focus. An even elaborate characterization can deliver it further justice. Our study includes a commentary on the geometry preserving nature of deep generative networks from a fairly new perspective. Being a novel representation, it creates new opportunities for inventive scrutiny. We also restrict our analysis to a non-parametric regime, which may encourage similar endeavours under a parametrized setup. Testing the regeneration capabilities of WAE for even general classes of distributions might be interesting too.

\section{Future Work}
Another WAE architecture that appears in \cite{tolstikhin2019wasserstein}, uses Maximum Mean Discrepancy (MMD) instead of JS divergence. To the best of our knowledge, there is no evidence of theoretical superiority that MMD enjoys. However, uniform convergence bounds on empirical distributions under MMD, as established by \cite{10.5555/2188385.2188410} in a different context, produces similar convergence rates. A comparative statistical analysis of the two architectures might be fruitful from a practitioner's viewpoint.

The concept of `disentanglement' in AE context remains somewhat unclear to this day. While it is not properly defined, to the best of our understanding, it can arguably be described as the process of achieving a factorial representation; one with 'expressive' and perhaps independent coordinates, which characterize the features. A somewhat gentle representation of the same might be a distribution with an axis-aligned covariance structure (diagonal dispersion matrix). Although our work allows for the latent distribution class to be $d$-dimensional axis-aligned Gaussians, a parametric approach might help in studying the relationship between regularization and disentanglement.

The non-parametric approach adopted in this work, does not take into consideration specific model architectures of the networks in use. An immediate investigation may look into the effect of network parameters such as, depth, width, activation functions etc. on regeneration and consistency in the latent space. This, in turn, may improve the theoretical results based on training plans as well.

\newpage

redpajama_set_name: RedPajamaArXiv


Registration Opens: 01/05/2012 Cost: $695 - Customers of Deltek and Lohfeld Consulting Group can receive a current customer discount of $145 for registering for this class. You can receive an additional $50 discount if you register three or more students or you register for the three classes in this educational seminar series. Additional classes in the series include Mastering Proposal Writing for Technical and Managerial Professionals and Learn Capture Management – How government contractors win business. Breakfast and lunch are included in the registration fee.


Nineteen-year-old Ben was at the cusp of a new life – he was a month away from joining the National University of Singapore (NUS) – when he was struck by shattering news: he had acute lymphocytic leukaemia (ALL), a blood cancer that usually occurs in children and rarely in adults. Overnight, Ben, who had been looking forward to the challenges of student life, was facing far graver challenges. "It was very upsetting. I had no idea if I could go back to a normal life," Ben recalls. To add on to his worries, the financial burden of seeking the complex treatment he needed was too high for his parents, who were both teachers.
Fortunately, help was at hand. Ben's plight was brought to the attention of Lee Foundation, which funded his treatment by Associate Professor Allen Yeoh, a paediatric cancer specialist and key paediatric cancer researcher at NUS Medicine. A/Prof Yeoh, who is one of the brains behind the NUH's 85-percent cure rate for childhood leukaemia, took Ben on as his first adult patient in his leukemia research trials. NUH is the principal teaching hospital of NUS Medicine and A/Prof Yeoh is Senior Consultant at University Children's Medical Institute's Division of Paediatric Haematology-Oncology at the NUH.
Ben eventually beat the odds – he graduated from NUS' School of Design and Environment, and became one of A/Prof Yeoh's and his team's many success stories.
A/Prof Yeoh, the Viva-Goh Foundation Associate Professor, attributes his department's success to a great extent to gifts from foundations such as the Lee Foundation, Children's Cancer Foundation , the Goh Foundation and Viva Foundation for children with cancer, whose seed funding allowed him to, as he says, "catch the winds" of progress in biomedical research in Singapore. Philanthropic funding such as these enabled him to build infrastructure and manpower, which is crucial to monitoring patients over the years and building data for his research.
A/Prof Yeoh and NUS Medicine have long been at the vanguard of paediatric leukaemia research and discovery. His mentor, Prof Quah Thuan Chong, Department of Paediatrics, NUS Medicine, was among the first in Asia to replace the traditional routine radiation therapy to the brain and testes, which affected children's IQ and left males sterile, with high doses of chemotherapy. The cure rate jumped from 50 to 75 percent in trials that ran from 1988 to 1996.
A/Prof Yeoh and his team continue to charge ahead with their pioneering work and are now exploring one of the most promising and exciting new treatments for cancer – immunotherapy. Unlike chemotherapy, it involves harnessing the power of the body's immune system to attack cancer cells. A key member of his team is Prof Dario Campana, described by Forbes magazine as a "real thought leader in the field of cancer cell therapies". Prof Dario, who holds the prestigious title of NMRC STaR Investigator, and is also the holder of the Mrs. Lee Kong Chian Chair in Advanced Cellular Therapy, has already developed at least 2 landmark cellular-based cancer therapies that have proven useful clinically.
It is with the help of dedicated scientists like Profs Yeoh and Campana and the generosity of donors like the Lee Foundation, the Goh Foundation, the Children's Cancer Foundation and the Viva Foundation for children with cancer that, as with the story of Ben, those of many other cancer-stricken children, continue to have the same happy endings to this day.
Your generous support will help to make medical education more accessible to the best and brightest from all levels of society, with diverse backgrounds.


text:
Q: How do I split and join a file that contains both numbers and strings? So for the following variable all I'm trying to do is replace the _ with ,:
Here is the code i'm trying:
Thefile= '''
"Name"_"Course"_"Grade"
"Norman White"_"Room 8-84, KMEC"_100
"Betty White"_"Room 8-85, KMEC"_98
"Student1"_"43 West Street, NYC"_95
"Student2"_"28 Mainstreet, Hoboken NJ"_93
"Student99"_"99 2nd. Ave., NYC"_85 '''

print(Thefile)
thefile_split = Thefile.split
[str(i) for i in thefile_split]
s = ","
s = s.join(thefile_split)
print(s)`

in which I get the following error:
TypeError: 'builtin_function_or_method' object is not iterable
Do I need to format the variable first or is my code wrong?

A: Since you only need to replace one character with another, you could simply use regular expressions to get the job done:
import re
Thefile= '''
"Name"_"Course"_"Grade"
"Norman White"_"Room 8-84, KMEC"_100
"Betty White"_"Room 8-85, KMEC"_98
"Student1"_"43 West Street, NYC"_95
"Student2"_"28 Mainstreet, Hoboken NJ"_93
"Student99"_"99 2nd. Ave., NYC"_85 '''

result = re.sub('_', ',', Thefile)
print(result)

re.sub looks for a '_' in the whole string and replaces it with ','
Output
"Name","Course","Grade"
"Norman White","Room 8-84, KMEC",100
"Betty White","Room 8-85, KMEC",98
"Student1","43 West Street, NYC",95
"Student2","28 Mainstreet, Hoboken NJ",93
"Student99","99 2nd. Ave., NYC",85

redpajama_set_name: RedPajamaStackExchange


We all grew up drinking milk and were told how important it was to our good health. However, when one looks closely at the scientific literature, it becomes clear that milk is nothing like the perfect food it's been marketed as. Not only is the research supporting the relationship between milk and bone health sketchy, a surprising number of additional health concerns are associated with milk consumption. Moreover, unbeknownst to most people, the production of milk has serious environmental and animal welfare impacts worthy of closer examination.
Listed below are a few highlights from the book. Visit the Resources section for additional information on these topics.
Infants and children fed cow's milk are more likely to suffer from colic, constipation, intestinal bleeding, iron deficiency anemia, ear infections, and weight problems, and are at greater risk of developing Type I diabetes.
The largest study of its kind, published in Archives of Pediatrics & Adolescent Medicine, reviewed more than 12,000 children aged nine to fourteen from every state in America. It found that the more milk the children drank, the fatter they became — even if they followed the current federal recommendation of three daily servings.
The California Farm Bureau Federation reported that when all dairy farming and milk processing water needs are taken into consideration, 48.3 gallons of water are used to produce one eight-ounce glass of milk.
Many people will be surprised to learn methane gas is twenty-three times more powerful than carbon dioxide as a greenhouse gas. Methane is the gas released by the flatulence and belches of beef and dairy cattle.
According to Michael Abberton of the British Department for the Environment, Food and Rural Affairs, an average dairy cow will release between one and two hundred liters of methane a day.
Cows in the United States produce an estimated hundred million tons of methane gas annually, which represents about 20 percent of the country's total annual emissions of the gas.
The average dairy cow produces 120 pounds of manure every day. A small dairy farm of two hundred cows can produce as much waste as a city of ten thousand people.
According to a Senate Committee on Agriculture, Nutrition and Forestry study, the quantity of urine and manure that California's dairy herds produce annually is equivalent to the waste produced by a city of twenty-one million people.
An EPA study estimated that in total, US dairy cows produce 54 billion pounds of manure annually. That's over two-and-a half times the amount produced by humans.
According to the United States Department of Agriculture (USDA), in routine samplings milk is shown to be contaminated with a variety of pesticide residues including carcinogens, neurotoxins, developmental and reproductive toxicants, and suspected hormone disruptors.
For a comprehensive list of detected pesticides and their effects, please visit this What's On My Food.
Perchlorate is the explosive ingredient in the fuel that propels missiles, rockets, and fireworks. Exposure to perchlorate may contribute to motor-skill defects, I.Q. deficit, and mental retardation in children, and may cause cancer.
The FDA reports that perchlorate has been detected in 217 of 232 samples of milk derived from supermarkets in fifteen states, including Arizona, California, Georgia, Kansas, Louisiana, Maryland, Missouri, North Carolina, Pennsylvania, South Carolina, Texas, Virginia, and Washington state.
Dioxin is a byproduct released into the environment during certain manufacturing processes including the production of certain pesticides, chlorine-containing chemicals, pharmaceuticals, plastics, and the chlorine bleaching of wood pulp to make paper products. Other major sources of dioxin are waste incinerators that burn plastic, paper, and chlorine-containing medical waste.
Dr. Arnold Schecter, a medical expert on dioxin and an advisor to the World Health Organization (WHO), has said even tiny amounts of dioxin have been shown to result in nervous system and liver damage. Dietary exposure to dioxin from milk is second only to that from beef consumption.


text:
CyrusIoC
========

Mini IoC container for mobile devices - optimized for xamarin forms shared PCL's

redpajama_set_name: RedPajamaGithub


Manchester City's FA Cup semi-final clash with Brighton will be played at Wembley Stadium on Saturday 6 April, kick-off at 5:30pm BST.
Pep Guardiola's side will be hoping for a repeat of our 2-0 Premier League win over the Seagulls back in September as we look to book our place in another cup final.
The game will be shown live in the UK on BBC One.
You can also follow live updates via our Matchday Centre or our Twitter: @ManCity.


text:
Monroe Police Department's Detective Sgt. Cindy Chessie stands along Main Street on Wednesday in Monroe. Chessie retired Thursday to become director of the Sky Valley Food Bank. (Andy Bronson / The Herald)
Monroe Detective Sgt. Cindy Chessie retires to lead food bank
by Eric Stevick
Friday, June 30, 2017 10:51pm
Local NewsMonroe
MONROE — When there was a major crime to solve, the Monroe Police Department would turn its training center into a war room.
Behind those doors was a study in contrasting methods.
Detective Sgt. Cindy Chessie would hang 2-by-3-foot Post-it notes and jot down critical information with colored highlighters. Photos of victims and suspects, quickly assembled biographies and lists of addresses and phone numbers lined the walls.
Detective Barry Hatch's first instinct, on the other hand, was to start with a Word document and Excel spreadsheet.
Their brains worked differently but their thoughts were remarkably in sync. When it would come down to strategy — say, approaching the victim or a witness first — their thoughts were always the same.
The way Hatch describes it, his boss wasn't one who could sit back and watch other people work. They would joke that she was more his partner than his supervisor.
"She was a fabulous detective before she was a supervisor," Hatch said. "Her investigative mind never quit. That's why she was such a great partner."
On Thursday, Hatch lost his partner when Chessie retired. She'd been with Monroe police for 25 years and is the first female officer to retire from the department. She reviewed all felony cases. Her team of detectives had been together for a decade. And she remained on call once a month on weekends.
"A big heart is leaving our department and that is going to be hard to replace," said Debbie Willis, an administrative director with the police agency.
Chessie, who grew up in the Skykomish Valley, isn't going far. She'll be the new executive director of the Sky Valley Food Bank, where she had been a volunteer for nearly five years.
While some detectives prefer anonymity, Chessie asks people how they're doing when she pushes her shopping cart down the grocery store aisle. She stays in touch with families years after a case has been resolved. She's been involved in Special Olympics for 20 years and led the Monroe Police Officer's Association in providing dinners for the homeless, organizing Easter egg hunts and shopping with needy children at Christmas.
She was among the law enforcement officers and athletes to carry the Special Olympics torch in China in 2007. She considers her leg on the Great Wall one of the highlights of her life.
"She always did so much for those people who are most vulnerable — the children, domestic violence victims who couldn't stand up for themselves, the murder victims, the mentally disabled," Hatch said.
One of the things Chessie stressed to her detectives was to make contact with the victim or their families first because they are often left in the dark. She wanted her team to go to hospitals and funerals.
Long before she was a detective, Chessie worked 12-hour graveyard shifts, often alone, as a patrol officer. Monroe was a smaller town back then. She would finish her shift, hustle home, make her sons a good breakfast and then check that they brushed their teeth before they headed off to school.
She somewhat stumbled into law enforcement. After high school, she did two stints on a fish processing boat in Alaska. She also worked at her parents' restaurant, the Dutch Cup, in Sultan. She remembers her dad, Don Newquist, would bring home customers on Christmas Eve if they had nowhere better to go.
She took a job as a part-time clerk with the Sultan Police Department where she used an electric typewriter. There, she was encouraged to become a reserve officer. She eventually became an officer and served in Sultan for just a few months when she was recruited to Monroe. She was in her early 30s at the time.
There was something about the new recruit that appealed to Colleen Wilson, then a Monroe lieutenant.
What Chessie lacked in experience, she made up for in passion and energy.
That's how Wilson saw it any way. Wilson later became the police chief and made Chessie a detective and promoted her to detective sergeant in 2001.
By then Wilson had seen deeper qualities, the connections Chessie made with abused children and assault victims and how hard she worked on their behalf.
"She cares from her very soul. That's how she is wired," said Wilson, who spent 40 years in law enforcement.
Chessie earned respect from outside law enforcement agencies.
When an inmate strangled corrections officer Jayme Biendl at the Washington State Reformatory chapel in 2011, the detective sergeant reached out to the Snohomish County Sheriff's Office Major Crimes Unit for extra help.
Detectives Dave Bilyeu and Brad Walvatne were assigned to help Hatch and Chessie. The investigation led to a guilty verdict and death sentence for Byron Scherf, a convicted rapist serving a life term. Over the years, Chessie stayed in contact with Biendl's family and has helped organize an annual memorial run in a tribute to the corrections officer.
"Cindy was more of a leader by example," Bilyeu said. "They appeared to be partners, but Cindy was in fact the supervisor. Cindy didn't have the need to have you know she was the supervisor … The community as a whole is a much better place because of Cindy's work over the years."
Chessie rattles off a list of memorable cases her unit investigated: convictions in the Biendl case, a whodunit homicide in which a key piece of evidence was a bottle of Frank's Red Hot hot sauce and a bank robber who used an inner tube as a means of escape down the river.
"Being a detective is really more about teamwork and being open to other people's opinions," she said.
Chessie worked on many child abuse investigations, including one of Snohomish County's early homicide-by-abuse cases in 1998. The suspect, a daycare owner, eventually pleaded guilty to killing Charlotte Wetzel who was just 16 months old.
There also are the unsolved cases that she still thinks about. Seventeen years after her remains were found in a swamp east of town, information seeking tips about the disappearance of Cynthia Rearden can be found on a bulletin board in the lobby of the Monroe Police Department.
Rearden last was seen in city limits in June 1999. Chessie knew her son and her stepdad.
Hatch always admired Chessie's close ties to her community, along with her compassion.
"Our little family is getting disrupted," he said the other day. "We have worked really, really well together for so many years and now we have a huge change."
On Tuesday night, Monroe Mayor Geoffrey Thomas gave Chessie a key to the city to a long and loud applause.
Monroe Police Chief Tim Quenzer said he will miss the detective sergeant.
When Chessie came into his office to break the news that she was retiring to work at the food bank, he wasn't surprised she would be keeping busy and doing something to help others.
"… I just thought this was perfect. I wish her absolutely the best."
Eric Stevick: 425-339-3446; stevick@heraldnet.com.
Everett ready to award bid work for pedestrian bridge
'Bipartisan' state budget averts partial government shutdown

redpajama_set_name: RedPajamaCommonCrawl


The I visa (foreign news media) allows employees working in the field of media, press, and radio to reside temporarily in the United States for journalistic purposes in the course of carrying out their jobs.
Journalists and correspondents who work for a newspaper in Germany, for example, or a U.S. media office can travel to the United States under an I visa for employment-related purposes to cover events in the United States for publications located outside of the U.S.
Journalists or (freelance) media staff, however, are not generally required to have an I visa. It is not whether someone works strictly as a journalist, but rather the specific purpose of the stay that decides whether media representatives need a visa and, if so, which one.
The applicant maintains their permanent residence outside the U.S. and intends only to travel to the U.S. for a temporary stay. Proof of sufficient financial resources to allow stay in the U.S. also plays a considerable role in the examination of the application, particularly if the applicant is self-employed.
In addition, applicants must demonstrate that they work for a foreign media company. This is usually done by presenting an employment contract that confirms that the person works as a reporter or editor, for example. Even freelance journalists or freelancers working for production companies can qualify for an I visa if they work for a media company outside the United States, they are on assignment in the United States under contract, and they can prove that they have received regular fees in the past. Usually the U.S. consulate also requires applicants to present their press pass.
The journalist must be exclusively engaged in the dissemination and collection of news and information while in the U.S. This means that the collected information (for example, the reportage and footage) must only be of a documentary character and intended for publication outside the United States.
Persons or companies that are producing commercial projects (advertising photography, etc.) and projects in the entertainment industry (docusoaps, entertainment programs, etc.) are excluded from the I visa category. Applicants seeking to engage in these activities must apply for an appropriate work visa (O, P or H), which provides a work permit from the U.S. Citizenship and Immigration Services (USCIS).
Applicants must remain employed/under contract with a media company located outside the United States during the entire stay. The visa holder must return to their original workplace after the stay, and they may not accept any fees from a U.S. company or entity. It is therefore essential to maintain a connection with a media company outside the United States.
The project cannot mostly be financed by an entity in the United States.
Usually I visas are granted for a period of 1 to 5 years (depending on the project) and allow a maximum residence status in the United States of up to 365 days per entry (several times consecutively or at a time).
Therefore, representatives of media companies who make frequent trips to the U.S. for journalistic purposes do not have to constantly reapply for the visa.
Spouses and unmarried children under 21 years of age receive a derivative I visa lasting the same period as the principal applicant's I visa. Once children reach the American age of majority, they must change their non-immigrant visa status or leave the country.
Family members are not allowed to accept employment. Persons with an I status are not allowed to apply for a general work permit (Employment Authorization Document EAD). However, I visa holders can attend a public/private educational institution.
Accompanying relatives of I visa holders can travel up to 90 days without a visa in the United States if their nationality permits visa free travel.


DUPLEX LOCATED CLOSE TO LOMITA PINES ON A QUIET STREET. WONDERFUL SPACIOUS LOT W/ OVER 8300 SQ. FT. OWNER HAS INSTALLED SOLAR PANELS ON BOTH UNITS. TAX RECORDS SHOW TWO - 1 BEDROOM W/ 1 BATH UNITS. CURRENTLY BOTH UNITS ARE NOW BEING USED AS 2 BEDROOMS W/1 BATH. WONDERFUL FRONT YARD. LONG DRIVEWAY IS PERFECT FOR R.V. PARKING OR SPACE FOR 4-6 CARS. LAUNDRY HOOK-UPS IN BOTH UNITS. BACK UNIT HAS FENCED IN YARD. HERE IS YOUR CHANCE TO LIVE IN ONE AND RENT THE OTHER OR 2 FAMILIES WHO WANT TO OWN. COPPER PLUMBING ( BUYER TO VERIFY). CLOSE TO FREEWAY AND SHOPPING.
Welcome to this 0 bedroom 0 bath Residential Income in the 121 - Lomita section of Lomita located at 25975 Appian Way built in 1949. It is 0 square feet and the current asking price is $ 659,000.


outdoor trash can storage cabinet full size of outdoor garbage can enclosure stainless steel garbage can under sink trash can garbage outdoor trash can storage cabinet.
outdoor trash can storage cabinet outdoor trash can gallon trash can 8 gallon trash can metal trash can with lid outdoor garbage can storage cabinet.
outdoor trash can storage cabinet trash can cabinet outdoor full size of outdoor trash shed outside garbage can storage outdoor storage cabinet outdoor cupboard plastic outdoor trash cabinet outdoor t.
outdoor trash can storage cabinet garbage can storage ideas garbage can storage kitchen garbage storage ideas garbage bin storage cabinet outdoor garbage storage ideas outdoor garbage can storage cabi.
outdoor trash can storage cabinet outdoor garbage bin storage outdoor garbage bin storage outdoor trash bin storage storage for garbage cans trash can storage outdoor outdoor garbage can outdoor trash.
outdoor trash can storage cabinet decoration outdoor trash bin storage cabinet ideas amazing can enclosure outdoor trash bin storage cabinet.
outdoor trash can storage cabinet trash container storage shed trash bin storage garbage storage cabinet outside garbage can storage outdoor garbage outdoor garbage storage cabinet.
outdoor trash can storage cabinet kitchen garbage can storage wooden kitchen trash cans for kitchen trash can storage cabinet small size outside trash storage cabinet.
outdoor trash can storage cabinet trash can storage medium size of outdoor garbage can outdoor trash can storage cabinet commercial garbage outdoor trash can storage cabinet.
outdoor trash can storage cabinet outdoor trash can storage cabinet kitchen garbage can storage trash can storage garbage can storage kitchen trash can storage cabinet small outdoor garbage outdoor tr.
outdoor trash can storage cabinet outdoor trash can enclosure trash can cabinet outdoor trash can storage cabinet outdoor trash can storage outdoor trash storage cabinet.
outdoor trash can storage cabinet outside trash can storage amazing outdoor trash can with wheels refuse storage shed garbage pail trash outdoor trash bin storage cabinet.
outdoor trash can storage cabinet trash can storage cabinet tilt out bin double hardware outdoor outdoor garbage can storage cabinet.
outdoor trash can storage cabinet trash can cabinet outdoor trash can storage cabinet outdoor trash can storage cabinet trash bin storage outside trash storage cabinet.
outdoor trash can storage cabinet trash can cabinet outdoor outdoor trash can enclosure prodigious attractive storage ideas home interior 4 outdoor trash can cabinet outdoor outdoor trash bin storage.
outdoor trash can storage cabinet trash can storage cabinet trash bin storage kitchen trash can storage cabinet mobile kitchen island w outdoor trash bin storage cabinet.
outdoor trash can storage cabinet outdoor garbage can storage bin picture of kitchen trash can storage cabinet garbage can storage cabinet outdoor trash can storage cabinet.
outdoor trash can storage cabinet outdoor trash can storage astonishing outdoor trash can holder outdoor garbage can storage outdoor trash can storage cabinet outdoor outdoor garbage storage outdoor t.
outdoor trash can storage cabinet awesome outdoor trash can storage cabinet outside trash storage cabinet.
outdoor trash can storage cabinet trash can storage outdoor garden storage garden storage garden storage shed garbage outdoor storage storage bin outside trash storage cabinet.
outdoor trash can storage cabinet outdoor trash can storage cabinet trash can storage bins small size of pallet garbage bin storage outdoor trash can storage cabinet outdoor garbage can storage cabine.
outdoor trash can storage cabinet outside trash can storage outdoor trash can storage cabinet garbage can storage plans storage cabinet for outdoor garbage can storage cabinet.
outdoor trash can storage cabinet garbage can holder outdoor trash can holder outdoor trash can storage awesome shed wood garbage within garbage can outdoor garbage storage cabinet.
outdoor trash can storage cabinet outdoor trash can storage cabinet kitchen trash can storage cabinet kitchen garbage can storage storage can storage cabinet outdoor bin garbage outdoor outdoor trash.
outdoor trash can storage cabinet related post outdoor garbage can storage cabinet.
outdoor trash can storage cabinet outdoor garbage storage medium size of garbage can outdoor trash can storage shed plastic outdoor storage outdoor garbage storage cabinet.
outdoor trash can storage cabinet outdoor trash can storage cabinet outdoor trash bin storage cabinet outdoor trash can storage cabinet outdoor garbage can storage cabinet.
outdoor trash can storage cabinet outdoor trash can storage cabinet elegant trash can outdoor trash can cabinet outdoor trash bin storage outside trash storage cabinet.
outdoor trash can storage cabinet outdoor trash can wheelie bin storage cabinets outside trash can storage trash bin storage for outdoor trash can storage cabinet.
outdoor trash can storage cabinet outdoor trash can enclosure kitchen garbage can storage garbage storage cabinet garbage outdoor storage storage wooden outdoor trash outside trash storage cabinet.
outdoor trash can storage cabinet indoor trash can storage garbage can storage outside outdoor trash bin storage cabinet outdoor outdoor trash outdoor garbage storage cabinet.
outdoor trash can storage cabinet outdoor trash can storage cabinet outdoor trash can storage cabinet home design ideas outdoor trash can storage cabinet outside trash storage cabinet.
outdoor trash can storage cabinet storage for trash cans outdoor trash can storage cabinet outside trash can storage trash storage cabinet outdoor trash can storage cabinet.
outdoor trash can storage cabinet garbage can storage outdoor trash can storage cabinet fresh outdoor trash can storage cabinet outdoor designs outdoor garbage can storage cabinet.
outdoor trash can storage cabinet outdoor trash bin outdoor trash bin storage cabinet outdoor outdoor trash bin storage trash can cabinet outdoor trash can storage cabinet.
outdoor trash can storage cabinet outdoor trash can storage cabinet garbage can storage shed trash cans bin outdoor cabinet outdoor trash can storage cabinet.


text:
Sonosite iViz Plays Key Role in POCUS Education in Panama
Emergency physician Matthew Lyon, MD FACEP, of the Medical College of Georgia at Augusta University is actively involved in an ultrasound education exchange with medical students and residents at the University of Panama. The school is currently teaching 1,000 medical...[Read more]
Michigan Doctors Bring Ultrasound Education to Madagascar
Pediatric and internal medicine resident Marie Arnaout, DO, and Assistant Professor of Internal Medicine Prashant Patel, DO, recently returned from Madagascar on a mission for Western Michigan University School of Medicine . They were accompanied by internal physician Richard Roach,...[Read more]
Ultrasound Guy Gives Back to Jamaica with Sonosite's Help
Trevor Dixon, MD, RDMS, is the Director of Emergency Ultrasound at Jacobi Medical Center in New York City and the founder and CEO of the JAHJAH Foundation (Jamaicans Abroad Helping Jamaicans At Home). Because of his specialty, Dr. Dixon is...[Read more]
Anesthesiologist Achieves Impressive Results with Nerve Blocks in Ecuador
Pain management specialist Joseph Marino, MD, is the chief medical officer at Long Island Jewish Valley Stream. He also serves as the director of regional anesthesia for Northwell Health and holds a faculty position at the Zucker School of Medicine...[Read more]
Ultrasound Provides Answers for Rebuilding Lives in Nepal
Dr. Jesus Casado Cerrada, Internist at the Hospital Universitario de Getafe and Professor at the Universidad Europea, Madrid, Spain, has travelled to the Rasuwa district of Nepal to help a local NGO rebuild the region's infrastructure following a severe earthquake...[Read more]
A Targeted Approach to Pain Management
Point-of-care ultrasound (POCUS) has a significant role to play in pain management for patients around the world. Dr. Victor Mayoral Rojals, Consultant Anesthetist and head of the pain unit at the Bellvitge University Hospital, in Barcelona, Spain, discusses the many...[Read more]
Renal Imaging and Blood Clotting for COVID-19 Patients: A Follow-Up Talk with Dr. Justin Kirk-Bayley
Dr. Justin Kirk-Bayley is the Clinical Director of Therapeutics at the Royal Surrey County Hospital in Guildford, England. Dr. Kirk-Bayley is an expert in the use of ultrasound in the ICU and anesthesiology. On April 30, Fujifilm Sonosite Chief Medical...[Read more]
Italian Hospital Makes Headway against COVID-19
Last month, we interviewed Dr. Enrico Storti , Anesthesia and ICU Director/Unit Coordinator of the Emergency Department at Maggiore Hospital, from the front line of the COVID-19 pandemic in Lodi, Italy. In this latest update from Lodi, BBC show Newsnight...[Read more]
Ultrasound Vital for Women's Healthcare in Rural Ghana
Dr. Carlye Jensen practices emergency and family medicine in Canada. She recently went on a mission for Ghana Rural Integrated Development (GRID) , whose purpose is to provide basic medical care to the rural areas of northern Ghana, where many...[Read more]
One Woman's Mission to End Preventable Infant and Maternal Deaths in Somaliland
Midwife Edna Adan has always dreamt of building her own hospital. After training in London in the 1950s, she qualified as a midwife and returned to her home in Somaliland. Unfortunately, a protracted civil war meant that it was many...[Read more]


text:
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical" >

    <TextView
        android:id="@+id/week_view"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:gravity="center"
        android:textSize="12sp"
        />

    <TextView
        android:id="@+id/weather_date_view"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:gravity="center"
        android:textSize="12sp"
        />

    <ImageView
        android:id="@+id/weather_picture_image_view"
        android:layout_width="42dp"
        android:layout_height="30dp"
        android:scaleType="fitCenter"
        android:layout_gravity="center_horizontal"
        />

    <TextView
        android:id="@+id/weather_info_view"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:gravity="center"
        android:textSize="12sp"
        />

    <TextView
        android:id="@+id/temperature_view"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:gravity="center"
        android:textSize="12sp"
        />

</LinearLayout>


text:
Сума-Шальская — река в России, протекает по Пудожскому району Карелии. Впадает в озеро Шальское, через которое протекает Шалица. Длина реки составляет 19 км, площадь водосборного бассейна — 146 км².

В среднем течении пересекает дорогу  («Вологда — Медвежьегорск — автомобильная дорога Р-21 "Кола"»). В нижнем течении принимает правый приток — Кобылий.

Данные водного реестра
По данным государственного водного реестра России относится к Балтийскому бассейновому округу, водохозяйственный участок реки — Водла, речной подбассейн реки — Свирь (включая реки бассейна Онежского озера). Относится к речному бассейну реки Нева (включая бассейны рек Онежского и Ладожского озера).

Код водного объекта в государственном водном реестре — 01040100412102000017071.

Примечания

Бассейн Шалицы
Реки Пудожского района

redpajama_set_name: RedPajamaWikipedia


text:
"Kelly Clarkson" hits five-week high, widening lead over all new shows
Review based on a national syndicated rating report for the week ending January 26.
In the week ending January 26, wherein much of syndication was stymied by four days of near-blanket coverage of President Trump's impeachment hearings, Kelly Clarkson (five-week high 1.4 live plus same day national Nielsen rating, up 8% from the week before) was a standout.
"Clarkson," which has already been renewed and has ranked as the number one new show of the year for 20 weeks in a row, attracted a crowd of nearly 2.0-million daily viewers, widening the gap between itself and everything else in first season syndication.
The popular rookie fronted by the former "American Idol" also churned out a 17% increase in the coveted W25-54 age group to its strongest demo performance since the week of September 30.
Also greenlit for year two, Tamron Hall (1.0, down 9%) hit a new season high the week before but this time around was broken out on three of the five days due to numerous preemptions and a corresponding loss of coverage.
The remaining talk newcomer, Mel Robbins (0.4, down 20%) trailed both "Hall" and "Clarkson" and reportedly will cease production in the spring.
Veteran talkers were led by powerhouse Live with Kelly and Ryan (2.1, up 5%) and Ellen DeGeneres (2.1, up 11%).
Usual talk leader Dr. Phil (2.0, down 23%) did not get to air its primary run on certain days in many markets but was still in the hunt.
Usual syndication leader Judge Judy (5.7, down 16%), was also blacked out all over the place, but still more than doubled anything else in daytime.
Also coming up big, Hot Bench (five week high 2.1, up 5%), which was created by Judge Judy Sheindlin, tied "Live" and "Ellen" as the second highest show in daytime after only "Judy."
Elsewhere Extra (0.9, steady) and TMZ (1.1, steady) were the only magazines in the big six to hang tough and avoid any losses in a very difficult frame. Entertainment Tonight (2.8, down 3%) led the magazines for the third straight week while all three of the other five shows in the top six dropped by double digits,
Games as a Service vs. Games as a Relationship
FTC moves to block merger of Harry's and Schick maker
"Live" Thrives in Christmas Week
"Barrymore" Boffo Amid Biden Buzz


text:
Crushers Conclude Season with Success at Nationals
HOUSTON- Matthew Stephens and Caroline Safford finished their climbing season by competing in the USA Climbing National Competition over the weekend. The event marked the first time in program history in which Brevard competed in the national event.
Stephens, who competed last year in the USA Climbing Youth Competition, finished 33rd out of 111 male climbers in the speed event while finishing 26th out of 200 climbers in the bouldering event. The top 25 competitors in each event moved on to the finals, as Stephens just missed the cut in the bouldering competition.
Safford, who has battled back from injury earlier in the year to qualify for nationals, carded a 6100 in the female bouldering event.
The Tornado duo competed against 93 other collegiate teams from across the country, and as a team finished ahead of collegiate squads from Virginia Tech, Alabama and Denver among others in the event.
Head coach Travis Gray also thanks all of his support from Brevard College over the course of the season.
"I would like to thank Jamie Atkinson, President Joyce, the board of trustees, Cameron and Rachael Austin of Brevard Rock Gym, as well as my fellow staff and colleagues here at Brevard College for their endless encouragement and support throughout the last eight months of our best season yet," said Gray.


text:
"Intense focus and meditation on the subtle element corresponding to this center of force is defined in the Yoga texts as the special, beatific expansion Akashi Darana. The person who practises this form of meditation will not die, nor disappear at the dissolution of the world. This person obtains the knowledge of the past, present and future as well as the control of the mysterious energies of time." (Swami Sivananda)
Location: the fifth energy center, Vishuddha chakra is located in the physical body in the area of the neck.
Represents: pure consciousness, creativity.
Symbolical representation: it is the dwelling of sound, and dreams, and the symbol of purity.
On the psychic level: it governs expression, awakening of inspiration, eloquence, archetypal models. This center of force is closely connected to the feeling of acceptance, superior receptivity, and subtle protection.
On the physical level: it coordinates the activity of the vocal cords, the hearing system, pharynges, oesophagus, as well as the spine. The corresponding glands are thyroid and parathyroid. This center of force is connected to the complex process of nourishment.
Specific reactions in the case of energetical disturbance: the person in which the energies characteristic to Vishuddha chakra are disturbed will have to face an imbalance at the level of the above-mentioned glands. The emotional factors are nonetheless extremely important in triggering this disease. These factors are translated in symptoms as insomnia, anxiety and depression.
Characteristic element: the subtle energies of ether.
Sense: hearing.
All the miraculous possibilities, the gifts, the powers named in the Eastern tradition siddhis are related to the activation of Vishuddha Chakra, and to be more specific, they are related to the fifth element, the ether.
The effects of meditating on this chakra are calm, serenity, purity, melodious voice, the awakening of artistic inspiration, the capacity of interpreting sacred writings, scriptures, of understanding the hidden message of dreams.
The person under the influence of Vishuddha Chakra no longer find attractive, nor important the distractions and pleasures caused by the senses and intellect.
Such a person will seek real knowledge beyond the limits of time, and beyond hereditary or cultural conditionings. Entering its field of energy equals the freedom from the whirlpools of this world and the control over our ego.
The idea of cosmic consciousness is related to this center of force. Both physiologically and psychologically, Vishuddha Chakra corresponds to the complex process of nurturing. The balanced development of the three centers of force leads to the formation of a balanced and harmonious personality.
The fourth center of force, Anahata Chakra is an intermediary stage, a central point that unifies the inferior levels with the upper levels, the sphere of survival, social living, and assertion of ones personality with the subtle, refined levels of the manifestation.
Not by a mere accident is Anahata Chakra the center of the heart, of love, of the awakening of the soul. The next stage is then the acquiring of a superior state of receptiveness, and an opening towards the divine inspirations.
In this sense, Vishuddha Chakra corresponds to the process of nurturing, both physically (the projection of Vishuddha Chakra is the neck) and subtly, referring to an energetical nourishment through the access in the superior worlds of the universe.
On this stage, the notion of "inner superior consciousness" becomes accessible, and the person may intuitively enter the mysteries of the divine manifestation.
TIME IS THE MIRROR OF ETERNITY
The state of receptiveness to the superior, elevated worlds makes possible the opening of some access gates to these universes. These experiences appear spontaneously in meditations, or they may be reached through the pouring of the divine grace. The psychologist Abraham Maslow named them "peak experiences".
For all the people who had such experiences, in their moments of ecstasy, intuitions and transcendence these experiences represent the most important thing in their lives, and are considered as the truly most precious aspects of life, that give life its value and measure.
They may be certain forms of overcoming ones individuality, or simply a sensation of being out of time.
The following quotation from "Memoir In the Defence of Caligostro" (from Serge Hutins book "The Secret Societies From Yesterday till Today") expresses very well a superior level of consciousness of a human being:
"I do not belong to any age, and I do not belong to any space. I feel permanently that I am outside time and space. My spiritual being experiences in its essence, the eternal existence and if sometimes if let myself sink into the cosmic thought climbing back up the course of ages, if then I reach my almighty spirit towards a superior way of living, different from that which you are familiar with, I experience the paradise-like ecstasy and I become that who I want to be. Participating fully and consciously to the reality of the Supreme Being, I conduct almost all my actions guided by the divine inspiration that helps and surrounds me."
THE ART MOULDS THE ARTIST AND THE ARTIST MOULDS THE WORLD
This superior inspiration appears only in the more or less conscious moments of superior receptiveness. The great composers, the people of art, even the scientists have had such moments of brilliancy.
If a person had this inner state of superior receptiveness and truly allows the resonance with elevated subtle realities, these realities will materialize into sublime ideal forms into the sphere of the consciousness.
Any kind of artistic experience or realization not only re-create art, but also they transform the receivers personality because through symbols received telepathically or created exteriorly, the art creator is capable of reintegrating certain aspects and of reaching a new elevated level of consciousness.
More than this, the act of creation and the act of being created are complementary parts of the same phenomenon. Through contrast, when the art and the inspiration are considerably embarrassed by the ego, creativity stagnates.
When the ego becomes exacerbated, art disappears, and when the ego disappears, universal art flourishes at its own will. When the artist is crushed down by the ego, allowing him to be dominated by petty calculations he begins to "imitate" himself and he is no longer capable of transmitting the sublime to the others. These inspirations then do not transform him and his personal inner growth stops.
THE GENIUS IS NURTURED WITH ESSENCES
In the genuine act of creation, free and manifested uninhibited, characterized by a "crazy" spontaneity, the artist nurtures himself first and most important of all. At this point, he offers a lot to the others.
When a person offers himself or herself to the world, they accept in fact the subtle guidance and inspiration of the inner superior consciousness. From this point of view, we may say that the superior act of nourishing is in fact equal with being nurtured. Through Vishuddha Chakra, the brilliant artist performs the ineffable identity with the inner voice, through his art.
In the case of a person who has the consciousness predominantly at this level, his or her role will be equally as distributor and receiver of the divine grace. Through the sublime experience of receiving the grace, the individual consciousness nurtured with the nectar of the divinity goes up to the next centre of force, Ajna Chakra.
Here the person realizes the mystery that the one who gives and the one who receives are in fact one, fused in ineffable unity, just as the drop that is now reunited with the ocean it used to be a part of.
For a better understanding of the 5th center of force, we explain a few key-words:
Akasha – subtle fundamental element that the yogis discovered for thousands of years as pertaining everything in the universe. In some secret texts, it is depicted as the space ether that comprises and records everything.
In the East, Akasha or the all-comprising ether is the subtlest of the five elements (earth, water, fire, air, and ether). Akasha is a fundamental substance that the wise advanced yogis say to fill and pertain all over the universe and it is the basic support of life and sound. In Akasha is registered instantaneously everything that occurs in the universe, and these images may be read by clairvoyants and are called akashic clichs.
Eternity – that which is truly divine and exists beyond time. The states of genuine spiritual elevation, which prompt us on the peaks of ecstatic consciousness are tangents to the perception of Reality.
This feeling of the eternity appears also as a state of satisfaction when we are completely into the present. The present moment is the mobile image of the eternity, and simultaneous experience, with depth and authenticity of the moment, together with extracting its essential significances, means in fact to transcend it through the access to the sublime sphere of force of authentic spiritual vision and experience.
In the spiritual realm, eternity is the spiritual counter-part of time in our phenomenal world. As Plato said, "time is a moving image of eternity".
Intuition – represents the direct, immediate knowledge of truth, through resonance and without involving the thought and the mind. C.G. Young defined intuition as one of the four fundamental psychological functions of the human being, the other three are thought, feelings and senses.
He describes intuition as a function of exploring the unknown, which offers possibilities and implications that are apparently difficult to be consciously perceived. Intuition is not an act of will; it is a state of attention and inspired reception.
Inspiration – state of resonance and inner communion that leads to a certain sublime experience, idea and/or action.
It is characterized through the revelation, on a subjective level, of some experiences, representing a psychic condition auspicious to the act of creation, reason that determined Plato to rightfully conceive it as a message of the divinity that pertains those who allow themselves be taken into ecstatic states of trance.
For Schopenhauer, the inspiration was the eternal subtle world of the divine ideas, and for Nietzche it was the expression of the Dionysian drunkenness giving birth to brilliant creations.
Time – the universal property of all forms to be comprised into a certain subtle energy of time, situating simultaneously or successively towards one another. The entire creation is subject to this becoming, so in other words to the subtle energy of time, only through the pure divine consciousness that transcends any becoming and time.
As it is in its essence a subtle energy, the dimensions of time are characteristically modulated by the structures of the individual and cosmic consciousness.
Thus, the information referring to past, present or future events are recorded or stored in a sui-generis magnetic support of the universe, reality that in yoga is named "akasha-ic recordings or clichs".
The advanced yogic can recall and re-live these traces deposited in the energies of time, after the experience occurred once. Also people that have natively this capacity of recalling the akasha-ic clichs can do it, but these cases are quite rare.
The presence and accessibility of such information in akasha tattva is implied even in the current theories in the modern physics, and examples of this would be the theory of the "implied order", issued by David Bohm, suggesting that there is a certain phenomenal and structural order in the universe and that this order has nothing to do with either becoming or temporality.
The scientists who lived at the end of the 20th century have proved that our sense of appreciating time is in fact a distortion of the reality.
God in Ten
Sexual Continence
12 Effects of Making Love with Continence
The Art of Making Love Divinely
Contemplative Sexuality
Intro to Contemplative Sex
Contemplative Sex, Part 2
© Copyright - SivaSakti.com Content Disclaimer - Medical Disclaimer


text:
Oxylobium cordifolium, commonly known as the heart-leaved shaggy pea, is a species of flowering plant in the family Fabaceae and is endemic to New South Wales. It is a small, prostrate shrub with long, wiry branches, heart-shaped leaves and orange-red flowers.

Description
Oxylobium cordifolium is a small, spreading shrub to about  high with branches up to  long and are densely covered with long, soft, straight hairs. The heart-shaped leaves are arranged opposite or whorled,  long,  wide, margins and apex curved downward, upper surface covered with warty protuberances, underside sparingly hairy. The orange-red flowers are borne at the end of branches in racemes, usually in groups of three, standard petal  long, bracts lance-shaped and taper to a point. The soft, oval-shaped seed pod is covered in soft, silky hairs, sessile,  long and tapering to a point. Flowering occurs from spring to early summer.

Taxonomy
Oxylobium cordifolium was first formally described in 1807 by Henry Cranke Andrews and the description was published in The Botanist's Repository for New, and Rare Plants. The specific epithet (cordifolium) means "heart leaved".

Distribution and habitat
Heart-leaved shaggy pea grows on damp, sandy soils in heath and coastal headlands south of Sydney, Tumut and Conjola districts.

References

Mirbelioids
Fabales of Australia
Flora of New South Wales
Taxa named by Henry Cranke Andrews