text:
\section{Introduction}

The idea of combining different models to obtain one that is better than any of its constituents (meta learning) has been explored extensively and it is currently used in many applications \cite{breiman1996stacked, leblanc1996combining, wolpert1992stacked}. Meta learning today, however, mainly consists of creating linear combinations of models (i.e stacking). Its purpose is to improve the accuracy of the individual models, albeit at the expense of interpretability. Related ideas are also explored for ensemble methods which create linear combination of simpler models. Two main ensemble methods can be highlighted: bagging and boosting \cite{breiman1996bagging,freund1996experiments}. In bagging, models are averaged to reduce the variance of individual models and improve accuracy. In boosting, simple models are sequentially learned reducing the bias of the estimator at each step \cite{friedman2001greedy}.

Usually thought independently from meta learning, the use of cross validation to select the best algorithm from a library (either different models or different hyperparameters) is widely popular \cite{efron1983estimating}. Establishing the theoretical basis for designing an oracle algorithm that will select the best from a library of models (using cross validation), Van der Laan et al demonstrated that cross validation can be used more aggressively than previously thought, terming the cross validation selector ``super learner" \cite{van2007super}. Specifically, it was shown that if the number of candidate estimators, $K(n)$, is polynomial in sample size, then the cross validation selector is asymptotically equivalent to the oracle selector --one that knows the best algorithm \cite{van2007super}. Similarly to the empirical use of cross validation, the super learner proposes to select one model from a library for all the observations. However, for complex functions and simple models in the library (e.g to afford interpretability for instance) it is possible that the model selected to be the best in one region of the covariates might not be the best in another.

In the present article,  we expand on this idea and investigate an algorithm that selects the best model from a library conditional on the covariates, called here Conditional Super Learner (CSL). This meta algorithm can be thought as learning in the cross validation space. With the CSL, therefore, we investigate a meta learning strategy that instead of forming a linear combination of models, it reduces the bias of the models in the library by selecting them conditional on the covariates. We show how the CSL has implications for both the accuracy of models and their interpretability. Specifically, in this article we:

\begin{enumerate}

\item Develop the theoretical foundations for the \emph{Conditional Super Learner}: An algorithm that selects the best model from a library conditional on the covariates.

\item Illustrate how the CSL is a generalized  partitioning algorithm that finds different boundary functions (not just vertical cuts as CART does) with  $\mathcal{M}$-estimators algorithms at the nodes.

\item Establish the connection between CSL and interpretability.

\item Show empirically how CSL improves over the regular strategy of using cross validation to select the best model for all observations.

\item Show empirically how CSL can give better $R^2$ than stacking in a set of regression problems.

\item Show empirically how CSL performs in the analysis of Hierarchical Data.

\end{enumerate}

\begin{figure}[t!]
 \includegraphics[width=0.5\textwidth]{Figure1.png}
\centering
\caption{This diagram shows an application of the CLS model. In this dataset we have 4 variables: number of bedrooms, bathrooms, latitude and longitude to predict house prices. The rectangular region shows how the oracle divides the latitude and longitude (normalized) in 3 regions. Each region has its own expert (using number of bedrooms and bathrooms), represented here by a diagram of a tree, to makes predictions.}
\label{Figure1}
\end{figure}

\section{Conditional Super Learner}

The algorithms that we discuss in this paper are supervised learning algorithms. The data are a finite set of paired observations $\mathcal{X}  = \{(x_i,y_i)\}_{1}^{N}$. The input vectors $x$, whose components are called here covariates, are assumed to be on $\mathbb{R}^p$, while $y$ can be either a regression or classification label.

We propose to solve supervised learning problems by 1) dividing the input space into a set of regions that are learned by an iterative algorithm and  2) fit simple interpretable models that we call ``experts" to the data that fall in these regions. The regions are learned by fitting a multi-class classification model that we call the ``oracle" which learns which expert should be used on each region. Given an oracle $o(x)$, region $k$ is defined as $\{o(x)=k\}$, that is, the set of points for with the oracle predict to use the function $f_k(x)$.

An example of an application of Conditional Super Learner (CSL) is shown in Figure \ref{Figure1}. In here we have 4 variables to predict houses prices: bedrooms,	bathrooms, latitude, longitude. The rectangular region shows how the oracle divides the latitude and longitude (normalized) in 3 regions.
Each region has its own expert, represented here by a diagram of a tree, to make predictions. Each of the experts has as input the 4 variables.

As with any meta-learning algorithm, the Conditional Super Learner algorithm for learning the oracle $o(x)$ (given the fits of the $K$ experts) will be applied to a cross-validated data set, using $V$-fold cross validation. That is, for each $Y_i$ falling in one of the $V$ validation samples, we have a corresponding training sample. We  couple each observation $Y_i$  with $K$ expert algorithms trained on subsets (from current best estimate of oracle) of its corresponding training data set, thereby creating a cross-validated data set of $N$ observations. In this section, for the sake of explaining the conditional super-learner algorithm, this formality of cross validation will be suppressed, but in our theoretical section we make the full conditional super-learning algorithm formal.

\subsection{Definition of CSL}

Given an oracle $o(x)$ and $K$ experts models $\{F_{k}(x)\}_{k=1}^{K}$ fitted on each of the corresponding regions $\{o(x)=k\}$, the CSL can be defined as:

\begin{equation}
\label{eqn1}
CSL(x) = \sum_{k=1}^{K} \mathbbm{1}\{o(x) = k\}F_k(x)
\end{equation}

where $o(x) \in \{1,2,..K\}$. $CSL(x)$ is the Conditional Super Learner that outputs the best model $F_{k}(x)$ from a library of $K$ models conditional on the covariate $x$. The idea is to find the $o(x)$ and  corresponding fits $\{F_{k}(x)\}_1^K$ that minimize a given loss function over the training data:

\begin{equation}
\label{eqn2}
\argmin_{o,\{F_{k}\}_1^K} \sum_{i=1}^{N} L\big(y_i,\sum_{k=1}^{K} \mathbbm{1}\{o(x_i)=k\} F_{k}(x_i)\big)
\end{equation}

\subsection{Fitting the oracle}

To find the solution to equation \ref{eqn2} we will employ a trick often used in machine learning. We will iterate between solving $o(x)$ and solving for $\{F_{k}(x)\}_1^K$. To solve for $o(x)$ we will assume that all $\{F_{k}(x)\}_1^K$ are known, the library, and that we also have  unbiased estimations (i.e., cross-validated) of the loss at each training point $L(y_i, F_k(x_i)) $. In this case, $CSL(x)$ will aim to find the best $o(x)$ that minimizes the loss function over the training data

\begin{equation}
\label{eqn3}
\argmin_{o(x)} \sum_{i=1}^{N} L\left(y_i ,\sum_{k=1}^{K} \mathbbm{1}\{o(x_i)=k\} F_{k}\left(x_i\right)\right)
\end{equation}

and using the definition of the indicator function, we can take the sum outside of the loss function and get Equation \ref{eqn4}:

\begin{equation}
\label{eqn4}
\argmin_{o(x)} \sum_{i=1}^{N} \sum_{k=1}^{K} \mathbbm{1}\{o(x_i)=k\} L(y_i , F_{k}(x_i))
\end{equation}

To introduce how we fit the oracle, we define a new dataset that we called ``extended" dataset. This is a dataset for a multi-class classification problem with $K$ classes, each class corresponding to one of the expert models. This dataset has $K\cdot N$ observations -- each $x_i$ appears $K$ times with corresponding labels $\{1, \dots K\}$ and a specific weight.\\

\begin{definition}
Given dataset  $\mathcal{X} = \{(x_i,y_i)\}_{1}^{N}$, expert functions $\mathcal{F}=\{F_{k}\}_1^K$ we define the {\bf extended dataset} $\mathcal{X} _{\mathcal{F}} = \{(\tilde{x}_i,z_i,w_i)\}_{1}^{K\cdot N}$ with $K\cdot N$ observations where:
\begin{itemize}
    \item $\tilde{x}_i = x_{\floor{i/K} + 1}$
    \item $z_i = i \mod K + 1$
    \item $w_i$ is a weight on observation $(\tilde{x}_i,z_i)$ and is defined in the following way:
\end{itemize}

Let $l_i$ be $K$ dimensional vector $l_i =(L(y_i, F_{1}(x_i)), \dots,L(y_i, F_{K}(x_i))$ where element $k$ is the loss of expert $k$ at point $(x_i, y_i)$. Let  $ONE_K$ a $K \times K$ matrix of all ones and $DIAG_K$  a $K \times K$ matrix with ones in the diagonal and zeros everywhere else.

\begin{equation}
(w_{iK +1}, \dots w_{iK + K})^T =  [ONE_K - DIAG_K]^{-1} l_i^T
\label{eqn5}
\end{equation}
\end{definition}

\begin{lemma}
Solving problem \ref{eqn4} is equivalent to finding the oracle $o(x)$ that minimizes the weighted miss classification error of a extended dataset $\mathcal{X} _{\mathcal{F}}$:

\begin{equation}
\argmin_{o(x)}  \sum_{i=1}^{N\cdot K} w_i \mathbbm{1}\{o(\tilde{x_i}) \neq z_i \}
\label{eqn6}
\end{equation}
\label{lemma1}
\end{lemma}

\begin{proof}

First note that a missclassification loss for a multi-class classification problem can be written as $L(y, f(x)) = \mathbbm{1}\{f(x) \neq y\}$. We want to write Equation \ref{eqn4} as a missclassification loss of a classification problem. For each observation $(x_i, y_i)$ consider the weighted dataset $\{(x_i, 1, w_1), (x_i, 2, w_2), \dots (x_i, K, w_K) \}$ with missclassification loss $\sum_{k=1}^{K} w_k \mathbbm{1}\{o(x_i) \neq k\}$. That is, we want to find for each observation $(x_i, y_i)$ weights $(w_1, \dots, w_K)$ such that:

\begin{equation}
\sum_{k=1}^{K} \mathbbm{1}\{o(x_i)=k\} L(y_i , F_{k}(x_i))  = \sum_{k=1}^{K} w_k \mathbbm{1}\{o(x_i) \neq k\}
\label{eqn7}
\end{equation}

Since $o(x_i)$ can just have values in $\{1, \dots, K\}$ we can consider all the options. For example, if $o(x_i) = k$, the equality in Equation \ref{eqn7} becomes
$L(y_i, F_k(x_i)) = \sum_{j=1}^K w_j - w_k$. If we consider all possible values for $o(x_i)$ we get the following set of equations:

\begin{align*}
L(y_i, F_1(x_i)) & = \sum_{j=1}^K w_j - w_1\\
&\vdots\\
L(y_i, F_K(x_i)) & = \sum_{j=1}^K w_j - w_K
\end{align*}

The previous equation can be written in matrix form $l_i^T = [ONE_K - DIAG_K] (w_1, \dots w_K)^T$. Which gives us:

$$(w_1, \dots, w_K)^T= [ONE_K - DIAG_K]^{-1} l_i^T$$

\end{proof}
\begin{comment}
\begin{matrix}
X_{ext} =
\begin{pmatrix}

  x_{11} & x_{12} &... &x_{1p} \\
  x_{11} & x_{12} &... &x_{1p} \\
         &  . &        &       \\
   x_{11} & x_{12} &... &x_{1p} \\
         &  . &        &       \\
         &  . &        &       \\
         &  . &        &       \\
  x_{N1} & x_{N2} &... &x_{Np}  \\
  x_{N1} & x_{N2} &... &x_{Np} \\
         &  . &        &       \\
  x_{N1} & x_{N2} &... &x_{Np}  \\

\end{pmatrix} &,&
Z_{ext}=
\begin{pmatrix}
  1 \\
  2 \\
  . \\
  K \\
  . \\
  . \\
  . \\
  1 \\
  2 \\
  . \\
  K
\end{pmatrix} &,&
W_{ext} =
\begin{pmatrix}
  $$w_{11} $$\\
  $$w_{21} $$\\
  . \\
  $$w_{K1} $$\\
  . \\
  . \\
  . \\
  $$w_{1N} $$\\
  $$w_{2N} $$\\
  . \\
  $$w_{KN} $$\\
\end{pmatrix}
\end{matrix}
\end{comment}

As a result of Lemma \ref{lemma1}, $o(x)$ is the solution of a multi-class classification problem on the extended dataset  $\mathcal{X} _{\mathcal{F}}$. We approximate o(x) by fitting any standard classification algorithm on $\mathcal{X} _{\mathcal{F}}$.

\subsection{Fitting the experts}

 Similarly to the previous section, in order to fit the experts we assume that $o(x)$ is known. Then Equation \ref{eqn2} becomes $K$ independent classification/ regression problems that minimize the empirical loss over observations $\{i:o(i)=k\}$, for each $k=1,\ldots,K$, which is generally already  solved by standard machine learning algorithms.

\begin{equation}
\label{eqn8}
\argmin_{F_k} \sum_{x_i:o(x_i) = k}  L(y_i ,F_k(x_i))
\end{equation}

\subsection{A two step algorithm}
 Finding  $\{F_k(x)\}_1^K$ indicates that  equation \ref{eqn2} can be minimized iteratively. Following the K-mean algorithm's philosophy, let us propose the minimization of equation \ref{eqn2} in two steps: one to fit the oracle and the other to fit the experts. Please note that if we take into consideration that at every time that  each step is applied the Loss function  decreases, the convergence to a local minimum is guaranteed. Of course, this is only true if we use, for each observation, the estimation of $(y_i ,F_k(\bf{x}_i)$ on the training data. This, however, will most likely result in overfitting. After this discussion we are ready to write  Conditional Super Learner pseudo code (see above).

\begin{algorithm}[t]
\SetAlgoLined
\LinesNotNumbered
\KwIn{
    $ \mathcal{X} = \{ (x_i, y_i)\}_1^N$; $\mathcal{F} = (F_1, F_2, \dots, F_K)$
}
Initialize: for each sample split $v=1,\ldots,V$, fit the experts $\mathcal{F} = (F_1, F_2, \dots, F_K)$  on initial subsets of the $v$-th training data set. For each $i$, let $F_{k,-i}$ be the $k$-th expert trained on training sample that excludes $Y_i$. Construct the corresponding cross-validated data set $(Y_i,F_{1,-i}(x_i),\ldots,F_{K,-i}(x_i))$, $i=1,\ldots,N$.

\For{$t=1$ : iterations}{
 For each point and each expert compute: $L(y_i ,F_{k,-i}(x_i))$\\
 Create extended dataset $\mathcal{X}_{\mathcal{F}}$\\
 Fit $o(x)$ on $\mathcal{X}_{\mathcal{F}}$\\
 Re-fit each expert $F_k$ on $\{o(x)=k\}$ for the $V$-training samples. \\
}
Based on final $o(x)$, refit each expert $F_k$ on $\{o(x)=k\}$ for total sample.
\KwResult{$\sum_{k=1}^{K} \mathbbm{1}\{o(x)=k\}F_{k}(x)$}
\caption{{Conditional Super Learner \textbf{(CSL)}}}
\label{algo:duplicate}
\end{algorithm}

\section{Theoretical Guarantees}

In this section we will formalize meta-learning in general and the Conditional Super Learner in particular. Specifically we will:

\begin{enumerate}

\item Formalize meta-learning, cross validation and the Conditional Super Learner.

\item Prove a rate of convergence theorem  for a general Super Learner that uses cross validation .

\item Highlight the bias variance trade-off for different meta-learning algorithms.

\item Give practical recommendations on how to best use aggressive meta-learning algorithms.

\item Connect meta-learning with a double super-learner.

\end{enumerate}

Due to the complexity of the notation this section will be self contained. The readers interested in a more practical use of the CSL can skip this section.

\subsection{Formalizing meta learning}

Let us start by specifying the data generating process, the type of candidate estimators considered, the loss function, cross validation and finally meta-learning. This section contains all necessary assumptions and definitions before we prove formal results for the performance of the Super Learner, in particular, w.r.t. its oracle choice.

\subsubsection{Data Generating Process}
Let $O_1,\ldots,O_n$ be $n$ i.i.d. copies of a random variable $O\sim P_0$ with data distribution $P_0$ that is known to be an element of a specified statistical model ${\cal M}$. In a regression or classification application we have $O=(X,Y)$ for a covariate vector $X$ 	and outcome  $Y$, where $Y$ could be categorial, ordered discrete, or continuous. Let $O\in [0,\tau_o]\subset \hbox{${\rm I\kern-.2em R}$}^{d_1}$ be a Euclidean valued random variable with realizations in a cube $[0,\tau_o]$. Let $P_n$ be the empirical probability measure of $O_1,\ldots,O_n$. Let $\Psi:{\cal M}\rightarrow {\bf \Psi}=\Psi({\cal M})$ be a particular functional parameter of $P_0$.  In our setting $\Psi(P_0)$ could be a  conditional mean $E_0(Y\mid X)$ (regression) or a conditional probability distribution $P(Y=\cdot\mid X)$ (classification).

\subsubsection{Restriction on the type of functional parameters considered}
Each possible value  $\psi$ of this functional parameter is a $d$-variate real valued function $x\rightarrow\psi(x)$, where we assume that $\psi:[0,\tau]\subset\hbox{${\rm I\kern-.2em R}$}^d\rightarrow \hbox{${\rm I\kern-.2em R}$}$ is defined on a $d$-dimensional cube $[0,\tau]$. We  assume that the parameter space ${\bf \Psi}\subset D[0,\tau]$ is contained in the space of multivariate real valued cadlag functions that are assumed to be right-continuous with left-hand limits \cite{neuhaus1971weak}. Another possible assumption we emphasize is that the so called sectional variation norm of each $\psi\in {\bf \Psi} $ is bounded \cite{gill1995inefficient, van2017generally}.

To define the sectional variation norm of a function $\psi\in D[0,\tau]$, for each subset $s\subset\{1,\ldots,d\}$, we define its $s$-specific section $\psi_s(x)=\psi(x_s,0_{-s})$, where for a vector $x$ $x_s=(x(j):j\in s)$ and $x_{-s}=(x(j):j\not \in s)$.
For each section $\psi_s$ we can compute its variation norm as $\int_{(0_s,\tau_s]}\mid  \psi_s(du)\mid$ which can be represented as a limit over a partitioning in $\mid s\mid$-dimensional cubes (left-open, right-closed) of the sum over the cubes in the partitioning of the absolute value of the measure $\psi_s$ assigns to this cube. Here we are reminded that the measure $f((a,b])$ a function $f$ assigns to a cube $(a,b]$ is defined as a generalized difference over the $2^{\mid s\mid}$ corners of $(a,b]$ where $\mid s\mid$ denotes the size of the set $s$: for example, for $d=2$ $f(((a_1,a_2),(b_1,b_2)])=f(b_1,b_2)-f(a_1,b_2)-f(b_1,a_2)+f(a_1,a_2)$.
We write $\parallel \psi_s\parallel_v =\int_{(0_s,\tau_s]}\mid \psi_s(du)\mid$.
The sectional variation norm of $\psi$ is defined as the sum over all the sections of the section-specific variation norm:
\[
\parallel \psi\parallel_v^*=\mid \psi(0)\mid+\sum_{s\subset\{1,\ldots,d\}} \int_{(0_s,\tau_s]} \mid \psi_s(du)\mid .\]
We also note that for any cadlag function $\psi$ with finite sectional variation norm $\parallel \psi\parallel_v^*$ we have the following representation \cite{van2006oracle, van2017generally}:
\begin{eqnarray*}
\psi(x)&=&\psi(0)+\sum_{s\subset\{1,\ldots,d\}} \int_{(0_s,x_s]} \psi_s(du) \\
&=& \psi(0)+\sum_{s\subset\{1,\ldots,d\}} \int \phi_{s,u}(x) \psi_s(du),\end{eqnarray*}
where $\phi_{s,u}(x)=I(x_s\geq u)$ is the tensor product of indicators $I(x_j\geq u_j)$ over $j\in s$ defined by knot point $u=(u(j):j\in s)$.
For simplicity, we will assume that there is a universal bound $\sup_{\psi\in {\bf \Psi}}\parallel \psi\parallel_v^*<\infty$ on the sectional variation norm over the parameter space, even though this assumption can be prevented by considering a sieve ${\bf \Psi}_{c_j}=\{\psi\in {\bf \Psi}:\parallel \psi\parallel_v^*<c_j\}$ where the variation norm bound $c_j$ is an increasing sequence converging to infinity, and selecting $j$ with cross validation.

\subsubsection{Loss Function}

Our goal is estimation of $\Psi(P_0)$.  Let $L(\psi)(O)$ be a loss function for $\psi_0=\Psi(P_0)$ so that
$\psi_0=\arg\min_{\psi\in {\bf \Psi}}P_0 L(\psi)$ minimizes the expectation of the loss (i.e., risk), where we use notation $Pf\equiv \int f(o)dP(o)$. Assume $M_1\equiv \sup_{o,\psi}\mid L(\psi)-L(\psi_0)(o)\mid $ and $M_2=\sup_{\psi}P_0\{L(\psi)-L(\psi_0)\}^2/P_0(L(\psi)-L(\psi_0))$ are both finite, where the suprema in these definitions are over $\psi\in \Psi({\cal M})$ and over  $o$  in a support of $P_0$.
The bounds $M_1,M_2$ guarantee that the cross validation selector is well behaved and satisfies oracle inequalities \cite{van2003unified, van2006oracle,van2007super}

We also assume that $L({\bf \Psi})=\{L(\psi):\psi\in {\bf \Psi}\}$ is contained in set of cadlag functions $D[0,\tau_o]$ and that $\sup_{\psi\in {\bf \Psi}} \parallel L(\psi)\parallel_v^*<\infty$.
In other words, for any $\psi$ $L(\psi)$ is contained in class of cadlag functions of $O$ with a universal bound on the sectional variation norm. This is a weak regularity assumption on the loss function $L()$ since ${\bf \Psi}$ is itself also contained in the class of cadlag functions with a universal bound on the sectional variation norm. The loss-function also implies a loss-based dissimilarity
\[
d_0(\psi,\psi_0)\equiv P_0L(\psi)-P_0L(\psi_0),\]
which will behave as a square of an $L^2(P_0)$-norm, due to $M_2<\infty$.

\subsubsection{Library of candidate estimators, cross validation , oracle selector and meta-learning}

An estimator can be represented as a mapping from the empirical measure $P_n$ into the parameter space. We let ${\cal M}_{np}$ be the set of al possible realizations of an empirical probability measure, including its limits as $n$ converges to infinity (i.e., a nonparametric statistical model), so that an estimator $\hat{\Psi}:{\cal M}_{np}\rightarrow {\bf \Psi}$.
We start out with a set $\hat{\Psi}_j:{\cal M}_{NP}\rightarrow {\bf \Psi}$ of $J$ candidate estimators, $j=1,\ldots,J$.

{\bf $V$-fold cross validation:} We define a $V$-fold cross validation scheme that maps an empirical probability measure $P_n$ into  an empirical measure of a training sample $P_{n,v}$ and corresponding empirical measure of the validation sample $P_{n,v}^1$,  across $v$-specific sample splits, $v=1,\ldots,V$.

{\bf Meta-learning model:}
Let ${\cal F}$ be a set of functions $f:\hbox{${\rm I\kern-.2em R}$}^J\rightarrow \hbox{${\rm I\kern-.2em R}$}$, so that  $\psi_{f,n}=\hat{\Psi}_f(P_n)\equiv f(\hat{\Psi}_j(P_n):j=1,\ldots,J)$ represents a candidate estimator of $\psi_0$ that combines the $J$-candidate estimators, where, as a function of $x$, it is evaluated as $\psi_{f,n}(x)=f(\psi_{jn}(x):j=1,\ldots,J)$. One would restrict $f$ to be such that $f(\psi_1,\ldots,\psi_J)\in {\bf \Psi}$ for any $(\psi_1,\ldots,\psi_J)\in {\bf \Psi}^J$. This set of functions ${\cal F}$ defines a class of candidate estimators $(\hat{\Psi}_f: f\in {\cal F})$, where each choice is a particular ensemble from the $J$  estimators in the library.

{\bf Cross validated risk for candidate estimator $\hat{\Psi}_f$:}
For any $f$-specific candidate estimator $\hat{\Psi}_f$ we can evaluate its performance by its cross-validated risk:
\[
CV(\hat{\Psi}_f,P_n)=\frac{1}{V}\sum_{v=1}^V P_{n,v}^1 L(\hat{\Psi}_f(P_{n,v})),\]
i.e. the average across sample splits of the empirical mean over validation sample of the losses of the candidate estimator based on training sample.

{\bf{${\cal F}$-cross validation selector:}}
Let
\[
f_n=\arg\min_{f\in {\cal F}} CV(\hat{\Psi}_f,P_n)\]
be the Super Learner defined by minimizing the cross-validated risk of $\hat{\Psi}_f$ over all $f\in {\cal F}$.
A simple variation of this is that $f_n$ is only an approximation of this minimum obtained by using some kind of iterative greedy algorithm.
We also refer to $f_n$ as the ${\cal F}$-specific cross validation selector.

{\bf ${\cal F}$-super-learner:}
The ${\cal F}$-super-learner $\hat{\Psi}_{\cal F}(P_n)$ is now defined by the corresponding estimator applied to the whole sample:
\[
\hat{\Psi}_{\cal F}(P_n)\equiv \hat{\Psi}_{f_n}(P_n),\]
or, the immediate available (not requiring rerunning $\hat{\Psi}_{f_n}$ on whole sample after having determined the cross validation selector $f_n$):
\[
\hat{\Psi}_{\cal F}(P_n)=\frac{1}{V}\sum_{v=1}^V \hat{\Psi}_{f_n}(P_{n,v}).\]
The latter choice assumes that ${\bf \Psi}$ is convex so that a simple average represents indeed a sensible estimator (and, assuming the loss-function is convex, it will satisfy the same oracle inequalities as presented below).  For the sake of theoretical analysis of the ${\cal F}$-specific super-learner, we assume that $f_n$ is the actual minimizer over ${\cal F}$ of the cross-validated risk .

{\bf ${\cal F}$-oracle selector:}
The cross validation selector $f_n$ is aiming to estimate the oracle selector
\begin{eqnarray*}
f_{0n}&=&\arg\min_{f\in {\cal F}} \frac{1}{V}\sum_{v=1}^V P_0 L(\hat{\Psi}_f(P_{n,v}))\\
&=&\arg\min_{f\in {\cal F}} \frac{1}{V}\sum_{v=1}^V d_0(\hat{\Psi}_f(P_{n,v}),\psi_0).
\end{eqnarray*}
That is, $f_{0n}$ selects the choice of candidate estimator among all $\{\hat{\Psi}_f:f\in {\cal F}\}$ whose realization on training samples $P_{n,v}$ is closest to the true function $\psi_0$ w.r.t. loss-based dissimilarity. Therefore, we refer to $f_{0n}$ as an oracle selector. Note that both $f_n$ and $f_{0n}$ are indexed by the choice of Super Learner model ${\cal F}$, so that we could also use notation $f_{n,\cal F}$ and $f_{0n,{\cal F}}$ as well.

\subsubsection{Rate of convergence}

After above definitions, in this section we will show that the Super Learner $f_n$ will converge to the oracle choice $f_{0n}$  at a rate at least faster than $O_p(n^{-1/4})$ being $n$ the number of observations. For a given class of functions ${\cal G}\subset D[0,\tau_o]$ of function of $O$, and probability measure $Q$, let  $N(\epsilon,{\cal G},L^2(Q))$ denote its covering number, defined as the minimal number of balls with radius $\epsilon$ needed to cover ${\cal G}$ in the Hilbert space $L^2(Q)$.

\begin{theorem}
Consider the class of functions
 ${\cal G}=\{L(\hat{\Psi}_f(P_{n,v})): f\in {\cal F}\}$ and let $\alpha=\alpha({\cal F})$ be such that $\sqrt{\sup_Q\log N(\epsilon,{\cal G},L^2(Q))}\ll \epsilon^{-(1-\alpha)}$. We have that $\alpha({\cal F})\geq \alpha(d_1)$, where $\alpha(d_1)=2/(2+d_1)$.
 Then,
\[0\leq \frac{1}{V}\sum_v P_0L(\hat{\Psi}_{f_n}(P_{n,v}))-P_0L(\hat{\Psi}_{f_{0n}}(P_{n,v}))=O_P(r(n)),\]
where $r(n)=O_P(n^{-1/2})$. If, analogue to our condition that $M_2<\infty$, we assume that for a universal $M_{2,cv}$
\begin{equation}\label{M2cv}
\frac{\frac{1}{V}\sum_v P_0\left\{ L(\hat{\Psi}_{f_n}(P_{n,v}))-L(\hat{\Psi}_{f_{0n}}(P_{n,v})) \right\}^2}{\frac{1}{V}\sum_v \{P_0L(\hat{\Psi}_{f_n}(P_{n,v})-P_0L(\hat{\Psi}_{f_{0n}}(P_{n,v}) )\}}\leq M_{2,cv}
\end{equation}

then, $r(n)=O_P(n^{-1/2-\alpha({\cal F})})$.

\end{theorem}
{\bf Proof:}
We have
\begin{eqnarray*}
0&\leq &\frac{1}{V}\sum_v P_0L(\hat{\Psi}_{f_n}(P_{n,v})-P_0L(\hat{\Psi}_{f_{0n}}(P_{n,v}))\\
&=&\frac{1}{V}\sum_v -(P_{n,v}^1-P_0)\{L(\hat{\Psi}_{f_n}(P_{n,v}))-L(\hat{\Psi}_{f_{0n}}(P_{n,v}))\}\\
&&+\frac{1}{V}\sum_v P_{n,v}^1 \{ L(\hat{\Psi}_{f_n}(P_{n,v}))-L(\hat{\Psi}_{f_{0n}}(P_{n,v}))\}\\
&\leq&\frac{1}{V}\sum_v -(P_{n,v}^1-P_0)\{L(\hat{\Psi}_{f_n}(P_{n,v}))-L(\hat{\Psi}_{f_{0n}}(P_{n,v}))\}.
\end{eqnarray*}
Consider the $v$-specific empirical process term, conditional on $P_{n,v}$.
By assumption, $\hat{\Psi}_{f_n}(P_{n,v})$ and $\hat{\Psi}_{f_{0n}}(P_{n,v})$ are elements of ${\bf \Psi}$ so that these are cadlag functions with a universal bound on their sectional variation norm. More precisely, conditional on the training sample, the ensembles cover the subspace $\{\hat{\Psi}_f(P_{n,v})): f\in {\cal F}\}$, which implies a corresponding class ${\cal G}=\{L(\hat{\Psi}_f(P_{n,v})): f\in {\cal F}\}$.
We also assumed that $L({\bf \Psi})$ is contained in cadlag functions $D[0,\tau_o]$ with a universal bound on the sectional variation norm.
Since this is a Donsker class, it follows that the empirical process term is $O_P(n^{-1/2})$. This proves that the left-hand side is $O_P(n^{-1/2})$.
Given (\ref{M2cv}), it follows that, for each $v$, the $L^2(P_0)$-norm of $L(\hat{\Psi}_{f_n}(P_{n,v}))-L(\hat{\Psi}_{f_{0n}}(P_{n,v}))$ is $O_P(n^{-1/4})$. As in \cite{van2017generally}, relying on the finite sample modulus of continuity bound for empirical processes indexed by a class of functions with covering number bounded in terms of $\alpha({\cal F})$ \cite{van2011local}, it follows that  the $v$-specific empirical process is $O_P(n^{-1/2-\alpha({\cal F})})$.
This proves the theorem. $\Box$

\subsection{The Conditional Super Learner}

{\bf Single step conditional Super Learner, treating the experts as known:}
The Conditional Super Learner presented in this article corresponds with a particular type of meta-learning model ${\cal F}$ consisting of the following type of functions
\[
f_{{\cal A},d}(\psi_1(x),\ldots,\psi_J(x))=\sum_{l} I_{A_l}(x) \psi_{d(l)}(x),\]
which are indexed by a partitioning ${\cal A}=\{A_l, l=1,\ldots,L)$ of $[0,\tau]$ (i.e., covariate space), and a classifier $d:\{1,\ldots,L\}\rightarrow \{1,\ldots,J\}$ for which
$d(l)$ represents the choice of function among the $J$ functions $\{\psi_1,\ldots,\psi_J\}$: its relation with $o(x)$ in previous section is that if $x\in A_l$, then $o(x)=d(l)$, so that $o(x)$ is determined by the partitioning $\cal A$ and the classifier $d$.
In this manner, ${\cal F}=\{f_{{\cal A},d}: {\cal A},d\}$ consists of such functions for varying choices of partitioning ${\cal A}$ and corresponding classifiers $d$.
Given $\hat{\Psi}_j$, $j=1,\ldots,J$, and thereby the cross-validated data set $(Y_i,(\hat{\Psi}_j\left(P_{n,v(i)} \right):j=1,\ldots,J))$
(here $v(i)$ is the sample split for which $i$ is in the $v$-th validation sample), the cross validation selector $f_n$ corresponds now with selecting a $({\cal A}_n,d_n)$ of partitioning ${\cal A}_n$ of the covariate space and classifier $d_n$ obtained with a particular algorithm aiming to approximately minimize the cross-validated empirical risk over all ensembles $f_{{\cal A},d}(\hat{\Psi}_j(P_n));j=1,\ldots,J)$ across possible choices ${\cal A}$ and $d$ (defined by meta-learning model ${\cal F}$). This Super Learner corresponds with the simple (single iteration) conditional super-learner which treats the candidate estimators $\hat{\Psi}_j$ as fixed, not affected by $d$.

{\bf Conditional Super learner:}

For each set $A$, we can define $\hat{\Psi}_{j,A}(P_n)=\hat{\Psi}_j(P_{n,A})$, where $P_{n,A}$ is the empirical probability measure of the subsample $\{i:X_i\in A\}$. We now define the ensembles \[
f_{{\cal A},d}(\hat{\Psi}_j:j=1,\ldots,J)(P_n)=\sum_l I_{A_l}(x)\hat{\Psi}_{d(l)}(P_{n,A_l})\] indexed by a partitioning ${\cal A}$ of rectangles and classification function $d$.
Let ${\cal F}=\{f_{{\cal A},d}: {\cal A},d\}$ for a set of possible partitioning and classifiers d. We could also index this ensemble by a single classifier function $o(x)$ by setting $A_l=\{x:o(x)=l\}$, and $d(l)=l$. The cross-validated risk for this ensemble $\hat{\Psi}_{f_{{\cal A},d}}$ of $(\hat{\Psi}_j: j=1,\ldots,J)$ is given by
\[
CV(\hat{\Psi}_{f_{{\cal A},d}},P_n)=\frac{1}{V}\sum_{v=1}^V P_{n,v}^1L(f_{{\cal A},d}(\hat{\Psi}_j:j=1,\ldots,J)(P_{n,v})).\]
Our conditional Super Learner is defined as the cross validation selector:
\[
f_{{\cal A}_n,d_n}=\arg\min_{f\in {\cal F}}CV(\hat{\Psi}_{f_{{\cal A},d}},P_n).\]
Our algorithm for determining or approximating this cross validation selector uses an iterative algorithm of the type: let $m=0$, initiate a partitioning ${\cal A}^m=\{A_l^m:l\}$, and minimize
\[
(d^{m+1},{\cal A}^{m+1})=\arg\min_{d,{\cal A}} {\cal A}\frac{1}{V}\sum_{v=1}^V P_{n,v}^1 L\left(\sum_l I_{A_l}\hat{\Psi}_{d(l)}\left(P_{n,A_l^m}\right)\right),\]
and, update ${\cal A}^{m+1}=\{A_l^{m+1}:l\}$, and iterate.

Either of the classes of functions satisfies the conditions of Theorem 3.1, if the partitionings are not too fine so that the sectional variation norm remains uniformly bounded.
{\bf Theorem 3.1} , therefore, also applies to our $CSL$ and as such this one will converge to the oracle choice at a rate faster than $O_p(n^{-1/4})$. In practicality, though, the CSL will be controlled by the variance bias trade-off as illustrated in the next section.

\subsection{Bias Variance Trade-off of the Super Learners}
Let \[
d_0(\hat{\Psi}_{f_n}(P_{n,v}),\hat{\Psi}_{f_{0n}}(P_{n,v}))=
P_0L(\hat{\Psi}_{f_n}(P_{n,v}))-P_0L(\hat{\Psi}_{f_{0n}}(P_{n,v})).\]
So we have
\begin{eqnarray*}
\frac{1}{V}\sum_v d_0(\hat{\Psi}_{f_n}(P_{n,v}),\psi_0)&=&
 \frac{1}{V}\sum_v d_0(\hat{\Psi}_{f_n}(P_{n,v}),\hat{\Psi}_{f_{0n}}(P_{n,v}))\\
 &&+
 \frac{1}{V}\sum_v d_0(\hat{\Psi}_{f_{0n}}(P_{n,v}),\psi_0)\\
 &\equiv& E_n({\cal F})+B_n({\cal F}).
 \end{eqnarray*}
 As the size of ${\cal F}$ increases, the "estimation-term"  $E_n({\cal F})$ will worsen, while the "bias term" $B_n({\cal F})$ will improve.  Therefore, a good choice of meta-learning model needs to carefully trade-off the variance term $E_n({\cal F})$ and bias term $B_n({\cal F})$ at the meta-learning level.
 The optimal choice of meta-learning model  ${\cal F}$ among a set of candidate meta-learning models will depend on this trade-off, and the precise trade-off depends on the candidate meta-learning models and the library estimators.
 By the previous theorem, we always have
 \[
 E_n({\cal F})=O_P(n^{-1/2-\alpha(d_1)}).\]
 Thus, the ${\cal F}$-super-learner will always achieve a rate of convergence $n^{-1/2-\alpha(d_1)}$ w.r.t. loss-based dissimilarity under the assumption that the oracle selected candidate estimator achieves at minimal that same rate of convergence.

If one considers relatively large meta-learning models, as is easily the case for our conditional super-learner, then there is a risk that one worsens the performance relative to simpler meta-learning models. Therefore, the most sensible strategy is to define a sequence of Super Learner models ${\cal F}_k$ with increasing complexity as $k$ increases, and use cross validation to select the best ${\cal F}_k$-specific super-learner among $k=1,\ldots,K$. Due  to the oracle inequality for the discrete cross validation selector this guarantees that the resulting super-learner will be asymptotically equivalent with the oracle selected super-learner among the $K$ ${\cal F}_k$-specific super-learners.
In this manner, one is guaranteed to outperform any given ${\cal F}$-specific super-learner by including this choice of meta-learning model in our collection of Super Learners.
We will refer to this as double super-learning. In the next section we study double super-learner in some detail.

\subsection{The Double Super-Learner to learn the best Super Learner}

Let ${\cal F}_k$, $k=1,\ldots,K$, be a collection of meta-learning models.
In  a typical setting, the size of the family ${\cal F}_k$ will be growing as $k$ increases, so that the corresponding meta-learning step is increasingly data adaptive as $k$ increases. Each family ${\cal F}_k$, defines a collection of candidate estimators $\hat{\Psi}_f(P_n)$ indexed by $f\in {\cal F}_k$, and the meta-learning step is defined by minimizing the cross-validated risk $CV(\hat{\Psi}_f,P_n)$ over all $f\in {\cal F}_k$.  Some of the families ${\cal F}_k$ might be discrete sets and linear models so that the meta learner corresponds with simply selecting the best estimator among the $J$ candidate estimators in the library and the best linear combination of these $J$ estimators, respectively.

{\bf ${\cal F}_k$-cross validation selector:}
Let
\[
f_{n,k}=\arg\min_{f\in {\cal F}_k} CV(\hat{\Psi}_f,P_n)\]
be the Super Learner defined by minimizing the cross-validated risk of $\hat{\Psi}_f$ over all $f\in {\cal F}_k$.

{\bf ${\cal F}_k$-super-learner:}
The ${\cal F}_k$-super-learner is now defined by the corresponding estimator applied to the whole sample:
\[
\hat{\Psi}_{{\cal F}_k}(P_n)=\hat{\Psi}_{f_{n,k}}(P_n),\]
or, the immediate available (not requiring rerunning $\hat{\Psi}_{f_{n,k}}$ on whole sample after having determined $f_{n,k}$):
\[
\hat{\Psi}_{{\cal F}_k}(P_n)=\frac{1}{V}\sum_{v=1}^V \hat{\Psi}_{f_{n,k}}(P_{n,v}).\]
For notational convenience, let's also use notation $\hat{\Psi}_k^{SL}(P_n)=\hat{\Psi}_{f_{n,k}}(P_n)$.

{\bf Cross validation selector among the $K$ ${\cal F}_k$-super-learners:}
Let
\begin{eqnarray*}
k_n&=&\arg\min_k \frac{1}{V}\sum_{v=1}^V P_{n,v}^1 L(\hat{\Psi}_k^{SL}(P_{n,v})).
\end{eqnarray*}
We also consider the oracle selector of $k$:
\begin{eqnarray*}
\tilde{k}_n&=&\arg\min_k \frac{1}{V}\sum_{v=1}^V P_0L(\hat{\Psi}^{SL}_k(P_{n,v})).\\
\end{eqnarray*}

{\bf Proposed double super-learner:}
Then, our double super-learner and proposed estimator is defined by
\[
\hat{\Psi}(P_n)=\hat{\Psi}^{SL}_{k_n}(P_n).\]

{\bf Inner $V_1$-fold cross validation within training sample $P_{n,v}$:}
Note that for computing $k_n$, and thereby $\hat{\Psi}^{SL}_k(P_{n,v})$ we need to apply a cross validation scheme to the training sample $P_{n,v}$ itself.
Consider a $V_1$-fold cross validation scheme where for each sample split $v_1=1,\ldots,V_1$ of a sample $P_{n,v}$   into validation sample $P_{n,v,v_1}^1$ and complementary training sample $P_{n,v,v_1}$. We can then apply our definition of $\hat{\Psi}^{SL}_k$ above to the data $P_{n,v}$ using this inner $V_1$-fold cross validation scheme (i.e., replace $P_n$ by $P_{n,v}$ and replace $P_{n,v}$, by $P_{n,v,v_1}$).

{\bf Double cross validation:} If we apply the inner $V_1$-fold cross validation scheme to a training sample $P_{n,v}$ from an outer cross validation scheme, then this $V_1$-cross validation splits $P_{n,v}$ into $P_{n,v,v_1}$ and $P_{n,v,v_1}^1$, $v_1=1,\ldots,V_1$. So $(V,V_1)$-double cross validation maps $P_n$ into $V\times V_1$-sample splits $(P_{n,v,v_1},P_{n,v,v_1}^1:v_1=1,\ldots,V_1, v=1\ldots,V)$.

\subsection{Oracle inequality for double super-learner showing asymptotic equivalence with super-learner using oracle choice of meta-learning model}
By \cite{van2006oracle} we have, for any $\delta>0$, and constant $C(M_1,M_2,\delta)=2(1+\delta)^2(M_1/3+M_2/\delta)$,
\[ \frac{E_0 \frac{1}{V}\sum_v d_0(\hat{\Psi}^{SL}_{k_n}(P_{n,v}),\psi_0)}{E_0\\min_k \frac{1}{V}\sum_v d_0(\hat{\Psi}^{SL}_k(P_{n,v}),\psi_0)+C(M_1,M_2,\delta)\frac{\log K_n}{np}}
\leq (1+\delta) \]
If  the number of Super Learners does not grow faster than a polynomial in sample size $n$, i.e, $K_n\ll n^p$ for some $p$, and
\[
\frac{n^{-1}\log n}{E_0\min_k \frac{1}{V}\sum_v d_0(\hat{\Psi}^{SL}_k(P_{n,v}),\psi_0)}\rightarrow 0,\]
then the  double super-learner $\hat{\Psi}^{SL}_{k_n}$ is asymptotic equivalent with the oracle selected super-learner $\hat{\Psi}_{\tilde{k}_n}$:
\[
\frac{E_0 \frac{1}{V}\sum_v d_0(\hat{\Psi}^{SL}_{k_n}(P_{n,v}),\psi_0)}{E_0\\min_k \frac{1}{V}\sum_v d_0(\hat{\Psi}^{SL}_k(P_{n,v}),\psi_0)}
\rightarrow 1\mbox{ as $n\rightarrow\infty$.}
\]

\subsection{A finite sample oracle inequality for an $\epsilon$-net double super-learner}
The above results show that a double super-learner will be asymptotically equivalent with choosing the best meta-learning algorithm among the sequence of Super Learners, and that, this oracle selected Super Learner approximates the oracle choice in its meta-learning model at a rate depending on its covering number, but either way, at least as fast as $n^{-1/4}$. In this subsection, we show a stronger result demonstrating that the double super-learner is able to optimally trade-off bias and variance across all possible ensembles across the meta-learning models. However, this requires generating a more refined set of meta-learning models, namely, for each meta-learning model, we define an  $\epsilon$-net (like sieve of increasing complexity) representing a resolution.

Let ${\cal F}_k=\{f_{k,\alpha}:\alpha\in {\cal E}_k\}$ for some parametrization $\alpha\rightarrow f_{k,\alpha}$ with the $\alpha$-parameter varying over a set ${\cal E}_k$.

For each $k$, let ${\cal E}_k(\epsilon)=\{\alpha_{k,j}: j=1,\ldots,N_k(\epsilon)\}\subset {\cal E}_k$ be a finite set of values in ${\cal E}_k$ so that the parameter space of $\{\hat{\Psi}_{\alpha}(P_n):\alpha\in {\cal E}_k(\epsilon)\}$
represents an $\epsilon$-net of $\{\hat{\Psi}_{\alpha}(P_n):\alpha\in {\cal E}_k\}$ w.r.t. some dissimilarity (chosen to dominate or be equivalent with the loss-based dissimilarity). Let $N_k(\epsilon)$ the number of elements in ${\cal E}_k(\epsilon)$.

{\bf $(k,\epsilon)$-th super-learner:}
For each $k,\epsilon$, given a sample $P_n$, let
\begin{equation}\label{kthMLt}
\alpha_{k,\epsilon}(P_n)\equiv \arg\min_{\alpha\in {\cal E}_k(\epsilon)}\frac{1}{V_1}\sum_{v_1=1}^{V_1} P_{n,v_1}^1 L(\hat{\Psi}_{k,\alpha}(P_{n,v_1}) ).\end{equation}
Then, $\hat{\Psi}^{SL}_{k,\epsilon}:{\cal M}_{NP}\rightarrow {\bf \Psi}$ defined by \[
\hat{\Psi}^{SL}_{k,\epsilon}(P_n)\equiv \hat{\Psi}_{k,\alpha_{k,\epsilon}(P_n)}(P_n)\]
 is a super-learner based on the $(k,\epsilon)$-th meta-learning algorithm defined by (\ref{kthMLt}), i.e., by minimizing the cross-validated risk over a $\epsilon$-net of the $k$-th specific family of the library of candidate estimators $(\hat{\Psi}_j:j=1,\ldots,J)$.

{\bf Set of candidate super-learners:}
Now, $\hat{\Psi}^{SL}_{k,\epsilon}:{\cal M}_{NP}\rightarrow {\bf \Psi}$, $k=1,\ldots,K$, and $\epsilon$ varying over a grid, represents a set of candidate estimators, each one of them being a super-learner based on $(k,\epsilon)$-specific meta-learning algorithm defined by minimizing the cross-validated risk over a parametric family.

{\bf cross validation selector of choice of meta learning, including resolution $\epsilon$:}
Let
\begin{eqnarray*}
(k_n,\epsilon_n)&=&\arg\min_{k,\epsilon} \frac{1}{V}\sum_{v=1}^V P_{n,v}^1 L(\hat{\Psi}^{SL}_{k,\epsilon}(P_{n,v}))\\
&=&\arg\min_{k,\epsilon} \frac{1}{V}\sum_{v=1}^V P_{n,v}^1 L(\hat{\Psi}_{k,\alpha_{k,\epsilon}(P_{n,v})}(P_{n,v})),
\end{eqnarray*}
where
\[
\alpha_{k,\epsilon}(P_{n,v})=\arg\min_{\alpha\in {\cal E}_k(\epsilon)}\frac{1}{V_1}\sum_{v_1=1}^{V_1} P_{n,v,v_1}^1 L(\hat{\Psi}_{k,\alpha}(P_{n,v,v_1}) ).
\]

{\bf Proposed double  super-learner for theoretical analysis:}
Then, our final super-learner and proposed estimator is defined by
\[
\hat{\Psi}(P_n)=\hat{\Psi}^{SL}_{k_n,\epsilon_n}(P_n).\]

Let $K_n$ be the number of values of $(\epsilon,k)$ over which we minimize in definition of $(k_n,\epsilon_n)$.
We have the following theorem.
\begin{theorem}\label{theoremepsnet}
Suppose
\begin{eqnarray}
Ed_0(\hat{\Psi}_{k,\alpha_{k,\epsilon}(P_n)}(P_{n,v}),\psi_0) &\leq&
E \frac{1}{V_1}\sum_{v_1} d_0(\hat{\Psi}_{k,\alpha_{k,\epsilon}(P_{n,v})}(P_{n,v,v_1}),\psi_0).
\label{aa}
\end{eqnarray}
Then, we have
\[
\begin{array}{l}
E \frac{1}{V}\sum_v d_0(\hat{\Psi}_{k_n,\epsilon_n}^{SL}(P_{n,v}),\psi_0)\leq (1+\delta)^2\\
\min_{k,\epsilon} \frac{1}{V}\sum_v \left \{
E  \min_{\alpha\in {\cal E}_k(\epsilon)} \frac{1}{V_1}\sum_{v_1} d_0(\hat{\Psi}_{k,\alpha}(P_{n,v,v_1}),\psi_0)+C(M_1,M_2,\delta) \frac{\log N_k(\epsilon)}{n(1-p)p_1}\right \}\\
\hfill +C(M_1,M_2,\delta)\log K_n /(np).
\end{array}
\]
\end{theorem}
{\bf Proof:}
As shown in previous section, regarding the cross validation selector $(k_n,\epsilon_n)$ of $(k,\epsilon)$ we have
\begin{eqnarray}
E\frac{1}{V}\sum_v d_0(\hat{\Psi}_{k_n,\epsilon_n}^{SL}(P_{n,v}),\psi_0)&\leq & (1+\delta)E \min_{\epsilon,k}\frac{1}{V}\sum_v  d_0(\hat{\Psi}_{k,\epsilon}^{SL}(P_{n,v}),\psi_0)\nonumber \\
&& +C(M_1,M_2,\delta)\frac{\log K_n}{np} \label{firstoracle}
\end{eqnarray}
Recall $\hat{\Psi}_{k,\epsilon}^{SL}(P_{n,v})=\hat{\Psi}_{k,\alpha_{k,\epsilon}(P_{n,v})}(P_{n,v})$.
Regarding the cross validation selector $\alpha_{k,\epsilon}(P_{n,v})$ we have
\begin{eqnarray}
E\frac{1}{V_1}\sum_{v_1} d_0(\hat{\Psi}_{k,\alpha_{k,\epsilon}(P_{n,v})}(P_{n,v,v_1}),\psi_0)&\leq& (1+\delta)E \min_{\alpha\in {\cal E}_k(\epsilon)} \frac{1}{V_1}\sum_{v_1} d_0(\hat{\Psi}_{k,\alpha}(P_{n,v,v_1}),\psi_0) \nonumber \\
&&+C(M_1,M_2,\delta) \frac{\log N_k(\epsilon)}{n(1-p)p_1}. \label{secondoracle}
\end{eqnarray}
Suppose (\ref{aa}) holds.
Then, first term on right-hand side of (\ref{firstoracle}) is bounded by
\[
(1+\delta) \min_{k,\epsilon} \frac{1}{V}\sum_v E \frac{1}{V_1}\sum_{v_1} d_0(\hat{\Psi}_{k,\alpha_{k,\epsilon}(P_{n,v})}(P_{n,v,v_1}),\psi_0).\]
Using the second oracle inequality (\ref{secondoracle}) yields that we can bound this by $(1+\delta)$ times
\[
\min_{k,\epsilon} \frac{1}{V}\sum_v \left \{
(1+\delta) E  \min_{\alpha\in {\cal E}_k(\epsilon)} \frac{1}{V_1}\sum_{v_1} d_0(\hat{\Psi}_{k,\alpha}(P_{n,v,v_1}),\psi_0)+C(M_1,M_2,\delta) \frac{\log N_k(\epsilon)}{n(1-p)p_1}\right \}.
\]
This proves the stated bound.
$\Box$

\paragraph{Discussion of Theorem \ref{theoremepsnet}}
Consider the displayed inequality in Theorem \ref{theoremepsnet}.
Note that the last term is negligible as long as $K=K_n$ is not growing faster than polynomial in $n$.
So it is all about the first term. For each $k$, view $\hat{\Psi}_{k,\alpha}(P_{n,v,v_1})$ as a $k$-specific data adaptive model with parameter $\alpha\in {\cal E}_k$. First consider the case that $\psi_0$ is contained in one of these $k$-specific models with probability tending to 1 as $n\rightarrow\infty$. For  the $k$-specific  data adaptive model,  $\min_{\alpha\in {\cal E}_k(\epsilon)}$ denotes a bias term while the covering number term  $\log N_k(\epsilon)/(n(1-p)p_1)$ represents a variance term, and it is known that optimizing over $\epsilon$ the sum of these two terms optimally trades off bias and variance (up till constant), resulting in a minimax rate of convergence for this $k$-specific data adaptive model \cite{van2003unified}

The outer $\min_k$ shows that we achieve the minimax rate of convergence corresponding with the smallest of the data adaptive models that contains the true $\psi_0$. So if one of the data adaptive models contains the true $\psi_0$ then we would achieve the minimax adaptive rate of convergence. For most choices of $k$-specific families, one does not expect that that $\psi_0$ is contained by any of them, so that the bias term is not just driven by the $\epsilon$-resolution, but also by the bias of the overall $k$-specific model. Either way, the leading term in this oracle inequality is still properly trading of the actual bias with the variance term.

\section{Practical Aspects: Initialization, Collapsing and Soft CSL}

We initialize CSL by picking the type of experts (e.g trees, linear models, etc) and a random subset of the data to fit each expert to introduce diversity. Please note that as its counterpart (K-means), CSL can get stuck in local minimum. Therefore we run the algorithm a few times in our experiments. We use a validation set (different than the test set) to select the best solution. While running CSL it is often the case that some of the experts will collapse (e.g model selection). This happens when the range of values predicted by $o(x)$ is less than $K$ or similarly when the size of $\{x: o(x)=k\}$ becomes very small. In these cases we re-adjust the size of $K$ as the algorithm runs.

Finally, please note that we can use the step when we are fitting the experts to introduce regularization. If the oracle $o(x)$ also estimates probability of an observation belonging to a model  (e.g. logistic regression), then we can use $p({0}(\bf{x}) = k,\bf{x})$ to introduce similarity among the experts when we are fitting them, specially around the boundaries defined by the oracle.

\section{Simulations}

In this section we describe our experiments. We consider the dataset of all regression problems from the Penn Machine Learning Benchmarks \cite{olson2017pmlb}. In the first set of experiments, dataset where the number of observations was between 200 and 500000 ($N=84$) were considered. In the second set of experiments, we selected a subset of those with at least 2000 points ($N=19$). We report $R^2$ as the performance metric. In the first experiments, datasets were split in $80\%$ training and $20\%$ testing sets. For the second experiment, we need a validation set, therefore data was split in $70\%/ 15\%/ 15\%$ for train/validation/test.

In all our experiments we use the following set of base algorithms or experts:

 \begin{enumerate}
\item Ridge: alphas = [1e-4, 1e-3, 1e-2, 1e-1, 1, 2, 4, 8, 16, 32, 64, 132]
\item ElasticNet: l1 ratio = 0 (Lasso)
\item ElasticNet: l1 ratio = 0.5
\item Decision Tree: max depth = 4
\item Decision Tree: max depth = 5
\item Decision Tree: max depth = 6
\end{enumerate}

Although the CSL can be used with more complex models as experts, using simple linear or tree models have two appealing: protects the algorithm for overfitting and has implications for its interpretability as discussed below.

\begin{figure}[t!]
 \includegraphics[width=0.6\textwidth]{Figure2.png}
\centering
\caption{One shot CSL vs cross validation}
\label{Figure2}
\end{figure}

\subsection{Single step CSL and its comparisson to cross validation}

First, we wanted to evaluate empirically how CSL (${\cal F}_3$ class of meta learners in the theoretical section) performs compared to the naive strategy of using the expert model selected through cross validation for all the points (${\cal F}_1$ class in the theoretical section). For this we used the library of experts described above and a decision tree algorithm with $max\_depth=1$ as our oracle. No partition was allowed if the terminal node did not have more than $2\%$ of observations belonging to it. This decision tree algorithm was selected as the oracle because if not partition is performed, then the minimization of equation \ref{eqn1} results in selecting the model that minimizes the cross validation error (equal to ${\cal F}_1$ class ). As such, our CSL in this case includes the possibility of just using cross validation and it should perform, on average, better than using cross validation to select one model. Please note that in this section the experts are obtained on all the training data and only one iteration is allowed for the meta. The empirical evaluation of this CSL was compared to the performance obtained if we use cross validation to select the best model from one of the sixth algorithms mentioned above.

 Figure \ref{Figure2} shows the results obtained when we compared $R^2$ for both CSL and cross validation. In $77.5\%$ of the datasets CSL had at least the same performance as cross validation and in $20 \%$ its $R^2$ was bigger by at least 1. Although these results show that CSL improves over naive cross validation, overfitting can happen and extra attention needs to be paid. In fact, we obtained an outlier where cross validation outperformed the CSL by 0.125. This problem is the synthetic dataset $658-fri-c3-250-25$ from the Friedman's regression datasets \cite{olson2017pmlb,friedman1991multivariate}. The same is a hard problem where algorithms are prompt to overfit since training only contains 200 points with 25 explanatory variables, several of them correlated to each other. Additionally, for those datasets where CSL was better, on average they had 3805 data points compared to 2368 in those datasets where cross validation did better. Therefore, as a general rule, CSL needs more than 2500 data points to perform better than cross validation. These empirical results corroborate our suggestions on the theoretical session and the need for a two tier Super Learner algorithm where the type of meta strategy, or class of functions as defined here, is also selected.

 In the next section, we will compare a full CSL  to stacking. ${\cal F}_2$ vs ${\cal F}_3$ meta learning strategies in our theoretical discussion.

\subsection{CSL versus Stacking}
\begin{table}
 \begin{tabular}{|l|c|c|c|c|}%
    \hline
    \bfseries Dataset & \bfseries CSL\_mean & \bfseries Stack\_mean & \bfseries Diff & \bfseries Test resul
    \csvreader[head to column names]{grade.csv}{
    {\\\hline \dataset & \islmean & \stackmean & \diffmean & \test}
    \tabularnewline\hline
    \end{tabular}
    \caption{Results comparing CSL and stacking on 19 regression datasets. In one dataset stacking is significantly better than CSL. In 8 problems both algorithms are statistically the same. In 11 problem CSL is better.}
    \label{table_stacking}
\end{table}

{\bf Stacking}  \cite{wolpert1992stacked, breiman1996stacked,SmythWolpert1997} is
a general procedure where a learner is trained to combine the individual learners. The base level models are trained on the original training set, then a meta-model is trained on the outputs of the base level model as features. The base level often consists of different learning algorithms. In our experiments, we use the same 6 base models defined in previous section. As meta-model in stacking we use a linear regression model which corresponds to ${\cal F}_2$.

{\bf CSL}. In this experiment, we use the same set of experts as in previous section and we use a two layer feed-forward neural network as the oracle. The oracle was written in PyTorch and fitted with Adam optimizer with learning rates of 0.15. The number of epochs at each iteration was a function of the sample size ($ \frac{3000}{\log(N)^2}$). The hidden layer was also set as a function of the sample size ($\min(2\log(N), 150)$). After the first linear layer and before the Relu activation function, batch normalization was used. A dropout layer with $p=0.2$ is used before the second linear layer.

In Table \ref{table_stacking} we show results from comparing CSL and stacking on regression datasets. For each dataset we run each algorithm 10 times by spliting training, validation and testing sets using different seed. \emph{CSL\_mean} shows the mean $R^2$ over all experiments. Similarly, \emph{Stack\_mean} shows the mean $R^2$ of the stacking experiments. \emph{Diff} show the difference between \emph{CSL\_mean} and \emph{Stack\_mean}. Column \emph{Test results} shows whether a t-test found the difference in mean to be significant. There were 19 datasets in this experiment. For one dataset stacking is significantly better than CSL. In 8 problems both algorithms are statistically the same. In 11 problem CSL is better. In the experiment for dataset \emph{574\_house\_16H}, one of the runs produces an outlier which is responsible for the mean difference.

\subsection{CSL on hierarchical synthetic data}

Hierarchical models are extremely important in fields like Medicine \cite{normand1997statistical}. CSL, due to its architecture, seems specially suited to analyze hierachical data. In this section, we evaluate the performance of CSL on hierarchical synthetic problems. Specifically, we wanted to investigate if:

\begin{enumerate}
\item CSL can discover hierarchical structures.
\item Compare its performance in these type of problems to gradient boosting and random forest.
\end{enumerate}

Given a real dataset $\{(x_i, y_i)\}^N_1$ from the Penn Machine Learning Benchmarks, we generate syntetic data by using the observations from the covariates but generating new labels. Here is how we generated the new ($\tilde{y}$):
\begin{enumerate}
\item Sample 70\% of the data.
\item Use the covariates to find $K=3$ clusters using K-means algorithm.
\item Fit a 2-layer neural network using the cluster id as a label.
\item Using the neural network, predict a cluster id ($l_i$) on each of the original observations, creating the dataset $\{(x_i, y_i, l_i)\}^N_1$.
\item Fit each subset $\{(x_i, y_i) \text{ such that } l_i=k \}$ for $k \in \{1, 2, 3\}$ to a regression model (Ridge). Use the prediction from the regression model as the new synthetic label ($\tilde{y}$).
\end{enumerate}

\begin{table}
 \begin{tabular}{|l|c|c|c|c|}%
    \hline
    \bfseries Dataset & \bfseries CSL & \bfseries Base Expert & \bfseries RF & \bfseries GBM \\
    \hline
    564\_fried & \textbf{0.97} (0.02) & 0.45 (0.17) & 0.82 (0.04) &  0.82 (0.06) \\
    574\_house\_16H & \textbf{0.98} (0.01) & 0.84 (0.04) & 0.87 (0.01) & 0.93 (0.01)\\
    294\_satellite\_image & \textbf{0.99} (0.01) & 0.88 (<0.01) & 0.98 (0.01) & 0.98 (<0.01)\\
    218\_house\_8L & \textbf{0.97} (0.02) & 0.75 (0.03) & 0.94 (0.01) & 0.93 (0.01)
    \tabularnewline\hline
    \end{tabular}
    \caption{Results on running CSL on synthetic data. Comparison are made with random forest, gradient boosting and the best preforming base expert. For each dataset the we generate 10 synthetic problems. For each method, the mean test $R^2$ and standard deviation is shown.}
    \label{table_synt}
\end{table}

For the experiments in Table \ref{table_synt}, gradient boosting was ran with the following parameters: min\_child\_weight=50, learning rate = 0.1, colsample\_bytree= 0.3, max\_depth= 15, subsample=0.8, and with 500 trees. For random forest, we used 1000 trees, max\_features='sqrt' and we found max\_depth with cross validation for each problem. The problems are a subset of the problems in Table \ref{table_stacking}, where the $R^2$ of the base expert is lower than 0.9. The CLS alrorithm was initialized using 11 linear models as based experts. It turns out that by using trees together with linear models on these problems the algorithm sometimes would get stuck in suboptimal local minima.

Table \ref{table_synt} shows the results ($R^2$) of CSL on synthetic data compared to random forest, gradient boosting or the best base expert (selected using cross validation). As it can be seen, CSL significantly outperforms all other algorithms as expected.

\subsection{Implications on interpretability}

In many high-stake domains like medicine and the law, interpretability of machine learning algorithms is highly desirable since errors can have dire consequences \cite{valdes2016mediboost, louzada2016classification, cabitza2018machine}. This fact has led to a renew interest for the development of interpretable models. Interpretability, however, can only be judge relative to the field of application as it is in the eyes of the beholders. In fact, there is a considerable disagreement on what the concept means and how to measure it \cite{lipton2016mythos,lipton2017doctor, doshi2017towards}. Different algorithms afford different degrees of interpretability and even  black boxes can be investigated to gain some intuition on how predictions are being made. For instance, variable importance or distillation can be used to interpret neural networks \cite{tan2018transparent}. This level of interpretability might be enough for applications that do not impose high risk. In other applications (e.g medicine), the need to understand the models globally rises \cite{valdes2016mediboost}. Without attempting to formally quantify and define interpretability here, we will illustrate below how the $CSL$ results in models that are highly transparent.

{\bf Predicting house prices.} To illustrate how CSL can be use as an interpretable algorithm we use a dataset of house rental prices in New York City. We have 4 input variables: latitude, longitude, number of bedrooms and number of bathrooms. Two make it really simple to visualize and interpet, the oracle was given two of the variables: latitude and longitude. The CSL model found a solution in which the oracle parition the space of latitude and longitude in 3 regions (see top of Figure \ref{Figure1}) and for each region a tree of depth 5 predicts the house prices. This simple solution get an $R^2$ of 0.68. As a comparison, the best single model of a tree of depth 5 has an $R^2$ of 0.62 and a random forest with 500 trees (of depth 9) has an $R^2$ of 0.72. To find the best random forest we did grid search on the number of variables and the depth of the trees.

The simple solution of a tree of depth 5 is interpretable since a tree of depth 5 can be easily examined. Also, 3 trees of depth 5 can be easily examined as well as the 2 dimensional space where the oracle split the restricted input regions. On the other hand, the random forest with 500 trees and unrestricted depth can not be called interpretable.

\subsection{Connection to other algorithms}

Our CSL is related to different algorithms and highlighting them here can give us additional intuitions about its performance, what problems are best suited for it and how to improve it. First, please note that $o(x)$ partitions the space in $K$ regions or subsets $\{\mathcal{R}\}_1^K$ where the models $\{F_{k}(x)\}_1^K$ are used for prediction; ergo establishing the connection between meta learning and generalized partitioning machines. Different from recursive algorithms like CART, MediBoost or the Additive Tree, CSL partitions defined by the oracle can have complex forms and are not forced to be perpendicular to the covariates. Additionally, $CSL(x)$ also generalizes the strategy of using cross validation to select the best model. Please note that if in equation \ref{eqn1} we force $o(\bf{x}) = c$ where $c$ is a constant $\in \{1..K\}$ then the solution to \ref{eqn4}, $\hat{o}(\bf{x})$, just selects the model that minimizes the cross validation error. As such, using cross validation to select the best model is the simplest case of $CSL(x)$ where the meta learner predicts a constant regardless of the covariate. CSL can also be thought of a generalization of the K-means algorithm. If the expert models are constant, and the oracle has infinity capacity to always be able to assign each observation to the best mean, then CSL becomes the K-mean algorithm. Finally, as shown above, due to its archictecure, CSL is a non parametric hierarchical algorithm and performs quite well for this type of problems.

\section{Conclusions}

In this work we introduced the CSL algorithm. We proved theoretically and empirically how we can extend the idea of meta learning and develop an algorithm that outperforms the naive use of cross validation to select the best model.We proved that the CSL has a rate of convergance faster than $O_p(n^{-1/4})$. More over, we have obtained very interesting and practical results. For instance, CSL outperformed stacking in the datasets analyzed. Additionally, it significantly outperformed Random Forests or Gradient Boosting in the analysis of Hierarchical Data. Finally, its connection to interpretability and other algorithms were highlighted to deepen our understanding of its performance. As such, the CSL is an algorithm suited for the analysis of medical datasets where hierarchical models and intepretability are of paramount importance.

\section{Acknowledgments}

Research reported in this publication was supported by the National Institute Of Biomedical Imaging And Bioengineering of the National Institutes of Health under Award Number K08EB026500 and by the National Institute of Allergy and Infectious Diseases under Award Number 5R01AI074345-09. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.Finally, we would also like to thank Dr Charles McCullow for initially suggesting to investigate this topic.

\bibliographystyle{tfs}

redpajama_set_name: RedPajamaArXiv


These underglazes are intense, opaque colors which are especially good for watercolor effects and fine detail. They are packaged in pans, like watercolors, and you need to use water and a brush to apply them. All colors may be fired to cone 10 except for Pink, Maroon, Rose and Yellow, which will only hold up until cone 5/6.


text:
Natalie Portman Hairstyles
Natalie Portman with her classy upstyle hairstyle at the Vanity Fair Oscar Party 2015. - Advertisement -
Artists have to make so many sacrifices for acting. They often need to come out of their own shell and be someone else. As a result, they need to bring in changes in their hairstyle and getup. Sometimes, they have to carry these styles into their off-screen lives as well.
Movie stars often change their hairstyles, and many times, it's due to the roles they play in the movie. Natalie Portman is a very talented artist who achieved a lot at a very young age. She has undergone several transformations in her hairstyle over her acting career.
Here's Natalie Portman's hairstyles over the years.
Pregnant Natalie Portman, who threw her hair up into a vintage updo, wore a majestic flowing white Dior dress, giving her an ethereal look at the 23rd Annual Screen Actors Guild Awards held at the Shrine Expo Hall in Los Angeles on January 29, 2017.
Check out more Natalie Portman's hairstyles here!
Natalie Portman with long upstyle hair at the Nazarian Center for Israel Studies fifth Annual Gala 2015. She looks elegant, as ever, in her short sequin-dress.
Natalie Portman with her upstyle hair seen in Midtown, New York 2016. Even in ordinary days, she looks very elegant with her usual hairstyle; natural-looking yet very classy.
Natalie Portman with her classy hairstyle at the 20th Annual Hollywood Film Awards 2016. She looks blooming in her black dress with her baby bump; very classy and elegant.
Natalie Portman with her straight brunette hair at the 73rd Venice Film Festival 2016. Her natural-looking style is perfect for her bird-printed dress; she looks simple yet very classy.
Natalie Portman with her long brunette upstyle hair at the premiere of the 'Knight of Cups' 2015. She just looks stunningly elegant in her black dress.
Natalie Portman with her hair pulled into an upstyle hairstyle at the gala premiere for "Sicario" 2015. She looks very elegant in her green long dress with perfect with her beautiful smile.
Natalie Portman with her upstyle hairstyle at the 68th annual Cannes Film Festival 2015. A natural-looking style that makes her very elegant, perfect for her black bead-studded dress.
Natalie Postman with her medium-length hair at the ceremony of the 68th Cannes Film Festival 2015. She looks stunningly gorgeous in her red dress; very elegant.
Natalie Portman with her elegant upstyle hairstyle at the premiere of 'Planetarium' during the 73rd Venice film festival 2016. She looks stunning in her white long dress; very elegant and classy.
Natalie Portman with her hair pulled into an upstyle hairstyle at the premier of 'Jackie' during the 73rd Venice film Festival 2016. She looks very elegant with black and white abstract dress.
Natalie Portman with her classy upstyle hairstyle at the Vanity Fair Oscar Party 2015. As ever, she looked elegant and stunning in her white dress.
During the early years, she had a bob cut style most of the time. She had a straight bob with fringes and blunt bangs on the front. She also had a curly bob. She experimented with her hair color as well. She had black, brown and even brunette shades. She was seen in the twenties curls at a party in 2003. She then moved to the mid-length hairstyle.
This style of hair suits her innocent face very well. It gives her the elegant feminine look. She has tried from straight to wavy mid-length hair. She has put on different shades as well. She chose highlights from brown, brunette, chestnut, etc. Up-dos good look on her as well. In several events, she had chosen this hairstyle. Sometimes, she went for the curly updos and sometimes the straight ones.
In her career, from the very beginning, she has proved herself to be a serious actress. She not only appeared in highly commercial films but also acted in films with serious stories that needed high-class acting skills. Her best performance so far was in the film Black Swan. Her first film was Leon: The Professional which was released in 1994. This film got her many other films offers including Heat, Mars Attacks!, Beautiful Girls, etc. Her acting is brilliant in all these films. For her, the real fame came from the first sequel to the movie Star Wars. She played the role of Queen Amidala in this movie.
She later acted in the following sequels of the same movie and gained huge popularity. She has won a number of awards. She is the youngest actress to win the Academy Award for Best Actress. She won this award for her performance in Black Swan. She also received the Golden Globe Award for the same movie. She got an Oscar nomination for Closer and Academy Award nomination for Jackie.
She was the youngest jury member of the Cannes Film Festival in 2008. Not many people know that she is also a Harvard graduate. This talented actress has a long way to go and lots more to achieve in future.
Return to the main women celebrity page.

redpajama_set_name: RedPajamaCommonCrawl


text:
package me.yoerger.geoff.edu.progClass.bookClasses;

import javax.swing.*;
import java.text.*;

public class SimpleOutput
{

  /**
   * Method to show a warning to a user
   * @param message the message to display
   */
  public static void showWarning(String message)
  {
    message = addNewLines(message);
    JOptionPane.showMessageDialog(null,message,"Warning Display",
                                  JOptionPane.WARNING_MESSAGE);
  }

  /**
   * Method to show an error to a user
   * @param message the message to display
   */
  public static void showError(String message)
  {
    message = addNewLines(message);
    JOptionPane.showMessageDialog(null,message,"Error Display",
                                  JOptionPane.ERROR_MESSAGE);
  }

  /**
   * Method to show information to the user
   * @param message the message to display
   */
  public static void showInformation(String message)
  {
    message = addNewLines(message);
    JOptionPane.showMessageDialog(null, message, "Information Display",
                                  JOptionPane.INFORMATION_MESSAGE);
  }

  /**
   * Method to add new line character if the message
   * is too long
   * @param message the input message
   * @return the message with new lines added if needed
   */
  public static String addNewLines(String message)
  {
    BreakIterator boundary =
      BreakIterator.getLineInstance();
    boundary.setText(message);
    int start = boundary.first();
    String result = "";
    String currLine = "";
    String temp = null;

    // loop till no more possible line breaks
    for (int end = boundary.next();
          end != BreakIterator.DONE;
          start = end, end = boundary.next())
    {
      // get string between start and end
      temp = message.substring(start,end);

      /* if adding that to the current line
       * would make it too long then add current
       * to result followed by a newline and
       * reset current
       */
      if (temp.length() + currLine.length() > 100)
      {
        result = result + currLine + "\n";
        currLine = temp;
      }
      // else add the segment to the current line
      else
        currLine = currLine + temp;
    }

    // if no line breaks use the original message
    if (result.length() == 0)
      result = message;
    // else add any leftover parts
    else
      result = result + currLine;

    return result;
  }

} // end of SimpleOutput class

redpajama_set_name: RedPajamaGithub


This stunning engagement ring is created with a sparkle that will last a lifetime! A luminous princess cut center stone accented by a dazzling pave setting that that classy bride-to-be will adore! This Engagement Ring is made to order in your choice of 14K White or 14K Yellow Gold and comes with a 1.50 Carat Center Simulated Diamond and a total of 1.85 Carat total weight. An optional Matching Band is available as the perfect complement for this ring and you can either choose 1 or 2 Bands from the drop-down menus above. According to your selection of precious metal and ring size, your order will ship within 7 working days.


I have a small collection of classic Nintendo Game and Watch devices, I once promised to put images online. I may add further old videogaming stuff I have.
Wikipedia page about Gamen and Watch is quite good, if you're looking for more info.
Today I saw that in the Rewe supermarket, they sell remakes of old Nintendo Game and Watch games (called Mini Classics). For those who don't know, Game and Watch where early Nintendo Games, only one game in a device, with an lcd-screen.
I bought the two classics Super Mario Bros. and Donkey Kong Junior (there were some others but none of them seemed to be of the original titles). I also have a quite nice collection of older Game and Watch titles, I think I might upload some pictures of them when I find time for it.


text:
Those Anarcho Punks Sure Put On A Fun Show | Against Me Live Review | The Beacham Orlando | February, 19 2015
by Jordan Shroyer • February 26, 2015
Against Me! took the stage by storm, jumping into "True Trans Soul Rebel" off of 2014's highly praised Transgender Dysphoria Blues. Followed by a career spanning set well mixed with cuts from their six album catalog.
Worriers, featuring Lauren Dentizio formerly of the Measure {SA} and Mikey Erg! (of The Ergs! / The Chris Gethard Show house band / at least a dozen FEST bands) on drums opened the show. They performed a set of new songs as well as material from their 2013 effort, Cruel Optimist, and debut 7", Past Lives.
While Philadelphia's Creeopid appeared to still be mid-soundcheck, they suddenly launched right into their set without warning. As the first song winded down, bassist Anna Troxell asked for the lights to be lowered. The venue went near black with the exception of the bright white glow coming from the kick drum, setting the mood for the rest of their set.
Creepoid's shoegaze-y meets Pixies vibes, with guitarist Sean Miller and Troxell sharing vocal duties, translated well to a bigger stage. Through the screaming guitar solos and pounding drums, they somehow managed to send the crowd into a lulled trance for 30 plus minutes of melodic noise. ranging from soft a pretty to loud and spacey. Instead of just playing songs from both of their stellar No Idea Record's full lengths, they graced us with a selection on brand new unheard material from a stage proclaimed "June release" which they finished tracking just before this tour began.
The Beacham is a nice sized room with big sound, but it can be a bit uncomfortable. If a show is sold out (it holds approximately 1,200), it's often too crowded to find a space to comfortably watch (this may only pertain to aging show-goers so you young-ins can ignore that). If a show is undersold, it can feel very uncomfortable, especially for the band. This show had the perfect amount of people. I didn't do a door count or anything, but by the time Against Me! hit the stage, I'd estimate around 800 or so were in the room. It was nice and full, but easy to find a spot near the stage to get a great view, dance, and sing along without getting thrown in a pit or feeling forced to stand in the same space for the rest of the night.
Against Me! have gone though many member and stylistic changes over the past 14 years. Starting as a solo acoustic act in 1997, Laura Jane Grace (then known as Tom Gabel), the teenage anarchist, eventually grew the project into a 4 piece. Former members include drummers George Robello (Hot Water Music) and Jay Weinberg (son of Max from Conan O'Brian and Springsteen fame) who's flashiness added a different kind of energy to AM!'s festival shows. But never really fit in with what the band was about or supposed to sound like. Just a bad fit.
Blasphemous as this may be to those who are devoted to the classic line up, with drummer Warren Oakes (02'-09' who quit to open Boca Fiesta, a Mexican restaurant in Gainesville, FL) and bass player Andrew Seaward (02'-13'), the current line-up of Grace, long time guitarist James Bowen, Inge Johansson (International Noise Conspiracy) on bass and Adam Willard (Rocket From The Crypt/Offspring/Social Distortion) on drums has spawned the most solid and musically engaging performances I've seen in the decade-plus I've been going to AM! shows. Not necessarily the most lively or energetic, but it's the perfect place for this band to be at this point in their career and their set at The Beacham proved this yet again.
They cranked out song after song without appearing to even take a breath. Within seconds of "Cliche Gueva's" final "waaaaaaaaaaaaay" being shouted from the lips of what appeared to be everyone in attendance, Willard was already pounding on the snare for the intro of "Reinventing Axl Rose." The energy was fierce and the band didn't stop for so much as a drink of water until who knows how many songs in. It was just back to back to back songs; a pretty impressive amount of endurance from both the band and the crowd. And, even though this has come to be expected by now, it's still impressive that after so many years together a band can crank it out the way Against Me! does for 23 songs a night, every night.
The set was closed out with the sing along (I mean, they're just about all sing alongs), "Sink, Florida, Sink." The band left the stage while the crowed screamed and Grace came back out with an acoustic, going into renditions of "Baby I'm An Anarchist," "Joy," and "Thrash Unreal" which the rest of the band joined her on. After this point I was pretty fuzzy (I blame Mitch and his whiskey shots). But, yeah, the show ruled.
Against Me! Setlist, February 19, 2015 – The Beacham Orlando:
True Trans Soul Rebel
Cliche Guevara
Reinventing Axl Rose
Those Anarcho Punks Are Mysterious
Osama Bin Laden as the Crucified Christ
Don't Lose Touch
I Was Teenage a Teenage Anarchist
How Low
T.S.R.
Walking Is Still Honest
Drinking With the Jocks
Transgender Dysphoria Blues
White People For Peace
You Look Like I Need A Drink
Black Me Out
Pints of Guinness Make You Strong
Sink, Florida, Sink
Baby I'm An Anarchist (acoustic)
Joy (acoustic)
Thrash Unreal (acoustic + full band)
FuckMyLife666
We Laugh At Danger and Break All the Rules
Against Me! Set List Pie Chart by Album:
Against Me Live Review by Jordan Shroyer.
Against Me Live Concert Photos by Carlo Cavaluzzi.
GIVEAWAY — Yacht Rock Revue @ House Of Blues Orlando (May 08)
TICKET GIVEAWAY + PREVIEW: Circa Survive | February 9, 2022 @ House of Blues Orlando
TICKET GIVEAWAY — CARAVAN PALACE (French Electro-Swing) coming to House Of Blues Friday, January 21
Spencer's TOP 5 New Years Eve Shows To Go To!


text:
Home » Intellectual Property Law » The Three C's – Content sharing, copyright and Creative Commons licenses
The Three C's – Content sharing, copyright and Creative Commons licenses
Provided by KISCH IP
With an entrenched history in Africa, KISCH IP has, for the last 145 years, assisted clients from small to large businesses in all sectors, in safeguarding their intellectual property rights. While acknowledging our establish... more
By Tammi Pretorius
Topics Intellectual Property Law
When a work, such as a video, is uploaded onto YouTube or a photograph uploaded onto Flickr, people often believe that because it is being made available on the internet, it is free for them to download, distribute or use. This is a common misconception and what many people don't realise, is that, more often than not, copyright vests in those particular works. Unless you have been authorised by the owner of the copyright (for example, in the form of a license), you cannot simply do what you want with the work.
It's vitally important to ensure that when you are using or downloading works from content sharing and knowledge-based platforms such as YouTube, Flickr and Wikipedia, you have the authorisation or license to do so (or, in the alternative, to ensure that the work or content is not protected by copyright). One way to do this is to see if a Creative Commons license has been assigned to the work or content, before downloading or using the work.
A Creative Commons license is, essentially a copyright-management regime which allows copyright owners to share their works in the public space, whilst still ensuring that their rights remain intact. By by-passing the normal negotiations between a copyright owner (licensor) and an authorised user (licensee) regarding rights and terms of use, Creative Commons licenses ensure copyright owners remain in control of their works (to some extent) whilst allowing the public free access to use these works – without the costs and hassle of having to negotiate and agree on various terms and conditions of use and without the threat of a copyright infringement claim against them.
There are just over a handful of different types of Creative Commons licenses. For example:
An "Attribution ShareAlike CC BY-SA" license allows the public to use, make changes to and distribute a work for both commercial and non-commercial purposes, provided the original creator or copyright owner is credited and any new or derivative work created is licensed under identical terms. Wikipedia uses this type of Creative Commons licence.
An "Attribution-NonCommercial CC BY-NC image" license allows the public to use, make changes to and distribute a work for non-commercial purposes, provided the original creator or copyright owner is credited, although any new or derivative work created doesn't need to be licensed under identical terms. The Brooklyn Museum uses this type of Creative Commons license on Flickr.
An "Attribution-NonCommercial-NoDerivs CC BY-NC-ND image" license only allows the public to download and share a work, provided the original creator or copyright owner is credited, the work is not altered in any way and the work is used for non-commercial purposes. TED Talks videos use this type of Creative Commons licence.
If you do not comply with the terms of the license, you will no longer have the right to use the work and will be open to a possible copyright infringement claim by the owner of the copyrighted work.
As a first step, if you are looking for particular content to use online, conduct a CC Search, check the Creative Commons directory or use the search filters on content-sharing platforms. Always double-check search results to ensure that you haven't been misled regarding the type of licence (if any) applied to the work.
What you should know about copyright as an author or creator
I own the copyright in my tattoo, right?
copyrightcopyright infringementCreative Commons licenseFlickrTED TalksWikipediaYouTube
Tammi Pretorius
Tammi Pretorius is an attorney at KISCH IP's trade mark department. Her experience includes: Trade mark searching and registration; Trade mark portfolio management; Trade mark litigation; Copyright advice and litigation; and domain name disputes
Send a legal query to Tammi Pretorius
Intellectual Property Law articles by KISCH IP
What is the purpose of an IP holding company?
Recent changes to Canada's patent regime
Kylie Jenner and the interplay between mistakes, memes, marketing and trade mark protection
Performing artists beware - Protection is needed
Intellectual Property Law articles on GoLegal
Karoo lamb, rooibos tea, Bapsfontein…What's in a name?
Tammi Pretorius is an attorney at KISCH IP's trade mark department. Her experience includes: Trade mark searching and registration; Trade mark portfolio management; Trade mark litigation; Copyright...
Experts in Intellectual Property Law
118 Relevant articles


Our world and universe continue to fascinate, intrigue and surprise. We can learn many lessons from plants and animals on how to solve common problems and gain deeper understanding of our place in the universe by studying the geology and natural processes of the planets. Neutron scattering is being used to offer new potential for crop breeding and medicines, and to bring new insights to planetary science.
Spider-silk is five times as strong as steel and absorbs three times more energy than the material used in bullet-proof vests. Spiders spin silk from a mix of water and proteins stored as a gel in specialized silk glands inside their bodies. As the gel is pulled through their spinning glands it becomes a very resilient solid that could have many potential uses in the industrial world. The strength and elasticity of silk could be harnessed for new plastics and biomedical implants if it could be made artificially.
Neutron scattering is playing a key role in discovering how silk can be made artificially. Research teams are using neutron beams tuned for studying biological materials to shine a light on the atomic scale structural changes as the gel transforms into solid fibre. Experiments have unlocked some answers, and continue to reveal more of nature's secrets.
We are asking how nature makes such amazing materials. Neutron scarttering is an excellent technique for understanding the spider's magic tricks.
Cold-blooded lizards have only limited ability to regulate their own body temperature. When temperatures fall in winter, so does their body temperature, putting tissues and cells at risk of irreparable damage from internal ice. To prevent lethal ice crystals forming in and between cells in their body, lizards use chemical compounds such as glycerol to reduce the freezing temperature of water. During prolonged exposure to sub-zero temperatures, cell activity is paused until temperatures rise again and normal activity can safely resume.
Molecular structure data collected with neutron scattering shows how mixing glycerol with water prevents rigid ice networks from forming. This new fundamental understanding of the role of glycerol will be helpful in a range of applications. Food storage, fertility treatment and transporting medicines could benefit from a new understanding of how lizards survive at low temperatures.
Improving our fundamental knowledge of lizard cryopreservation may lead to improved storage and recovery of tissue for fertility treatment, better storage of drugs in the pharmaceutical industry and transport of organs for surgery, and better storage of food in the agricultural industry.
Food security is becoming a major concern in the UK and across the world, as harvest yields are challenged by climate change, pests, diseases and the demands of arising world population. A quarter of the world's crops are lost to pests and disease. Understanding how plants defend themselves could be one way to reduce losses. Common crops like rye, barley, oats, and wheat make antimicrobial proteins to defend themselves against disease, fungi and bacteria. In wheat, the defence proteins play an additional role in giving the endosperm texture, an economically important quality that determines the milling characteristics of the wheat.
Food scientists are using neutron scattering to learn about the molecular action of defence proteins and their interaction with the cell membranes of invaders. They can watch defence proteins punch their way through a cell membrane to kill hostile bacteria or strip vital components from its surface.
Anti-microbial plant defence proteins could be used in transgenic crop species to increase disease resistance and food yield. As regional climates change, this knowledge will also help farmers and breeders to adapt plants to counteract shifting weather patterns.
Satellite missions to the giant gas planets Jupiter and Saturn have revealed that our Solar System displays a rich variety of bodies, each with a complex and diverse evolutionary history. Therefore, understanding the evolution of the planets and moons presents one of the major challenges in Earth and planetary sciences.
Geologists have developed novel high-pressure neutron scattering experiments to model the Earth's interior or predict the geology of the icy moons of the Solar System. Unique equipment developed by university groups in the UK and France for neutron scattering instruments can squeeze rocks and other materials to very high pressures. These high pressures reproduce the conditions found inside Titan, Saturn's largest moon, or inside the mantle of the Earth at depths of up to 700km. The precise data derived from neutron scattering experiments allows planetary scientists to better interpret the geology seen in surface images taken from spacecraft, or create robust interpretations of seismic data recorded on Earth.


I could not stop watching this over and over! This animation is from the movie "Up" and the song is by Alan Jackson!
Can anything be any cuter and touching than this!
I made these visiting cards or thank you tags or whatever from the left over paper. Aren't they cute! I am sure not any cuter than the video!
Thanks for looking, Hope you all have a great week ahead!
thankyou tags shows ur passion& creativity.Ilove ur thoughts ur too good in ur work.
Oh my goodness ~ that video made me cry! How sweet!
But thankfully your cards brought a smile back to my face. Lovely!


text:
Yep, It's Your Weekly Tarot Card Reading
Ahem, Here's Jen Reacting to Brad Pitt's Wife Joke
A Dark Secret in Amish Country
The 10 Best At-Home Face Peels of All Time
Two Experts Explain Exactly How to Eat Your Way to Abs AND a Booty
It's harder than it sounds — but filling!
By Elizabeth Narins
Because most people need to consume more food to build a bubbly booty, and less to see sculpted abs, it's not exactly easy to sculpt an amazing ass and six-pack simultaneously, according to registered dietitian Christopher Tuttle, a certified personal trainer and International Federation of Bodybuilding and Fitness coach.
His best advice for those who seek results ASAP is to focus on one goal at a time: "First build muscle, then cut body fat at a later date while you maintain what you've worked for with exercise," he says, citing the same advice he gives his clients leading up to body-building competitions. Here's how to get own your goals on track:
Eating for An A+ Ass
"If you want to build a butt, you have to lift something," Tuttle says, referring to body-weight workouts and, once you're strong enough, squats and lunges with weights. "But you also need to eat adequate calories."
Amp up your butt workouts without ample food intake, and you're wasting your time: "You can't build a butt out of nothing," says registered dietitian Robin Barrie Kaiden, who's also a certified personal trainer. "You need to eat protein to build muscle."
Although protein isn't dose dependent — you don't get bigger muscles from eating more meat — Kaiden says if you work out about an hour a day, you need about 1.8 grams of protein per kilogram of body weight — about one gram more than is recommended for the average person, but less than the 2.3 to 3.1 grams of protein per kilogram the International Society of Sports Nutrition (ISSN) recommends for professional body builders. (Just divide your weight in pounds by 2.2 to get your weight in kilograms.)
The source of protein is less important than the quantity: "You just need the grams," she says, insisting protein bars and powders, which can be convenient, are no superior to whole foods such as fish and chicken.
In addition to protein, which should make up between 15 and 40 percent of your daily calories, you might need more carbs than you think, Kaiden says.
That's because your muscles only grow bigger when they have enough fuel to both soldier through weight-training workouts and repair the resulting muscle damage — and carbs are your body's favorite power source. "You won't have enough energy to exercise without eating enough carbs," she says, adding that 40 to 65 percent of your daily calories should come from whole grains like brown rice and oatmeal, which beat processed carbs like white bread and sugary snacks.
Healthy fats — think avocado and olive oil — should make up the last 20 to 40 percent of your calories, since they're essential to every cell in the body regardless of your gym goals.
If you can't handle percentage points, just keep it simple. "Eat real food — and more of it than you'd think," Kaiden says.
And remember that glute gains are made in the gym, not the kitchen. "Even if you eat a perfect diet, your body won't change," Tuttle says. For that, you need exercise — and patience. "It could take five months to put on five pounds of muscle," he says. It's ever the more reason to start lifting heavy now.
Eating for Abs
Once you've built your foundation — strong glutes and hamstrings — you're ready for phase two: #OperationAbs!
To really see your core muscles pop, Kaiden says, you may need to minimize belly fat. "You can change the shape of your body from exercise alone, but fat loss is mostly about diet," she says — mostly because it takes so much time and effort to burn enough calories to make a dent in your body's fat stores.
Although amping up your cardio can help create a calorie deficit, you'll also need to gradually cut back on starchy foods. (For his clients, Tuttle prescribes meals that consist roughly of 40 percent protein, 30 percent carbs, and 30 percent fat, but recommendations are individualized and range widely, even among professional body builders.)
That said, don't eliminate carbs altogether — you need it to fuel workouts! — and remember your best intensions to lose belly fat could all be for naught, since genetics determine where body fat is stored (and sacrificed): "You can never spot reduce," he says.
Lucky for carb lovers, another way to help your core muscles come out is to nip superficial bloating in the bud. Gastrointestinal issues such as constipation and insufficient water intake can also impact the appearance of your belly, Kaiden says.
Eating enough fiber from foods like fruits, vegetables, and whole grains can help you sidestep distention. Most women should shoot for about 25 grams per day, which you can get from about 3 cups of veggies (4 grams each), two pieces of fruit (4 grams each) and one cup of whole grains such as oatmeal or barley (about 5 grams).
If your diet falls short in this department, gradually increase fibrous foods as you amp up your fluid intake to three to four liters per day to minimize any initial bloating that proceeds improvements in digestion, she suggests.
It's also smart to avoid inflammatory foods like processed foods with added sugars, saturated fats, artificial sweeteners, and refined carbs, according to Kaiden. To that end, she recommends keeping sodium intake below 2,000 milligrams per day to prevent fluid retention. "The more real foods you eat, the better off you'll be," she says.
The good news: There's no need to outlaw any specific foods for the sake of awesome abs. "When you can't eat something, you obsess over it," says Tuttle, who tells his clients it's cool give into cravings for carbs or other treats on the weekends. (It's actually helpful to mix things up, lest the body get used to low-carb, low-calorie living, and slow your metabolism in response, he says — and research on intermittent fasting appears to support the theory.)
If dietary changes don't appeal to you, there's always the gym: "Regular, vigorous training can make your core muscles bigger, which expands the surface area of your midsection to distribute fat better and make you look more muscular," Tuttle says. He recommends moves like hanging leg raises, medicine ball crunches on a Swiss ball, and cable rope crunches. As soon as you can complete more than 50 consecutive reps of any one move, add weight to keep up the intensity, he says.
Timing Meals for Results
Strategically timing meals can keep your energy levels up while strength training FTW or creating a calorie deficit. It all begins with breakfast, which helps you stockpile energy for the rest of the day, Tuttle says.
To that end, he suggests eating four to six small meals rather than just three larger ones: "Eating similarly sized meals with equal amounts of protein at least four times a day can help you maintain muscle mass and promote recovery," he says. "The gaps between breakfast, lunch, and dinner are just too long," he says. And as for snacking? "When you fill your car with gas, you don't get one-fourth of a gallon, you fill up and go." The same applies to fueling your body with frequent, balanced meals.
When it comes to scheduling food intake around workouts, don't stress too much. "Eating within a certain time period after your workout won't make a difference unless you're training more than once a day," he says, since evidence of benefits for non-athletes is mixed. "If every meal has roughly the same amount of protein and calories, you don't have to worry about pre- and post-workout nutrition."
If all of this overwhelms you, take it one step at a time. "If you're doing nothing, then start doing something," Kaiden says — even if it's admitting the body you've got ain't all that bad.
Follow Elizabeth on Instagram and Twitter.
Elizabeth Narins Senior fitness and health editor Elizabeth Narins is a Brooklyn, NY-based writer and a former senior editor at Cosmopolitan.com, where she wrote about fitness, health, and more.
Behold: The 7 Best Pillows for Neck Pain
Amazon's Having a Big Fitbit Sale Right Now
I Bought the *Best* Amazon Workout Gear
This Fitness App Broke Me and I'd Do It Again
A Lazy Girl's Review of the Fitbit Versa 2
These Diets You're Considering RN Are Dumb
Lululemon's Post-Christmas Sale Is Here
This 30-Day Abs Challenge Will Light Your Core Up
18 Expert-Approved Ways to Prevent a Hangover
These Gym Bags Will Actually Hold All Your Stuff
Here's How Much You Can Boost Your Butt in 2 Weeks
4 Exercises to Tone and Tighten Your Abs and Butt
The Newest Way to Get a Great Booty Is to Electric Shock Your Butt
Exactly How to Get Kayla Itsines's Sick Abs in 7 Minutes
5 Ways to Tighten Your Abs and Butt ~*With Bae*~
A Part of Hearst Digital Media Cosmopolitan participates in various affiliate marketing programs, which means we may get paid commissions on editorially chosen products purchased through our links to retailer sites.


text:
Barcelona Wax Museum
Mon-Fri: 10 a.m.-1:30 p.m. and 4 p.m.-7:30 p.m.; Sat, Sun and holidays: 11 a.m.-2 p.m. and 4:30 p.m.-8:30 p.m. Summer: 10 a.m.-10 p.m.
Passatge de la Banca, 7, Barcelona , Catalonia, 08002
Given its prime position on Las Ramblas, you can easily visit the Barcelona Wax Museum during a day of sightseeing in downtown Barcelona. Save time during your visit with skip-the-line entry or make the most of your stay in the city with a Barcelona City Pass—this pass typically provides access to the Barcelona Wax Museum, alongside over 20 other attractions such as Barcelona Zoo and Casa Vicens. Alternatively, visit the Barcelona Wax Museum on a Gothic Quarter walking tour to learn more about the area.
For added benefits—such as an audio guide, photo souvenir, and more—opt for an all-inclusive ticket or save time with skip-the-line entry.
Guided tours are typically held every Monday, Wednesday, Friday, and Sunday at 12pm and 5pm.
Dramatized night visits are also held every Saturday in June, July, August, and September; check in advance for specific timings.
There are two offbeat on-site cafés, one modeled after a fairy-tale forest and the other littered with origami animals.
Barcelona Wax Museum is mostly accessible for wheelchair users thanks to elevators and ramped access points.
Barcelona Wax Museum is situated at the waterside end of Las Ramblas in downtown Barcelona and is easily accessible on foot from most points in the Gothic Quarter and El Raval neighborhoods. You can also access the Barcelona Wax Museum using the Barcelona Metro system—depart at the Drassanes stop (Line 3, Green) and leave via the Rambla exit.
Spanish Tapas Tours in Barcelona
Skip the Line at La Sagrada Familia
Skip the Line at Park Güell
Must-See Museums in Barcelona
Barcelona Wax Museum is a great year-round, rainy-day option for visitors in Barcelona. The museum typically closes for a few hours in the mid-afternoon but enjoys longer opening hours during the summer months, when you can expect bigger crowds.
Other Museums in Barcelona
If you're not keen on straightforward art galleries or run-of-the-mill history museums, Barcelona has a wealth of alternative institutions you can explore instead. Learn about the history of perfume at the beautifully curated Perfume Museum (Museu del Perfum); delve into eroticism at the Erotic Museum of Barcelona (Museu de l'Eròtica); or get magical at the King of Magic Museum (El Rei de la Magia).
Barcelona Cruise Port
Columbus Monument (Mirador de Colom)
Barcelona Maritime Museum (Museu Marítim de Barcelona)
Tablao Flamenco Cordobes
George Orwell Square (Plaça de George Orwell)
Barcelona Pavilion (Pabellon Mies van der Rohe)
Güell Palace (Palau Güell)
Placa del Rei
Port of Barcelona
Hash Marihuana & Hemp Museum
Ciutat Vella (Old Town)
Barcelona City Hall (Casa de la Ciutat)
Maremagnum