text:
\section{Introduction}
\label{intro}

Systematic single-cell studies of live cell imaging from microfluidic single-cell cultivation~(MSCC) works with high spatial and temporal resolution of cellular behavior.
So far, analysis of images like these has mostly been performed manually or is assisted by technological aiding systems, yet requiring human experts and therefore extensive human labor to create annotations of images; clearly, this procedure is not feasible in many cases, and it creates the need for different, more affordable and automated computer vision solutions~\cite{Theorell2019}.

The current state of the art for computer vision tasks and image processing that does not require human labor are convolutional deep neural networks~\cite{Ioannidou2017}. These are also  used extensively in the biomedical domain~\cite{Razzak2018}. Especially, approaches to track cells in images~\cite{Moen2019} have been an ongoing field of study in recent years. However, optimization for this task has proven to be a very cumbersome challenge which remains prone to errors.

In recent years few shot learning has become a large research area. As an example interesting approaches constitute: ~\cite{transferlearning.xyz,NovelRec38:online,FewShotL77:online}. Popular surveys on this topic \cite{DeepAI-survey} categorize few shot learning into single modal and multi-modal learning. Here, single-modal learning can be further divided into Transfer Learning, Data Augmentation and Meta Learning. This work can be classified as a few shot learning approach that applies Data Augmentation and Transfer Learning for the specific use case of microscopy cell counting.

The proposed benchmark suite~\cite{Ulman2017} allows comparing different imaging technologies and extrapolation of the strengths and limitations of diverse methods for cell tracking, none of which are deemed as final solution for this task, even those with added interaction by bioimage analysis experts \cite{Berg2019} or distributed work of manual labeling~\cite{Hughes2018}.

In this contribution, we address a challenging task in biomedical image analysis by means of specific and adapted transfer learning technologies.
Related work in this field includes
Brent et al.~\cite{Brent2018} which used transfer learning to predict microscope images between different imaging technologies, however without sufficient incorporation of the vast diversity of cell imagery and characteristics. The approach by Falk et al.~\cite{Falk2019} provides one of the few toolboxes for cell tracking, albeit adherent, rather than suspension cells.  It allows transfer learning based on given models and novel data, whereby data set enrichment technologies limit the number of required samples.

In contrast to already reported single-cell cultivation studies \cite{Carlo2006} and \cite{Kolnik2012}, where adherent growing cell lines are the focus of investigation, we address  the scenario of more complex suspension cells, with their circular basic shape but ever-changing contour due to vesicle secretion and additional challenges like cell movement and floating within the experiment chamber, which renders analysis tools of adherent cells deficient. These cells growing in suspension comes with different and challenging obstacles to achieve automation of analysis, which will be described in \autoref{sec:data}.

Siamese networks have been used for a variety of tasks as they can help facilitate few shot learning or clustering of the data space by generalizing from unlabeled data. This is done in \cite{muller-etal-2022-shot} for text data and in \cite{10.3389/fcell.2021.767897} for genome sequencing. These presented architecture are however specific to their domains and not applicable to image processing. Furthermore, we extend the classical siamese network by including transfer learning and automatic data augmentation.

In this work, we want to make use of a network trained on one microscopic image type and adapt it to provide sufficiently accurate cell counting for a different microscopy technology, where no trained network exists due to the lack of annotated data. We particularly focus on mitigating human labor for annotations.
Our previously introduced deep twin auto-encoder architecture \method{Twin-VAE}{} \cite{TwinVAE} is trained on data stemming from one imaging modality and thereafter transferred to the similar yet different domain of the other microscopy technology.
This training procedure greatly reduces the need for natural, labeled data, by using synthetic, auxiliary training data, for which the ground truth is known and which is easy to obtain in this setting, since the \method{Twin-VAE}{} does not require the images to be rendered realistically in every regard, such as morphological details.

In the following, we will first describe the specific application domain from biotechnology, the underlying machine learning challenge, and the deep Siamese network architecture which will be used for transfer learning. Afterwards, we elaborate the details of the proposed transfer learning scheme, as well as perform an analysis of how the unique architecture used affects the transfer learning procedure. Its performance is evaluated for real data sets and using ablation studies, as well as comparison to state-of-the-art alternatives and baselines. A discussion concludes the contribution.

The application area in question is a prime example of a domain, where the state-of-the-art Image processing techniques do not work sufficiently well due to very little texture and other visual characteristics of the images, described in \autoref{sec:data}.
In addition, there exists no Deep Learning Models which easily and efficiently solve the task, as shown in \cite{TwinVAE} by comparing to EfficientNet \cite{Tan2019}, and Watershed methods \cite{Watershed2013} and shown here later by comparing to BigTransfer \cite{DBLP:journals/corr/abs-1912-11370} and our previous work.
The emergence of more data in such specialized domains like this makes it important to provide an easy-to-use system which has a high performance and enables automation of processes involving this kind of data.
\newline
Thus, the contribution and novelty of our work is as follows:
\begin{itemize}
    \item We improved performance and lowered computational complexity (outperforming the original work \cite{TwinVAE})
    \item We build an efficient transfer pipeline and showed on two microscopy datasets empirically that it outperformed a variety of methods, including state-of-the-Art Image processing.
    \item By performing extensive ablations studies we gained insight into
    which parts of the architecture contains representations which are beneficial for the transfer learning ability of the network. Thus, we are contributing to the debate how deep neural networks represent information. \cite{natureNNunderstanding}
\end{itemize}

\section{Material and Methods}

\label{sec:data}
\subsection{MSCC and live cell imaging data}

\begin{figure}[t!]
\centering
\includegraphics[width=\linewidth]{sample_micro}
\caption{\ Samples of data from the two microscopy technologies. Bright-field microscopy (left), phase-contrast microscopy (right).
The data has been preprocessed in the form of chip drift removal and orientational stabilization (translational and rotational) and a crop to square the images that allows further cropping by data augmentation techniques.
The cell counting module has to differentiate between the cells, smudges, chamber and background.}\label{fig:sample}
\end{figure}

The image data which is used in this study was obtained by single-cell cultivation of mammalian suspension cells as shown before~\cite{Schmitz2020}. CHO-K1 cells were cultivated in polydimethylsiloxane (PDMS)-glass-chips and constantly provided with nutrients by perfusion of the microfluidic device.
The goal of an automated analysis of such data is an automated extraction of important parameters of the observed dynamics, such as cell growth. Since many important parameters can be estimated based on the number of cells at a specific time point, the number of cells constitutes a key quantity and are taken as target labels.
The data used in this work consists of multiple parts, characterized 1) by the according microscopy technologies, bright-field microscopy and phase-contrast microscopy, 2) by the type of data, natural or synthetic, i.e., the original data or data which are generated and added to the original one within the learning pipeline, as described later, 3) by existence of a label and 4) the usage of that data for training or testing. Example images of both modalities are shown in \autoref{fig:sample}.

\autoref{Tab:01} shows an overview statistic of all data sets. The \texttt{Nat} set contains the aforementioned natural images, \texttt{Syn} the synthetic ones. The \texttt{BF} tag declares bright-field microscopy images, consisting of \num{12}~experiment scenarios with \num{956} overall images, of which a label (i.e.\ cell count) exists for \SI{7.5}{\percent} of the training images.
The \texttt{PC} tag is denoting phase-contrast microscopy images which consist of \num{37}~experiment scenarios, accumulating to \num{3976}~used images with a labeling rate of \SI{6.2}{\percent} for the training data.
The labels were created by hand in a nearly regular interval over the experiment scenarios for all natural data sets, however images were removed beforehand, if they had more than \num{30}~cells, since the expected outcome of the cultivation experiment is already determined at this point.

In the upcoming analysis, we focus on the transfer from the larger data set \dataset{Nat-PC}{} to the smaller set \dataset{Nat-BF}{}, since this is the common way to apply transfer learning. The phase-contrast imagery also contains more variation of the biological processes, which makes phase-contrast microscopy arguably more popular than bright-field microscopy. Our experiments also contains transfers from bright-field microscopy to phase-contrast microscopy to show the robustness of the technique.
\autoref{fig:distribution} shows the distribution of images against the cell counts in them for \dataset{Nat-PC}{}. A clear trend towards images with low cell counts can be seen.

This can be taken into account for optimization of the transfer learning methodology, since it can be assumed that data of this type has a similar distribution, particularly in the light of exponential growth rates and the presence of failing cultivations.
Most of the labeled images from (\dataset{L-Te}) are used for testing the cell count prediction and the reconstruction, rather than used during training (\dataset{L-Tr}). This is done, because we focus on a method that reliably works on small amounts of labeled data.
Further unlabeled test data (\dataset{U-Te}) is used to evaluate the reconstruction only, since (\dataset{L-Te}) remains too small to include a broad overview of the different chamber situations (clumping, overlapping, escaping etc.) to be confident about the stability of the performance for a convolutional network.

\begin{table}[!b]
\caption{Overview of data sets used. The \dataset{Nat} tag indicates natural data, \dataset{Syn} represents synthetic data. \dataset{BF} classifies the bright-field images, \dataset{PC} the phase-contrast ones. \dataset{L} and \dataset{U} marks labeled and unlabeled data and lastly \dataset{Tr} and \dataset{Te} separate the data into training and test data.}
\label{Tab:01}
{
\centering
\setlength{\tabcolsep}{0.5em}
\begin{tabular}{@{}cccccS[table-format=4.0]@{}}\toprule
Abbreviation & Type      & Technique      & Label & Usage      & {Size} \\\midrule
\dataset{Nat-BF-L-Tr}  & natural   & bright-field   & yes     & Training &    281 \\
\dataset{Nat-BF-U-Tr}  & natural   & bright-field   & no      & Training &    2188 \\
\dataset{Nat-BF-L-Te}  & natural   & bright-field   & yes     & Testing  &    290 \\
\dataset{Nat-BF-U-Te}  & natural   & bright-field   & no      & Testing  &    224 \\
\dataset{Nat-PC-L-Tr}  & natural   & phase-contrast & yes     & Training &    209 \\
\dataset{Nat-PC-U-Tr}  & natural   & phase-contrast & no      & Training &    2943 \\
\dataset{Nat-PC-L-Te}  & natural   & phase-contrast & yes     & Testing  &    398 \\
\dataset{Nat-PC-U-Te}  & natural   & phase-contrast & no      & Testing  &    394 \\
\dataset{Syn-BF-L-Tr}  & synthetic & bright-field   & yes     & Training &    2469 \\
\dataset{Syn-BF-L-Te}  & synthetic & bright-field   & yes     & Testing  &    514 \\
\dataset{Syn-PC-L-Tr}  & synthetic & phase-contrast & yes     & Training &    3152 \\
\dataset{Syn-PC-L-Te}  & synthetic & phase-contrast & yes     & Testing  &    792 \\\bottomrule
\end{tabular}
}
\end{table}

\subsection{Synthetic data}
As our task is reliable cell counting for suspension cell microscopic images and given data is often limited and with only few manual annotations, retraining a deep neural network for every new set of data is inadequate and delivers deficient accuracies for the task. To overcome this limitation, we transfer a trained model which achieves high accuracies on its original task to the newly presented task.

Since the learning methodology is semi-supervised, our formerly introduced Twin-VAE \cite{TwinVAE} will be used as a basis to propose a novel transfer learning method to mitigate the aforementioned complications. Here, synthetic data is used as an auxiliary training set \dataset{Syn}{} which is also used for transfer learning. We evolve on the Siamese architecture, which inherently solves the task of abstraction from the synthetic nature of data set enrichtments.

The synthetically generated data is visually simplified (constant background, ellipsoidal cells) to allow the loss construction to focus on the regression task rather than the intricate reconstruction of arbitrary visual cell membranes and organelles. This is done by drawing cells as ellipsoids, varying some attributes like their brightness, size and blurriness of edges. For further detail, see the original Paper \cite{TwinVAE}. Reconstructions of real appearances from synthetic data, while interesting to suggesting inherent stability, are not of importance for a high accuracy on the task. Ground truth labels are known for synthetic data, because it is based on pre-defined geometric style modeling, neglecting texture and complex morphology. Thereby, geometric heterogeneity of this data is simplified compared to real data, examples of which can be seen in \autoref{fig:sample_syn}.

Synthetic data allows for creation of a large variety of independent image samples that are correlated but not identical to the appearance of natural data. Unlike popular data set enrichment technologies, the amount of data can freely be determined since it is independent of the amount of real data, and representatives of any type of underlying label can easily be generated.
We show that our \method{Twin-VAE}{} architecture is successfully trained and improved in accuracy like this in \autoref{sec:model}.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{distribution}
\caption{Visualization of distribution of images by cell count for the merged data sets \dataset{Nat-PC-L-Te}{} and \dataset{Nat-PC-U-Te}{}. We discard data with higher cells counts than \num{30}, because they are irrelevant for the cultivation experiments.}\label{fig:distribution}
\end{figure}

\autoref{Tab:01} lists the synthetic data as \dataset{Syn}{}, concatenated by the microscopy technology category \dataset{BF}{} or \dataset{PC}{} accordingly. The \dataset{U}{} or \dataset{L}{} declaration tells if the data is labeled and the table separates them between training (\dataset{Tr}{}) and test (\dataset{Te}{}) images.
The cell distribution in these images was chosen to be close to that of the \dataset{Nat}{} data sets for the sake of resemblance and because of the criticality of correct cell counts on low cell count imagery.
Synthetic images are generated with seed consistency, i.e. in such a manner that ensures reproducibility and can be generated in arbitrary amounts, however larger amounts of synthetic data will increase training time nearly linearly, while improving performance with diminishing returns as will be shown in \autoref{sec:evaluation}. For the sake of computational time, the default \dataset{Syn}{} set sizes roughly match the corresponding number of natural images. However, sets with different amounts have also been created for accuracy comparisons (\autoref{fig:syndata}).
The background is created by the mean value of the entire natural training data set showing less than \num{5} cells to assure high visibility of empty background and to remove the smudges that appear in only a few experiment scenarios.
The working resolution the synthetic data is generated in is \num{128}~by \num{128}~pixels, matching the resolution of the architecture's input size described in \autoref{sec:model} and their file size is about \SI{8}{MB} per \num{1000}~images.

The virtual generator is highly adjustable, creating images with a given distribution of cell counts,  overlapping of cells, variations of  the brightness of the cell's inner organelles, their membrane silhouette and the background.
More complex visual fidelity of natural data such as ongoing cell divisions can also be mirrored by a combination of these mechanics, e.g., by creating a small overlap together with more noisy cell borders. Smudges, as in \autoref{fig:sample}, have not been inserted, since they are an interference factor and likely only hinder the training process.
The cells' shape has been simplified to deformed ellipsis to roughly match the shape of the natural cells. Noise, individual luminance per cell, and multiplication with Gaussian filters of random strengths have been added to increase the variety of cells in the data.
We ensure easy adjustability of the generation mechanism to natural cells in other data sets, that have different shape characteristics.

\subsection{Network Architecture \& Training}
\label{sec:model}
\subsubsection{Siamese Architecture}
We use a novel deep Siamese twin architecture that separates the input data for training depending on its origin, thus circumventing the problem of differences in appearance of synthetic and natural images.
This approach requires that the architecture creates a tightly coupled shared inner representation of the different data sources to achieve low training losses and good generalization ability for semi-supervised setups.

For this, two identical variational autoencoders (VAE) are created for the two data sets. They share the weights of their last encoding layer, the first decoding layer and the small hidden layer in-between (see \autoref{fig:architecture}).
VAEs constitute a state-of-the-art solution for generalized few-shot learning~\cite{Schonfeld2019} and weight-sharing has been used to reduce neural network sizes and to improve test performance beforehand~\cite{Karen2017}.

\begin{figure}[t!]
\centering
\includegraphics[width=\linewidth]{synth}
\caption{Random samples of synthetic data from the sets \dataset{Syn-BF-L-Tr}\ and \dataset{Syn-PC-L-Tr}. Sample of bright-field microscopy on the left and of phase-contrast microscopy on the right. The images are dimensionless and therefore do not show scale bars. The theoretical cell sizes are identical to natural data cells, but is of no importance for the work.}\label{fig:sample_syn}
\end{figure}

Specifically, in our setup, one of the VAEs works on synthetic data only (\textit{VAE-syn}), while the other one uses natural data only (\textit{VAE-nat}).
The non-shared outer layers account for the different visual characteristics of synthetic and natural data, while the shared inner layers are enforced to create a common representation of relevant image characteristics. By adding a two-layer deep fully connected neural network regression model for the cell counting task, the architecture works in a supervised manner for data for which the label is known, based on the shared representation of the VAEs.
Cell detection by regression has been shown to work well for other (less demanding) tasks~\cite{Yuanpu2015,Weidi2018}.
Our architecture therefore addresses two objectives simultaneously:\\
(i) Mostly unsupervised encoding and decoding of natural and synthetic input images using a shared representation.\\
(ii) Supervised counting of the cells for both natural and synthetic images.

\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth]{architecture}
\caption{Visualization of the \method{Twin-VAE}{} architecture.
The blue elements handle synthetic data, while the yellow elements handle natural data.
The green elements are shared between the two VAEs and contain the inner representation of the cell imagery, while the purple elements result in an estimation of the cell count.}\label{fig:architecture}
\end{figure*}

\subsubsection{Loss}
Given an input image \(x\) of pixels, a label (i.e., cell count) \(l\) between \num{1} and \num{30} and a type \(t \in \{\mathfrak{n}, \mathfrak{s}\}\), representing the fact whether the image is natural or synthetic, we obtain a reconstruction loss \(\recocost(x)\) of the VAE, a regression loss \(\regrcost(x, l)\) of the task at hand such as cell counting, and a distributional regularization loss \(\regucost\), which aims for a homogeneous representation of synthetic and real data in the embedding space of the VAE.
We combine these losses to form our twin loss \(\twincost(x, l, t)\) with weighting factors \(C_{\recocost}^{t}\), \(C_{\regrcost}^{t, l}\), and \(C_{\regucost}^{t}\), respectively, which allows us to balance image reconstruction fidelity (\(C_{\recocost}^{t}\)), regression performance (\(C_{\regrcost}^{t, l}\)) and distributional stability (\(C_{\regucost}^{t}\)) and therefore to maximize the impact of regression errors on the loss. Furthermore, it allows us to gracefully handle input images without known cell counts by setting \(C_{\regrcost}^{t, l}\) to zero:
\begin{equation}\label{eq:twinloss}
\begin{split}
    \twincost(x, l, t) = \\
    C_{\recocost}^{t} \cdot \recocost(x)
    + C_{\regrcost}^{t, l} \cdot \regrcost(x, l)
    + C_{\regucost}^{t} \cdot \regucost(x)
\end{split}
\end{equation}

During our experiments, the mean-squared error (MSE) \(||x-d(x)||^2\), where \(d(x)\) is the reconstruction of the input image \(x\) and \(||l-r(x)||^2\), where \(r(x)\) is the estimated cell count, yielded the best results respectively, when used as \(\recocost(x)\) and \(\regrcost(x, l)\) for training on phase-contrast data, and as \(\regrcost(x, l)\) for bright-field data. However, for bright-field data the binary cross entropy (BCE) \(-l\cdot log(r(x))+(1-l)\cdot log(1-r(x))\)
turned out to be the superior choice for \(\recocost(x)\) and was often resulting in just slightly worse results than the MSE for phase-contrast data.
The \(\regucost\) is applied as the Kullback-Leibler divergence (KLD) of the standard VAE model (\cite{Kingma2013}) and is obligatory to enforce generation of latent vectors with sufficient similarity to a normal distribution.
The weighting factors \(C_{\recocost}^{t}\), \(C_{\regrcost}^{t, l}\), and \(C_{\regucost}^{t}\) are carefully chosen for training, punishing incorrect cell count predictions especially on natural data, while relaxing the importance of visual reconstruction.
Details on this are provided in the following section.

\subsubsection{Neural Network Structure}
\label{subsec:nn}
The outer, non-shared part of the encoder is composed of four two-dimensional convolutional layers with kernel size \num{5} and a stride of \num{2}, initialized with an orthogonal basis~\cite{Saxe2014}.
Inbetween the layers, leaky rectified linear units (ReLU) with a leakiness of \num{0.2} are activated, together with a dropout of \num{0.1}.
Channel amounts used for the convolutions are in order: \num{32}, \num{64}, \num{128} and \num{256} for the encoders.
The inner, shared part of the encoder consists of an additional two-dimensional convolutional layer with identical properties and \num{512} channels.
The layer is followed by the bottleneck, consisting of three fully connected layers of sizes \num{512}, \num{256} and \num{512} again, each with a dropout of \num{0.1}.
The inner, shared part of the decoder has \num{256}~channels, contains a two-dimensional transposed convolutional operator layer with identical kernel size and stride as in the encoder, and is followed by a batch normalization over a four-dimensional input and a leaky ReLU with the same leakiness.
The outer, non-shared part of the decoder consists of five layers of kernel sizes \num{5}, \num{5}, \num{5}, \num{2}, \num{6}, following the convention of a small penultimate followed by a bigger last layer, keeping the stride of \num{2} except for the fourth layer using a stride of \num{1}, the same leaky ReLUs and a sigmoidal activation function at the end.
Additionally, a branch of fully connected neurons for the regressor consisting of two layers of sizes \num{256} and \num{128} is being fed by the output of the shared part of the decoder, uses linear layers and a constant dropout of \num{0.1}.

The architecture is using the Adam optimizer for phase-contrast microscopy data, and the rectified Adam (RAdam)~\cite{Radam2020} optimizer for bright-field data.
The combination of the decoder loss factor \(C_{\recocost} = 100\), the regressor loss factor \(C_{\regrcost}=3\) and the KLD factor \(C_{\regucost}=2\) yields the best results for phase-contrast data.
For the BCE, the decoder loss factor is not constant, but decays over time with a rate of \num{3e-5} per epoch, since the BCE does not decrease significantly within the training process, but needs to decrease over time to amplify the importance of low regression losses \(\regrcost(x, l)\).

While it seems counter-intuitive that \(C_{\recocost}\) is bigger than \(C_{\regrcost}\) and \(C_{\regucost}\), it is caused by the MSE for pixel data getting very small on normalized images. KLD is supposed to  stay relatively small. While it is required to enhance the quality of the distributions, it should not impact the training of cell predictions and image reconstructions too much by unfortunate sampling from the latent vector, however it has to be impactful enough to enforce natural and synthetic data into similar representations in the inner layers.

Since training is done over thousands of epochs, a soft weight decay of \num{1e-5} per epoch is added, combined with a fixed learning rate of \num{1.3e-4}. A delayed start for the regressor is used to allow for pure image reconstructions to contain meaningful images, ensuring the representation of information of existent cells in the representation before the regressor has to extract that information. A delay of \num{100}~epochs has been used to achieve the results presented in \autoref{sec:evaluation}.

A batch size of \num{128} for the phase-contrast images and \num{64} for the bright-field images works best, and the training runs for up to \num{50.000} epochs, unless early stopping conditions abort it.

\subsubsection{Data Augmentation}
\label{subsec:da}
To maximize the use of the limited amounts of natural data, multiple data augmentation techniques are combined and applied to the data.
Randomly occurring horizontal and vertical flips, possibly combined with a random crop of the image of scale \num{0.9} combined with a resize to its original, meaning the images get randomly cropped to \num{115} pixels in width and height, then scaled back to \num{128} pixels.

The crop adds difficulty to the cell detection process by partially cropping cells out, however it proved helpful as long as the crop is not too strict and cuts away cells completely.
Then, a \num{90} degree rotation is applied at random and a zero-centered noise map is generated and added to the image with a small amplitude factor.
Additionally, small rotations of \num{0} to \num{5} degrees are added before the crop, to spread cell occurrence even more. The crop will then mostly remove the undefined parts of the image, that are created when rotating non-circular images.

\subsection{Image Reconstruction}
Although our goal is automatic counting of cells, our loss from \autoref{eq:twinloss} includes a term for image reconstruction. The reasoning behind this decision is that analysis of the reconstruction abilities of \method{Twin-VAE}{} is only possible with this loss. Furthermore, the loss enables us to check if the learned shared representation is meaningful by checking the correlation between the visual existence of cells in the image reconstructions and the actual cell count.

During training of \method{Twin-VAE}{}, the natural input images are first processed by a specialized encoder, followed by a shared encoder and decoder of the two twins, and finally reconstructed by a specialized decoder (see \autoref{fig:architecture}). Synthetic data is handled equivalently.
The learned inner representation must be shared and similar between the two types of data for 1) the regression to work as intended and 2) the cell counting in natural images to benefit from synthetic data as much as possible.
Verification of this is done by encoding a natural image with the appropriate encoder but performing the reconstruction with the decoder that is intended and trained for synthetic images and vice versa. In the following, we demonstrate exactly this.

In \autoref{fig:optimal_pc} and \autoref{fig:optimal_bf} we show examples of perfect translations, where a natural image is encoded and subsequently decoded as a synthetic image.
The cell count is unchanged, the cell prediction matches the actual existence of cells and the position and size of cells are also retained, while the overall appearance is simplified, however \method{Twin-VAE}{} has learned to remove noise and condense the information down to what is required and helpful to count cells.

\begin{figure}
\centering
\includegraphics[width=\linewidth]{optimal_pc}
\caption{Example of a perfect synthetic-looking reconstruction (right) of a natural image (left) from \dataset{Nat-PC-L-Te}.
The cell counts match exactly and the position as well as size of cells are preserved.
While the smudge on the left is recreated visually, it does not lead the regressional part of the \method{Twin-VAE}{} to a wrong cell count.}
\label{fig:optimal_pc}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\linewidth]{optimal_bf}
\caption{Example of a perfect synthetic-looking reconstruction (right) of a natural image (left) from \dataset{Nat-BF-L-Te}.
The cell counts match exactly and the position as well as size of cells are preserved. For this data set, where smudges are more faint, they don't get reconstructed usually.}
\label{fig:optimal_bf}
\end{figure}

Even when \method{Twin-VAE}{} does not translate an image perfectly,
the reconstruction can be useful to understand where an error occurs.
In \autoref{fig:suboptimal} we show an example where two cells that are very close together are interpreted and reconstructed as a single cell.
As well as translating images from natural to synthetic-looking,
\method{Twin-VAE}{} can perform the inverse translation from synthetic to natural-looking as well.
We provide an example in \autoref{fig:syntonat}.

\begin{figure}
\centering
\includegraphics[width=\linewidth]{nat_suboptimal}
\caption{Example of a faulty synthetic reconstruction (right) of a natural image (left) from \dataset{Nat-PC-L-Te}.
The human expert determined the cell count to be \num{10}, the prediction differs by one. The reconstruction shows a merge of the top two cells in the bottom-left triple of cells. The two cells clump together in such a way, that there is almost no visual indication of a border between them, especially missing the usual bright boundary around cells that can be seen around the rest of the cells.}
\label{fig:suboptimal}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\linewidth]{syntonat}
\caption{Example of an accurate, natural-looking reconstruction (right) of a synthetic image (left) from \dataset{Syn-BF-L-Tr}. Cell counts match exactly, position and size of the cells are preserved. While these conversions are not mandatory for the transfer process, they ensure representational consistency on a visually comprehensible level.}
\label{fig:syntonat}
\end{figure}

\subsection{Baselines}
\label{sec:baselines}
We implement two different methods to serve as baselines for our evaluation.

The first is \method{Twin-VAE}{} which already outperforms classical baselines like \emph{EfficientNet}~\cite{Tan2019} and \method{Watershed}{}~\cite{Watershed2013}. We compare our new \method{Transfer Twin-VAE}{} to \method{Twin-VAE}{} and BigTransfer, commonly shortened to \method{BiT}{} \cite{DBLP:journals/corr/abs-1912-11370}. \method{Twin-VAE}{} is a previous work of ours upon which \method{Transfer Twin-VAE}{} and the variation \method{double Transfer Twin-VAE}{} build. It consists of the same architecture but is trained upon a single dataset consisting of natural and synthetic images. Furthermore, less extensive hyperparameter tuning was performed on \method{Twin-VAE}{} due to longer training times.\\
The second method we compare to is a transfer learning pipeline from Kolsenikov et al. called \method{BiT}{} that produces state-of-the-art classification results on Cifar-100 and similar datasets in the few shot case (1-10 examples per class). \method{BiT}{} consists of the classical ResNet \cite{DBLP:journals/corr/HeZRS15} architecture but with very long pre-training times on large image corporas and a custom hyperrule that determines the training time and learning rate during transfer dependent on the new dataset size.
Since the data augmentation applied during \method{BiT}{} is not immanently applicable in the cell counting case, we used the same data augmentation as in our own method.
We tested changes to the hyperrule presented in \method{BiT}{} but did not find any significant improvements, therefore used the values provided by the authors.

\section{Transfer learning methodology}
In this chapter, we will describe various experiments to determine which transfer methodology is suited best to the \method{Twin-VAE}{} architecture and the final transfer learning methodology used.
\subsection{Transfer Method}

\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth]{Overview.png}
\caption{Comparison of a variety of hyperparameter choices. These choices include: KLD loss factor (top left), rlf loss factor (bottom middle), dlf loss factor (bottom left), crop size (bottom right), noise factor (top right), length of pre-train (top middle)}\label{fig:hyper}
\end{figure*}

When we write about freezing a section of the network in this work, mathematically, we multiply the gradient update $\Delta$ of the frozen part of the network $f$ with weights $w_f$ by zero.
Hence, frozen weight update refers to the rule
$ w_f = w_f + \Delta w_f * 0 $
instead of the normal weight update
$ w_f = w_f + \Delta w_f $
The gradient is still passed through to non-frozen parts of the network, enabling them to learn.
Since no standard procedure exists in the literature for applying transfer learning to a Twin-VAE which is trained with synthetic data augmentation, four different possible methods of transfer learning are proposed and compared.
These methods are:

\emph{Frozen Outer layers.} A popular observation in convolutional networks is that the early layers consist of universal edge processing masks and the later layers are more specialized for the task at hand \cite{Yamashita2018}. In a Twin-VAE architecture, these later layers of a standard convolutional network correspond to the shared inner layers and the early convolutional layers correspond to the outer layers of the Twin-VAE. Based on this analogy, we try to train only the inner layers of the network.
Everything which is not part of the shared elements of the network pictured in \autoref{fig:architecture} is not trained.

\emph{Frozen Core.} A common view on VAEs is that the produced embedding space should be highly sensitive in regard to the variance in the training set. Since the imaging method is not changed during normal training the VAE embedding should not encompass this variable, rather it should be highly sensitive to cell count and cell position in the images, which were the main things varied in the original training set. Since the task of cell counting remains the same and the only difference between tasks is the imaging method used, we tried to keep this shared inner representation frozen during training.
Everything that is part of the shared elements of the network pictured in \autoref{fig:architecture} is not trained.

\emph{Simultaneous transfer.} In the third series of tests, we were not freezing any layers at all. This has the potential problem of the initial transfer period with high losses destroying useful information in intermediate layers.

\emph{Thawing layers.} Last, we experimented to start with frozen inner or outer layers and gradually unfreeze them during training. This is done explicitly to prevent the potential problem described in Simultaneous transfer, but to still be able to fine-tune these layers appropriately to the new task.

\subsection{Hyperparameter tuning}

The original Twin-VAE needed \num{50000} epochs to converge to satisfactory results, which equates to a near \num{100} hours on an NVIDIA Tesla P-100 16G. This made hyperparameter tuning using standard methods computationally costly.

By using transfer learning to converge significantly faster to similar or even better results, we were able to conduct more extensive hyperparameter searches.
Since a full grid search over all possible hyperparameters is still not computationally feasible, instead, an iterative search was performed by tuning a single hyperparameter finding the best value and proceeding to the next hyperparameter, recapturing obscured parameter choices in later repetitions.
The hyperparameters and training options tuned were:

\begin{itemize}
\item Training time,
\item transfer method,
\item image noise ratio,
\item image crop size,
\item relative ratio of KLD loss, regression loss and reconstruction loss
\item learning rate
\item learning rate schedule
\end{itemize}
Plots for most of these are provided in \autoref{fig:hyper}.

\subsection{Results}
We see when tuning the hyperparameters that most of the parameters show only small improvements to the final accuracy (1-2 percent), cumulatively the performance of the network can be significantly improved (5 percent). Our experiments show that for most hyperparameters, large performance degradation can be observed if they are poorly chosen.

The choice of transmission method revealed which parts of the network have learned transferable information and which need to be retrained. We conclude that, unlike in a typical convolutional network, the outer layers need the most retraining.
If these layers remained frozen, the network could not successfully transfer its knowledge to the new imaging method. Conversely, when the inner layers remained frozen, the network achieved only 1-2 percent less performance than when everything remained unfrozen.

This could be due to the rather unique circumstance of the final task being the same, just on pictures taken with a different imaging method.

Not freezing any layers achieved the best performance overall, we attribute this to the postulated effect of the initial transfer window scrambling information  not being observed.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{Transfer_method.png}
\caption{Comparison of test accuracy between different transfer learning options. The transfer was performed from the datasets \dataset{Syn-PC-L-Te}{}, \dataset{Nat-PC-L-Te}{} transferring to the datasets \dataset{Syn-BF-L-Te}{}, and \dataset{Nat-BF-L-Te}{}. In this application, the simplest option (not freezing any layers at all) performs the best, while freezing only the inner layers performs only slightly worse. Freezing the outer layers greatly impact the ability of the network to adapt to the new imaging method. Gradual thawing of the frozen layers does not have a large impact on performance.}\label{fig:testacc}
\end{figure}

Another interesting effect observed was that when trained for very long training times (\num{150000} epochs) the network did not show any signs of double descent \cite{DBLP:journals/corr/abs-1912-02292} and achieved convergence after only \num{10000} epochs. Compared to the non pre-trained network where convergence was achieved at the earliest after \num{50000} epochs, this represents a speed up of at least 5 times.

\section{Twin-VAE during transfer learning}
In this section we systematically investigate the effect the unique architecture of the Twin-VAE has on the transfer process. We choose to investigate whether the decoder part of the network is needed during transfer learning, and whether the synthetic data used for training is needed during transfer learning. Through this we ask if the Twin-VAE architecture is only necessary for the pre-training part and if it can be simplified to a normal convolutional network for transfer learning.
\subsection{Decoder}
To investigate whether the Decoder is important during transfer learning, we perform multiple transfer training runs where we successively lower the reconstruction loss factor \(C_{\recocost}^{t}\). \autoref{fig:decoder} shows that the loss factor does not seem to have any impact on performance. The variance between runs is small enough to be within normal statistical deviations observed between runs with the same parameters and can not be clearly attributed to the different loss factors. Based on this, we conclude that the Decoder part of the network does not have a positive impact during the transfer procedure and can be set inactive to speed up transfer computing time even further.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{Decoder_loss.png}
\caption{Comparison of test accuracy during transfer between different reconstruction loss factors \(C_{\regucost}^{t}\). The transfer was performed from the datasets \dataset{Syn-PC-L-Te}{}, \dataset{Nat-PC-L-Te}{} transferring to the datasets \dataset{Syn-BF-L-Te}{}, and \dataset{Nat-BF-L-Te}{}. All values tried have little to no effect on the accuracy.}\label{fig:decoder}
\end{figure}

\subsection{Synthetic Data}
To assess the relevance of the synthetic data during transfer learning, we vary the ratio of synthetic data to natural data. In the original paper\cite{TwinVAE} the ratio of synthetic data to natural data was maintained at 1:1 to prevent the architecture from projecting the different data types to distinct embeddings in the VAE bottleneck. A point of note is that a large part of the natural data is unlabeled, while the synthetic data is fully labeled, subsequently the synthetic data had a large contribution to the training of the regressor.

During our experiments we vary the ratio of synthetic to natural data in the range of 0.25-10:1.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{ratio_line_plot.png}
\caption{Comparison of test accuracy during transfer between different synthetic data to natural data ratios. The transfer was performed from the datasets \dataset{Syn-PC-L-Te}{}, \dataset{Nat-PC-L-Te}{} transferring to the datasets \dataset{Syn-BF-L-Te}{}, and \dataset{Nat-BF-L-Te}{}. Synthetic data ratio below 0.5:1 negatively impact the network's performance, ratios above 5:1 also have a negative impact upon performance. The best performance was achieved with a ratio of 2:1.}\label{fig:syndata}
\end{figure}

\autoref{fig:syndata} suggests that the performance of the network is negatively affected when the synthetic data ratio is especially small (<0.5) or large (>5). The optimal ratio found was 2:1. Since higher amounts of training data generally lead to better performance, there seems to be a problem generalizing from the synthetic data to the natural data.

We suggest that the KLD loss \(\regucost\) is still able to keep the distribution of the natural and the synthetic data the same in cases where the ratio is close enough to 1:1 but for more extreme ratios the KLD loss \(\regucost\) alone is insufficient. To validate this hypothesis we show 3 different UMAPs \cite{mcinnes2018uniform} in \autoref{fig:UMAPratio} that depict the distribution of natural and synthetic data in the embedding layer of the VAE.

\autoref{fig:UMAPratio} shows that the VAE distribution seems to regard the number of cells in an image as a more important aspect, the higher the synthetic data ratio. However, it does not show a separation of synthetic (blue dots) and natural data (red dots), so it is not clear why the performance of the model decreases for higher synthetic data ratios. We leave this for future work.

\begin{figure}
\centering
\includegraphics[width=\linewidth]{CombinedUMAPcropped.png}
\caption{Different UMAPs of the embedding dimension of the Twin-VAE after transfer. On the dataset \dataset{Syn-PC-L-Te}{}, \dataset{Nat-PC-L-Te}{} transferring to the datasets \dataset{Syn-BF-L-Te}{}, and \dataset{Nat-BF-L-Te}{}. Blue dots represent synthetic data, red dots natural data and green dots natural unlabeled data. The color gradient of the dots represent the number of cells in the image, where lighter indicates a higher cell count. In all cases some outliers of red and green dots are found. In the lowest ratio these outliers seem to be most severe. The direction of low to high cell count is clearly recognizable in all plots. The high ratio visualization has the strongest correlation of cell count to embedding distance.}\label{fig:UMAPratio}
\end{figure}

\section{Results and Discussion}
\label{sec:evaluation}
We present the final results of all methods on the four data sets \dataset{Syn-PC-L-Te}{}, \dataset{Nat-PC-L-Te}{}, \dataset{Syn-BF-L-Te}{}, and\\ \dataset{Nat-BF-L-Te}{} in \autoref{table:results-cell-counts}.
Our \method{Transfer Twin-VAE}{} consistently outperforms all other methods \method{Twin-VAE}{} and \method{BiT}{} by a clear margin on the \dataset{Syn-BF-L-Te}{}, and \\ \dataset{Nat-BF-L-Te}{} data sets.
On the \dataset{Syn-PC-L-Te}{} and \\ \dataset{Nat-PC-L-Te}{} data sets, where more natural data is available for training, the stronger initialization by \method{Transfer Twin-VAE}{} does not have as strong of an impact. It has a good performance on the dataset with very little training time, but does not easily achieve the same performance as \method{Twin-VAE}{} on \dataset{Nat-PC-L-Te}{}. On the \dataset{Syn-PC-L-Te}{} it outperforms all other methods handily, this is most likely due to the transfer working even better on synthetic images than on natural images. Further reasons for \method{Transfer Twin-VAE}{} not performing better than \method{Twin-VAE}{} on \dataset{Nat-PC-L-Te}{} could be that the performance of the initialization on the original dataset is not as high. To remedy this, we used the \method{Transfer Twin-VAE}{} transferred to \dataset{Nat-BF-L-Te}{} and \\ \dataset{Syn-BF-L-Te}{} as a starting point to transfer back to \dataset{Nat-PC-L-Te}{} and \dataset{Syn-PC-L-Te}{}, using this configuration we obtained the best results on \dataset{Nat-PC-L-Te}{} and \dataset{Syn-PC-L-Te}{}, we call this configuration the\\ \method{double Transfer Twin-VAE}{}.

\begin{table*}[htbp]
\caption{Evaluation of all methods on the data sets \dataset{Syn-PC-L-Te}{}, \dataset{Nat-PC-L-Te}{}, \dataset{Syn-BF-L-Te}{}, and \dataset{Nat-BF-L-Te}{}.
For each method and data set, we report the mean absolute error (MAE), the mean relative error (MRE), and the accuracy.
Ultimately, only the performance on natural data (\dataset{Nat}{}) is important,
but we also report the performance on synthetic data (\dataset{Syn}{}) to provide further context.
We use an upward arrow \(\uparrow\) to indicate that higher is better
and a downward arrow \(\downarrow\) to indicate that lower is better. Training times are reported on an NVIDIA Tesla P-100 16G GPU for all models.}
\label{table:results-cell-counts}
\centering
\sisetup{detect-weight,table-auto-round}
\setlength{\tabcolsep}{0.5em}
\begin{tabular}{cS[table-format=2.2]S[table-format=2.2]S[table-format=2.2]|S[table-format=2.2]S[table-format=2.2]S[table-format=2.2]|S[table-format=2.2]}\toprule
Method & \dataset{Syn}{} {MAE  $\downarrow$} & {MRE / \%  $\downarrow$} & {Acc. / \%  $\uparrow$} & \dataset{Nat}{}  {MAE $\downarrow$} & {MRE / \%  $\downarrow$} & {Acc. / \%  $\uparrow$} & {training Time / sec $\downarrow$}\\\midrule
\multicolumn{7}{c}{{\dataset{PC}{} (phase-contrast microscopy)}} \\
\method{Twin-VAE}{} (\dataset{Nat}{} only)         &        n/a &        n/a &        n/a &       1.07 &       20.1 &       39.8 & 180000  \\
\method{Twin-VAE\textsubscript{max-acc}}{}                   & \best  0.09 &       0.68 & 68.2       &       0.60 &       5.92 & \ 57.8 & 400000 \\
\method{Twin-VAE\textsubscript{min-dev}}{}                   &       0.14 &       0.73 &       62.1 &  0.59 & \ 5.66 &       57.0 & 400000\\
\method{BiT}{}                               &        n/a &        n/a &        n/a &      2.203 &        n/a &      26.13 & 9000 \\
\method{Transfer Twin-VAE}{}(\dataset{Nat}{} only) &        n/a &        n/a &        n/a &       1.01 &      14.11 &       44.2 & 24000 \\
\method{Transfer Twin-VAE}{}                &       0.15 &  0.43 & 85.0 &       0.66 &      6.46 &     53.7 &  40000  \\
\method{double Transfer Twin-VAE}{}                &       0.12 & \best 0.43 & \best 85.0 &       \best0.58 &      \best 5.56 &     \best  58.7 &  71000  \\
\midrule
\multicolumn{7}{c}{{\dataset{BF}{} (bright-field microscopy)}} \\
\method{Twin-VAE}{} (\dataset{Nat}{} only)          &        n/a &        n/a &        n/a &       0.91 &       13.3 &        23.4  & 150000 \\
\method{Twin-VAE\textsubscript{max-acc}}{}                    &       0.48 &       4.27 &       60.1 &       0.68 &       7.60 &        53.2  &  310000 \\
\method{Twin-VAE\textsubscript{min-dev}}{}                    &       0.52 &       4.63 &       58.2 &       0.63 &       7.31 &        51.9 &  310000 \\
\method{BiT}{}                                &        n/a &        n/a &        n/a &       1.03 &        n/a &        43.1 & 5400\\
\method{Transfer Twin-VAE}{}(\dataset{Nat}{} only)  &        n/a &        n/a &        n/a &       0.72 &       7.88 &       51.36 & 20000 \\
\method{Transfer Twin-VAE}{}                 & \best 0.40 & \best 3.87 & \best 66.6 & \best 0.52 & \best 5.47 & \best 60.74 &  31000  \\
\bottomrule
\end{tabular}
\end{table*}

The most useful long term information transfer seems to be happening from better performing microscopy methods (phase contrast) to worse performing methods (bright field). In conclusion, a magnitude shorter training times, a better starting point and some hyperparameter tuning always outperforms random weight initialization.
Interestingly, the resulting transfer performance on the \dataset{Syn-BF-L-Te}{}, and \dataset{Nat-BF-L-Te}{} datasets is better than either microscopy method alone (\num{60.74} $\leftrightarrow$ \num{53.20}/\num{57.80}).

In summary, the factors gained with our best methodology \method{double Transfer Twin-VAE}{} are: 19\% better accuracy compared to BiT on the \dataset{Nat-BF-L-Te}{} dataset, 32\% accuracy better accuracy compared to BiT on the \dataset{Nat-PC-L-Te}{} dataset. We achieved even higher accuracy gains compared to EfficientNet and Watershed. We gained about 1.1\% accuracy compared to our previous \method{Twin-VAE}{} on the \dataset{Nat-PC-L-Te}{} dataset and about 7.5\% accuracy compared to \method{Twin-VAE}{} on the \dataset{Nat-BF-L-Te}{} dataset.

\begin{figure}
\centering
\includegraphics[width=\linewidth]{Comparison}
\caption{\ The mean relative error (MRE) for \method{BiT}{}, \method{Transfer Twin-VAE}{}, and \method{Twin-VAE\textsubscript{min-dev}}{} on \dataset{Nat-BF-L-Te}{} on a logarithmic scale. Horizontal bars indicate the average MRE of their respective color and method.}
\label{fig:deviation}
\end{figure}

\section{Conclusion and Outlook}
In this paper, we present a significant improvement over the original Twin-VAE by using transfer learning methods to improve the accuracy and training times of the original architecture using pre-trained checkpoints of the original paper. Utilizing these shortened training times, we perform extensive hyperparameter tuning and improve the accuracy even further.
Furthermore, we research which parts of the original Twin-VAE architecture are necessary for the transfer learning case.
We determine that the synthetic data still plays a key role in achieving high performance and can not be removed without significant performance loss.
The Decoder part of the network does not contribute to achieving higher accuracies during transfer learning and is therefore only necessary for pre-training on the original datasets.

\emph{Limitations:}
The transfer procedure only performs well if the initial starting point has a high performance on the original dataset. If the starting point has low performance on the original dataset the transfer procedure might achieve lower performance than a normally trained network.
In practice this is not a big problem since good starting points (in our case the \method{Twin-VAE\textsubscript{max-acc}}{}) can be chosen easily based upon their performance on their original dataset.

The Twin-VAE architecture can use synthetic data to improve performance on natural data, currently this is limited to ratios of up to 2:1. It would be desirable if the network could abstract even further and possibly not need any labels on the natural data at all. Current methods to increase regularization (Higher \(\regucost\), Dropout, Weight Decay) are not able to force the network to project the different data types onto the same embedding.

\emph{Potential and future work:}
The methods described in this paper enables the automation and remote surveillance of various previously tedious and labor-intensive laboratory experiments. To use its full potential it would be interesting to implement this method as edge computing on modern microscopy hardware.

To enable edge computing computational efficiency is key, here different techniques to prune network weights or similar methods could be explored to facilitate even faster computation times.

Other possible future work includes, using active learning~\cite{Cohn1994} to further refine the algorithm's predictions and enable entirely new areas of prediction, such as the survival probability of an entire cell culture.

\section{Authors' information}
\label{sec:8}
\textbf{Data Availability}
The datasets generated during and/or analysed during the current study are available in the Natural and synthetic CHO-K1 time-lapse suspension cell microscopy images (bright-field and phase-contrast) v2 repository, \url{https://pub.uni-bielefeld.de/record/2960030}.

\begin{acknowledgements}
\thanks{We gratefully acknowledge funding by the BMWi witin the project KI-Marktplatz, grant number 01MK20007E (PK), and by the European Commission within the project  ICU4Covid, Grant Agreement number 101016000 —  H2020-SC1-PHE-CORONAVIRUS-2020-2 / H2020-SC1-PHE- CORONAVIRUS-2020-2-CNECT (DS).}
\end{acknowledgements}

\section*{Conflict of interest}
The authors declare that they have no conflict of interest.

\bibliographystyle{spmpsci}

redpajama_set_name: RedPajamaArXiv


text:
Bidding war over Australian gas
By Angela Macdonald-Smith | February 10, 2009, Tuesday | Print Edition
BG Group Plc, the United Kingdom company seeking to develop a natural gas export project in northeastern Australia, offered to buy Pure Energy Resources Ltd for A$796 million (US$536 million), topping an offer from Arrow Energy Ltd.
The cash offer of A$6.40 a share is 19 percent higher than the value of Australian Arrow's cash and stock bid, based on the last closing price, BG said yesterday in a statement. Arrow jumped 12 percent to A$2.48 and Metgasco Ltd rose 13 percent to 38 cents in Sydney as investors bet that other coal-seam gas stocks may attract bids.
BG and Arrow, Royal Dutch Shell Plc's Australian partner in coal-seam gas, are among companies seeking more reserves to feed planned liquefied natural gas projects. Australia's industry to extract gas from coal seams attracted more than A$17 billion in investment last year as companies such as ConocoPhillips tapped into ventures that could meet demand in Asia for cleaner fuels.
"This underlines that even in this period of doom and gloom there's still strong interest in the energy sector, particularly in the LNG space," said Gavin Wendt, senior resources analyst at Fat Prophets Funds Management in Sydney.

redpajama_set_name: RedPajamaCommonCrawl


First day at a new job for Todd Mills, having a bit of the jitters, fearing he has lost his edge. He has an interview with an up and coming political figure that could be a presidential candidate.Before the interview is finished he is taken Hostage by Two men and Women, they succeed in getting away. somehow Steve Rawlings appears to be dragged into it. This is a book dealing with the politics behind AIDS funding. The political figure is arguing government funding is no longer necessary, the kidnappers agenda is to make a statement. Can Todd Solve this, how is Steve Rawlings involved?


text:
using System.Collections.Generic;

namespace Twitch.Base.Models.NewAPI.Schedule
{
    /// <summary>
    /// Information about a broadcaster's schedule
    /// </summary>
    public class ScheduleModel
    {
        /// <summary>
        /// The ID of the broadcaster.
        /// </summary>
        public string broadcaster_id { get; set; }
        /// <summary>
        /// The name of the broadcaster.
        /// </summary>
        public string broadcaster_name { get; set; }
        /// <summary>
        /// Login of the broadcaster.
        /// </summary>
        public string broadcaster_login { get; set; }
        /// <summary>
        /// If Vacation Mode is enabled, this includes start and end dates for the vacation.If Vacation Mode is disabled, value is set to null.
        /// </summary>
        public ScheduleVacationModel vacation { get; set; }
        /// <summary>
        /// Scheduled broadcasts for this stream schedule.
        /// </summary>
        public List<ScheduleSegmentModel> segments { get; set; } = new List<ScheduleSegmentModel>();
    }
}

redpajama_set_name: RedPajamaGithub


text:
Weinberg: String Quartets Nos.
Arcadia Quartet
For this the second volume in their series of Weinberg's string quartets, the Arcadia Quartet again presents three quartets from contrasting periods in the stylistic development of the composer. The first quartet of the self-taught teenager, written in 1937, in what Weinberg later described as his 'neo-impressionist' style, was heavily revised later in his life, and eventually re-published as Op. 141, in 1985. (It is this revised version that has been recorded here, the original version surviving only in manuscript form, in places virtually illegible.) The seventh quartet dates from 1957, after a gap of twenty turbulent years that had witnessed the emigration of Weinberg from Poland to Russia, his introduction to Shostakovich, and his experience of censorship and imprisonment in 1953. In contrast to his earlier quartets, the mood is more intimate and withdrawn, yet defiant. The eleventh quartet was composed between 13 October 1965 and 25 December 1966, at a time when Weinberg was mulling over the composition of his first opera, The Passenger. It is dedicated to his first daughter, Victoria, and was premièred by the Borodin Quartet on 13 April 1967 in the Chamber Hall of the Moscow Conservatoire.


This document gives a recommendation for the payload configuration for performing dual soundings with two radiosondes. General guidelines are specified which aides the operator to prepare a payload for dual soundings. The recommendation is partly based on the results of Vaisala RS92-RS41 dual soundings performed at Lindenberg observatory using various configurations of the payload. In addition, general rules and guidelines are proposed for rigs with multiple-device payloads.
Reference quality upper-air measurements: GRUAN data processing for the Vaisala RS92 radiosonde, Atmos. Meas. Tech., 7, 4463–4490, doi:10.5194/amt-7-4463-2014, available at www.atmos-meas-tech.net/7/4463/2014/ (last access 26 July 2016).
Fast-response high-resolution temperature sonde aimed at contamination- free profile observations, Atmos. Meas. Tech., 3, 1673-1681, doi: 10.5194/amt-3-1673- 2010, available at www.atmos-meas-tech.net/3/1673/2010/amt-3-1673-2010.html (last access 26 July 2016).


text:
'use strict';

describe('Service: style', function () {

  // load the service's module
  beforeEach(module('angularAppApp'));

  // instantiate service
  var style;
  beforeEach(inject(function (_style_) {
    style = _style_;
  }));

  it('should do something', function () {
    expect(!!style).toBe(true);
  });

});


Today we ask the age old question: Will It Pancake? GMM #889!
bone marrow... prepared properly... is delicious!
I watch these videos to stay with my diet.
Ok I eat bone marrow all the time so I'd be down for the marrow pancake.
LINK IS SUCH A STUPID HOEEE .
I was just in Fuquay Varina the other weekend!
Bone marrow is tasty .
What type of bagels do you eat where they're rubbery ?????
LMAO so how does one get a hold of guinea pig balls?
Western Blinis are tiny pancakes typically served with stuff on top.
I put bacon in my pancakes!!
Dude bone marrow is good.
13:09 Will It Guinea Pig Testical?
The dog wouldn't eat bones if humans aren't so selfish and stubborn.
That pink stuff you eat after sushi is what humans eat to freshen breathe. Maybe wasabi to?
Breathe mints you said there's nothing.
Sry for bad English I'm from America.
"is there anything humans can eat that actually cleans their mouth besides gum?"
Those have sugar in them. Not really cleaning.
I miss it when they say will it pancake after saying what they mixed with the pancake.
Actually, in europe you eat bone marrow, it's called os à moelle and i absolutely looove it!
I could write a story on links forehead!
They should've done bacon pancakes.
Tbh bone marrow is pretty good.. it's like butter.
bone marrow is good if cooked properly!
Nothing you guys do is considered normal that's why I love you guys so much also I'm born and bred in the South just like you guys GO DAWGS.
In chiense cultures, the inside of the bone marrow is considered the sacred part of the bone because there is so little of it. I've tasted it and it isn't that bad.
Good mythical morning, a breakfast restaurant that's sells everything they like and invent from the show!
Made my guinea pigs watch this to remind them who's boss.


Nulo FreeStyle knows that our feline friends require nutrition that helps maintain lean muscles without excess calories. That's why their Grain-Free Turkey & Duck Recipe Cat & Kitten Dry Cat Food is formulated with 84% animal-based proteins and no white potatoes, grains, or tapioca, making it perfect for growing kittens to adult cats. Nulo's patented BC30 probiotic is a pure probiotic strain that supports your cat's gastrointestinal health. This delicious formula features small, disk-shaped pieces with a taste and texture cats love.


A Study in Scarlet is the first of the Sherlock Holmes stories. Watson narrates his first meeting with the eccentric Holmes, who appears almost genius in some fields of study and completely ignorant in others. This, as Holmes explains, is because he believes that brain space is limited, and one must be careful what one puts in. He gradually reveals his method and thinking to an astonished Watson. We see Holmes languishing at home, scratching at the violin he can play to perfection. We see the erratic way he conducts business, and finally, after a trip across half the world into a completely different landscape and story, we see him unravel the mystery at the heart of the novel in an apparently effortless manner.


text:
Cricket Horrible Cricket - The Ashes Down Under, 2013/14
putitintheblockhole
prayforrain
By Alexlotl,
November 3, 2013 in Sport
Alexlotl
EDIT: Updated thread title.
*straightens tie, coughs, waits for studio lights to come up*
PLAY THEME MUSIC
http://www.youtube.com/watch?v=CY_3SGNRvMw
Hello, and welcome to another edition of Cricket Lovely Cricket. You join us for the second leg in this year's Ashes one-two - England's tour of Australia. After an eventful home series in which both sides looked a bit scrappy, and dubious umpiring decisions stole the limelight more than once, what can we expect from the Australian tour? Have the teams, and the officials, learnt from their mistakes?
England, of course, won that first leg 3-0, thanks to an indomitable performance from Ian Bell and fairly consistent excellence from Anderson, Swann and Broad. The Aussie cupboard wasn't entirely bare, with Ryan Harris outbowling everyone else in the series, but just not getting the backup he needed to keep the pressure up.
The other problem for the Aussies was the fragility of their bowlers - such a problem, that one wonders if the reputation of Troy Cooley is going to undergo a negative re-evaluation in future. Starc, Pattinson and Bird were all afflicted with (suspected) stress fractures, Harris had hamstring problems towards the end of the tour. Of the quicks, only Peter Siddle, rumoured to be a golem constructed of tinned meat and coconut matting, made it through the tour without injury. Add to that the fact that Pat Cummins is still crocked, and Shane Watson limped off the field in India today, and the Aussies have to be worrying.
No squad announced for Australia yet, but it sounds like Clarke will be fit, and like George W Bailey might have played his way into the team in the India runfest.
For England, things look much better. Bresnan came down with a stress fracture during the last series, but is rumoured to be potentially ready to be flown into the squad if his recovery continues at its current pace. Cook also has a bad back, but should be fit. Monty Pee (can't wait for the bouncer jokes), who can probably be considered psychologically injured at this point, is also picked.
A few interesting names in the England squad. Michael Carberry, who I remember being the Next Big Thing about five years ago, and has put in a good knock in the first (rather lame) warm up. Boyd Rankin, our latest stolen Irishman, who is a big thing, and I believe came over from Ireland via a causeway of some sort. Bairstow's grip on the #6 slot still doesn't look secure, with yet another Yorkshireman (c/o Harare) in the form of Gary Ballance competing for that place. Given as his England scores so far consist of a 2 ball duck followed by a golden quacker in the warm up, Bairstow may be feeling a bit better. Tremlett returns from injury, but I'd be surprised to see him make an impact (mind you, I was surprised last time). Ben Stokes (born right here in sunny Christchurch NZ), who I'm very eager to see play, joins as a potential all-rounder, filling the ginger Dunelmian slot left vacant since Collingwood's retirement, and is probably more of a threat to Bairstow's place.
Two interesting names that aren't on the England list are Nick Compton, who we can presume forever discarded at this point, and Graham Onions, who despite dominating in the County Championship didn't get a call-up, and will be playing state cricket in South Africa. From my perspective, it looks like England see Onions as a like-for-like swap for Anderson, and if Jimmy gets injured I have no doubt Onions will be straight on the plane to Aussie, match fit and with plenty of competitive games under his belt.
Schedule can be found here. Once again, Tasmania misses out on a test, but gets probably the best warm-up match. TMS will be doing live commentary for the next two warm-up matches, as well as the full test series.
Two captains with bad backs competing to lift the lightest trophy in international sport is rather amusing. I think if it was the Stanley Cup, they'd both give up all hope.
Sounds like a lovely day in Hobart, and Cook and Carberry are making hay while the sun shines. Both sound in very good nick, although Cook's not running too well. Carberry's first 50 pretty slow, but he's exploded out of the blocks since. Both in the 90s as I type, heading for a double-century opening stand.
Big news is that Carberry is opening, with Root slotting in at #5. Suggests that England aren't so sure about their #6 position, and may play Carberry at the top and push Root back down. If Ballance gets some runs this match, then the selectors have a very nice headache.
This is bizarre radio - it's TMS, but about 80% of the commentators are Australian. It's not going out on Australian radio, and the feed is geolocked so it's only available in the UK (thank you, Tunnel Bear), so it seems like a weird choice. The Aussie Commentators have been making salty jokes about "mighty erections" (cranes), probably safe in the knowledge that no bugger is hearing them.
kiroquai
You know the best thing about all this?
Christmas Day. Wine. Too much food. A strange selection of possibly naff gifts to occupy your time. No work!
And then? Everyone else buggers off to bed, you finally get control of the TV remote, cricket on, snacks and wine at hand, sorted. A Boxing Day Ashes test. The feeling of slightly drunk excitement and satisfaction as the bowler runs in to bowl the first ball.
I've had an immensely busy year, but at that one precise moment I'm going to be as happy and stress-free as a pig in poo.
Good to see Carberry get some runs - coming back from the blood clot in his lungs was a major recovery.
I'm gonna have a newborn baby to selflessly volunteer to look after night shifts with. Can't wait. Get him learning fielding positions and the art of a good leave, nice and early.
We seem to have taken the weather on tour with us, let's hope the bowlers get some time in next week's warmup match...
Would rather have Prior behind the stumps than Bairstow, but Vaughan's comments have been pretty poor. I don't give much time to the theory that he doesn't like him due to management issues (Bairstow not being managed by the same company as Vaughan) but I think he's definitely got some sort of problem with him. Bairstow is an attack minded batsman, and I think playing in that slot in the order will suit him, but with Swann not having left handsets to bowl to there will be a lot more emphasis on catching behind the wickets from the quicks.
England would choose Prior over Bairstow for certain, but if he's an injury risk they'll not rush him back for this test I'm sure. Apparently Australia haven't lost on this ground since about 1988 or something mad like that, I can't wait for it all to start.
So Peter Siddle is saying they'll be trying to get Cook out. Pretty wild thinking there! I was also interested in Mitchell Johnson saying he's aiming for the throat, which suggests bowling a bit shorter than their bowling coach was talking about. Worrever.
Tremlett or Finn for me, their extra height can get more out of the pitch - but they need to bowl the right line.
The only reason to put Bairstow behind the stumps is to bring in another bowler.
Aussies need Watson fit to bowl, he can be deadly at first/second change and in keeping the pressure on new batsmen.
Great analysis here from Simon Hughes:
Summary: England's top order is strong, Australia's middle order is soft, Anderson is superb where Harris is fragile, Cook and Flower are stronger tactically than Clarke and Lehman.
Wow that's quite optimistic!
VIc Marks was a bit more cautious: http://www.theguardian.com/sport/blog/2013/nov/16/england-ashes-australia-first-test
I can't bloody wait, me. I think England will win the series relatively comfortably but I dunno about this first test and I don't think it'll be as straightforward as last time in Australia seemed to be. Though even then I remember that first innings with Strauss going early, and then getting beat easily in Perth.
Draw first Test for me, then England to take the series 2-1.
Minutes to go! I need sleep so badly but I gotta watch at least a bit to start with. England to bowl first then.
Not bad, need to break this partnership though.
Good knock from Johnson. If he could bowl, he'd be an all-rounder
On paper a good day for England despite the revival from Haddin and Mitchlol, but obviously we'll have to see how our batsmen get on first to really judge. The pitch did look good for batting.
feltmonkey
The hypocrisy of the Australian fans booing Stuart Broad is hilarious.
Hah yeah. Good thing is though, he certainly let his bowling do the walking.
Er, talking.
Interesting to see that the Aussies are favourites after the first day, think that's crazy myself.
That really depends on whether England find themselves six down before tea on day 2...
Yeah it's all about the England innings. Last tour we won because the top order scored heavily, but that was against a weaker attack than Australia have now. In the summer our top order wasn't great, and if we hadn't had Bell digging us out of trouble time and again the overall result could have been very different.
Agreed. In theory, this pitch suits our top order but can they be as patient and judicious as they need to be? And it being first test there aren't many Aussie bowlers injured yet so they look quite strong.
Be funny if they troll them again by getting out quickly in the first innings and then batting forever in the second.
82-3 at the moment. Looks like Cook and Trott are still in extremely average test form.
87-5! We're I big trouble. Bowlers being let down by an awful display from the batsmen again.
Parksey
At least in the summer we were saved by a solid middle order (largely Pietersen and Bell, as Prior's average was pretty bad).
Now, our top order are underperforming, but our middle order is just outright collapsing. Always a bit tentative to declare an opponent's innings as awful when you've yet to have a go yourself. Suddenly the Australian total looks like it's miles away.
I was going to go to bed at 4am, or when we got to 100. I thought that would be at a loss of two wickets. They started falling just as I was about to shut my laptop down.
Yup, it's been pretty nuts. 91-8.
Just like in the summer we once again got stuck in a slow run rate and then this time wickets tumbled. Same old problems. If Australia go on to win this test I'd put money on the momentum carrying them through to regaining the Ashes.
We had a pretty terrible first innings at the Gabba in 2010, didn't we? Wiki says we were about 140 runs behind, which is roughly where we will probably end up this time.
Clearly we are going to be some distance behind them now - and after scoffing somewhat at their performance yesterday. They at least had a decent stand to elevate their total. Broad is handy with the bat for a lower-order batsmen, but he's not going to have them looking over their shoulders at their lead. I mean, I think we are yet to even avoid the follow on?
It's easy to blame England's batsmen, but the Australian bowlers have been great. Johnson in particular has been terrifically hostile, and we've gone from steady and composed, to shell-shocked and apprehensive. And that isn't "apprehensive" as in "careful" but rather looking nervous and frightened.
We're going to be behind whatever happens, so it's a case of how much. Just trying to get runs here and there and drag this out, edging as close as we can. I mean, even 150 is a lofty ambition, and something I am going to bed now dreaming about.
Some stats from Sky and the BBC: 39 runs for 6 wickets in that session and in the last 27 balls, England scored four runs at the cost of five wickets.
I can't help but fell that Trott needlessly going in that extra over before lunch started it. We'd have gone into lunch in the ascendancy with him still at the crease, though even then, we were still probably on a par with how the Australians did yesterday.
Still, it's only the first innings of the first test. They've had a very good bowling performance, so we have to hope that this is the exception rather than the rule.
It's worrying that it was a batting pitch and we've done particularly badly, mind. Given how we were over here a few months ago.
136. Oh dear.
Nice cameo from Broad, but that was balls. Interesting stat from the radio - the last time we won the first match in an away test series was against Bangladesh in 2010.
Four wickets for less than 50 before the close would make me feel better.
Don't think there is any chance of that.


Join Vicky Barzey from Expedia CruiseShipCenters in Olds every Sunday at 7:45am for another edition of Let's Talk Travel.
Get tips, tricks and deals on all things travel right from the source. You'll get the inside industry scoop that will have you ready to jet set in no time!


Whether you're fixing fajitas, ribs, bbq, or steaks, one of these specially selected Texas dinnerware sets from our Texas Dinnerware collection will complete the experience. Choose from our Sky Ranch dinnerware, made of microwave and oven safe western stoneware dishes. These casual, ranch dinnerware dishes have a quality and style that reflects the best of Texas: toughness, beauty and pride.


text:
A rare case of COVID-19 vaccine-induced thrombotic thrombocytopaenia (VITT) involving the veno-splanchnic and pulmonary arterial circulation, from a UK district general hospital
Online download statistics by month: September 2021 to January 2023
Sep 2021 1679 1675 170
Feb 2022 99 97 36
Aug 2022 94 93 16
Sep 2022 95 97 39
Oct 2022 88 85 19
Jan 2023 75 75 14
Total 4573 4563 767


text:
Q: Map to an Entity with fields instead of properties in Entity Framework Is it possible to tell EF to map to a Class with fields instead of a class with Properties?
The Entity is then used by Filehelpers to create File, That's why I'm unfortunately bound to fields.
We have a lot of Entities, with a lot of Properties and a lot of Data, so remapping them or working with a backing field is not really an option.
Any suggestions or tricks?
public class MyUnmappedEntity
{
  public string Name;
}

var data = app.Database.SqlQuery<MyUnmappedEntity>("SELECT * FROM MyTable").ToList();

redpajama_set_name: RedPajamaStackExchange


Believe it or not, there are plenty of intriguing museums emphasizing the rich history of the city scattered between all the flashing neon lights and giant billboards. For those curious about learning more about the origins of Las Vegas' gaming industry without having to step foot inside a casino, make sure to check out the Neon Museum. This is where you can tour around the city's largest collection of the famous lights and signs accumulated from the golden days of Las Vegas gambling during the 1950s, and learn a few stories regarding the construction and business backgrounds of these once-iconic attractions.
Speaking of Sin City, one of the more edgy museum experiences you can indulge in during your Las Vegas holidays is a trip to the Mob Museum. For those unaware, the mafia had quite a presence and influence upon the origins of Las Vegas, and the Mob Museum is the place to be for learning more about the history of both famous mob families and law enforcement names alike. This museum is included on the list of the U.S. National Register of Historic Places, and is the headquarters for learning about some of the most notorious criminals in U.S. history, such as Al Capone, Bugsy Siegel, Frank Costello and more.
There are also more conventional forms of education available to explore in a few Las Vegas museums. The Nevada State Museum, Las Vegas is an excellent resource for learning more how the entire state of Nevada came to be, from learning of the original Native American inhabitants to the state's involvement in the testing of atomic bombs in deserted deserts during World War II. The Las Vegas Natural History Museum is a great place to tour around with the kids, as exhibits range from exploring the history of ancient Egypt and its mummies to replicas of giant dinosaurs that used to roam the state of Nevada millions of years ago.
Hoover Dam is one of the iconic sightseeing attractions just outside of Las Vegas.
While it may be hard to tell when taking a stroll down the strip, there are actually plenty of incredibly scenic locations to explore around Las Vegas. One of the most beautiful areas you can explore without travelling too far away from Sin City is the Red Rock Canyon National Conservation Area. This National Conservation Area is roughly a 25 kilometre drive away from downtown Las Vegas, and features endless opportunities to engage in an afternoon full of hiking, rock climbing, horseback riding, off road biking or even having a picnic while watching an incredible Nevada sunset. There are more than 800 square kilometres of government protected land to explore around these parts, providing visitors a perfect getaway from all the hecticness of downtown Las Vegas.
If you're up for a little more of a trip, you can venture out to Hoover Dam, which is just under 50 kilometres away from Las Vegas. Millions of visitors flock to this attraction every year to see the largest manmade reservoir in the U.S. in person, and there are guided tours throughout the year that allow visitors to not only get a peek of what it's like inside the giant facilities, but also learn more about the rich history that went into the construction of the Hoover Dam. Tours tend to be one hour long, and tickets do sell out quickly, so plan on getting there early if you're curious to witness this massive attraction in all its architectural glory.
Once you've properly soaked in all the wondrous nature surrounding Las Vegas, make sure to get a bird's eye view of the vast array of sparkling lights and shimmering signs of downtown by taking a trip to one of several aerial view opportunities. For starters, there are a few companies that offer helicopter rides that will guide you all over the downtown area, providing you with a one-of-a-kind view you'll be hard pressed to experience anywhere else. You can also go to the top of the Stratosphere Casino, Hotel and Tower, which is the tallest freestanding observation tower in the entire U.S., standing at more than 7,400 square metres high.
Helicopter tours provide the most scenic opportunity to get a bird's eye view of downtown Las Vegas.
Of course, not all the forms of entertainment in Las Vegas have to do with gambling. The big-time casinos in this city are also home to some of the world's most sought after concerts and shows, catering to all types of entertainment tastes. If you've never before treated yourself to the spectacular performances of O Cirque du Soleil, then you'll certainly need to enlighten yourself to some of the most jaw-dropping displays of acrobatic athleticism on the planet.
Even if you don't believe in magic, you'll be convinced your eyes are deceiving you while watching a Penn and Teller performance. The famous duo blends humor with illusions during nightly shows that will have you laughing your heart out and pondering how exactly did they just do that.
Finally, one of the hottest tickets on the strip is to see legendary singer Celine Dion in concert. Throughout most of the year, you can catch the famous "My Heart Will Go On" singer at Caesar's Palace, where her voice is more golden than the slot machine tokens.
The Alps mountain range is one of the most magnificent and well known in the world, and the best way to be properly introduced to these spectacular boulders is a scenic drive straight through them. Klausen Pass is an awe-inspiring mountain pass taking you right into the heart of the Alps that can be accessed when headed south from Zurich. A great feature about the Klausen Pass is that there is a highway option nearby that tends to attract more traffic, so this strip of pavement is usually pretty deserted, allowing you for optimal cruise time. Typical sights you'll see along the Klausen Pass include cruising along elevated roads reaching nearly 2,000 metres as you swerve through various tunnels leading you directly past the enchanting mountains. You'll also go through evergreen forests, see dazzling waterfalls and there are a few cafes to stop at along the way, in case you need a quick break from all the wonderful scenery.
The Alps mountain passes provide one incredible view after another along the Klausen Pass.
This famous route in Norway is just over 100 kilometres in length and is an ideal scenic introduction into the vast natural beauty of west Norway. Deep fjords, captivating mountainsides and plenty of sparkling cascades are common encounters along the Trollstigen National Tourist Route, as taking this highway nestled within the the wondrous countrysides of Norway's Rauma Municipality will provide countless photo opportunities. This road has been open to travellers since 1936, and frequent renovation of the roads has allowed for sufficient conditions when driving through steep mountain passes. The Trollstigen National Tourist Route will eventually lead you to Ørnesvingen, an excellent observation stopping point where you can park the car and get a glimpse of the alluring Geirangerfjord, home of the truly impressive Seven Sisters waterfalls. You can also head over to the Norwegian Fjord Centre, which is where you can learn more about the history of the famous fjords of Norway, as well as where to go to spot them all.
Another essential European scenic drive can be experienced on Alpine Road, a 450 kilometre stretch of highway that takes travellers from the coasts of Lake Constance to the elegant landscapes surrounding Lake Königssee in the German state of Bavaria. With the incredible Berchtesgaden Alps looming in the distance, each turn along Alpine Road escorts you through countless charming villages that are all ideal to stop at and indulge in some exquisite traditional Germany meals. Your eyes will dart from all the array of entrancing colours that burst with every twist of the road, and once you reach Lake Königssee, prepare yourself for even more once in a lifetime views.
There are three primary highways that guide you alongside the immaculate French Riviera coastlines.
Corniches essentially translate to a road that runs along a coastline that's cut into the edge of a cliff. Les Corniches roads are the essential route for exploring all the natural treasures that are abundant along the French Riviera in the southeastern region of the country. These three distinct corniches are divided among altitude, ranging from 35 metres above sea level up to 365 metres above the Mediterranean Sea. The higher road arguably provides the most chances to see the incredible glimpses of everything from the Riviera beaches to the towering Alps mountains over yonder, and you can even stop at the small town of La Turbie, where you can still see the Trophy of Augustus, still standing since 6 B.C. The lower altitude highways will guide you through more of the various cities and villages surrounding the French Riviera, providing one excellent resting point after another.
While you're probably aware of the historical significance of these iconic landmarks, it will certainly make your trip to the ancient Pyramids of Giza more awe-inspiring knowing some of the rich history behind its construction. For starters, the oldest and largest pyramid, and only one that's officially considered one of the original Seven Wonders of the Ancient World, is the Pyramid of Khufu, more commonly referred to as the Great Pyramid of Giza. Built during 26th century B.C., it has been estimated to have taken 10-20 years to construct the Great Pyramid of Giza, and at its original height of 146.5 metres, it was the tallest manmade structure in the entire world for nearly 4,000 years. With its estimated volume of 2.5 million cubic metres, experts have reported it would have taken installing roughly 800 tonnes of stone every day over the course of 20 years to complete.
The Sphinx is one of the most awe-inspiring sights to see in person.
Inside the Great Pyramid of Giza, visitors can tour the three distinct chambers that were created to honour the pharaoh Khufu and his wives. In addition to the Great Pyramid of Giza, the other two primary pyramids in this region are the Pyramid of Khafre, the second oldest and largest of the three, and the Pyramid of Menkaure, which rounds out the royal pyramids. You'll also be able to see the famous Great Sphinx of Giza statue that lies directly in front of the three great pyramids. Be prepared to plan out an entire day around this area, as there is plenty to see and cover throughout all of these structures.
It's just under a 20 kilometre drive out to the Great Pyramids of Giza from downtown Cairo, and there are several options to consider for cruising out to the area. You can always take a taxi or other commercial driving service from Cairo to Giza, but be aware that most of these vehicles don't have meters inside, so you should always agree upon a price prior to taking off for the pyramids. If you're feeling fancy, you can take a limousine service out to the pyramids, and most of these companies have fixed rates, so you won't have any negotiation surprises once you finally arrive in Giza. If you're planning on visiting the pyramids with a professional tour group, they'll most likely have shuttle buses ready to take you back and forth between Cairo and Giza. Of course, a much cheaper route to consider is taking the public buses, which can be arranged at the airport or around downtown near wherever you're staying.
In case you're unaware, Egypt is one of the hottest countries in the world to visit during its summertime. If you're planning on travelling to Cairo between May and September, prepare yourself for average high temperatures of at least the low 30s. There is also minimal shade provided around all of the pyramids, unless you're inside touring the premises of course. With all this in mind, water packs, sunscreen and hats are essential gear to bring along with you. Even during the other seasons, the humidity can still be extreme, so always take necessary precautions before arriving.
Camel rides are a popular way to travel back and forth between each of the pyramids.
You should try to get to the pyramids as early as you can, which will help you avoid both the crowds and the highest temperatures of the day. Be aware that while tour guides offer opportunities to check out all the smaller pyramids, there are only 300 tickets distributed daily for each of the three great Pyramids of Giza. Each ticket to individual pyramids will also cost extra to purchase, so be sure to bring extra cash prior to departing from Cairo. This means you should choose wisely which one to tour, as they tend to sell out fast. It's also important to note that getting back and forth between all the pyramids can be quite a trek, and there are other options than walking to all of the locations. Camel rides are extremely popular to do, but be warned – the camels, and their owners, tend to not be the friendliest encounters you'll experience. If you do have any problems negotiating with any workers around the pyramids, be sure to flag down an Antiquity Policeman to resolve any predicaments.
It's important to remember that these ancient structures are thousands of years old, and are not necessarily all that accommodating for touring visitors. Keep in mind that entering the pyramids entails hours of walking around steep and narrow steps, and anyone who gets nervous from claustrophobia will certainly have a tough time dealing with the interior tours. If darkness and tight spaces may be a problem for you, it's probably best to staying outside of the premises. You should also make an effort to stick around at night, as every evening around the Great Pyramids of Giza concludes with a sound and light show centered around the Sphinx. The dazzling display is quite a way to finish an epic day of journeying through one of the most marvelous showcases of human achievement in the world.
Anyone considering a trip to Berlin anytime soon should also mull over relishing the opportunity to spend the night in one of the most preserved night skies in all of Europe. Westhavelland Nature Park, located roughly 70 kilometres west of Berlin, was recently declared as the latest addition to the International Dark Sky Reserve list for providing more than 750 square kilometres of protected land completely unhindered by light pollution. This wetland is the largest protected area in the German Federal State of Brandenburg, and once the sun sets, prepare yourself for a dazzling array of stars in every direction you turn your head toward the heavens.
Each of these parks offer excellent chances to see the Milky Way galaxy above.
Another member of the International Dark Sky Reserve is Kerry International Dark Sky Reserve, located in the southwest region of County Kerry, Ireland. The Kerry International Dark Sky Reserve is one of three members of the prestigious Gold Tier Dark-Sky Reserves in the world and is also the only Gold Tier Reserve in the Northern Hemisphere at this time. What does all that mean? That means visitors who stay the night will be treated with more than 700 square kilometres of nothing but pure starry skies. Nestled between the Kerry Mountains and the Atlantic Ocean, this is a pristine place to see the Milky Way, along with its nearest galaxy neighbor, the Andromeda Galaxy, only a mere 2.5 million light years away. Everything from nebulae to meteors can be spotted around the Kerry International Dark Sky Reserve, making it a must-stop experience when travelling to Ireland.
Frank Sinatra once sang "let me see what life is like on Jupiter and Mars." While it's hard to say when mankind will officially answer that inquiry, you can certainly see Jupiter and Mars themselves during a starry night sky spent at Brecon Beacons National Park in South Wales. You can spot both of these planets without even using a telescope, and if you do have some equipment, look around Jupiter to see if you can spot the group of Jupiter's largest moons, known as the Galilean Satellites. The best part about attending this park is that there is even more outdoor opportunities available to partake in during the day. Brecon Beacons National Park is home to splendid mountain trails, forest hikes and wondrous scenery in just about every direction you look. But once the sun sets come the real natural treasures, as you'll be hard pressed to find a more starry night sky anywhere else around the United Kingdom.
Be prepared to spot dozens of constellations during your stay at these places.
The state of Utah is known as one of the most scenic areas to visit in all of the U.S., and one of the biggest reasons for that is the chance to see the Natural Bridges National Monument in person. This park gets its name from the beautiful three naturally formed bridges named Kachina, Owachomo and Sipapu that were formed from water erosion that flowed in the stream bed of these canyons. If you travel to the Owachomo bridge at night, you'll be able to see a perfect circle around thousands of stars clustered together, along with the enchanting Milky Way galaxy just above the bridge. Each of the natural bridges also has no traces of light pollution near them, so you'll be guaranteed to get an unfiltered look at the incredible star show above.
Dog sledding as a form of transportation is a long practiced Alaskan tradition that allows travellers to encounter the northern wilderness by using man's best friend as the tour guide. No matter what time of year it is, you'll find plenty of dog sled tour companies available throughout all the major cities from Anchorage to Fairbanks that specialize in offering newcomers to the area this once-in-a-lifetime opportunity. These companies offer a variety of different excursions, ranging from family tours across the frozen tundra to more gentle journeys on land that cater to those travelling with younger members. A typical dog sledding trip covers anywhere between 20-50 kilometres, depending on how long you choose to go, and will take you through one scenic mountain pass or countryside after another.
Introduce yourself to the Alaskan wilderness with a dog sledding excursion.
While the thought of getting anywhere close to the glacier waters of Alaska might sound a tad frigid to most travellers, it's hard to find a more awe-inspiring way to get in touch with Alaskan nature than through a kayak or canoe tour. Generally, the best places to embark on a canoe or kayak expedition in Alaska are any of the cities that border the Gulf of Alaska on the state's Southeastern coast, which includes anywhere from Anchorage to Juneau. You can find anywhere between one-day tours to week-long voyages, which can take you through the maze of magnificent glaciers while peering out onto the towering mountains and evergreen forests in the distance. Kayaking or canoeing is also one of the best ways to go wildlife spotting, as certain tours promise a high likelihood of encountering whales in the water, or at least bears or buffalo herds near the shoreline.
If you're looking for a means to encounter the Alaskan wilderness at your own relaxed pace and leisure, than you'll probably want to consider taking on a hiking trail. The best and safest spots for going for a stroll amidst the pristine natural beauty of Alaska are any of the national parks that are spread throughout the state. If you're visiting Anchorage, you should definitely check out Wrangell-St. Elias National Park, which has dozens of scenic hiking trails and itself is bigger than the entire country of Switzerland. One the most popular and beautiful hiking trails in the entire state of Alaska is the Chilkoot Trail, a 53 kilometre trail that ultimately spans from Dyea, Alaska, to Bennett, British Columbia. Besides all the impressive scenery you'll encounter along the way, there's plenty of rich history that went into the makings of this trail. More than 30,000 people braved their way along this route during the height of the Klondike Gold Rush in the Yukon Territory during the late 19th century, with many ultimately falling victim to the treacherous conditions along the way. But fear not, travellers, as there are many resting points along the way as well as multiple national parks to stop by and explore if you ever need a break from the trail.
Starting April 1, 2015, passengers will no longer be able to wander into a Qantas lounge fresh from the seashore in a wet swimsuit and jandals since the Australian airline will begin applying a dress code for their airport lounges.
The lounge staff will enforces the dress code and customers that are underdressed do risk the possibility of being turned away in the doorway whenever they don't fulfill the standards.