NGO KITAP - Tatar and Turkic Association in Prague. The word "KITAP" in Turkic languages means "a BOOK", which is symbolic - the culture and traditions of each nation - are as a new book to explore. One of the goals of our Association KITAP is to show and promote our culture, to inform interested people about our traditions and identity, which we do since year 2013, by organizing different cultural events.
Tomáš Boukal is a "wild card" of the event. Would you expect from a Czech professor of anthropology and ethnology to sing Turkic songs and play traditional Altai instruments? Tomáš is researching the indigenous people of Russia, especially those living in Siberia. He spent several years of his research life in Altai region, he performed traditional Turkic songs, which you can also find: on the page of his band Kanum. http://bandzone.cz/kanum.
Lilia Gerber-Khousnutdinova & Elmira Lyapina, together with Dania Altynbaeva-Lyapina, chairpersons of the NGO – KITAP, Tatar and Turkic Association in Prague, are spreading and promoting Tatar and Turkic culture in the Czech Republic. Lilia and Elmira performed traditional Tatar songs.
Server Abkerimov is reasonably called "the Golden Accordion", his mastery is worldly known. His virtuosic and heartfelt play made to applaud the audiences from different parts of the planet, in small European churches, as well as in huge Tokio Opera and Osaka Symphony Hall in Japan. The artist shows very impressive and original performances, attracting a lot of (his talent) admirers. He played his own compositions based on traditional Crimean Tatar motives.
Gulsara Bekbagysheva, is a professional dancer with academic education in the dance field. Gulsara is a member of the Central Asian Ensemble Jagalmay, which is performing in the Czech Republic since 2006. Gulsara is regularly participating in the international festivals, as International Folklore Festival Prague – heart of nations, or Chinese New Year with Fen Jun Song, or Refufest. She performed traditional Kyrgyz Dances.
Atman is an authentic Darbuka player from Turkey. He learned the split finger technique back at home from the source and developed the technique by crossing Europe, Asia and Australia. Rhythm is a spiritual path for Atman. Now he is inviting you to join him to discover the rhythmic realm.
Concert was moderated by Dania Altynbaeva-Lyapina.


Stigler High School was the first in our series on the top students from each area school.
The Haskell County Commissioners and others rolled out the UNITED plan for improvements at the Haskell County Courthouse.
Stigler and Quinton held their proms on April 1. Click here to subscribe and check out the photos.
City of Stigler employees relocate conduit and wiring on Broadway in downtown Stigler.


Nalou is a 19 year old pop/R&B singer from Montreal. She started performing in high school and since has engaged in performances in a variety of venues from local bars & street corners to "la place des arts. Most recently, she performed at the 2010 Miss Eternity Model Competition and Fashion Show held at the Bloor Theate, Toronto.
Her performance at Miss Eternity impressed one of the sponsors Toronto Exclusive Magazine www.torontoexclusive.com and the publisher invited her to receive more exposure on their media site.
After spending some time in Montreal and New York City, Nalou has settled in Toronto and continues to be inspired as an artist and performer.
She is currently working on her EP.
The release date is undisclosed for the moment. Fortunately her fans can visit her facebook band page or www.nalouchalifoux.blogspot.com and www.myspace.com/nalouchalifoux for updated information on upcoming shows ,events and news.
Greg Susoreny, is steadily climbing the National charts.


"What makes you smile?" is a question that we at Bynum Aesthetic Dentistry have asked our patients and friends for a long time now. Nearly 15 years to be exact! You see, we all have reasons for doing what we do and being who we are, but how often do we really sit down to take the time to really evaluate what it is that makes us smile? For some it is a loved one, a vacation or a feeling of being part of something bigger.


text:
Review: David Crosby hoping for a little more time in elegiac doc 'Remember My Name'
David Crosby and wife, Jan Dance, in "David Crosby: Remember My Name."
(Sony Pictures Classics)
Two-time Rock and Roll Hall of Fame member David Crosby confesses his sins and seeks atonement in this frank film
By Glenn WhippEntertainment Columnist
David Crosby, a two-time Rock and Roll Hall of Fame member for his tenure with the Byrds and Crosby, Stills & Nash, has talked freely about his addictions and missteps over the years. He wrote about his lurid adventures in his 489-page 1988 autobiography and then revisited the subject in a 2006 memoir that also detailed the health problems — liver failure, arterial disease and diabetes — that put him at death's door.
Crosby, 77, is still alive. But as we learn at the beginning of the new documentary "David Crosby: Remember My Name," he doesn't think he's long for this world. With three heart attacks and eight stents in his heart — the maximum number — Crosby figures another heart attack will claim him in the next year or two.
Of course, he said this in an interview conducted a couple of years ago. Crosby continues to be with us, which is almost as remarkable as the fact that his voice remains pure and soaring. It's his calling card. It's his life. And even with the health risks that come with travel, his love of singing trumps his fear of dying. "No music?" he asks, when questioned if he'd give it up for a stable, fulfilling personal life. No way.
A.J. Eaton's documentary captures Crosby's commitment to making his final years count ("time is the final currency," Crosby says) as well as the singer's candid admission that he has failed a great many people over the years. If you've read Crosby's two memoirs — or, really, any in-depth interview with him since he got sober during a 1980s prison stint — the substance of his regrets and self-inventory isn't particularly illuminating. But that doesn't mean there aren't moments in "Remember My Name" when this old narcissist makes you feel a twinge of empathy over his failings.
Crosby loves to point out — and he does so again in this film — that he's the guy in Crosby, Stills & Nash (and Crosby, Stills, Nash & Young) who never had a hit song. It's a little bit of self-deprecation, but mostly a boast. Graham Nash, Stephen Stills and Neil Young may have written all the songs you remember from the supergroup's 1970s heyday ("Ohio," "Our House," "Carry On"). Counterculture figure Crosby was the man behind the weird stuff. (Translation: He didn't compromise, man.)
The cantankerous glory of David Crosby, who hates Spotify, wants to reunite CSNY and saved at least $25,000 growing pot
David Crosby knows he's going to die soon. He's diabetic.
The documentary, with interviews conducted by Eaton and producer Cameron Crowe, who goes back with Crosby to his adolescent days as a Rolling Stone journalist, covers the career high points. But the filmmakers are more interested in how Crosby survived (he continues to record, releasing four solid albums this decade), and they're aided in this respect by the subject, a frank and gifted storyteller unafraid to delve into the most sordid moments of his life. Crosby's spirit remains vital, and he's determined to fly that freak flag into that good night.
"Remember My Name" relies almost solely on its subject. There are interviews and scenes with his wife, Jan, herself a recovering addict, and archival accounts from Crosby's collaborators, none of whom, tellingly, participated in this project. Stills, Nash and Young won't speak to him, period, Crosby says. The latest rift with Young came after Crosby called Daryl Hannah, now Young's wife, a "purely poisonous predator" five years ago. Nash couldn't even stand to look at Crosby during their last tour. Byrds frontman Roger McGuinn calls him "insufferable."
You don't really see that side of Crosby in "Remember My Name." He gets a little grouchy when the filmmakers take him to the Laurel Canyon Country Store, but that's about it. (When Crosby spies a picture of Jim Morrison at the famed "store where the creatures meet," he calls the Doors singer a "dork." He also makes the dubious claim that he was the first musician to move to Laurel Canyon, proving drugs do, indeed, cloud one's memory.)
No, the Crosby in "Remember My Name" is the wise hippie bard, reflective and repentant. Perhaps it's all a ruse, designed to spur one last musical reunion.
"Maybe I conned you into it," Crosby says to the camera at the end of the movie. "Maybe this is all a clever ploy."
Footage of Crosby, Stills & Nash singing a turgid version of "Silent Night" at the National Christmas Tree Lighting in 2014, complete with the Obamas' horrified reaction, should signal that particular (wooden) ship has sailed. This elegiac doc would make for a much better career coda.
'David Crosby: Remember My Name'
Rated: R, for language, drug material and brief nudity
When: Now playing
Where: Angelika Carmel Mountain, Landmark Hillcrest
Entertainment video playlist
glenn.whipp@latimes.com
Twitter: @glennwhipp
Arts | CultureThings To Do
Glenn Whipp
Glenn Whipp covers film and television for the Los Angeles Times and serves as columnist for The Envelope, The Times' awards season publication.

redpajama_set_name: RedPajamaCommonCrawl


text:
\subsection{Supervised Learning for Cloud Classification}

Most work to date on automated cloud classification, and in particular early work, has involved supervised learning,
whereby a classifier is provided
with many images of clouds labeled with the classes that one seeks to recognize
(e.g., cumulus, cirrus).
The classifier then learns the relevant features associated with those labels.
Such approaches have the significant advantage of being easy to evaluate,
assuming that a reliable and sufficiently large collection of ground truth data is available.
Collections of cloud image manually labeled with ISCCP classifications are widely used as ground-truth data for supervised cloud classification methods.

Early work on machine-based cloud classification algorithms integrated simple statistics with machine learning algorithms~\cite{Bankert1994CloudCO,Lee1990ANN, Welch1992PolarCA, Welch1988CloudFC}, and combined textual and cloud physical features~\cite{cloudsat2008,Schiffer1983TheIS}.
More recent work~\cite{Rasp2019CombiningCA, Xie2019SegCloudAN, Ye2017DeepCloudGC, Zhang2018CloudNetGC, marais2020leveraging} takes advantage of state-of-the-art deep learning technologies reinforced by increasingly powerful modern computing hardware
to achieve high classification accuracy in extracting relative features from images in cloud classification.

In particular, recent machine learning work has leveraged convolutional neural networks (CNNs),
a method that has exhibited impressive classification performance in image recognition~\cite{Simonyan2015VeryDC,krizhevsky2017imagenet}.
Shi et al.~\cite{Shi2017DeepCA}
using a pre-trained CNN plus a support vector machine to
classify two ground-based cloud observation products into both five (different pattern and thickness) and seven categories (WMO categories).
This approach demonstrated that the use of a deep neural network with convolutional layers can improve cloud classification performance.
Zhang et al.~\cite{Zhang2018CloudNetGC} developed CloudNet, a CNN inspired by a VGG architecture~\cite{Simonyan2015VeryDC}, to classify the 10 WMO cloud classes plus contrails from ground-based observations.
This work established that deep neural networks are capable of discriminating among the
10 WMO classes
without the use of customized features.

Two recent studies employed new labeled cloud datasets plus CNNs to study shallow marine clouds, which while commonly classified as stratocumulus or stratus  according to the standard cloud classification actually have rich diversity and different physical feedbacks.
Rasp et al.~\cite{Rasp2019CombiningCA} leveraged pixel-based CNNs to classify shallow marine clouds
into new four categories;
Stevens et al.~\cite{stevens2019} identified four recurrent categories of mesoscale organization in data from MODIS Terra and Aqua satellites.
Both studies used human labelers to classify cloud images according to texture, and found that the resulting custom annotations revealed physically meaningful cloud regimes when they then compared temperature, relative humidity, and vertical velocity across the four classes.

\subsection{Limitations of Supervised Learning Approaches}

Supervised cloud classification has drawbacks.
First, it requires a large annotated dataset.
Second, because human-defined cloud classes are only well defined for classic examples,
which account for just a small fraction of large satellite datasets~\cite{rossow1993comparison, hahn2001isccp},
supervised approaches fall short when used to classify diverse real-world data.
Third,
as labels are restricted to prior assumptions,
they cannot identify cloud types that were not specified in training data but that might be relevant to climate research.

These difficulties have been highlighted by several studies.
When Wood and Hartmann~\cite{Wood2006SpatialVO} investigated the relationship between the complex textures of low clouds and the physical characteristics found within common mesoscale (1--100~km) cloud organizations, they found that four frequent mesoscale cloud patterns associated with the existence of open- and closed-cell structures, often classified as stratus or stratocumulus
in the ISCCP classification, occur in different geographical regions and have different distributions of liquid water path, suggesting that the spatial variability of low clouds is underrepresented in the standard cloud types.

\subsection{Unsupervised Learning Approaches}\label{sec:unsup}

Unsupervised learning for cloud classification was first explored by
Visa et al.~\cite{Visa1995NeuralNB} and
Tian et al.~\cite{Tian1999ASO}
as an alternative to orthodox classification methods that depended on ad hoc thresholds and decision trees.
Those early works applied a self-organized map~\cite{kohonen1990self}, which preserves a topological relation when mapping a high-dimensional space to a low-dimensional space, to classify extracted features by singular value decomposition (SVD), and selected features by the input statistic.
Tian et al.~\cite{Tian1999ASO} concluded that unsupervised learning may not always match the standard cloud types because its classifications smooth boundaries between independent cloud classes,
leading to a classification accuracy (relative to human-defined cloud classes) that falls behind that of supervised learning.
Their results rejected the use of unsupervised learning as a promising method.

More recently, concern about the inability of artificial cloud classes to capture the diversity of clouds has spurred increased interest in the use of state-of-the-art unsupervised learning using CNNs to discover novel patterns.
As the convolutional filters used in CNNs can learn spatial patterns,
it seems probable that a CNN-based classifier should be able to capture more relevant physics than the classifiers used in earlier work~\cite{Visa1995NeuralNB,Tian1999ASO},
which eliminated spatial information by flattening and learned representative statistical values based purely on mean cloud properties.

Kurihana et al.~\cite{kurihana2019cloud} used a convolutional autoencoder to capture both the spatial structure of cloud images and relevant physics in the latent representation.
That preliminary study trained the autoencoder with a compound loss function (see Appendix~\ref{sec:CNRI}) using visible to thermal bands of MODIS radiances, and proposed an early version of the five criteria described in Section~\ref{sec:protocols} as a basis for evaluation.
Results showed that both the physical association and spatially cohesive cloud cluster criteria were satisfied, but the work did not address rotation dependency to input orientation and did not fully explore associations with established cloud categories.

In another relevant study,
Denby~\cite{denby2019unsuper} applied representation learning techniques to classify mesoscale cloud organizations.
In representation learning,
CNNs are trained to embed meaningful input data features in the latent representation, which is then used for a clustering or regression task.
Denby trained a residual neural network, ResNet-34, on GOES-16 images
to formulate the representation,
and then applied hierarchical clustering to the latent representation of unseen GOES-16 images.
This work differs from Kurihana et al.~\cite{kurihana2019cloud} and the present paper, in its approach to learning spatial features.
Denby uses only the encoder and loss function parts of an autoencoder to learn how to embed spatial features in the original image into the latent representation.
By contrast, we train a full encoder--loss function--decoder triad to learn differences in shapes of input images.
The resulting clusters differentiate cloud images based on the strength of radiation in visible and infrared channels (GOES-16 bands 1 and 9); the triplet loss applied to the ResNet-34 network encourages the separation of dissimilar texture of images.
However, their example classes still distribute similar cloud images across several clusters.
For example, they report (Figure 2 in Denby~\cite{denby2019unsuper}) that cellular stratocumulus clouds are distributed among at least four classes,
indicating a limitation in their ability to classify small patches of large cloud structures.
Their approach is also limited in its evaluation,
in that it does not investigate whether the mapping of clouds to clusters is aligned with physical properties---a major focus of our work.

\subsection{Use of MODIS Satellite Data}\label{sec:modis}

We use MODIS products~\cite{justice1998moderate} from the Terra satellite instruments
to train and test our autoencoder-based unsupervised learning framework for cloud clustering.
We choose MODIS as its instruments have captured data with approximately 2330~km by 2030~km spatial coverage every five minutes since 2000, providing a variety of products for analysis.
In Sections~\ref{sec:physicalreasonableness} and~\ref{sec:experiment}, we also use data
from the Aqua satellite instrument in a comparison with another
supervised cloud classification study~\cite{Zantedeschi2019CumuloAD}.

MODIS provides a variety of products, from raw radiance observations to derived cloud physical parameters, all aligned to the same geolocation points on the ground.
We use three such products in this work, MOD02, MOD35, and MOD06:
\begin{itemize}
    \item
MODIS level 1B calibrated radiance imagery at 1~km resolution (MOD021KM and MYD021KM; hereafter \textbf{MOD02})~\cite{mod021km}, which we use for autoencoder training and testing;
\item
MODIS35 level 2 cloud flag product at 1~km resolution (MOD35L2 and MYD35L2; hereafter, \textbf{MOD35})~\cite{Ackerman2008CloudDW,Frey2008CloudDW} aligned on MOD02 data,
which we use for quality control; and
\item
MODIS06 level 2 cloud product at 1~km resolution (MOD06L2 and MYD06L2; hereafter, \textbf{MOD06})~\cite{baum12, steven17}, which we use to evaluate to what extent clustering results obtained via our autoencoder, when trained only on radiance data, reflect cloud physical properties.
\end{itemize}

Note that we do not use the MOD06 variables as inputs to our autoencoders.
This is because
we want to perform clustering free from any assumptions captured in those variables.
Furthermore, the MOD06 product is estimated at pixel resolution from MOD02 via various theoretical bases and thermodynamics,
which can truncate spatial information concerning certain scales of cloud organization.

\begin{comment}
\begin{table*}
\centering
\caption{The elements of three MODIS products used to train and evaluate our autoencoders and clustering method. Source: NASA Earthdata}\label{tab:products}
\begin{small}
\begin{tabular}{C{1.3cm}p{6.9cm}C{1cm}p{5.3cm}}
\hline
 Product & Description &Band& Primary Use   \\ \hline
 MOD02   & \emph{Shortwave infrared (1.230-1.250 $\mu$m)}         & \emph{5} & \emph{Land/cloud/aerosols properties}  \\
         & Shortwave infrared (1.628-1.652 $\mu$m)         & 6 & Land/cloud/aerosols properties  \\
         & Shortwave infrared (2.105–2.155 $\mu$m)         & 7 & Land/cloud/aerosols properties   \\
         & Longwave thermal infrared (3.660–3.840 $\mu$m)  & 20& Surface/cloud temperature   \\
         & Longwave thermal infrared (7.175–7.475 $\mu$m)  & 28& Cirrus clouds water vapor   \\
         & Longwave thermal infrared (8.400–8.700 $\mu$m)  & 29& Cloud properties   \\
         & Longwave thermal infrared (10.780–11.280 $\mu$m)& 31& Surface/cloud temperature   \\ \hline
  MOD35  & Cloud mask             &  & Cloud pixel detection   \\ \hline
  MOD06  & Cloud optical thickness&  & Thickness of cloud   \\
         & Cloud top pressure     &  & Pressure at cloud top   \\
         & Cloud phase infrared   &  & Cloud particle phase   \\
         & Cloud effective radius &  & Radius of cloud droplet \\ \hline
\end{tabular}
\end{small}
\end{table*}
\end{comment}

\begin{table*}
\centering
\caption{The 12 MODIS products used to train and evaluate our autoencoders and clustering method. Source: NASA Earthdata.}\label{tab:products}
\begin{small}
\begin{tabular}{C{1.1cm}p{6.2cm}C{0.9cm}p{4cm}p{0.1cm}p{3.1cm}}
\hline
 Product & Description &Band& Primary Use  && Our Use \\ \hline
 MOD02   & Shortwave infrared (1.230-1.250 $\mu$m)         & 5 & Land/cloud/aerosols properties & & Replaces band 6 in \S\ref{sec:experiment} \\
         & Shortwave infrared (1.628-1.652 $\mu$m)         & 6 & Land/cloud/aerosols properties & \rdelim\}{6}{3cm}& \multirow{6}{3cm}{Six bands used to train autoencoders} \\
         & Shortwave infrared (2.105–2.155 $\mu$m)         & 7 & Land/cloud/aerosols properties &&  \\
         & Longwave thermal infrared (3.660–3.840 $\mu$m)  & 20& Surface/cloud temperature  && \\
         & Longwave thermal infrared (7.175–7.475 $\mu$m)  & 28& Cirrus clouds water vapor   &&\\
         & Longwave thermal infrared (8.400–8.700 $\mu$m)  & 29& Cloud properties  && \\
         & Longwave thermal infrared (10.780–11.280 $\mu$m)& 31& Surface/cloud temperature   &&\\ \hline
  MOD35  & Cloud mask             &  & Cloud pixel detection   &&\\ \hline
  MOD06  & Cloud optical thickness&  & Thickness of cloud   &\rdelim\}{4}{*}& \multirow{4}{3cm}{Used to evaluate clusters based on MOD02-trained autoencoder } \\
         & Cloud top pressure     &  & Pressure at cloud top   &&\\
         & Cloud phase infrared   &  & Cloud particle phase   &&\\
         & Cloud effective radius &  & Radius of cloud droplet &&\\ \hline
\end{tabular}
\end{small}
\end{table*}

\subsubsection{Details on MOD02 Bands}
The MOD02 product has 36 spectral bands in the range 0.4--14.4 $\mu$m (i.e., visible to thermal infrared) from the year 2000 to the present.
We list the MOD02 radiance bands used in this work in Table~\ref{tab:products}.
Except as noted in the next paragraph, we use
in our patches the six bands listed in rows 2--7.
We use these bands,
which range from the shortwave to longwave,
because they are important for the retrieval algorithms used to compute the MOD06 variables listed in Table~\ref{tab:products}.
Specifically, bands 6, 7, and 20 are used to estimate cloud optical properties (e.g., cloud optical thickness and effective radius) and bands 28, 29, and 31 are used to separate high and low clouds and detect cloud phase;
thus, we hope that a model trained on these six bands may learn associations of cloud physical parameters, especially cloud optical thickness and cloud top pressure used for ISCCP cloud classification.
As is common in neural network training,
we normalized the values in each selected band to the range [0, 1],
so as to prevent different scales of values in different bands from biasing training.

Due to a known stripe noise issue at band 6 (1.628--1.652 $\mu$m) in Aqua~\cite{rakwatin2007stripe,gladkova2011quantitative}, we use band 5 (1.230--1.250 $\mu$m: the first row in Table~\ref{tab:products}) as an alternative to band 6 for the study reported in Section~\ref{sec:experiment}.
We select that band because it is primarily sensitive to cloud optical thickness (the 1.6 $\mu$m and 3.7 $\mu$m bands are used for cloud optical thickness retrievals~\cite{steven17}) and is the closest to band 6.

The MODIS instrument
generates hierarchical data format (HDF) image files of
2030 pixels $\times$ 1354 pixels $\times$ 36 bands.
To enable efficient learning of cloud features on neural networks, we use a smaller geographical unit, a \emph{patch}, as the element on which we perform clustering.
Each 128 pixel $\times$ 128 pixel $\times$ 6 band subset is cropped from a MODIS image.

\begin{figure}
    \centering

    \includegraphics[width=0.9\columnwidth,trim=3mm 3mm 6mm 4mm,clip]{Figures/datasubsets3.pdf}
    \caption{The five shaded ovals (not to scale) each denote a different subset of the full 19-year MODIS MOD02 1~km calibrated radiances dataset that we use in our work.
    For each, we indicate the number of patches.
    Training subsets are on the left and test subsets on the right.
    Of the three test subsets, \texttt{TestCUMULO} and \texttt{Phys} are composed of patches that are adjacent in space and time, while \texttt{Test} is not.}

    \label{fig:data}
\end{figure}

\subsubsection{MOD02 Subsets Used in Evaluation}\label{sec:testdata}
The full 19-year (2000--2018) Terra MOD02 dataset corresponds to about 100 million
valid 128 pixel $\times$ 128 pixel patches of the form just described.
From this set, we extract the following subsets (see also Fig.~\ref{fig:data})
for training and testing.
\begin{itemize}

    \item \texttt{Train}: \num{1075994} patches used to train the NRI autoencoder. This is about 1\% of the full 19-year (2000--2018) MOD02 (Terra) dataset.
    \item  \texttt{TrainCR}: A \num{60000}-patch subset of  \texttt{Train}, used to train the RA and RI autoencoders.
    \item  \texttt{Test}: \num{2000} patches selected at random from the 19-year MODIS dataset minus those patches already selected for \texttt{Train},
    used to test the various autoencoders.
    \item  \texttt{TestCUMULO}: \num{58734} patches corresponding to all daytime swathes of MOD02 (Aqua) on January 1, 2008, which we use in Section~\ref{sec:physicalreasonableness} to investigate associations with cloud physical parameters, and in Section~\ref{sec:experiment} to compare the resulting clusters with a labeled dataset.
    \item \texttt{Phys}:
    493 patches selected from a region off the coast of California (11--44$^\circ$ N, 144--112$^\circ$ E) on December 1st 2015, which we use in Sections~\ref{sec:spatialinfotest} and~\ref{sec:test:separable} to investigate associations with cloud physical parameters in a spatially continuous region.
\end{itemize}

We select the latter region as it features marine stratocumulus clouds that form in the stable air off the west coast of continent, where upwelling of cold water prevents deep convection from sea surface and stabilize low cloud~\cite{woodStCreview}, and the specific date as it exhibits both low marine stratocumulus, which occur frequently in winter, and high cloud, and thus provides a good test as to whether our framework clearly separates these different cloud objects. Following our quality--control scheme, we remove patches comprised of invalid pixels and patches containing less than 30\% of cloud pixels as detected by the MOD35 cloud flag product.

\subsection{Autoencoder Network Architecture}

\begin{figure}
    \centering
    \includegraphics[width=.8\linewidth]{Figures/autoencoder.pdf}
    \caption{Autoencoder cartoon, showing input $X$, encoder network \texttt{E}, latent representation \texttt{h}, decoder network \texttt{D}, output $\hat{X}$. Adapted from Wikipedia.
    }
    \label{fig:autoencoder}
\end{figure}

The autoencoder~\cite{kramer1991nonlinear,goodfellow2016deep,Hinton2011TransformingA,Hinton94AEscience} is a widely accepted unsupervised learning method that
learns to convert input images first to a compact lower-dimensional ``latent space" representation
and then from that representation to an output, in a way that minimizes the difference between input and output: the reconstruction loss.
An autoencoder typically uses a symmetric encoder-decoder architecture (see Fig.~\ref{fig:autoencoder}), with the encoder $E$ extracting essential information from high-dimensional inputs into a lower dimensional intermediate layer, the \emph{latent representation},
and the decoder $D$ then approximating the inputs from the latent representation.
The decoder restores the original data from $h$, a latent representation at the bottleneck layer, where the input data $X = \{x_1, \cdots, x_n\}$ are encoded as $z = E(X)$, such that $\hat{X} = D(z)$.

Training an autoencoder network generates a model capable of restoring the principal patterns of the original structure in an unsupervised manner.
Because a trained autoencoder retains only relevant features in the latent representation, omitting noise in the inputs, such networks are commonly used as a preprocessing tool~\cite{Caron2018DeepCF} prior to image processing
tasks such as clustering or regression, anomaly detection~\cite{Sakurada14Anom}, denoising~\cite{Vincent2008ExtractingAC}, and inpainting~\cite{Pathak2016ContextEF,Suganuma2018ExploitingTP}.
In addition,
autoencoders have been applied successfully to a wide variety of image
recognition problems~\cite{Caron2018DeepCF} via
clustering and classification of the compact representation.

\begin{figure}
		\centering
	\includegraphics[width=\textwidth,trim=4mm 3mm 3mm 3mm,clip]{Figures/arch_cnn_label.pdf}
	\caption{The RICC (and L2 and RA: see below) encoder consists of L blocks, each with three convolutional layers (orange) activated by leaky ReLU, and with batch normalization (red) applied at the final convolutional layer in each block before activation.}\label{fig:network:ricc}
\end{figure}

We summarize the detailed configuration of our encoder in Fig.~\ref{fig:network:ricc} and Table~\ref{model-table}.
The decoder (not shown) has the same structure, but mirrored.
We follow standard practice in integrating CNNs into our autoencoder architecture
so as to preserve spatial structure in the input images and to detect the edges of input cloud objects.
We stack the convolutional layers into \textit{blocks},
each consisting of three convolutional layers of the same size, plus batch normalization~\cite{Ioffe2015BatchNA} at the end of the block.
(Three layers are required to allow the bottleneck representation to learn the structure of an entire input image.)

We use a 3$\times$3 filter, following Simonyan et al.~\cite{Simonyan2015VeryDC} who proposed that the smaller odd-number kernel size with multiple convolutional layers can learn entire features in its inputs.
We halve the resolution of the incoming image but double the filter size each time that we move from one convolutional block to the next,
using max pooling with stride 2 in the NRI and RA autoencoders.
However, as the pooling operation can trigger a loss of spatial structure in CNNs~\cite{Hinton2011TransformingA},
we alternate the max-pooling layer with a stride convolution layer, as stride convolution with increased stride performs as accurately as max pooling in image recognition tasks~\cite{springenberg2014striving}.
We use a leaky rectified linear unit (leaky ReLU), a refinement of ReLU~\cite{Nair2010RectifiedLU}, for the activation function, $f(x) = \max(0.3x,x)$.
We use stochastic gradient descent~\cite{lecun1998gradient} as our optimization method.

\begin{table}[thb]
\caption{The layers used in the RICC autoencoder's encoder}
\label{model-table}
\centering
   \begin{tabular}{l|r|r|r}
    \hline
    Layer & Resolution & Filters & Operation\\
    \hline\hline
    Input   &  128 x128&     6& Resize\\ \hline
    Conv    &   32 x 32&     6& Conv x 1\\ \hline
    Block 1 &   16 x 16&    32& Conv x 3\\ \hline
    Block 2 &   16 x 16&    64& Conv x 3\\ \hline
    Block 3 &     8 x 8&   128& Conv x 3\\ \hline
    Block 4 &     4 x 4&   256& Conv x 3\\ \hline
    Block 5 &     2 x 2&  512& Conv x 3\\
    \hline
    \end{tabular}
\vskip -0.1in
\end{table}

\subsection{Rotation-Invariant Loss Function}\label{sec:ri:loss}

The loss function is used during autoencoder training to quantify the difference between an original and restored image;
the optimization process then works to minimize that difference.
In a conventional autoencoder, this function might be:
\begin{equation} \label{standardloss}
    L(\theta) = \sum_{x \in S} || x - D_{\theta}(E_{\theta}(x))  ||^p_p ,

\end{equation}
where $S$ is a set of training inputs;
$\theta$ is the encoder and decoder parameters, for which values are to be set via training;
$x$ and $D_{\theta}(E_{\theta}(x))$ are an input in $S$ and its restoration, respectively;
and $|| \cdot ||^p_p$ denotes the L$_p$ loss, the $p$th power of the $p$-norm of the inputs and restorations.
However, as we shall see below, this simple loss function does not preserve important features in cloud images.

RICC instead adapts the loss function used in the shifted transform invariant autoencoder of Matsuo et al.~\cite{Matsuo2017TransformIA} to obtain a loss function $L$ that
combines both a rotation-invariant loss, $L_{\text{inv}}$,
to learn the rotation-invariance needed to map different orientations of otherwise identical input images into an uniform orientation, and
a restoration loss, $L_{\text{res}}$, to learn
the spatial information needed to restore structural patterns in inputs with high fidelity:
\begin{equation}\label{eq:ri_loss}
    L = \lambda_{ \text{inv}} L_{\text{inv}} + \lambda_{ \text{res}} L_{\text{res}},
\end{equation}
where the scalars $\lambda_{\text{inv}}$ and $\lambda_{\text{res}}$ are the weights of the respective loss terms. We describe the two loss terms in turn.

\textit{Rotation-Invariant Loss}:
Matsuo et al.~\cite{Matsuo2017TransformIA} define a transform variance term that sums,
over both the minibatch $S$ and a set of shift parameters,
the differences between restored input and restored translated input:
\[
     L_{\text{matsuo}}(\theta) = \sum_{x \in S}\sum_{T \in {\cal T}} {|| D_{\theta}(E_{\theta}(x)) - D_{\theta}(E_{\theta}(T(x)))||^2_2},
\]
where ${\cal T}$ is a set of shift transform operators,
each of which translates objects in an input by a different shift vector

We modify their formulation
by replacing their shift transform operator $T$
with a scalar rotation operator $R$ and
by computing the average, rather than the sum, of the differences for each set of $N$ rotated images. The loss term is then:
\begin{equation}\label{tinv}
     L_{\text{inv}}(\theta) = \frac{1}{N}\sum_{x \in S}\sum_{R\in {\cal R}} {|| D_{\theta}(E_{\theta}(x)) - D_{\theta}(E_{\theta}(R(x)))||^2_2},

\end{equation}
where ${\cal R}$ is a set of rotation operators,
each of which rotates an input by a different angle

$L_{ \text{inv}}$ computes, for each image in a minibatch,
the difference between the restored original and all restored rotations.
Thus, the minimization of Equation~(\ref{tinv}) by the optimization process
produces similar latent representations for an image, regardless of its orientation.

The second modification facilitates tuning of the $\lambda$ parameters
(see Appendix~\ref{sec:valid_cloud})
by balancing the rotation-invariant loss term,
which is now a sum of differences over the minibatch $S$,
and the restoration loss term,
which as we define next, computes the sum of minimum differences over the minibatch.

\begin{comment}
\textit{Rotation-Invariant Loss}:
Matsuo et al.~\cite{Matsuo2017TransformIA} minimize the sum of differences between decoder output and the restored image with translated inputs.
We replace the shift transform operator $T_{\theta_i}$ used in Matsuo et al.,
an operator that translates the position of objects in an input, with a rotation operator. This involves two modifications.
First, we replace the shift vector $\theta_i$ for the $i$-th shift parameters with a scalar operator $T$ that rotates an image by a supplied $\theta^{\circ}_i$.
\begin{equation}\label{tinv}
     L_{\text{inv}} = \frac{1}{N}\sum_{x \in S}\sum_i {|| \hat{x} - D(E(T_{\theta_{i}}(x)))||^2_2}.
\end{equation}

Second, we compute the average of the sum of differences for respective images,
an approach that we believe facilitates tuning of the $\lambda$ parameters (see Appendix~\cite{sec:valid_cloud}) and gives us the intuitive balance of rotation-invariance and restoration loss.
Recall that Equation~(\ref{tinv}) computes the sum of differences,
whereas the restoration loss term, presented next, computes the sum of minimum differences among images in a minibatch.
Considering that the combination of two $\lambda$ parameters is critical in learning, as discussed in the next subsection,
the rotation-invariant and restoration loss terms should be computed over the minibatch average.

$L_{ \text{inv}}$, as defined by Equation~(\ref{tinv}), computes the difference for various rotated inputs for each image in a minibatch.
Thus, the combination of encoder and decoder during the optimization minimizes Equation~(\ref{tinv}), producing similar latent representations regardless of input rotation.
\end{comment}

\textit{Restoration Loss}: To enable a network to learn spatial substructure in images, we also adapt the restoration error term of Matsuo et al.~\cite{Matsuo2017TransformIA}
to minimize the difference between the restoration, $D_{\theta}(E_{\theta}(x))$,
of an original image $x$, and the original image when rotated by an operator $R \in {\cal R}$,
so that optimization obtains values $\theta$ that preserve spatial structure
in the input while allowing for transformation to a canonical orientation.
\begin{equation}\label{tres}

    L_{\text{res}}(\theta) = \sum_{x \in S} \displaystyle \min_{R \in {\cal R}} {|| R(x) - D_{\theta}(E_{\theta}(x)) ||^2_2} .

\end{equation}

Matsuo et al.~\cite{Matsuo2017TransformIA} also include a sparsity term, with the goal of minimizing the number of neurons required to achieve translation invariance in their latent representation. For simplicity of optimization, we exclude that term.

\subsection{Clustering Method}\label{sec:HAC}

The final step in our method development is to cluster the latent representations produced by the encoder for a set of cloud images.
We apply hierarchical agglomerative clustering (HAC)~\cite{Johnson1967HAC} to merge pairs of clusters from bottom to top, minimizing at each step the linkage distance among merging clusters.
We use Ward's method~\cite{Ward1963HierarchicalGT},
which minimizes the variance of merging clusters, as the linkage metric.
Suppose that we have two clusters $C_A$ and $C_B$ containing data points $X = \{x_{i_1}, \cdots, x_{i_N}\}$ for cluster $i \in \{A,B\}$. HAC with Ward's method computes the linkage distance as:
\begin{equation}
    d(C_A, C_B) = E(C_A \cup C_B ) - E(C_1) - E(C_2),
\end{equation}
where $E(C_i) =\sum_{x \in C_i} (d(x, c_i))$ for the centroid $c_i = \sum_{X \in C_i} \frac{x}{|C_i|}$, denotes the sum of the weighted square distances between all data points in the cluster and the centroid $c_i$.

We use HAC as our clustering method because its initialization strategy of repeatedly merging data points has been shown to give more stable results than other methods that use random starting centroids~\cite{macqueen1967kmeans}.
We have shown in previous work~\cite{Jenkins2019DevelopingUL} that HAC clustering results outperform those obtained with other common clustering algorithms.

\subsection{Criterion 1: Physically Reasonable Clusters}\label{sec:phys}

We expect the clusters produced by a clustering method to be \emph{physically reasonable}, with scientifically relevant distinctions.
To this end, we define a test that
examines quantitative differences among cloud physics parameters.
Results are in Section~\ref{sec:physicalreasonableness}.

\textbf{Test \ref{test11}: Cloud physics parameters.}\mylabel{test11}{1}
We define reasonableness in terms of whether each cluster produced by the unsupervised clustering method is associated with distinct distributions of four selected retrieved cloud physics parameters from the MOD06 product:
cloud optical thickness (COT), cloud top phase (CPH), cloud top pressure (CTP), and cloud effective radius (CER).
We select COT and CTP because these two parameters are used for ISCCP cloud classification to evaluate compatibility between novel types of clouds and the established ISCCP cloud classes~\cite{isccp1991}.
We select CPH and CER in order to consider differences in cloud properties that are not considered in the nine ISCCP classes.
For instance, comparing CTP and CPH allows us to determine whether high CTP values and the CPH liquid phase appear concurrently in the same cluster, as we would expect, because high CTP represents lower altitude clouds for which cloud particles should be in the liquid phase.
We expect our clustering system to learn complex combinations of radiances, enabling out trained autoencoders to reflect information from more physical variables than does ISCCP's classification.

We apply the reasonableness test as follows.
We first verify that the values for each parameter are not randomly distributed,
which would contradict cloud physics.
Then, for each parameter,
we examine whether different clusters show different distributions.
We do this by computing for each cluster pair the inter-cluster correlation
and then computing the median of the resulting inter-cluster correlation coefficients.
If for at least one of the four parameters this median is less than 0.6, a cutoff commonly used to indicate no strong correlation, then we conclude that the clusters are indeed grouping patches based on physical properties.

Note that we train our autoencoders only on the MOD02 input radiances,
not on the MOD06 COT, CPH, CTP, and CER parameters.
Thus, this test determines whether our training and clustering process is able to embed into the latent representation the distinctions that are recorded by the MOD06 parameters.
Recall from Section~\ref{sec:modis} that there are six radiance bands.
Three of these bands (6, 7, 20, at 1.6, 2.1, 3.7 $\mu$m, respectively) are used in the algorithm that estimates cloud optical properties (e.g., cloud optical thickness and effective radius), and the other three (28, 29, 31, at 7.3, 8.5, 11 $\mu$m) are used to separate high and low clouds and to detect cloud phase.

\subsection{Criterion 2: Spatial Distribution}\label{sec:test:spatial}
We expect the clusters produced by the unsupervised clustering method to capture information on cloud \emph{spatial distributions}. This means that they should not be reproducible by using only mean properties over the target area.
To this end, we define three tests, with results in Section~\ref{sec:spatialinfotest}.

\textbf{Test 2.1: Spatial coherence.} \mylabel{test21}{2.1}
We evaluate whether the clusters produced by an autoencoder demonstrate more spatially coherent assignments than those obtained by clustering patch-mean cloud parameters.
When clustering without an autoencoder,
we apply HAC to the patch-mean values of COT, CTP, CPH, and CER.

\textbf{Test 2.2: Smoothing.}\mylabel{test22}{2.2}
We examine how the cluster assignments for cloud images change when we alter the spatial resolution of the images via smoothing.

\textbf{Test 2.3: Scrambling.} \mylabel{test23}{2.3}
We examine how cluster assignments scramble pixels in patches so as to remove spatial patterns while preserving the distribution of values.

If the cluster assignments do \emph{not} change in the latter two tests,
we conclude that our autoencoder is not learning spatial information,
because the encoder generates similar representations when the input images
are transformed to remove spatial information.

\paragraph{Adjusted mutual information (AMI) score}
We use the AMI score~\cite{Nguyen2014AMI},
which adjusts the mutual information (MI) score to account for chance,
to measure the extent to which cluster assignments agree for inputs of different spatial resolutions.
Given two clustering assignments $U$ and $V$, the AMI is computed as:
\begin{equation}
    \text{AMI}(U,V) =\frac{\text{MI}(U,V)-\mathbb{E}(\text{MI}(U,V))}{\text{avg} \{H(U), H(V)\}- \mathbb{E}(\text{MI}(U,V)) } ,
\end{equation}
where $H(\cdot)$ depicts entropy, which formally defines with probability $P(u)$ as
\begin{equation}
    H(U) = -\displaystyle \sum_{u\in U} P(u) \log{}P(u)
\end{equation}
and $\text{MI}(U,V)$ is the mutual information between clustering assignments $U$ and $V$, as determined by their joint distribution $P(u,v)$ and their respective probabilities $P(u)$ and $P(v)$:
\begin{equation}
    \text{MI}(U,V) = \displaystyle \sum_{u\in U} \displaystyle \sum_{v\in V} P(u,v) \log{}\frac{P(u,v)}{P(u) P(v)} .
\end{equation}

The AMI of two sets of cluster assignments is 1 if the assignments match perfectly,
regardless of the assigned labels, and 0 if there is no match.
If the AMI score between the clustering results obtained for two different spatial resolutions is low, we conclude that the trained autoencoder has learned spatial information in its latent representation;
if the agreement score is high, we conclude that it
has failed to capture spatial patterns and thus is likely instead encoding information
about the distribution of pixel values in input images.

\paragraph{Smoothing test implementation}
Fig.~\ref{fig:smooth} provides the pseudo code for the smoothing test.
We use the \texttt{Test} dataset described in Section~\ref{sec:testdata},
which contains \num{2000} holdout patches unseen during training.
Let $k$ be the kernel size used to convolve the image with a boxcar filter to produce the smoothed version.
The smoothing process computes an average value for each pixel region in a patch.
For instance, for $k = 2$, it sets each pixel
at $(i,j)$, $(i+1,j)$, $(i,j+1)$ and $(i+1,j+1)$,
for $i \in \{1, 3, \cdots N-1\}$ and $j \in \{1, 3, \cdots N-1\}$, where $N$ is the patch size, to be the average value of those pixels,
leaving any remaining border pixels
(e.g., if $N=5$ and $k=2$, those with $i=5$ or $j=5$) unchanged.

Let $P_k$ be the patches obtained when each of
the holdout patches $X_{\text{holdout}}$ is smoothed over $k \times k$ pixels.
We then encode each smoothed patch in $P_k$ with the trained encoder, giving $Z_k$ as the corresponding set of latent representations, and apply HAC to those latent representations to obtain a set of 12 clusters $C_k$.
(We work with 12 clusters here as we have found that AMI scores typically stabilize
for 10 or more clusters.)
Lower AMI scores then indicate that the autoencoder has mapped different spatial structures within input images into different latent representations.
Finally we determine agreement between $C_1$, obtained from the unsmoothed $X_{\text{holdout}}$, and $C_k$ by computing AMI$(C_1, C_k)$.
We perform this process for each $k\in\{2,\cdots,9\}$.

\begin{figure}[htbp]
\centering
\hrule
\vspace{1ex}
\begin{minipage}[c]{0.70\textwidth}
\begin{singlespace}
\begin{tabbing}
\=aaa\=aaa\=aaa\=\kill
\> \blue{\# Smooth a single image}\\
\> \texttt{smooth(x, k):}\\
\>\> \texttt{p$_{\{(i,j),(i,j+1), ..., (i+k-1,j+k-1)\}}$}\\
\>\>\>\> \texttt{$\leftarrow$ Average(x$_{\{(i,j),(i,j+1), ..., (i+k-1,j+k-1)\}}$}\\
\>\> \texttt{return(p)}\\
\\
\>\blue{\# Compare cluster assignments for different kernel sizes}\\
\>\texttt{for kernel size k in 1..9:}\\
\>\> \blue{\# Smooth each patch for kernel size $k$}\\
\>\> \texttt{P$_k$ = \{ smooth(x, k) for x in X$_{\text{holdout}}$ \}}\\
\>\> \blue{\# Encode each smoothed and scrambled patch}\\
\>\> \texttt{Z$_k^{sc}$ = \{ encode(x) for x in P$_k$ \}}\\
\>\> \blue{\# Cluster resulting latent representations}\\
\>\> \texttt{C$_k^{sc}$ = Cluster(P$_k$, \#cluster)}\\
\>\> \blue{\# Determine agreement between C$_1$ and C$_k$}\\
\>\> \texttt{compute AMI(C$_1$, C$_k$)}
\end{tabbing}
\end{singlespace}
\end{minipage}
\begin{minipage}[t]{0.35\textwidth}
\includegraphics[width=\linewidth,trim=198mm 57mm 53mm 135mm,clip]{./Figures/smooth.png}
\end{minipage}
\vspace{1ex}
\hrule
\vspace{-1ex}
    \caption{Pseudocode for the smoothing protocol. The \texttt{smooth} function computes an average over a local window $k \times k$, as depicted in the figure.
    }
    \label{fig:smooth}
\end{figure}

\paragraph{Scrambling test implementation}
Fig.~\ref{fig:scramble} presents the scrambling test protocol in pseudo code form.
As with the Smoothing Test, we compare the cluster assignments that result when the autoencoder is trained
with different parameters and
on images that are smoothed with different kernel sizes, $k\in\{1,2,\cdots,9\}$.
However, in this test, we also scramble the pixels of each image after smoothing
by applying a random permutation to the smoothed pixels,
a process that we repeat (with the same random permutation) for each channel.

Again, suppose that we use holdout patches $X_{\text{holdout}}$ in evaluating the smoothing protocol.
For each kernel size $k\in\{1,2,\cdots,9\}$,
we smooth each image in $X_{\text{holdout}}$ by $k$
and then scramble its pixels,
giving $P_k$.
We then compute the latent representation on $Z^{sc}_k$ by encoding $P^{sc}_k$ with a trained encoder,
and cluster the latent representations obtained for the different images to obtain clusters $C^{sc}_k$.
We assess the agreement of $C_k$ and $C^{sc}_k$
by computing AMI$(C_k, C^{sc}_k)$.
\begin{figure}[htbp]
    \centering
\hrule
\vspace{1ex}
\begin{minipage}[c]{0.7\textwidth}
\begin{singlespace}
\begin{tabbing}
\=aaa\=aaa\=aaa\=aaa\=\kill
\> \blue{\# Scramble a single image}\\
\> \texttt{scramble(x):}\\
\>\> \texttt{indicesA = [(1,1), (1,2), ...,}\\
\>\>\>\> \texttt{(height(x), width(x))]}\\
\>\> \texttt{indicesB = random\_shuffle(indicesA)}\\
\>\> \texttt{for (i,j),(m,l) in (indicesA,indicesB):}\\
\>\>\> \texttt{p$_{\text{(i,j)}}$ = x$_{\text{(l,m)}}$}\\
\>\>\> \texttt{p$_{\text{(l,m)}}$ = x$_{\text{(i,j)}}$}\\
\>\> \texttt{return(p)}\\
\\
\>\blue{\# Compare cluster assignments for different kernel sizes}\\
\>\texttt{for kernel size k in 1..9:}\\
\>\> \blue{\# Smooth each patch for kernel size $k$: see Fig.~\ref{fig:smooth}}\\
\>\> \texttt{P$_k$ = \{ smooth(x, k) for x in X$_{\text{holdout}}$ \}}\\
\>\> \blue{\# Scramble each smoothed patch}\\
\>\> \texttt{P$_k^{sc}$ = \{ scramble(x) for x in P$_k$ \}}\\
\>\> \blue{\# Encode each smoothed and scrambled patch}\\
\>\> \texttt{Z$_k^{sc}$ = \{ encode(x) for x in P$_k^{sc}$ \}}\\
\>\> \blue{\# Cluster resulting latent representations}\\
\>\> \texttt{C$_k^{sc}$ = Cluster(Z$_k^{sc}$, \#cluster)}\\
\>\> \blue{\# Determine agreement between C$_1$ and C$_k^{sc}$}\\
\>\> \texttt{compute AMI(C$_1$, C$_k^{sc}$)}
\end{tabbing}
\end{singlespace}
\end{minipage}
\begin{minipage}[t]{0.3\textwidth}
\includegraphics[width=\linewidth,trim=202mm 5mm 38mm 152mm,clip]{./Figures/scramble.jpg}
\end{minipage}
\vspace{1ex}
\hrule
\vspace{-1ex}
    \caption{Pseudocode for the scrambling protocol.
    The scramble function shuffles the values of randomly selected pixel pairs,
    as depicted in the figure.
    }
    \label{fig:scramble}
\end{figure}

\subsection{Criterion 3: Separable Clusters}\label{sec:test:separable}
We expect the clusters produced by an unsupervised clustering method to be \emph{separable}:
(i.e., both cohesive in latent space and separated from each other).
We define one test, with results in Section~\ref{sec:results:sep}.

\textbf{Test \ref{test31}: Spatial organization.}\mylabel{test31}{3}
We use t-distributed Stochastic Neighbor Embedding (t-SNE)~\cite{Maaten2008VisualizingDU} (see Appendix~\ref{sec:tSNE} for formulation)
to examine the spatial organization of the latent representation when projected onto a two-dimensional map, and thus to determine whether similar (dissimilar) clusters, as defined by the physical parameter distributions, are projected to closer (more distant) locations in the embedding space.

\subsection{Criterion 4: Rotation Invariance}\label{sec:test:rotate}

We expect the classes produced by an unsupervised clustering method to be \emph{rotationally invariant},
(i.e., to place a cloud image into the same cluster regardless of its orientation).
regardless of the orientation of the image in which they appear.
We define two tests of increasing difficulty, with results presented in Section~\ref{sec:spatialinfotest}.

\textbf{Test 4.1: MNIST rotation-invariance.} We evaluate rotation-invariance on simple images containing well-defined structures, namely, the MNIST database~\cite{deng2012mnist}.\mylabel{test41}{4.1}

\textbf{Test 4.2: Multi-cluster rotation-invariance.} We evaluate whether our RI autoencoder groups similar types of clouds into the same cluster regardless of image orientation.\mylabel{test42}{4.2}

\paragraph{MNIST rotation-invariance}
This test evaluates rotation-invariance on simple images containing well-defined structures: specifically, on images from the MNIST database~\cite{deng2012mnist}.
This database of \num{70000} images of handwritten digits,
partitioned into an \num{60000}-image training set and a \num{10000}-image test set,
is commonly used for training image processing systems~\cite{deng2012mnist}.
Each image is a 28$\times$28 array of 8-bit gray-scale levels,
plus an associated integer label.

We train the autoencoder on the MNIST training set,
apply the trained autoencoder to images from the test set,
and evaluate whether the resulting outputs
reproduce the input images with mapping to a single canonical orientation.
Specifically, we sample 40 images for every digit class,
create four replicas of each image,
and rotate each of the resulting 10$\times$40$\times$4=1600 images
at random.
We then verify whether
the 40$\times$4=160 outputs for each digit class map to a consistent orientation.

\paragraph{Multi-cluster rotation-invariance}
We use the 2000 \texttt{Test} patches (see Section~\ref{sec:testdata}) not considered during training as holdout patches.
We make 11 copies of each patch in this set.
We rotate every 30$^{\circ}$; thus, the ideal result should return the same cluster label for both the original patches and the rotated copies.
We then implement HAC clustering for from \num{4} to \num{2000} clusters.
The AMI score should be close to \num{1} for \num{2000} clusters, because our \texttt{Test} dataset has \num{2000} patches, meaning that each patch and its replications can be placed in an unique cluster.

\subsection{Criterion 5: Stable Clusters}

We expect the clusters produced by an unsupervised clustering method to be \emph{stable}, (i.e., to be identical or at least similar to those produced) when the method is trained with different subsets of the training data.
Cluster stability is a difficult problem,
and there is not yet agreement in the community as to how to define it~\cite{ben2006sober,von2010clustering}.
Thus, we do not consider this criterion in this study.

\subsection{Results for Criterion 1: Physical Reasonableness}\label{sec:physicalreasonableness}
\begin{figure*}
    \centering

     \includegraphics[width=\linewidth,]{Figures/hist_Imgaes_ri_phys_12_CUMULO.pdf}
    \caption{\emph{Test~\ref{test11}, cloud physics parameters, applied to RI autoencoder}. The first four columns show histograms of the patch-mean values of four derived cloud physical parameters---COT (no unit), CPH (clear sky, liquid, or ice), CTP (hPa), and CER ($\mu$m)---for each of the 12 clusters produced by the RI autoencoder for the \texttt{TestCUMULO} dataset. The last two columns show, for the two patches closest to the centroid, the raw visible image (band 5), with white to grey indicating cloud and black either ocean or land without cloud.
    We make the following observations about these results.
    1) The distributions of each parameter \emph{within} clusters do not contradict cloud physics (i.e., they are not random) and furthermore show distinct differences \emph{across} clusters, a result supported quantitatively by median inter-cluster correlation coefficients, as discussed in the text.
    2) In clusters for which the COT histogram shows more larger values, the sample images are cloudier, as we would expect.

    3) Clusters \#7 and \#8 both have roughly similar proportions of liquid and ice phases, but differ in COT and CTP, which in \#8 are wider and narrower, respectively, than in \#7. It seems likely that \#7 is a mixture of two cloud types--low altitude liquid and high altitude ice crystal clouds---while \#8 aggregates more than two cloud types.

    }
    \label{fig:physicalreasonable}
\end{figure*}

\begin{table}
\centering
\caption{Test~\ref{test11}, median of correlation coefficients for four cloud physics parameters and for RI, RA, and NRA autoencoders. Correlation coefficients $<$0.6 in boldface. AE=autoencoder.}\label{tab:stats:physicalreasonable}
\centering
\begin{tabular}{|p{0.6cm}| C{2.6cm}|l*{3}{C{.7cm}}|}
\hline

 AE & {Parameter(s)} & COT &  CPH & CTP & CER \\ \hline\hline
 RI & $\lambda_{\text{inv}}, \lambda_{\text{res}} = 32,80$ & 0.842 & \textbf{0.591} & \textbf{0.461} & 0.692\\
 RA& $\lambda = 1 $ & 0.835 & \textbf{0.588} & \textbf{0.438} & 0.661\\
 NRI& -- & 0.829 & 0.629 & \textbf{0.425} & \textbf{0.484}  \\  \hline
\end{tabular}
\end{table}

\begin{figure}
    \centering
    \includegraphics[width=\textwidth,]{Figures/dendrogram2_20.pdf}
    \caption{
    Dendrogram representation of the HAC tree structure generated by RICC for the \texttt{TestCUMULO} dataset of January 1, 2008 with 12 clusters. Cluster numbers are shown at the dendogram leaf nodes; the column below each number shows
     MOD02 raw visible (band 5) images for the 20 patches closest to the centroid of that column's cluster,
     ordered according to their distance to the centroid, with the topmost patch being the closest.
     (Note that these clusters are the same as in Fig.~\ref{fig:physicalreasonable}, and
     the top two patches for each cluster are identical to those shown in last two columns of Fig.~\ref{fig:physicalreasonable}.)
    White-to-grey colors indicate cloud pixels; black denotes a non-cloud (ocean or land) pixel; and patches with a brown frame contain land clouds.
    We see that RICC clusters differentiate between ocean clouds (cluster \#6 is 96.6\% ocean cloud patches) and land clouds (cluster \#10 is 61.0\% land cloud patches), a differentiation based on texture and radiances that may capture different cloud formation processes.
    Furthermore, the example patches shown for the different clusters show evidence of rotation invariance, of grouping based on similarity of texture, and
    distinct differences in the density of cloud pixels and in circular/uniform/streak structure.
    Comparison with Fig.~\ref{fig:physicalreasonable} shows that clusters with sparse clouds (\#6, \#12) have lower standard deviation in their optical thickness than do clusters with homogeneous texture (\#1, \#2, \#3, \#5).
    }
    \label{fig:dendrogram}
\end{figure}

We apply Test~\ref{test11}, introduced in Section~\ref{sec:phys},
to evaluate the physical reasonableness of the cluster labels assigned by our trained autoencoders.
Specifically, we investigate whether cloud physical parameters (i.e., the retrieved physical parameters computed by MODIS scientists from cloud properties), are both consistent within clusters and distinct among clusters. We use the largest test dataset, \texttt{TestCUMULO}, for this purpose.

Fig.~\ref{fig:physicalreasonable} shows, for each of 12 clusters generated by RICC for the \texttt{TestCUMULO} dataset, the distributions of patch-mean values for four derived physical parameters, plus the two closest patches to the cluster's centroid.
We work with 12 clusters because we have seen previously that when clustering \texttt{Phys}, AMI scores stabilize~\cite{kurihana2019cloud} for cluster counts of 10 and higher.
Examination of these distributions shows that clusters are clearly associated with meaningful physical cloud attributes.
For example, high-altitude cirrus clouds have low cloud top pressure and are composed of ice crystals,
features that are captured in clusters \#3 and \#10, which both have a dominant ice phase, lower peak values for cloud top pressure (CTP), and relatively larger cloud effective radius (CER).
Cluster \#3 shows similar distributions for cloud optical thickness (COT) and CPH to cluster \#10,
but differs in CTP (narrower range) and CER (a single, smaller peak),
suggesting subtle physical differences.
As a second example of parameter value distributions having a reasonable association in terms of cloud physics, stratocumulus are low-altitude clouds with liquid droplets and a middle range of optical thickness: features seen in cluster \#9.

As discussed in Section~\ref{sec:phys}, we use the median inter-cluster correlation as a quantitative measure of the diversity of parameter distributions across clusters.
Our results are in Table~\ref{tab:stats:physicalreasonable}.
For the RI autoencoder (first row), this measure has values 0.591 and 0.461 for the CPH and CTP parameters, respectively,
suggesting that these parameters are differently distributed in different clusters, as we desire.
For the RA and NRI autoencoders (rows 2 and 3), this measure is also below 0.6 in the case of the CTP parameter, with values of 0.438 and 0.425, respectively.
In summary, we conclude that all three autoencoders, while trained on data that contain only information about cloud textures and radiances, produce clusters that reflect intrinsic differences in physical parameters.

We can also see in Fig.~\ref{fig:physicalreasonable} a clear association between COT distributions and displayed patches.
In clusters \#1, \#2, and \#3, the displayed patches show dense clouds (i.e., dominant white and grey pixels) and COT has a long tail distribution;
in clusters \#6 and \#12, the displayed patches show sparse clouds (i.e., mostly black pixels) and COT has a narrow distribution and smaller peak values.

To examine cluster-texture associations in more detail, we show in Fig.~\ref{fig:dendrogram} a hierarchical \emph{dendrogram}, created by merging pairs of clusters that are close in latent space. In the resulting graph, the height of each node is proportional to the intergroup dissimilarity between its two offspring subgraphs or leaf nodes.
We also show, below each cluster number, images for the 20 patches that are closest to that cluster's centroid.
We find that clusters with dense cloud textures (\#1, \#2, \#3, \#5) also have large COT values with large standard deviations, while
clusters \#6 and \#12 again show lower COT peak values and standard deviation and example patches with primarily sparse clouds over ocean.

Interestingly, while clusters \#7 and \#8 have similar distributions in Fig.~\ref{fig:physicalreasonable}, they are some distance apart in Fig.~\ref{fig:dendrogram} and show different textures in the example images.
To examine this phenomenon further, we computed the percentage of ice and liquid pixels within each patch in the two clusters.
We find that while cluster \#8 has a broad range [0\%--100\%] of ice and liquid percentage, in cluster \#7 most patches have $>$80\% of either ice or liquid pixels, suggesting that cluster \#8 amalgamates multiple cloud types with different physical and texture features.

\subsection{Results for Criterion 2: Spatial Distribution}\label{sec:spatialinfotest}

\begin{figure*}
    \centering
    \includegraphics[width=\textwidth,trim=0mm 28.mm 0mm 32.5mm,clip]{Figures/test2_coherence_label2.pdf}
    \caption{\emph{Test~\ref{test21}: Spatial coherence, applied to NRI, RA, and RI autoencoders}.
    On the same MODIS swath image (\texttt{Phys}, with 493 patches), we show:
    Left, center: Clusters produced by NRICC, RACC, and RICC, respectively;
    right: Clusters produced by HAC applied to patch-mean values of COT, CTP, CWP, and CER cloud physical parameters.
    The background raw visible image (band 1) is provided for context only; it is not used in training the autoencoders.
    In each case, 12 clusters are produced.
    The color bar shows cluster number; white means no data due to orbital coverage gaps or invalid data, and black means patches with $<30\%$ cloud pixels. All three autoencoder-based approaches produce spatially coherent cluster assignments.}
    \label{fig:coherence}
\end{figure*}
We apply the three tests described in Section~\ref{sec:test:spatial} to the NRI cloud clustering (NRICC), RA cloud clustering (RACC), and RICC frameworks.

\paragraph{Results for Test \ref{test21}, Spatial Coherence}
We use the 493-patch MODIS swath \texttt{Phys} to investigate spatial coherence.
We show in Fig.~\ref{fig:coherence} four different clusterings of this dataset:
three with (left, centers) and one without (right) an autoencoder.
We see that all three autoencoder-based approaches
(none of which is aware of patch locations in the swath)
produce more spatially cohesive cluster assignments than does a clustering based only on mean physics parameters.

The \texttt{Phys} dataset used here to verify spatial coherence is disjoint from the \texttt{TestCUMULO} dataset used in Section~\ref{sec:physicalreasonableness} to verify physical reasonableness. Thus as a further test,
reported in Appendix~\ref{sec:appned_c1physdata}, we also verify that the clusters produced by RICC for \texttt{Phys} are physically reasonable.

\paragraph{Results for Test \ref{test22}, Spatial Resolution}
This test examines whether and how cluster assignments change when we alter the spatial resolution of images via smoothing.
Table~\ref{tab:spacialtest} summarizes the results obtained.
Each row in the table shows results for a different autoencoder,
while the columns correspond to different kernel sizes.
The notation tells that the horizontal resolution of the inputs for the RI autoencoder is 32$\times$32 and that of inputs for the RA and NRI autoencoders are 128$\times$128, so that we need to apply different kernel sizes to smooth the same percentage of pixels in the images.
We see that the best RI autoencoder,
obtained with $\lambda_{\text{res}}$~=~80, $\lambda_{\text{inv}}$~=~32,
achieves the lowest agreement score, 0.577,
showing that it is indeed capturing spatial patterns.

\begin{table*}
\small\centering
\caption{Spatial distribution tests \ref{test22} and \ref{test23}, applied to NRI, RA, and RI autoencoders.
AMI scores are given for different kernel sizes, with the largest value in each column is in boldface.
}\label{tab:spacialtest}
\scriptsize\centering
\begin{tabular}{|p{1.2cm}| C{2cm}|l*{7}{C{.4cm}}|l*{8}{C{.4cm}}|}
\hline
 \multicolumn{2}{|c|}{}   &  \multicolumn{8}{c|}{Test \ref{test22}: Smoothing} &  \multicolumn{9}{c|}{Test \ref{test23}: Scrambling} \\ \hline
 \multirow{2}{*}{{Autoencoder}} & \multirow{2}{*}{Parameter(s)}  &  \multicolumn{17}{c|}{Kernel size}  \\
 &  &2(8)&3(12)&4(16)&5(20)&6(24)&7(28)&8(32)&9(36) &1(1)&2(8)&3(12)&4(16)&5(20)&6(24)&7(28)&8(32)&9(36) \\ \hline\hline
 \multirow{3}{*}{RI} & $\lambda_{\text{inv}}, \lambda_{\text{res}} = 32,80$&0.630&\textbf{0.607}&0.664&\textbf{0.604}&\textbf{0.624}&\textbf{0.577}&\textbf{0.580}&\textbf{0.582}&0.636&0.612&\textbf{0.601}&\textbf{0.612}&\textbf{0.584}&\textbf{0.619}&0.600&0.625&\textbf{0.589} \\
 & $\lambda_{\text{inv}}, \lambda_{\text{res}} = 0,80$ &0.669&0.665&0.675&0.631&0.661&0.624&0.597&0.611&\textbf{0.631}&0.629&0.637&0.641&0.626&0.678&\textbf{0.585}&\textbf{0.590}&0.603 \\
 & $\lambda_{\text{inv}},\lambda_{\text{res}} = 3.2,8$ &\textbf{0.626}&0.658&\textbf{0.640}&0.644&0.654&0.661&0.608& 0.610&0.642&0.624&0.627&0.646&0.608&0.633&\textbf{0.585}&0.606&0.593\\
 {RA}& $\lambda = 1 $ &0.743&0.748&0.705&0.648&0.760&0.696&0.727&0.691&0.709&0.704&0.722&0.741&0.661&0.770&0.676&0.758&0.749  \\
 {NRI}& -- & 0.645&0.657&0.692&0.658&0.673&0.693&0.674&0.627&0.654&\textbf{0.610}&0.652&0.618&0.637&0.623&0.668&0.702&0.642  \\  \hline
\end{tabular}
\end{table*}

\paragraph{Results for Test \ref{test23}, Scrambling}
This text examines how cluster assignments change when we scramble image pixels so as to remove spatial patterns while preserving the distribution of pixel values.
We evaluate clustering agreements for inputs of original and scrambled patches.
A low agreement score indicates that the trained autoencoder is encoding information about spatial patterns in the latent representation;
a high agreement score shows that it is not.
Table~\ref{tab:spacialtest} shows results.
We see that the RI autoencoder, under the optimal parameter combination, outperforms other autoencoders in learning spatial patterns,
achieving the lowest score, 0.584.

\subsection{Results for Criterion 3: Separable Clusters}\label{sec:results:sep}
\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{Figures/test3_sep_ri_phy_12.pdf}
    \vspace{-2ex}
    \caption{\emph{Test \ref{test31}, spatial organization, applied to RI autoencoder}. Top: t-SNE visualization of the latent representations of \texttt{Test} patches.
    Patches in each cluster are projected near to each other and distant from patches in other clusters.
    Bottom: Histograms of path-mean values of four derived cloud physics parameters---optical thickness, phase, cloud top pressure, and effective radius---for the three clusters circled in the top panel.
    (We select these three clusters to highlight the association between distance in 2D tSNE space and physical parameters; see Fig.~\ref{fig:physicalreasonablePhysdata} in Appendix~\ref{sec:appned_c1physdata} for histograms of the other nine clusters.)
    Closer clusters (\#2, \#4) demonstrate similar distributions in their four histograms than do distant clusters (e.g., \#2, \#3), indicating that RICC clusters reflect nonlinear interrelations among selected spectral bands.
    }
    \label{fig:separable}
\end{figure}
We apply Test~\ref{test31}, described in Section~\ref{sec:test:separable}, to RICC,
applying t-SNE to the latent representations produced by the RI autoencoder for the 493 patches in \texttt{Phys}.

Results are in Fig.~\ref{fig:separable}.
The upper panel shows the two dimensional t-SNE projection,
with each color representing one of 12 clusters.
We see that the clusters are, for the most part,
homogeneous and distinct,
indicating that clustering achieves good separability.
The lower panel provides histograms for four physical properties for clusters \#2, \#3, and \#4.
We choose to highlight these clusters here because clusters \#2 and \#3 capture the common physical features of high-altitude cirrus and stratocumulus, respectively (demonstrated in Appendix~\ref{sec:appned_c1physdata} in Fig.~\ref{fig:physicalreasonablePhysdata}),
and \#2 is one of the three clusters closest to \#4.
We see that the two highlighted clusters that are adjacent in the projected map (\#2 and \#4) have similar physical properties, whereas the first highlighted cluster (\#3), which is distant from both, has more dissimilar physical properties.
We observed in previous work a similar trend for NRI autoencoder on the same dataset~\cite{kurihana2019cloud}, concluding that NRICC and RICC system can approximate non-linear physical relationships across the selected six bands.

Recall that each cloud physical parameter provided in the MOD06 product is estimated via theoretically based algorithms~\cite{Ackerman2008CloudDW,baum12,steven17} that combine radiance and brightness values from spectral bands that are sensitive to physical features.
For example, bands 6 and 7 correspond to the atmospheric absorption range of H$_2$O, and thus are used for the computation of COT and CER~\cite{baum12}.
Similarly, the separation of the liquid and ice phases in CPH is computed from the brightnesses in bands 29 and 31,
due to those bands' sensitivity to the ice phase.

\begin{figure*}
	\centering
	\begin{minipage}{.33\columnwidth}
		\centering
		\subfloat[NRI autoencoder.\label{fig:mnist-img1}]{
		    \includegraphics[width=.9\textwidth]{Figures/mnist_NRI_7.pdf}
		}
	\end{minipage}%
	\begin{minipage}{.33\columnwidth}
		\centering
		\subfloat[RA autoencoder.\label{fig:mnist-img2}]{
		    \includegraphics[width=.9\textwidth]{Figures/mnist_clf_7.pdf}
		}
	\end{minipage}
	\begin{minipage}{.33\columnwidth}
		\centering
		\subfloat[RI autoencoder.\label{fig:mnist-img3}]{
		    \includegraphics[width=.9\textwidth]{Figures/mnist_RI_7.pdf}
		}
	\end{minipage}
	\caption{A single instance of the digit class `7' from MNIST (top row), four random rotations of that image (middle row), and restorations of those rotated images (bottom row).
    (a) The NRI autoencoder yields images that display no canonical rotation.
    (b, c) The RA and RI autoencoders both produce restored images in a uniform canonical rotation.}\label{fig:mnist_images}
\end{figure*}
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\columnwidth,trim=1mm 1mm 1mm 1mm,clip]{./Figures/multi_cluster.pdf}

    \vspace{-2ex}
    \caption{\emph{Test \ref{test42}, multi-cluster, applied to NRI, RA, and RI autoencoders}. Clustering agreement scores (AMI)

    on the \texttt{Test} dataset,
    for from 4 to 2000 clusters. The AMI curve for RICC converges at 0.9, meaning that RICC autoencoder produces rotation-invariant latent representation.
    }
    \label{fig:ri_multi}
\end{figure}
\subsection{Results for Criterion 4: Rotation-Invariance}\label{sec:multicluster}

We apply the two tests described in Section~\ref{sec:test:rotate}.

\paragraph{Results for Test \ref{test41}, MNIST Rotation-Invariance}
Fig.~\ref{fig:mnist_images} shows restorations of a digit image `7' as produced by three different autoencoders: the NRI autoencoder of Equation~(\ref{eq:cnri}); the best RA autoencoder with $\lambda$ = 0.1 (Fig.~\ref{fig:mnist-img2}); and the best RI autoencoder with $\lambda_{\text{inv}}$ = 10, $\lambda_{\text{res}}$ = 10 (Fig.~\ref{fig:mnist-img3}).
We see that in this MNIST test, the RA and RI autoencoders both produce outputs with a single canonical rotation, as well as preserving input patterns with high fidelity.

\paragraph{Results for Test \ref{test42}, Multi-cluster}
This test is designed to verify
whether RICC groups similar clouds into the same cluster regardless of image orientation.
We use the 2000 \texttt{Test} patches (see Section~\ref{sec:testdata}) as holdout patches.
We make 11 copies of each patch in our holdout set, with and without rotations.
We rotate every 30$^{\circ}$; thus, the ideal result should return the same cluster label for both the original patches and the rotated copies.
We then implement HAC clustering for from \num{4} to \num{2000} clusters.
We would expect the AMI score to be be close to \num{1} for \num{2000} clusters because our \texttt{Test} dataset has \num{2000} patches, meaning that each patch and its replications can be placed in an unique cluster.
Fig.~\ref{fig:ri_multi} plots AMI scores as a function of the number of clusters.
The AMI curve for the RICC (blue) converges to 0.9, indicating that the clustering result is agnostic to the orientation of clouds in the holdout set.
In contrast, the AMI curve for RACC (green) achieves 0.88 at number of cluster 128 but decreases to 0.64, telling us that the latent representation insufficiently learns spatial features in inputs, rendering clustering assignments at random among similar structure of cloud images.
We see that the agreement score for NRICC decreases as the number of clusters increases, suggesting that the NRI autoencoder's latent representation is influenced by the rotation of images even if they are for the same types of clouds.
Therefore, we conclude that RICC is functional to process a real image dataset when input orientation does not matter for pattern recognition.

\section{RI Autoencoder Hyperparameter Search}\label{sec:valid_cloud}
The performance of the RI autoencoder on a particular training set is sensitive to the values assigned to the $\lambda$ parameters.
We formulate in Fig.~\ref{gridsearch_protocol} a grid search process for finding an optimal combination of $\lambda_{ \text{inv}}$ and $\lambda_{ \text{res}}$.
(We also test a number of learning rates, for 10$^{-4}$ to 10$^{-2}$, for stochastic gradient descent.)
This process proceeds as follows:

\begin{enumerate}
    \item Fix $\lambda_{\text{res}}$ and learning rate \emph{lr}. Set $\lambda_{ \text{inv}}$ = 0.
    \item Train the autoencoder to obtain a baseline trained model \textbf{A}; measure its restoration loss $L_{\text{res}}$, on a holdout set $X_{\text{holdout}}$:
    $L_{\text{res}}(\textbf{A}, X_{\text{holdout}})$.
    \item Set $\lambda_{\text{inv}}$ = 0.1.
    \item Train the autoencoder with the new $\lambda_{\text{inv}}$ to obtain a new trained model \textbf{B}; measure its restoration loss on the same holdout set, giving $L_{\text{res}}(\textbf{B}, X_{\text{holdout}})$.
    \item If the restoration loss of the newly trained model \textbf{B} is less than 20$\%$ larger than that of the baseline \textbf{A}, i.e., if $L_{\text{res}}(\textbf{B}, X_{\text{holdout}}) \leq 1.2 L_{\text{res}}(\textbf{A}, X_{\text{holdout}})$, then double $\lambda_{\text{inv}}$.
    \item Compare the results produced by \textbf{B} for images $x_i\in X_{\text{holdout}}$, each replicated $N$ times, and with each replicate rotated by a different $\theta$. Observe whether, for each image, all $\textbf{B}(\texttt{rot}(x_i, \theta))$ have the same canonical rotation.
    (In our implementation, we assume that \emph{N} = 12 and $\theta\in \{0, 30, \cdots, 330\}$.) We verified $\textbf{B}(\texttt{rot}(x_i, \theta))$ based on the cosine similarity and eye-ball observation.
    \item If for all $x_i \in X_{\text{holdout}}$, all $\textbf{B}(\texttt{rot}(x_i, \theta))$ are the same, terminate the search.
    \item Otherwise halve $\lambda_{\text{inv}}$.
\end{enumerate}
\begin{figure}[htb!]
    \begin{center}
        \includegraphics[width=\textwidth,trim=1mm 1mm 1mm 1mm,clip]{Figures/workflow_gridsearch.pdf}
    \end{center}
    \vspace{-2ex}
    \caption{Flowchart of the parameter search method used to determine values for $\lambda_{ \text{inv}}$ and $\lambda_{ \text{res}}$ for cloud images.}
    \label{gridsearch_protocol}
\end{figure}

\begin{figure*}
	\centering
	\begin{minipage}{.225\columnwidth}
		\centering
		\subfloat[Ratio of restoration losses\label{fig:gridsearch_lossratio}]{
		\includegraphics[width=\textwidth]{Figures/div_param_search_cloud1.pdf}}
	\end{minipage}%
	\hspace{5mm}
	\begin{minipage}{.225\columnwidth}
		\centering
		\subfloat[Similarity of restored images. \label{fig:gridsearch_cossim}]{\includegraphics[width=\textwidth]{Figures/div_param_search_cloud2.pdf}}
	\end{minipage}%
	\hspace{5mm}
	\begin{minipage}{.45\columnwidth}
		\centering
		\subfloat[Training losses for different $\lambda$ combinations.\label{fig:gridsearch_loss_cloud}]{\includegraphics[width=\textwidth]{Figures/ri_loss_cloud.pdf}}
	\end{minipage}
	\caption{Results of grid search for RI autoencoder. (a) Ratio of the two restoration losses $L_{\text{res}}(\textbf{B}, X_{\text{holdout}})/ L_{\text{res}}(\textbf{A}, X_{\text{holdout}})$ for different $\lambda$ combinations. (b) Standard deviation of cosine similarity computed on the restoration images from RI autoencoder, when feeding various transformations rotated by $\{0^{\circ}, 30^{\circ},\cdots, 330^{\circ}\}$. We highlighted the optimal combination by a red circle. (c) Training losses for the optimal $\lambda$ combination (blue) and two suboptimal combinations (orange and green) examined in the grid search.  }\label{fig:gridsearch_result_all}
\end{figure*}

We then repeat this procedure for a new $\lambda_{\text{inv}}$ value.

As shown in Fig.~\ref{fig:gridsearch_result_all},
this grid search reveals that the optimal parameter combination is $\lambda_{\text{inv}}$ = 32 and $\lambda_{\text{res}}$ = 80 for \emph{lr} = 10$^{-2}$.
Other parameter combinations fail to satisfy the restoration error ratio criterion [$L_{\text{res}}(\textbf{B}, X_{\text{holdout}}) > 1.2 L_{\text{res}}(\textbf{A}, X_{\text{holdout}})$] (Fig~\ref{fig:gridsearch_lossratio}): i.e., the restoration loss in the newly trained model B produces restoration errors more than 20$\%$ larger than those of our baseline, and/or the rotation-invariant criterion: i.e., $\textbf{B}(\texttt{rot}(x_i, \theta))$ is not identical for all $\theta\in\{0, 30, \cdots, 330\}$ (Fig.~\ref{fig:gridsearch_cossim}).
Fig.~\ref{fig:gridsearch_lossratio} shows the ratio of restoration loss  $L_{\text{res}}(\textbf{B}, X_{\text{holdout}}) / L_{\text{res}}(\textbf{A}, X_{\text{holdout}})$ where hatched elements indicate $L_{\text{res}}(\textbf{B}, X_{\text{holdout}}) > 1.2 L_{\text{res}}(\textbf{A}, X_{\text{holdout}})$.
In Fig.~\ref{fig:gridsearch_lossratio}, navy elements are more likely to show at lower $\lambda_{\text{res}}$ ($\lambda_{\text{res}}=1, 10, 20$) and larger $\lambda_{\text{inv}}$ ($(\lambda_{\text{res}}, \lambda_{\text{inv}})  = (50,25.6), (80,51.2)$), suggesting that a larger $\lambda_{\text{res}}$ decreases the restoration error ratio, allowing for larger $\lambda_{\text{inv}}$ that satisfy the criterion.

To enable quantitative investigation of rotation-invariance,
we show in Fig.~\ref{fig:gridsearch_cossim} the standard deviation of the cosine similarities for outputs from the RI autoencoder as a function of $\lambda$ values.
This measure of the similarity of two vectors $X$ and $X^{\prime}$ is defined in terms of
the cosine of the angle between them:
\begin{equation}\label{eq:cossim}
    \text{Cosine similarity} = \frac{<X, X^{\prime}>}{\|X\| \cdot \| X^{\prime}\|} .
\end{equation}

Equation~(\ref{eq:cossim}) gives 1 when elements of two vectors are exactly matched.
As we assume $\texttt{rot}(x_i, \theta)$ for $\theta\in$ \{0, 30, $\cdots$, 330\}.), we computed the similarity metric among the output for $\theta = 0$ and the other angles, and then evaluated the standard deviation.
A standard deviation closer to 0 in Fig.~\ref{fig:gridsearch_cossim} means that cosine similarities in restored images with and without rotation obtain identical representation, which verifies the autoencoder maps images with different transformations into a single canonical orientation.
Studying the result of standard deviations, we see that increasing $\lambda_{\text{res}}$ needs increasing $\lambda_{\text{inv}}$ to achieve the rotation-invariant criterion, while increasing $\lambda_{\text{res}}$ leads to better-quality restorations.
Fig.~\ref{fig:gridsearch_loss_cloud} shows the evolution of training loss over iterations with different parameter values.
The lowest convergence of the blue line in Fig.~\ref{fig:gridsearch_loss_cloud} proves that the optimal combination ($\lambda_{\text{inv}}$ = 32, $\lambda_{\text{res}}$ = 80) achieves the lowest rotation-invariant loss of the combinations shown.
(We do not consider $\lambda$ parameter values higher than those shown,
as that causes the minimization to be unstable.)

We note that the selection of learning rate is also key to discovering the optimal configuration.
Learning rates lower than the 10$^{-2}$ considered here allow the network to take larger $\lambda$ values, which results in the latent representation not being able to achieve rotation-invariance.

\section{Dimensionality reduction technique}\label{sec:tSNE}
t-SNE is a probabilistic nonlinear dimensionality reduction technique that maps each data point in a high-dimensional space to a lower-dimensional point in such a way that, with high probability, similar data are placed near to each other and dissimilar data far apart.
Suppose that we have $N$ latent representations $Z = \{z_1, \cdots, z_i, z_j, \cdots, z_N\}$ produced by an autoencoder.
Let $P$ be a joint probability distribution in the high-dimensional input space,
and $Q$ a joint probability distribution in the low-dimensional projection space.
The t-SNE optimization minimizes the Kullback-Leibler (KL) divergence between $p_{j|i}$ and $q_{j|i}$.
The conditional probability $p_{j|i}$ for the M-dimensional latent representation produced by our autoencoders is:
\begin{equation}
    p_{j|i}  = \frac{\exp(-\| z_i - z_j\|^2/2\sigma_i^2)}{\displaystyle \sum_{k\neq i}\exp(-\| z_i - z_k\|^2/2\sigma_i^2) },
\end{equation}
where $\sigma_i$ denotes the variance of a Gaussian distribution for data point $z_i$,
while for a two-dimensional map:
\begin{equation}
    q_{j|i}  = \frac{ \left(1 +\| z^{\prime}_i - z^{\prime}_j\|^2 \right)^{-1}}{\displaystyle \sum_{k\neq l} \left(1 +\| z^{\prime}_l - z^{\prime}_k\|^2 \right)^{-1} }.
\end{equation}
The cost function to be minimized by gradient descent is then
\begin{equation}\label{tsne-formula}
    \text{KL}(P||Q) = \sum_i \sum_j p_{j|i} \log{}\frac{p_{j|i}}{q_{j|i}} .
\end{equation}
Note that this joint distribution assumes a Student t-distribution with one degree of freedom in the low-dimensional map.

\section{The NRI Autoencoder}\label{sec:CNRI}
The
\textit{non-rotation-invariant} (NRI) autoencoder~\cite{kurihana2019cloud}
introduces a more sophisticated loss function to address the reconstruction image issue shown in Fig.~\ref{fig:collapse}.
This loss function combines four metrics:
\begin{equation}\label{eq:cnri}
    L = L_1 + L_2 +  L_{\text{high-pass}} + L_\text{MS-SSIM}.
\end{equation}

The first two terms are the L1 and L2 loss, respectively, corresponding to $p = 1$ and $p = 2$ in Equation~(\ref{standardloss}).

The third term, a high pass filter to reduce noise in the reconstructed image,
is the high frequency error norm after passing data through a Sobel filter~\cite{sobel19683x3} to detect cloud edges in input images:
\begin{equation}
    \begin{split}
    L_{\text{high-pass}}(\theta)
    &=  \sum_{x \in S} | g_X(x) - g_X(D_{\theta}(E_{\theta}(x)))| \\
    & \quad + | g_Y(x) - g_Y( D_{\theta}(E_{\theta}(x)) ) |,

    \end{split}
\end{equation}
where $g_X, g_Y$ denotes the gradients on the X and Y axes of the input patch, respectively.

The fourth term is the multi-scale structure similarity index (MS-SSIM)~\cite{wang2003multiscale}, a multi-band version of the SSIM~\cite{wang2004image} index often used in computer vision to assess image similarity.
This constraint rewards the production of similar latent representations for similar textures in cloud images:
\begin{equation}
    L_\text{MS-SSIM}(\theta) = \sum_{x \in S} \frac{(2\mu_{x} \mu_{\hat{x}_{\theta}+C_1)} (2\sigma_{x \hat{x}_{\theta}}+C_2)}{(\mu^2_{x} + \mu^2_{\hat{x}_{\theta}} + C_1)(\sigma^2_{x} + \sigma^2_{\hat{x}_{\theta}}+C_2)  },

\end{equation}
where $\mu$ is the mean of the patch, $\sigma$ is its variance; $C_1$, $C_2$ are variables to stabilize the division; and
$\hat{x}_\theta$ represents $D_{\theta}(E_{\theta}(x))$.
We use $C_1 = (0.01\cdot L)^2$ and $C_1 = (0.03\cdot L)^2$, where $L$ denotes the dynamic range for the sample data, i.e., the difference between maximum and minimum pixel values.

The NRI autoencoder also has a slightly different network architecture: see Fig.~\ref{fig:cnri}.

\begin{figure}
	\centering
	\includegraphics[width=\columnwidth,trim=4mm 3mm 3mm 3mm,clip]{Figures/arch_res_label.pdf}
	\caption{The NRI encoder architecture is identical to that of Fig.~\ref{fig:network:ricc},
	except that it has two rather than three convolutional layers per block, and skip connections (the arrows that skip over the convolutional layers).}\label{fig:cnri}
\end{figure}

\section{The Rotation-Aware Autoencoder}\label{sec:RA}
The NRI autoencoder
produces different representations for images that differ only in their orientation.
This behavior is problematic for cloud clustering, because
cloud formation is driven not by wind direction but by mechanisms such as adiabatic or non-adiabatic cooling, convection, advection, and terrestrial effects.
Furthermore, satellites image clouds from different angles.
Thus, any particular type of cloud can occur in different orientations in images.

To address this problem, the \textit{rotation aware} (RA) autoencoder
optimizes simultaneously for both
an image reconstruction task (to learn spatial patterns) and
a bottleneck representation task
(to produce identical latent representations for an image and various rotations of that image).
Its loss function thus combines two terms:
\begin{eqnarray}\label{eq:RA}
    L_\text{RA}(\lambda) &=& L_{\textrm{\textrm{agn}}} + \lambda \cdot  L_{{\textrm{inv}^{\prime}}},
\end{eqnarray}
where $\lambda$ is a parameter used to balance the two terms.

\textit{Rotation-Agnostic Loss:}
The first loss term works to ensure that there
is at least one rotation angle for which the input image and its rotated
reconstruction are similar.
To this end, it minimizes the difference between an input image, $x$,
and the restored image $D_{\theta}(E_{\theta}(x))$ when rotated by $R \in {\cal R}$.
It is defined as:
\begin{eqnarray}\label{eq:agnostic}

    L_{\textrm{\textrm{agn}}}(\theta) & = &
    \sum_{x \in S}{\min_{R \in {\cal R}} || x - R(D_{\theta}(E_{\theta}(x))) ||_{2}^{2}} .

\end{eqnarray}

\textit{Bottleneck Loss:}
The second loss term works to ensure that any rotated version of the image input maps to a similar latent representation at the bottleneck layer.
It is defined as:
\begin{eqnarray}\label{eq:RA:bottleneck}

   L_{\textrm{inv}^{\prime}}(\theta) &=&  \sum_{x \in S}{\max_{R \in {\cal{R}}}|| z_{\theta}(x) - z_{\theta}(R(x))||_{2}^{2}} ,

\end{eqnarray}
where the $x$ and $R(x)$ are as for the rotation-agnostic loss, and
$z_{\theta}(\cdot)$ is an intermediate dimensionally reduced representation, meaning $z_{\theta}(x)$ is a bottleneck representation of the image $x$ and
$z_{\theta}(R(x))$ is that of the image rotated by $R \in {\cal R}$.

The key differences between $L_{\textrm{inv}}$ in Equation~(\ref{tinv}) and $L_{\textrm{inv}^{\prime}}$ in Equation~(\ref{eq:RA:bottleneck}) are that the latter computes a sum, rather than a max, over rotations ${\cal{R}}$, and computes distance in the original image space rather than bottleneck space.
Suppose that $z \approx E(D(z))$ for any latent representation $z$,
and the autoencoder reconstructs high-fidelity images such that $D_{\theta}(E_{\theta}(x)) \approx x$.
Thus we can rewrite the application of $E$ inside the norm in Equation~(\ref{tinv}) as follows,
showing the similarity between $L_{\textrm{inv}}$ and $L_{\textrm{inv}^{\prime}}$.
\begin{equation}\label{eq:isometry}
\begin{split}
    x - D(E(R(x)) &\approx  E(x) - E(D(E(R(x)))) \\
    &\approx E(x) - E(R(x)) \\
    &= z(x) - z(R(x)) .
\end{split}
\end{equation}

In implementing the bottleneck loss, $L_{\textrm{inv}^{\prime}}$,
we need not apply the maximum operator
to all variations of $R$ in [$0^{\circ}$, 360$^{\circ}$).
Instead, as we describe below,
we implement a data pipeline that for each image in a training set
first creates a fixed number of copies and then applies a random rotation to each copy,
to create a set $I$ of rotated versions.
Then, for each such set $I$, we
compute the sum of squares difference between the latent representation for each image $i$ in $I$ and those for all other images in $I$:
\begin{equation}\label{bottleneck_imp}
    L_{{\textrm{inv}}^{\prime}}^{i}(I) = \sum_{j \neq i} ||z_i - z_j ||^2_{2}.
\end{equation}

The bottleneck loss for each such set of rotated images in a minibatch is then:
\begin{equation}
    L_{\textrm{inv}^{\prime}}(I) = \sum_{i\in I} L^{i}_{\textrm{inv}^{\prime}}(I) \hspace{2mm}.
\end{equation}

Recall that the RA loss function (Equation (\ref{eq:RA})) contains a bottleneck loss term that is simplified by Equation~\ref{bottleneck_imp}.
To learn a similar representation of an image across the different rotations in similar types of images,
we need to introduce a data augumentation scheme, allowing images in minibatch to consist of several transformations.

We adjust our training minibatch to make our autoencoder learn the rotation-invariant features efficiently.
For instance, in learning MNIST set, we input all 10 digit classes.
To implement Equation~(\ref{bottleneck_imp}) with diversity of transformation of training images in each minibatch, as illustrated in Fig.~\ref{minibatch.fig},
we randomly select eight digits from MNIST;
replicate each image four times for a minibatch size of 32;
and then rotate each of the 32 images by an angle selected at random from [0$^{\circ}$,  360$^{\circ}$) to produce a training minibatch.

\begin{figure}[htbp!]
    \begin{center}
        \includegraphics[width=\textwidth,trim=20mm 22mm 23mm 48mm,clip]{Figures/minibatch_creation.pdf}
    \end{center}
    \vspace{-2ex}
    \caption{Cartoon of the minibatch creation concept in our data pipeline. 1) We select eight images randomly from MNIST. 2) We create four identical copies of each image. 3) We apply a random rotation in the range [0$^{\circ}$, 360$^{\circ}$)
    to each of the 8$\times$4 = 32 images to obtain a minibatch of 32 images.}
    \label{minibatch.fig}
\end{figure}

\section{Physical reasonableness in \texttt{Phys}}\label{sec:appned_c1physdata}
\begin{comment}
then you have your spatial organization test. You did this with a smaller dataset. Now you no longer need to show optical properties / physical reasonableness since you already proved that, but for the reader's convenience you would say the test for spatial contiguity, is done with a different training set but these clusters are also physically reasonable in the same way as shown above, see supp docs.
\end{comment}
To verify the physical reasonableness of the clusters produced by RICC on the \texttt{Phys} dataset used in Sections~\ref{sec:test:spatial} and~\ref{sec:test:separable},
we examine histograms of physics parameter values in different clusters (they are not randomly distributed:
see Fig.~\ref{fig:physicalreasonablePhysdata}), and compute the median inter-cluster correlation coefficient values (all are below our threshold of 0.6: 0.47 for cloud optical thickness, 0.45 for cloud phase infrared, 0.20 for cloud top pressure, and 0.39 for cloud effective radius).

\begin{figure}[htbp!]
    \centering

     \includegraphics[width=\linewidth,trim=0mm 1mm 0mm 1mm,clip]{Figures/hist_ri_phys_12.pdf}
    \caption{\emph{Test \ref{test11}--extension, cloud physics parameters for RICC applied to \texttt{Phys}}. Histograms of patch-mean values for four derived cloud physical parameters, cloud optical thickness (COT; no unit), cloud phase infrared (CPH; clear sky, liquid, ice), cloud top pressure (CTP; hPa), and cloud effective radius (CER; $\mu$m) show that these values are not randomly distributed within the 12 clusters produced by RICC.
    }
    \label{fig:physicalreasonablePhysdata}
\end{figure}

\end{appendices}

\section{Introduction}
\input{Narrative/01introduction}

\section{Approaches to Cloud Classification}\label{sec:approach}
\input{Narrative/02background}

\section{The RICC Autoencoder Method}\label{sec:ricc}
\input{Narrative/03ricc}

\section{Evaluating Unsupervised Cloud Clustering}\label{sec:protocols}
\input{Narrative/04eval_approach}

\section{Alternate Autoencoders}\label{sec:alternates}
\input{Narrative/05alternate}

\section{Evaluation Results}\label{sec:evals}
\input{Narrative/06eval_results}

\section{Comparison to a Labeled Dataset}\label{sec:experiment}
\input{Narrative/07experiment}

\section{Conclusions}\label{sec:conclusion}
\input{Narrative/08conclusion}

\section*{Acknowledgments}

The authors thank the University of Chicago's Research Computing Center for access to computing resources, and Valentina Zantedeschi and co-authors for access to CUMULO labeled dataset and scripts.

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


text:
Ilyse Morgenstein Fuerst: After the Rebellion: Religion, Rebels and Jihad in South Asia
This talk is part of the Humanities Futures Critical Islamic Studies Working Group's speaker series.
Religion, rebels, and jihad were redefined in the aftermath of the 1857 Rebellion in South Asia. What it meant to belong to a particular religion—specifically Islam—came to signify one's political leanings. In turn, religious concepts with long, multifaceted histories—specifically jihad—came to be synonymous with a religion and its religious community. This talk addresses how the events of 1857-1858 minoritized and racialized Indian Muslims, with particular attention to the use of jihad as a rhetorical concept in the colonial period. Professor Ilyse Morgenstein Fuerst specializes in religions of South Asia. Her research deals with Islam in South Asia, historiography, and the development of theories of religion. Other areas of interest include how religion has been defined and relates to both nationalism and colonialism. She earned her B.A. in Religion from Colgate University in 2005, an M.T.S. at Harvard Divinity School in 2007, and a Ph.D. from the University of North Carolina at Chapel Hill in the Islamic Studies concentration in the department of Religious Studies in 2012.
Working Group: Critical Islamic Studies
After the Rebellion: Religion, Rebels and Jihad in South Asia