\section{Word clustering}
\label{sec:analysis-of-word-embeddings-word-clustering}
In this section, we will apply cluster analysis on the word embeddings of the SGNS-enwiki, to search for deeper structures within the data. In particular, we will compare clustering algorithms on the word embeddings of the SGNS-enwiki in \cref{sec:comparing-clustering-algorithms}, and following, we will look at clustering of distinct groups of words in \cref{sec:clustering-word-groups}.

\subsection{Comparing clustering algorithms}
\label{sec:comparing-clustering-algorithms}
In this subsection, we compare clustering algorithms on the word embeddings of the SGNS-enwiki. Due to a large number of words in the vocabulary of the SGNS-enwiki (roughly 4.4 million, see \cref{sec:word2vec-hyperparameter-choices} for more details), we restrict the analysis to the 10000 most common (i.e most frequently occurring) words. This way, we speed up the computation by reducing the computational requirement. Also, we note that we should still get reliable results, as the most common words yield good quality vector representations due to the nature of their word frequencies.

To perform the cluster analysis, we used all clustering algorithm from \cref{sec:clustering-algorithms}, except for Spectral clustering (\cref{sec:spectral-clustering}), as it was too computationally expensive to run. In particular, we used the following algorithms: k-means clustering (\cref{sec:k-means-clustering}), mini-batch (MB) k-means clustering (\cref{sec:mini-batch-k-means-clustering}), k-medoids clustering (\cref{sec:k-medoids-clustering}), GMMs (\cref{sec:gmm-clustering}), hierarchical clustering (agglomerative) (\cref{sec:hierarchical-clustering}), HDBSCAN (\cref{sec:hdbscan-clustering}) and ToMATo (\cref{sec:tomato-clustering}). We used the \path{scikit-learn} \cite{ScikitLearn2011}, \path{scikit-learn-extra} \cite{ScikitLearnExtra2021}, \path{gudhi} \cite{gudhi2021} and \path{hdbscan} \cite{mcinnes2017hdbscan} Python packages to perform clustering. Furthermore, we trained the clustering algorithms using a grid-search manner, i.e. by trying all combinations of hyperparameters. \cref{table:hyperparameters-clustering-algorithms} shows the hyperparameters used to train each clustering algorithm. By forming a grid of hyperparameters for each clustering algorithm, we get a rough sense of the best set of hyperparameters. For the initial grid-search, we let \path{n_clusters_range}=2, 3, 4, 5, 10, 50, 100, 150, 200, 300, 400, 500, 750, 1000, 1500, 2000, 3000, 4000, 5000, 6000, 7000, 8000 be the cluster numbers used in algorithms where applicable. We let \path{n_clusters_range} range from 2 to 8000 clusters, using varying step sizes, to investigate the effect of the number of clusters for each algorithm, where it was applicable. To train the clustering algorithms, we used the standard word embeddings if the algorithm supported cosine similarity (or distance) and normalized word embeddings if the algorithm required Euclidean distances. After training the clustering algorithms, we validated them using the internal cluster validation methods from \cref{sec:cluster-validation}. In particular, we used the mean Silhouette Coefficient (SC) (\cref{sec:silhouette-coefficient}), the Davies-Bouldin Index (DBI) (\cref{sec:davies-bouldin-index}) and the Caliński-Harabasz Index (CHI) (\cref{sec:calinski-harabasz-index}). We used the \path{scikit-learn} Python package to perform internal clustering validation.
\begin{table}[H]
    \centering
    \begin{tabular}{@{}lll@{}}
    \toprule
    Clustering algorithm                           & Hyperparameters & Values \\
    \midrule
    \trcolor K-means clustering & \path{n_clusters} & \path{n_clusters_range} \\
    \multirow{2}{*}{Mini-batch k-means clustering} & \path{n_clusters} & \path{n_clusters_range} \\
                                                   & \path{batch_size} & 100 \\
    \trcolor K-medoids clustering & \path{n_clusters} & \path{n_clusters_range} \\
    GMM clustering & \path{n_components} & \path{n_clusters_range} \\
    \trcolor & \path{n_clusters} & \path{n_clusters_range} \\
    \trcolor \multirow{-2}{*}{Agglomerative clustering} & \path{linkage} & \path{single}, \path{average}, \path{complete}, \path{ward} \\
    \multirow{2}{*}{HDBSCAN} & \path{min_cluster_size} & 2, 4, 8, 16, 32, 64 \\
                             & \path{min_samples} & 1, 2, 4, 8, 16, 32, 64\\
    \trcolor                         & \path{density_type} & \path{DTM}, \path{logDTM}, \path{KDE}, \path{logKDE} \\
    \trcolor \multirow{-2}{*}{ToMATo} & \path{k} & 2, 3, \ldots, 10, 20, \ldots, 50, 100, \ldots, 250 \\
    \bottomrule
    \end{tabular}
    \caption{Hyperparameters used to train each clustering algorithm for the cluster analysis.}
    \label{table:hyperparameters-clustering-algorithms}
\end{table}

We visualize the result from the initial grid-search in \cref{fig:cluster-analysis-comparison-internal-cluster-validation}, where we see that the agglomerative clustering algorithm performs the best (close to k-means clustering) and k-medoids clustering performs the worst. For this reason, we will now focus on the agglomerative clustering algorithm and search for the best set of hyperparameters. Particularly, we will search for the best linkage criterion and number of clusters.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-comparison-internal-cluster-validation.pdf}
    \caption{Comparison of internal cluster validation results from clustering algorithms trained on word embeddings of the SGNS-enwiki model. The red dot in each plot denotes the most optimal value.}
    \label{fig:cluster-analysis-comparison-internal-cluster-validation}
\end{figure}

We will now look at the initial grid search result using the agglomerative clustering algorithm to deepen our understanding of the results. We visualize the results in \cref{fig:cluster-analysis-agglomerative-internal-cluster-validation}, and we notice that by using the single linkage criterion, we get relatively poor results. For the remaining criteria, we observe that we get more or less the same results, with the ward criterion being slightly ahead of the rest. By inspecting the best value for the number of clusters for each internal cluster validation method in \cref{fig:cluster-analysis-agglomerative-internal-cluster-validation}, we observe that the DBI and the CHI gives misleading results, while the SC is more meaningful. In particular, the DBI prefers to have the largest number of clusters, i.e. 8000. We inspected the words falling into the clusters, and from these, we observed that 6350 of the words were in clusters of size 1. This result indicates that the DBI is not particularly well suited for choosing the number of clusters, as it prefers to have clusters consisting of exactly one word. Using the CHI, we observe that it prefers to have the least number of clusters, i.e. 2. We inspected this result and noticed that in the first cluster, there was only a single word, while the second cluster had the remaining 9999 words. In other words, this indicates that the CHI is also not particularly well suited for choosing the number of clusters. Finally, using the SC, which we show in \cref{fig:cluster-analysis-agglomerative-internal-cluster-validation} (a), we observe that the preferred number of clusters lie around 3000 to 6000. We inspected the number of clusters as preferred by average, complete and ward linkage clustering and observed that they made sense, as there was more variety in the cluster sizes and the number of clusters having the specific cluster sizes. This indicates that the most preferable number of clusters (using SC) should lie in this range (3000 to 6000), and following, we will narrow down the search for the best number of clusters. For the next experiment, we will not include the single linkage clustering criterion, as it performed poorly.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-agglomerative-internal-cluster-validation.pdf}
    \caption{Internal cluster validation results using agglomerative clustering on word embeddings of the SGNS-enwiki model.}
    \label{fig:cluster-analysis-agglomerative-internal-cluster-validation}
\end{figure}

By narrowing the search to the range of 3000 to 6000 clusters, we find the best number of clusters for each criterion of agglomerative clustering. The narrowed search for the number of clusters is visualized in \cref{fig:cluster-analysis-agglomerative-internal-cluster-validation-narrow}, and we observe that ward linkage clustering with 4104 clusters results in the best clustering. In other words, these results indicate that, by using the ward clustering criterion, we obtain the best clustering of the 10000 most common words from the SGNS-enwiki model.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-agglomerative-internal-cluster-validation-narrow.pdf}
    \caption{The number of clusters plotted against the SC scores. The number of clusters is in the range of 3000 to 6000 clusters, and we use the average-, complete- and ward-linkage criteria. Here we see that the ward linkage criterion results in the highest SC score.}
    \label{fig:cluster-analysis-agglomerative-internal-cluster-validation-narrow}
\end{figure}

To deepen our understanding of the clustering result using agglomerative clustering and ward criterion on the word embeddings from SGNS-enwiki, we investigated the words falling into the 4104 clusters. In particular, we looked at the 10 largest and smallest clusters. We restricted the smallest clusters to contain at least 2 words, ensuring that we do not get clusters consisting of single words. In the top 10 largest clusters, we mostly saw names such as "Smith", "Wilson", or "Taylor" clustered together. We also saw the clustering of words representing numbers, e.g. "forty-five", "thirty-two" or "fifty-one", and the clustering of family-related words, e.g. "father", "son" and "brother". The top 10 smallest clusters mostly consisted of words that were strongly related to one another, such as "Adam" and "Noah", "card" and "cards", or "interior" and "exterior". We visualize some of the largest and smallest clusters in \cref{fig:cluster-analysis-agglomerative-2d-umap-top-clusters}, using a 2-dimensional UMAP (\cref{sec:umap}) embedding. To create the UMAP embedding, we used the \path{umap-learn} Python package \cite{mcinnes2018umap-software} and the default hyperparameterization. In \cref{fig:cluster-analysis-agglomerative-2d-umap-top-clusters}, we see that the clusters are widely spread all over the UMAP embedding. In addition to this, the UMAP embedding suggests that there are more clusters throughout the word embeddings, which the clustering algorithms were unable to pick up when evaluated using internal cluster validation methods. We will investigate this further, and in the next subsection, we will look at the clustering of distinct word groups. In particular, we will see if bigger sets of words cluster together in the UMAP embedding, suggesting that the word embeddings contains a deeper structure.
\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-agglomerative-2d-umap-top-clusters.pdf}
    \caption{2-dimensional UMAP embedding of the 10000 most common words from the SGNS-enwiki model, with some of the largest and smallest clusters outlined.}
    \label{fig:cluster-analysis-agglomerative-2d-umap-top-clusters}
\end{figure}

\subsection{Clustering word groups}
\label{sec:clustering-word-groups}
In this subsection, we will investigate the effect of clustering in the 2-dimensional UMAP embedding of the 10000 most common words of the SGNS-enwiki model, using distinct groups of words. In particular, we will cluster words related to countries/capitals, numbers, names (forenames and surnames) and food. Before performing the clustering, we first prepare the data used for the analysis. The countries/capitals data was retrieved from \cite{GeoNames}, where we used their API to fetch countries and their capital, resulting in 217 pairs of countries and capitals. The number data was generated by converting numbers to their string representation. We converted the numbers from zero to one trillion, resulting in 105 number-related words. The forenames data was retrieved from \cite{SSABabyNames}, where we used the top 1000 baby names from 2019. The surnames data was retrieved from \cite{CensusSurnames}, and we used the top 1000 surnames from 2010. Finally, the food data was retrieved from \cite{FoodIngredientList}, where we used the 250 most common ingredient words. We visualize the largest word group clusters falling into the 10000 most common words of the word embeddings from the SGNS-enwiki in \cref{fig:word-cluster-all-groups}, where we use a 2-dimensional UMAP embedding. In \cref{fig:word-cluster-all-groups}, we observe that two well-separated clusters are forming in the UMAP embedding, being the names and numbers word groups. We also see that the countries and food groups are more spread out in the embedding.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/word-cluster-all-groups.pdf}
    \caption{2-dimensional UMAP embedding of the 10000 most common words from the SGNS-enwiki model, with word groups outlined.}
    \label{fig:word-cluster-all-groups}
\end{figure}

In \cref{fig:word-cluster-all-groups}, we outline the largest clusters of the word groups and discard words falling out of the largest clusters. By including words that are outside the largest clusters, we saw that, in particular, the names word group is spread throughout the word embedding, as the data we used contained forenames and surnames of common words, such as "joy", "page" or "good". We illustrate this behaviour in \cref{fig:word-cluster-all-groups-emphasis-plots}, where we outline the four different word groups. In \cref{fig:word-cluster-all-groups-emphasis-plots} (a), we see that the country and capital words are mostly clustered to the middle left, with some capitals falling out of the bigger cluster. The "Stanley" and "Hamilton" capital cities are also used as names, as indicated in \cref{fig:word-cluster-all-groups-emphasis-plots} (c). For the numbers, we observe that most number-related words are clustered to the right, clearly separated from the rest of the words. However, we also observe that words such as "million", "billion" and "trillion" are clustered together outside the numbers cluster to the right. By inspection, we observed that the "million", "billion" and "trillion" words were, in fact, close to other financial words, such as "banks", "wealth" or "economics". In \cref{fig:word-cluster-all-groups-emphasis-plots} (c), we see that the forenames and surnames are clustered to the top left, in addition to being spread throughout the UMAP embedding. We also observe a small cluster of women names forming, containing the names "Diana" and "Isabella". Finally, we see that food-related words in \cref{fig:word-cluster-all-groups-emphasis-plots} (d) are slightly clustered around the words "egg" and "cheese", but also spread around the UMAP embedding. An interesting observation is the word "apple", which is both a fruit and a technology company. In this case, the word apple refers to the company Apple Inc., as we also saw earlier in \cref{table:word2vec-nearest-neighbours-words}.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/word-cluster-all-groups-emphasis-plots.pdf}
    \caption{2-dimensional UMAP embeddings of the 10000 most common words of the SGNS-enwiki model. Here we see four plots, and for each of them, we outline the four different word groups.}
    \label{fig:word-cluster-all-groups-emphasis-plots}
\end{figure}

To further develop our understanding of the SGNS-enwiki word embeddings, we will analyze two of the previous word groups. In particular, we will perform cluster analysis of the word embeddings of countries/capitals and numbers, where we will use clustering algorithms to cluster the words. We will use the same clustering algorithms specified in \cref{sec:comparing-clustering-algorithms}, in addition to Spectral clustering. To visualize the results, we will use dimensionality reduction algorithms to create 2-dimensional embeddings. We will also use latitude/longitude coordinates of countries to visualize the clustering results using countries/capitals word embeddings.

We analyzed the countries and capital word groups separately, as we choose to either identify a country by its name or its capital. Starting with the country word group, we performed cluster analysis. The result of the cluster analysis is summarized in \cref{fig:cluster-analysis-country-word-group-internal-cluster-validation}, where we see a similar result to the result in \cref{fig:cluster-analysis-comparison-internal-cluster-validation}, i.e. agglomerative clustering is the preferred clustering algorithm.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-country-word-group-internal-cluster-validation.pdf}
    \caption{Comparison of internal cluster validation results from clustering algorithms trained on country word embeddings from the SGNS-enwiki model. The red dot in each plot denotes the most optimal value.}
    \label{fig:cluster-analysis-country-word-group-internal-cluster-validation}
\end{figure}

Following, we inspected the scores from the DBI and CHI methods. We observed a similar pattern to the analysis from \cref{sec:comparing-clustering-algorithms}, namely that DBI prefers every word to be in its cluster and CHI prefers to have the smallest number of clusters (i.e. 2). For this reason, we will mainly focus on the results using SC. Using agglomerative clustering, we visualize its result in \cref{fig:cluster-analysis-agglomerative-country-word-group-internal-cluster-validation}. In \cref{fig:cluster-analysis-agglomerative-country-word-group-internal-cluster-validation}, we see similar results to \cref{fig:cluster-analysis-agglomerative-internal-cluster-validation}, namely that ward criterion gives the best clustering when using agglomerative clustering.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-agglomerative-country-word-group-internal-cluster-validation.pdf}
    \caption{Internal cluster validation results using agglomerative clustering on country word embeddings of the SGNS-enwiki model.}
    \label{fig:cluster-analysis-agglomerative-country-word-group-internal-cluster-validation}
\end{figure}

The best clustering using SC with agglomerative clustering and ward criterion resulted in 47 clusters. We visualize this result using latitude/longitude coordinates of each country in \cref{fig:cluster-analysis-agglomerative-country-word-group-top-clusters}, where we see that the five largest clusters correspond well with the continent of the countries.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-agglomerative-country-word-group-top-clusters.pdf}
    \caption{A comparison of countries divided into six continents (a) and the top 5 largest clusters from clustering of country word embeddings of the SGNS-enwiki model, using agglomerative clustering and ward criterion. Here we can see that the top 5 largest clusters in (b) correlate well with the continent of the respective countries.}
    \label{fig:cluster-analysis-agglomerative-country-word-group-top-clusters}
\end{figure}

Furthermore, we repeat the cluster analysis using capital to identify each country, i.e. we use the word embeddings of the capital words instead of the previously used country word embeddings. We show the result of the cluster analysis in \cref{fig:cluster-analysis-country-capitals-word-group-internal-cluster-validation}, where we see a similar result to the results in \cref{fig:cluster-analysis-comparison-internal-cluster-validation,fig:cluster-analysis-country-word-group-internal-cluster-validation}, namely that agglomerative clustering is the preferred choice of clustering algorithm.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-country-capitals-word-group-internal-cluster-validation.pdf}
    \caption{Comparison of internal cluster validation results from clustering algorithms trained on capital word embeddings from the SGNS-enwiki model. The red dot in each plot denotes the most optimal value.}
    \label{fig:cluster-analysis-country-capitals-word-group-internal-cluster-validation}
\end{figure}

We inspected the scores from the DBI and CHI methods, and similar to the results from \cref{sec:comparing-clustering-algorithms} and the cluster analysis using country word embeddings, we saw that DBI prefers every word to be in its own cluster and CHI prefers to have the smallest number of clusters (i.e. 2). This further strengthens the motivation to use SC over the other methods, and we mainly focus on the results using SC. Using agglomerative clustering, we visualize the results using capital word embeddings in \cref{fig:cluster-analysis-agglomerative-country-capitals-word-group-internal-cluster-validation}, where we see similar results to \cref{fig:cluster-analysis-agglomerative-internal-cluster-validation,fig:cluster-analysis-agglomerative-country-word-group-internal-cluster-validation}, namely that ward criterion gives the best clustering when using agglomerative clustering.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-agglomerative-country-capitals-word-group-internal-cluster-validation.pdf}
    \caption{Internal cluster validation results using agglomerative clustering on capital word embeddings of the SGNS-enwiki model.}
    \label{fig:cluster-analysis-agglomerative-country-capitals-word-group-internal-cluster-validation}
\end{figure}

The best clustering using SC with agglomerative clustering and ward criterion resulted in 21 clusters. We visualize this result using latitude/longitude coordinates of each country in \cref{fig:cluster-analysis-agglomerative-country-capitals-word-group-top-clusters}, where we see that we get larger clusters than by using country word embeddings in \cref{fig:cluster-analysis-agglomerative-country-word-group-top-clusters}. Furthermore, we observe that in \cref{fig:cluster-analysis-agglomerative-country-capitals-word-group-top-clusters} (b), the first cluster (green) consists of capitals where the countries are Spanish talking, as outlined by the "Madrid" (Spain), "Mexico City" (Mexico) and "Santiago" (Chile) boxes.
The second cluster (blue) in \cref{fig:cluster-analysis-agglomerative-country-capitals-word-group-top-clusters} (b) also correlates well with the Oceanic continent in \cref{fig:cluster-analysis-agglomerative-country-capitals-word-group-top-clusters} (a), while the third (red) and forth (purple) clusters in \cref{fig:cluster-analysis-agglomerative-country-capitals-word-group-top-clusters} (b) seem to capture the African continent adequately. Finally, we see that the (yellow) cluster consists of capitals from Eastern Europe and some capitals from Asia. For the record, Dakar is the capital of Senegal, Pretoria is one of the capitals of South Africa, and Suva is the capital of Fiji.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-agglomerative-country-capitals-word-group-top-clusters.pdf}
    \caption{A comparison of countries divided into six continents (a) and the top 5 largest clusters from clustering of capital word embeddings of the SGNS-enwiki model, using agglomerative clustering and ward criterion. In (b) we can see that Spanish speaking countries are clustered together in the first cluster (green), while the other clusters are well clustered with respect to their continents.}
    \label{fig:cluster-analysis-agglomerative-country-capitals-word-group-top-clusters}
\end{figure}

Next, we will perform cluster analysis of number word embeddings in a similar manner to how we performed the cluster analysis of country/capital word embeddings. First, we show a comparison of clustering algorithms using internal cluster validation methods in \cref{fig:cluster-analysis-numbers-word-group-internal-cluster-validation}, where we see that, overall, the agglomerative clustering algorithm performs the best, when evaluated using internal validation methods.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-numbers-word-group-internal-cluster-validation.pdf}
    \caption{Comparison of internal cluster validation results from clustering algorithms trained on number word embeddings from the SGNS-enwiki model. The red dot in each plot denotes the most optimal value.}
    \label{fig:cluster-analysis-numbers-word-group-internal-cluster-validation}
\end{figure}

Furthermore, we will use the agglomerative clustering algorithm to cluster number word embeddings. To find its best criterion and number of clusters, we first visualize its results in \cref{fig:cluster-analysis-agglomerative-numbers-word-group-internal-cluster-validation}. In \cref{fig:cluster-analysis-agglomerative-numbers-word-group-internal-cluster-validation}, we see the internal validation methods prefer different amount of clusters and linkage criteria. In particular, SC prefers complete linkage criterion with 2 clusters, DBI prefers single linkage criterion with 6 clusters, and CHI prefers ward linkage criterion with 3 clusters. In other words, we observe a different behaviour than in the results from the internal cluster validation methods in \cref{sec:comparing-clustering-algorithms} and the country/capital cluster analysis, namely that SC prefers the least amount of clusters, DBI does not prefer the most amount of clusters and CHI does not prefer the least amount of clusters.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-agglomerative-numbers-word-group-internal-cluster-validation.pdf}
    \caption{Internal cluster validation results using agglomerative clustering on number word embeddings of the SGNS-enwiki model.}
    \label{fig:cluster-analysis-agglomerative-numbers-word-group-internal-cluster-validation}
\end{figure}

Next, we visualize the best agglomerative clustering result of the number word embeddings as ranked by the internal clustering validation methods in \cref{fig:cluster-analysis-agglomerative-numbers-word-group-internal-cluster-validation-best-2d-pca}, where we use a 2-dimensional PCA embedding. We used PCA instead of UMAP here as there were few points to embed in 2 dimensions. In \cref{fig:cluster-analysis-agglomerative-numbers-word-group-internal-cluster-validation-best-2d-pca}, we see that it is not entirely clear how to cluster the number word embeddings.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/cluster-analysis-agglomerative-numbers-word-group-internal-cluster-validation-best-2d-pca.pdf}
    \caption{Comparison of the best clustering results using agglomerative clustering on the number word embeddings of the SGNS-enwiki model. Here we see that it is not clear which clustering is the best.}
    \label{fig:cluster-analysis-agglomerative-numbers-word-group-internal-cluster-validation-best-2d-pca}
\end{figure}

We further investigated the structure of the 2-dimensional PCA embedding of the number words and noticed an interesting relationship. We illustrate this relationship in \cref{fig:ordered-number-word-embeddings-2d-pca}, which shows that by assigning an increasing label from the smallest and to the largest number, we get that the colour of the label gradually increases from the smallest to the largest. In other words, there seems to be an underlying sequential relationship in the number word embeddings. Furthermore, this suggests that the underlying structure of number word embeddings may contain information that we have not been able to find yet.
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{thesis/figures/ordered-number-word-embeddings-2d-pca.pdf}
    \caption{2-dimensional PCA embedding of the 105 number word embeddings, where each word embedding has an increasing label assigned to it. Here we see that as we increase the number, we see a possible underlying sequential relationship.}
    \label{fig:ordered-number-word-embeddings-2d-pca}
\end{figure}


\documentclass[12pt, letterpaper]{article}
\usepackage[english]{babel}
\usepackage{amsmath}
\usepackage[margin=0.7in]{geometry}

  %%%%%%%%%%%%
  % PREAMBLE %
  %%%%%%%%%%%%
\begin{document}
\selectlanguage{english}

\section{The rotation matrix, W, within matrix/\_\_init\_\_.py}
  The rotation of a vector $\mathbf{r}$ through a counterclockwise angle $\theta$ about the
  normalized axis $\mathbf{\hat{e}}$ is treated geometrically, \textit{e.g.}, by Goldstein (1980).  Eq. 4-92
  in that text gives the Rodrigues rotation formula for the rotated vector,

  \begin{equation}
    \mathbf{r'} = \mathbf{r} \cos \theta + \mathbf{\hat{e}}(\mathbf{\hat{e}}\cdot\mathbf{r})
    \left[ 1 - \cos \theta \right] + (\mathbf{\hat{e}} \times \mathbf{r}) \sin \theta
    \text{.}
    \label{eqn:rodrigues}
  \end{equation}

  Note that Eq. 1 is for a rotating vector in a fixed laboratory frame.  We want to express
  the rotation in the form of a matrix operator $\mathbf{W}$ (Fischer \& Koch, 1996), such that
  $\mathbf{r'} = \mathbf{W}\mathbf{r}$.  We expand the formula in $3\times3$ matrix notation and
  rearrange:

  \begin{equation}
    \mathbf{r'} = (\mathbf{I} \cos \theta +
    \left[
    \begin{array}{c c c}
       \hat{e}_{x}^2  & \hat{e}_{x}\hat{e}_{y} & \hat{e}_{x}\hat{e}_{z} \\
       \hat{e}_{y}\hat{e}_{x}  & \hat{e}_{y}^2 & \hat{e}_{y}\hat{e}_{z} \\
       \hat{e}_{z}\hat{e}_{x}  & \hat{e}_{z}\hat{e}_{y} & \hat{e}_{z}^2 \\
    \end{array}
    \right]
    \left[ 1 - \cos \theta \right] +
    \left[
    \begin{array}{c c c}
       0  & -\hat{e}_{z} & \hat{e}_{y} \\
       \hat{e}_{z}  & 0 & -\hat{e}_{x} \\
       -\hat{e}_{y}  & \hat{e}_{x} & 0 \\
    \end{array}
    \right] \sin \theta)\mathbf{r}
    \text{,}
    \label{eqn:expansion}
  \end{equation}
  where $\mathbf{I}$ is the $3\times3$ identity matrix.  Eq. 2 is identical to one given in
  Boisen \& Gibbs (1990).  Let's simplify it a little bit by introducing notation for the
  outer product $\mathbf{\hat{e}\otimes\hat{e}}$ and the operator representation of the
  cross product matrix $\left[\mathbf{\hat{e}}\right]_\times$,

  \begin{equation}
   \mathbf{r'} = (\mathbf{I} \cos \theta +
   (\mathbf{\hat{e}\otimes\hat{e}})
   \left[ 1 - \cos \theta \right] +
    \left[
      \mathbf{\hat{e}}
    \right]_{\times} \sin \theta)\mathbf{r}
    \text{.}
    \label{eqn:sabbrev}
  \end{equation}

  Now with this compact notation it's convenient to express the partial derivative of
  the expression with respect to $\theta$:
  \begin{equation}
    \frac{\partial \mathbf{r'}}{\partial \theta} =
     (-\mathbf{I} \sin \theta +
   (\mathbf{\hat{e}\otimes\hat{e}})
     \sin \theta  +
    \left[
      \mathbf{\hat{e}}
    \right]_{\times} \cos \theta)\mathbf{r}
    \text{.}
    \label{eqn:gradient}
  \end{equation}

  We could also specialize Eq. 2 for \textit{e.g.} 2-fold rotations by taking $\theta=180^\circ$,
  giving the matrix operator
  \begin{equation}
    \mathbf{W} =
    \left[
    \begin{array}{c c c}
       2\hat{e}_{x}^{2}-1  & 2\hat{e}_{x}\hat{e}_{y} & 2\hat{e}_{x}\hat{e}_{z} \\
       2\hat{e}_{y}\hat{e}_{x}  & 2\hat{e}_{y}^{2}-1 & 2\hat{e}_{y}\hat{e}_{z} \\
       2\hat{e}_{z}\hat{e}_{x}  & 2\hat{e}_{z}\hat{e}_{y} & 2\hat{e}_{z}^{2}-1 \\
    \end{array}
    \right]
    \text{.}
    \label{eqn:gradient_of_rc}
  \end{equation}

  It should be stressed that $\mathbf{W}$ and $\mathbf{\hat{e}}$ are expressed in Cartesian laboratory
  coordinates rather than crystallographic coordinates.

\section{References}
\setlength{\parindent}{0cm}

  Boisen, M.B. Jr \& Gibbs, G.V. (1990).  \textit{Mathematical Crystallography, Reviews in Minerology,}
  Vol. 15 (revised edition).  Washington, DC: Mineralogical Society of America.
  \\
  \newline
  Fischer, W. \& Koch, E. (1996).  In \textit{International Tables for Crystallography, Volume A:
  Space-Group Symmetry}, $4^{th}$ revised edition, Hahn, T., ed.  Dordrecht: Kluwer Academic Publishers.
  \\
  \newline
  Goldstein, H. (1980).  \textit{Classical Mechanics}, $2^{nd}$ edition.  Reading, MA: Addison-Wesley,
  pp. 164-166.

  \end{document}


\paragraph{}
After the geometry information is exported from the IGES file, it can be feed into the quad-tree algorithm to generate mesh of the problem domain.
As an algorithm based on computational geometry, it requires great amount of numerical operations and hence the result may be sensitive to the tolerance.
An absolute tolerance may not be able to handle problems with very large or small geometric size.
As a consequence, the first step is to normalize the geometry into a uniformed space ($10\times10$ is used in this chapter).

\subsection{Background mesh}
\label{qdt_sc:qdt_background_mesh}
\paragraph{}
Background mesh describes a mesh in the background. %fig
Its density is decided by the geometry.
This section will introduce the procedure to generate the background mesh.
\paragraph{}
First of all, we start with one square which is the root of the tree.
The size of it will be slightly larger than the normalized geometric input and it is selected as $16 \times 16$ in this chapter.
An adaptive mesh generation based on numerical error can be found in Sec.~\ref{adp_sec:main}
After that, the root square will be divided into millions (defined by resolution, defined as $2^{res} \times 2^{res}$) smaller ones like pixels in the image.
Then, $2^{s_{max}} \times 2^{s_{max}}$ ``pixels'' will be group into the first level of the tree, or initial background mesh as shown in Fig.~\ref{qdt_fig:qdt_initial_mesh}.
It is used to control the maximum allowable mesh size globally or separately for different material regions.

\begin{figure}[h!]
    \centering
    \scalebox{0.6}{
        \includegraphics{quadtree/images/qdt_initial_mesh.eps}
    }
    \caption[Initial background mesh]{An example of the background initial mesh: $16 \times 16$ square are divided into $2^4 \times 2^4$ pixels (dashed lines, $res=4$) and $2^2 \times 2^2$ pixels form the initial mesh (solid lines, $s_{max}=2$)}
    \label{qdt_fig:qdt_initial_mesh}
\end{figure}

\paragraph{}
% generating the initial mesh without balance
The criteria to decided whether each individual square in the initial mesh need to be refined or not is based on the concept of seed point.
The curve will be uniformly discretized into a given number so called seed points uniformly and the mesh will be refined until the number of the seed points within the square is less than the threshold.
However, finer mesh is expected at the region where geometry with high curvature appears.
It can be solved by treating each segment of the polylines as an individual curve.
Due to the fact that algorithm described in .~\ref{qdt_section:iges_output} guarantees the chord length to arc length ratio, polylines ought to have finer segments at the position where curvature is significant.

\paragraph{}
% only two intersections allowed
Although seed points provides a good guide on the mesh density, situations where high density mesh is required while few seed points appeared may happen as plotted in Fig.~\ref{qdt_fig:qdt_seed_point_problem}.
The geometry limits the seed points due to the lack of curvature.
While, it is expected that the square element will be refined at least once as one layer of mesh may not be appropriate to formulate a thin shell structure.
    \begin{figure}[h!]
        \centering
        \scalebox{0.8}{
            \includegraphics{quadtree/images/qdt_seed_points_problem}
        }
        \caption[Limitation of the seed points]{
            Limitation of the seed points: few seed points will be generated over a straight line and few seed points will be included in the square element which leads to unexpected behavior.
            \tikz\draw[black,fill=black] (0,0) circle (.7ex);
            stands for the seed points.
        }
        \label{qdt_fig:qdt_seed_point_problem}
    \end{figure}
As a result, another restriction will be adopted together with the seed points to prevent this kind of situation from happening.
Element with more than two unique intersections will be tagged as ``need refinement'' no matter how many seed points it contains.
In numerical calculation, two points may be regarded as one if they are close enough to each others.
Normalization of the geometry described at the beginning of this section helps to define a meaningful tolerance to handle numerical error.

% balance the initial mesh
\paragraph{}
% balance in tree data structure
Self-balancing is adopted by most of the tree data structure such as AVL, B/B+, Red-black tree and so on.
Fig.~\ref{qdt_fig:tree_balance_avl} illustrates a self-balancing operation of the AVL tree.
Balancing by rotation is performed because difference in height of the leaf B and L is greater than the threshold.
    \begin{figure}
        \centering
        \scalebox{0.25}{
            \includegraphics{quadtree/images/qdt_avl_balance.png}
        }
        \caption{Balance of the AVL tree \citep{Roura2013}}
        \label{qdt_fig:tree_balance_avl}
    \end{figure}
% balance in quadtree
Same idea is adopted in the quadtree as well.
A refinement will be performed to achieve the balance if the difference in the height of the leaf (Cell A and B in Fig.~\ref{qdt_fig:tree_balance_quadtree} for example) is larger than one.
    \begin{figure}[h!]
        \begin{subfigure}[b]{0.5\linewidth}
            \centering
            \scalebox{0.7}{
                \includegraphics{quadtree/images/qdt_balance_before.eps}
            }
            \caption{Before balance operation}
        \end{subfigure}
        \begin{subfigure}[b]{0.5\linewidth}
            \centering
            \scalebox{0.7}{
                \includegraphics{quadtree/images/qdt_balance_after.eps}
            }
            \caption{After balance operation}
        \end{subfigure}
        \caption{Balance of quadtree: cell A is refined in order to balance the quadtree.}
        \label{qdt_fig:tree_balance_quadtree}
    \end{figure}
% reason behind balancing
The reason why balancing is predominately adopted in tree data structure lies in the guarantee of an $O\left(log(n)\right)$ time complexity for searching in any case.
Even thought computational cost on searching seems not to be significant during mesh generation using quadtree, a balanced tree provided some other attractive features that can be utilized in numerical analysis.
One of the advantages is to improve the mesh quality.
Any extremely small or large angle between the element and the scaling center may result in a bad quality mesh.
Chances are that these poor quality mesh may appear without self-balancing, shown in Fig.~\ref{qdt_fig:sbfem_adv_1}.
    \begin{figure}[h!]
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_balance_sbfem_adv_1.eps}
        }
        \caption{Small angle between element and scaling center may reduce the mesh quality}
        \label{qdt_fig:sbfem_adv_1}
    \end{figure}
%
Another reason for that is to kept the pattern of the cells.
If the threshold of the self-balancing is set to one ($2:1$ ratio), only six kinds of the cells will appear before the cutting as illustrated in Fig.~\ref{qdt_fig:sbfem_adv_2}.
Thanks to the geometric similarity, local stiffness matrix can be calculated and scaled directly when same kind of the cell appears which significantly reduce the computational cost.
Hanging nodes in Fig.~\ref{qdt_fig:sbfem_adv_2} can be a problem for traditional finite element to handle the displacement compatibility \citep{Tabarraei:2009:XFE,NME:NME3070,NME:NME2900}.
Solution including triangulation \citep{4037344,BERN1994384,ijeas251083}, using special shape function \citep{NME:NME1620120104} and other methods are available but special treatment is required.
As a comparison, SBFEM provides greater flexibility on the element, n-sides polygons with hanging nodes or curved edges can be treated natively.
    \begin{figure}[h!]
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_balance_sbfem_adv_2.eps}
        }
        \caption[Types of the cell in self-balancing quadtree]{
            Types of the cell when $2:1$ ratio is applied,
            \tikz\draw[black,fill=black] (0,0) circle (.7ex);
            stands for the hanging node
            }
        \label{qdt_fig:sbfem_adv_2}
    \end{figure}
%=====================================================================================================================%
\subsection{Hard point treatment}
\paragraph{}
%introduction
Hard points is a kind of point in the geometry that must be meshed as a node or scaling center.
When more than two materials are involved, it is common to have some hard points to make sure the point shared by three material can be properly formulated as shown in Fig.~\ref{qdt_fig:qdt_hard_point_demo}
    \begin{figure}[h!]
        \begin{subfigure}[b]{0.5\linewidth}
            \centering
            \scalebox{0.8}{
                \includegraphics{quadtree/images/qdt_hard_point_demo.eps}
            }
        \caption{Hard point is meshed as node}
        \end{subfigure}
        \begin{subfigure}[b]{0.5\linewidth}
            \centering
            \scalebox{0.8}{
                \includegraphics{quadtree/images/qdt_hard_point_demo_2.eps}
            }
        \caption{hard point is meshed as scaling center}
        \label{qdt_fig:qdt_hard_point_demo_sc}
        \end{subfigure}
        \caption[Example of a hard point]{Example of a hard point, elements around the point shared by three material must be properly divided into three.}
        \label{qdt_fig:qdt_hard_point_demo}
    \end{figure}
The difficulty in treating a hard point lies in its position in the background mesh.
The further it is away from the geometric center of the background mesh, the poorer the quality of mesh will be generated.
If the hard point in Fig.~\ref{qdt_fig:qdt_hard_point_demo_sc} is located somewhere that is very close to the left boundary of the background mesh, the mesh for material one after cutting may be quite elongated and twisted.
Besides, the requirement of the scaling center in the SBFEM can always be satisfied no matter where it is located in a convex polygon.
The opposite is true for a concave polygon, special treatment must be adopted to fullfil this requirement.
As a consequence, generally speaking, a convex mesh may be preferred over a concave one and hard point can be the only source that will introduce concave polygons in most of the situations.
Although algorithm finding qualified scaling center in a concave polygon exists, the quality of the mesh may not be satisfactory even the scaling center is located on the vertex.

\paragraph{}
% make sure the hard point is close to the center of the background mesh
The first step to treat the hard point will be trying to adjust the background mesh so that the hard point is close enough to the cell's geometric center in the background mesh.
The ideal size of the background mesh shall ensure that only one hard point is contained and that no points from any other curves should be located in it.
As a consequence, the size of the containing square $box\_size$ is set to be one third of the distance to the nearest curves or half of the distance to the nearest hard point, whichever is smaller.
These parameter usually result in a valid and large enough background mesh that can treat the hard point easily.
When building the background cell containing hard points in the algorithm, it will be implemented by having a considerably fine mesh within the range of the hard point and merge all cells in that range into a larger one to be the background cell.
Size of the ``considerably fine'' mesh $size\_field$ will be defined with the adjacent vertexes of the hard point derived in Eq.~\ref{qdt_eq:qdt_hard_point_size_field}
    \begin{equation}
        size\_field = \frac{box\_size}{
            2^{
                round(
                    \log(\frac{2 \pi}{ min(\alpha)})-1
                )
            }
        }
    \label{qdt_eq:qdt_hard_point_size_field}
    \end{equation}
where $\alpha$ is the minimal angle $min(\alpha_1, \alpha_2, \dots, \alpha_n)$ in Fig.~\ref{qdt_fig:qdt_hard_point_setp_1}.
As is exponentially related to the minimal angle $\alpha$, the $size\_field$ may always be small enough to capture thin shell.
    \begin{figure}
        \centering
        \scalebox{0.5}{
            \includegraphics{quadtree/images/qdt_hard_point_step_1.eps}
        }
        \caption[Hard point treatment step 1]{Hard point treatment step1: find $box\_size$ and $size\_field$}
        \label{qdt_fig:qdt_hard_point_setp_1}
    \end{figure}
%
After the first step of the hard point treatment is done, the background cell shall have the following properties
    \begin{enumerate}
        \item Distance between hard point to geometric center of the background cell must be smaller than $\sqrt{2}size\_field$
        \item Element after cutting share the hard point as the node or it will be one element with scaling center located at the hard point
    \end{enumerate}
With these two properties, the cell can be cut by simply connecting the intersections with the hard point later.

%=====================================================================================================================%
\subsection{Bucket sort algorithm}
\paragraph{}
When cutting the boundary, it is necessary to find the intersections between the polylines and the background mesh.
Finding geometric relationship between each cell and each curve may be necessary.
However, it could be computational expansive when quite a few curves are involved and a fine mesh is targeted.
As a result, an algorithm inspired by buck sort is introduced in order to reduce the computational cost of this process.
\paragraph{}
% introduce bucket sort algorithm
Bucket sort algorithm is a sorting algorithm that put an array of objects into different buckets and then sort the objects in each bucket.
Objects in each bucket then are sorted by other sorting algorithm or by applying the buckets sort recursively.
Although the sort algorithm can not be adopted directly in finding the intersection, the idea ``divide and conquer'' can be utilized in order to reduce the computational cost in finding the intersections.
\paragraph{}
% bucket sort in quadtree
First step of this method would be very similar to the first step in generating the background mesh.
The entire background square or the root of the quadtree will be divided evenly into $2^n \times 2^n$ smaller ones.
A larger $n$ will result in an increasing number of buckets and will detect more irrelevant curves but take more time.
A trade off normally is not necessary and $n=8$ usually provides a good enough result so it can be fixed as $8$ unless special need is posed on.
Fig.~\ref{qdt_fig:qdt_bucket_sort} illustrates an example of the algorithm ($n$ is taken as $4$ in this example in order to have a clear figure, it should usually be taken as no less than $6$).
The end points of the curve are located in gird D2 and I5, as a consequence, the curve will be put in bucket ``D2-I5''.
In other words, cells that are not in this region shall not have any intersections with the curve.

\begin{figure}[h!]
    \centering
    \scalebox{0.8}{
        \includegraphics{quadtree/images/qdt_bucket_sort.eps}
    }
    \caption[Bucket sort in quadtree]{Bucket sort in quadtree}
    \label{qdt_fig:qdt_bucket_sort}
\end{figure}

%=====================================================================================================================%
\subsection{Cutting with boundary}
\paragraph{}
% 3cases of the cutting
Generally, there are three situations where cutting is involved with one curve and they are sorted by the number of the intersections as shown in Fig.~\ref{qdt_fig:qdt_cutting}.
Due to the fact that the background cell is refined properly and hence the curves can be regarded as straight lines.
Case 3 in Fig.~\ref{qdt_fig:qdt_cutting_3} is an exception as it shall not happen unless even the background cell with the minimal allowable size may not be able to catch the feature.
Unfortunately, result from case 3 may not be as expected but increasing the resolution or decreasing the minimal allowable cell size can help to prevent case 3 from happening.
\begin{figure}[h!]
    \begin{subfigure}[b]{0.3\linewidth}
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_cutting_case_1.eps}
        }
        \caption{Case1: two intersections}
        \label{qdt_fig:qdt_cutting_2}
    \end{subfigure}
    \begin{subfigure}[b]{0.3\linewidth}
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_cutting_case_2.eps}
        }
        \label{qdt_fig:qdt_cutting_1}
        \caption{Case2: one intersectoin}
    \end{subfigure}
    \begin{subfigure}[b]{0.3\linewidth}
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_cutting_case_3.eps}
        }
        \caption{Case3: more than two intersections}
        \label{qdt_fig:qdt_cutting_3}
    \end{subfigure}
    \caption[Situations in cutting]{Situations in cutting: one, two and more than two intersections}
    \label{qdt_fig:qdt_cutting}
\end{figure}
% situation a (two intersections)
% sup an alg
In situation a (Fig.~\ref{qdt_fig:qdt_cutting_2}) where two intersections are observed, cell can be split by connecting points $EF$.
To start with, any vertex among $A$, $B$, $C$ or $D$ can be selected as the starting point.
After the point is selected, point A for example, another point from any of the line connected it will be chosen, point B for line AB for example.
This procedure will be repeated recursively until the intersection points $E$ or $F$ is found and next point will be $E$ or $F$ whichever is not selected at the moment.
Recursion will be continued until the starting point is found again.
A closed region then will be formed by all selected points in sequence.
Another cell can also be easily found by finding the implement set.
It should be noted that all cells split in this situation must be convex.

\paragraph{}
% situation b (one intersection)
% sup an alg
In case 2 where only one intersection will be detected, multiple curves with the intersections should be observed as in Fig.~\ref{qdt_fig:qdt_cutting_hard_point} and the hard points must be involved (point $H$ in Fig.~\ref{qdt_fig:qdt_cutting_hard_point}).
    \begin{figure}
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_cutting_hard_point.eps}
        }
        \caption[Hard point treatment in cutting]{Hard point treatment in cutting}
        \label{qdt_fig:qdt_cutting_hard_point}
    \end{figure}
Handling case 2 would be similar but slightly tricky compared to that in case 1.
At the beginning, the hard point is selected (point $H$ in the example).
After that, any intersection point is selected ($F$ in this example).
Then, the other point of any line segment that has $F$ will be selected ($D$ in this example).
The former step will be performed recursively until another intersection point is found.
Once the hard point is detected again, a closed region is found.
Repeating the procedure can help to find all closed regions cut by the hard point and multiple curves.

\paragraph{}
% situation c (more than two intersection)
% no alg needed
In case 3 where more than two intersections are found, split cells may not be as expected as this situation shall not happen normally.
If case 3 does happen, the first and the last intersection points will be selected and all other intersection points will be ignored.
After that, method in case one can be applied directly.

%=====================================================================================================================%
\subsection{Points moving}
\label{qdt_sc:point_moving}
\paragraph{}
% poor mesh quality
Although all types of cells with intersections are able to be treated by the algorithm described above, quality of the mesh sometimes may become out of control as example shown in Fig.~\ref{qdt_fig:qdt_cutting_poor_mesh_quality}.
    \begin{figure}[h!]
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_cutting_point_moving.eps}
        }
        \caption[Poor quality of mesh]{Poor quality of mesh: triangle element ABC is too small compared to its neiboughre}
        \label{qdt_fig:qdt_cutting_poor_mesh_quality}
    \end{figure}
\paragraph{}
% point moving
In order to improve the mesh quality, the point moving is adopted.
Generally there are two ways of moving, one is moving the closest points to the line (move point $C$ to line $AB$ in Fig.~\ref{qdt_fig:qdt_cutting_poor_mesh_quality}) and another is moving points on the line to the closing point, e.g. point $B$ will be moved to $C$ in Fig.~\ref{qdt_fig:qdt_cutting_poor_mesh_quality}.
Both methods has its own advantages and disadvantages and they will be discussed later.
\paragraph{}
% mark node as intersection
The recommended method is to mark the nodes that are close enough to the curves as intersections and project them back onto the curves at the end.
One significant advantage is that only convex cells may appear during cutting but special treatment need to be added in generating the background mesh.
The algorithm for cutting a convex cell into two will always yield valid result in existing algorithm.
For example, point $C$ in Fig.~\ref{qdt_fig:qdt_cutting_poor_mesh_quality} will be marked as the intersection and point $A$ and $B$ will be ignored.
Point $C$ will be projected back using algorithm introduced in Sec.~\ref{qt_sc:projection} to the curve when the cutting is finished.
The threshold usually is taken as a proportion to the length of the background cell size.
When finding the intersection, another detection will be posed after.
If the distance of the intersection to a node is smaller than $\epsilon a$ where $\epsilon$ is tolerance and $a$ is the size of the background cell, the intersection will be ignored and the nearest node will be marked as intersection instead.
\begin{figure}
    \centering
    \scalebox{1}{
        \includegraphics{quadtree/images/qdt_cutting_point_moving_1.eps}
    }
    \caption[Treatment of cell cutting when point is moved]{Treatment of cell cutting when point is moved}
    \label{qdt_fig:qdt_cutting_point_moving_1}
\end{figure}
Fig.~\ref{qdt_fig:qdt_cutting_point_moving_1} illustrates an example when points are moved toward one of the nodes.
If both intersections are moved then cutting may not be necessary.
When one is moved and another is retained, algorithm described in previous section can be adopted directly as the topology remain the same.
The disadvantage is that situation in Fig.~\ref{qdt_fig:qdt_cutting_point_moving_problem} when generating the background mesh need to be taken into consideration.
Points $A$, $B$ and $C$ will be marked as intersections and it becomes situation 3 in Fig.~\ref{qdt_fig:qdt_cutting_3}.
However, background cells like it shall be refined otherwise unexpected behavior resulted from cutting algorithm may appear and such unexpected behavior shall be only allowed to happen when minimal allowable cell size in setting is too large to capture the local feature.
As a consequence, when refining background mesh in section.~\ref{qdt_sc:qdt_background_mesh}, not only the number of intersections between curves with the origin cell need to be counted, but also that from the enlarged box need to be counted as well as shown in Fig.~\ref{qdt_fig:qdt_cutting_enlarged_box_point_moving_problem}.
Either more than two intersections are found in origin cell or the enlarged one in Fig.~\ref{qdt_fig:qdt_cutting_enlarged_box} will result in the refinement of the cell.
\begin{figure}
    \begin{subfigure}[b]{0.5\linewidth}
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_cutting_point_moving_1_dis.eps}
        }
        \caption{Problem in marking intersection}
        \label{qdt_fig:qdt_cutting_point_moving_problem}
    \end{subfigure}
    \begin{subfigure}[b]{0.5\linewidth}
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_cutting_point_moving_1_enlarge.eps}
        }
        \caption{Enlarged box (dashed line)}
        \label{qdt_fig:qdt_cutting_enlarged_box}
    \end{subfigure}
    \caption[Problem in marking intersection]{Adopting enlarged box to ensure the background mesh is refined properly}
    \label{qdt_fig:qdt_cutting_enlarged_box_point_moving_problem}
\end{figure}
% move points before cutting
Another approach is projecting the nodes in background mesh on to the curves before the cutting using algorithm introduced in Sec.~\ref{qt_sc:projection}.
Special treatments in generating the background mesh is no longer necessary but cutting concave cells may be hard to treat.
Fig.~\ref{qdt_fig:qdt_cutting_point_moving_2_concave} illustrates an example of a concave cell resulted from point moving.
An intersecting curve that is not a straight line (even with a very small curvature) leads to a possibility that the distance $a$ from the hanging node $A$ to the curve is slightly smaller than the tolerance while $b$ and $c$ are slightly larger than tolerance.
    \begin{figure}[h!]
        \begin{subfigure}[b]{0.5\linewidth}
            \centering
            \scalebox{1.2}{
                \includegraphics{quadtree/images/qdt_cutting_point_moving_2_concave_before.eps}
            }
            \caption{Before moving}
            \label{qdt_fig:qdt_cutting_point_moving_2_concave_before}
        \end{subfigure}
        \begin{subfigure}[b]{0.5\linewidth}
            \centering
            \scalebox{1.2}{
                \includegraphics{quadtree/images/qdt_cutting_point_moving_2_concave_after.eps}
            }
            \caption{After moving}
            \label{qdt_fig:qdt_cutting_point_moving_2_concave_after}
        \end{subfigure}
        \caption[Concave cell resulted from points moving]{Concave cell resulted from point moving}
        \label{qdt_fig:qdt_cutting_point_moving_2_concave}
    \end{figure}
As a result, points $A$ will be moved on the curves but the other two nodes will not and hence cutting a concave cell need to be treated.
This method is not recommended because special treatment on cutting concave cells need to be implemented and it could be much more complicated than that in previous method.
First of all, more than two intersections will be observed in Fig.~\ref{qdt_fig:qdt_cutting_point_moving_2_concave_after} and it may be mixed with case 3 in Fig.~\ref{qdt_fig:qdt_cutting_3}.
Moreover, the concave cell may be cut into three instead of two which create extra difference.
As a consequence, the sequence of all intersections may need to be recorded in order to distinguish it from case 3 in Fig.~\ref{qdt_fig:qdt_cutting_3} and special treatment in cutting concave into more than three parts need to be implemented as well.
Generally speaking, this approach may be much more complex than solving background mesh issue when moving points before cutting.
%=====================================================================================================================%
% \subsection{Scaling center for concave elements}
% later

%
%=====================================================================================================================%
\subsection{Color the region}
\paragraph{}
% intro
Multiple material properties and holes may appear in the input and the programme is expected to assign each mesh with correct material id, or color.
Each different material will be related to one color and the hole can be regarded as void material.
There are two ways to achieve this goal: method based on geometry and that based on topology.
Using geometric location to find the color is easy to implemented and not sensitive to the correctness of the mesh but may introduce errors in coloring.
While using topological property requires more computational cost and is sensitive to the validity of the mesh but usually provide correct coloring if former steps are correct.
Considering the possibility of coloring becoming the bottle neck of the whole programme is rarely possible and the fact that the mesh are expected to be valid, topological property may be preferred in most of the cases.

\paragraph{}
% geometry
Geometric location can provide a good reference of the color in most of the case but may fail in some situation.
Calculating geometric geometric center and finding which bounded area is it located is a computational cheap work and can be parallelize to utilize multiple thread or map-reducer using powerful clouding computing.
However it can fail in some cases especially for a concave polygon whose geometric center is outside of itself as shown in Fig.~\ref{qdt_fig:qdt_coloring_geo_problem}.
\begin{figure}[h!]
    \centering
    \scalebox{1.2}{
        \includegraphics{quadtree/images/qdt_coloring_geo.eps}
    }
    \caption[Coloring problem using location of geometric center in concave polygon]{Coloring problem using location of geometric center in concave polygon}
    \label{qdt_fig:qdt_coloring_geo_problem}
\end{figure}
%
Using the scaling center which is always inside of the polygon instead of geometric center can not prevent it from happening.
Fig.~\ref{qdt_fig:qdt_coloring_geo_problem_sc} discusses the possibility of coloring failure in convex cell where the geometric center is coincide with scaling center.
The coordinate of the scaling center of the rectangle in the left is $(\frac{l-a}{2},\frac{l}{2})$.
If the x coordinate of point $C$ is less than $\frac{l-a}{2}$, angle $\alpha$ is so significant that the point $C$ is not recognized as the hard point and $a$ is not so large compared to $l$ that the point $A$ and $B$ will not be merged to the node of the rectangle.

    \begin{figure}[h!]
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_coloring_geo_sc.eps}
        }
        \caption[Coloring problem using location of scaling center]{Coloring problem using location of scaling center}
        \label{qdt_fig:qdt_coloring_geo_problem_sc}
    \end{figure}
%
In other words, coloring failure may happen if
    \begin{equation}
        \begin{aligned}
            C_x  &\leq \frac{l-a}{2} \\
            \alpha &> \epsilon_{1} \\
            \frac{l-a}{l} &> \epsilon_{2}
        \end{aligned}
    \end{equation}
where $C_x$ refer to the $x$ coordinate of the point $C$, $\epsilon_1$ is the tolerance used in hard point definition and $\epsilon_2$ is the tolerance used for point moving in section.~\ref{qdt_sc:point_moving}.
Assuming the point $C$ is located on the geometric center of the rectangle in the left, or $C_x = \frac{l-a}{2}$, it is concluded that
    \begin{equation}
        \begin{aligned}
            2 \arctan \frac{l}{l-a} &> \epsilon_1 \\
            \frac{l-a}{l} &> \epsilon_{2}
        \end{aligned}
    \end{equation}
%
Result of minimal $\epsilon_1$ with different $\epsilon_2$ is summarized in table.~\ref{qdt_tab:qdt_coloring_ep1_ep2}.
Setting $\epsilon_1$ greater than $150^\circ$ may result in large amount of unexpected hard points and a $\epsilon_2 > 0.35$ could lead to significantly increased points moving.
A combination of $\epsilon_1 = 150^\circ$ and $\epsilon_2 = 0.3$ seems to be the only possible solution.
However, sticking on the parameters that produces some unexpected hard points and points moving may not surpass the advantage of this method.

    \begin{table}[h!]
        \centering
        \caption{Minimal $\epsilon_2$ with different $\epsilon_1$}
        \label{qdt_tab:qdt_coloring_ep1_ep2}
        \begin{tabular}{ccccccccc}
            \toprule
            $\epsilon_1$(Deg) & 100    & 110      & 120     & 130      & 140     & 150   &   160  &  170   \\
            \midrule
            $\epsilon_2$ & 0.84 & 0.70 & 0.58 & 0.47 & 0.36 & 0.27 &    0.18 &  0.09\\
            \bottomrule
        \end{tabular}
    \end{table}
% topology
A method based on the topology, on the other hand, provides correct coloring in theory if the mesh generated is valid.
In other words, the mesh satisfies the following requirements:
    \begin{itemize}
        \item The background mesh shall be refined properly so that situation in Fig.~\ref{qdt_fig:qdt_cutting_3} will not appear
        \item The Hard points are set properly
        \item The input has ben properly scaled so that there is a few outer margin
    \end{itemize}
The coloring algorithm will utilize a breadth-first search algorithm to find color for each of the cell.
Any polygon can be picked as the root element of the search tree.
For convenience, the most bottom-left element is defined as the root of the search tree.
All neighbor polygons will be added as child nodes recursively until a cell whose boundary is shared by another color is found.
Once such a cell is found, the algorithm then consider the whole branch share same color.
The loops stops when there are not any unvisited polygons found under all available parent nodes.
Any uncolored polygons will have the same color as its youngest parent's.
Take Fig.~\ref{qdt_fig:qdt_color_example} as an example,

\begin{figure}[h!]
    \begin{subfigure}[b]{0.5\linewidth}
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_coloring_example_geometry.eps}
        }
        \caption{Geometry of an example for coloring}
    \end{subfigure}
    \begin{subfigure}[b]{0.5\linewidth}
        \centering
        \scalebox{0.8}{
            \includegraphics{quadtree/images/qdt_coloring_example_mesh.eps}
        }
        \caption{Mesh of an example for coloring}
    \end{subfigure}
    \caption{Example for coloring}
    \label{qdt_fig:qdt_color_example}
\end{figure}
% step 1
The root element will be the polygon $0$ and its neighbor polygons $1$ and $4$ will be appended as child node.
Since no boundary of polygon $1$ and $4$ is the border between two materials, the recursion goes on.
The polygon $15$ then will be found as the child node of the polygon $1$, the recursion will pause before polygon $5$ being append to polygon $1$.
The hypotenuse of the polygon $15$ is the border between the color $1$ and $-1$, considering the fact that the known color of its youngest parent (the polygon $0$) is $1$, whole branch (the polygon $0,1$ and $15$) will be marked as color $1$.
By knowing the hypotenuse of the polygon $15$ can only be contained in one other polygon, next child node, polygon $2$, will be marked as color $-1$ in next recursion.
    \begin{figure}[h!]
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_coloring_example_tree_0.eps}
        }
        \caption{Example for coloring: step 1}
        \label{qdt_fig:qdt_color_tree_0}
    \end{figure}
% step 2
The recursion resumes for the polygon $1$ and the polygon $5$ shall be appended and the polygon $4$ find $5$ and $12$.
Due to the fact that the polygon $5$ is visited by the polygon $1$, it will not be added to the polygon $4$'s child.
After that, the polygon $15$ find the polygon $2$, whose bottom boundary is the border between the color $1$ and $-1$ (background color is set as $1$ and hence it is assumed color outside of the bonding box is $1$).
The oldest node with known color is itself, the polygon $2$ and as a result the branch with single element will be marked as color $-1$.
Bottom edge of the polygon $2$ is contained in itself only, which implies the termination of the branch.
As another child node of polygon $15$, the polygon $6$ , whose hypotenuse is border between two colors and the youngest parent with known color will be polygon $15$ which is marked as color $1$.
The polygon $13$ will be marked as color $-1$ in next recursion.
%
    \begin{figure}[h!]
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_coloring_example_tree_1.eps}
        }
        \caption{Example for coloring: step 2}
        \label{qdt_fig:qdt_color_tree_1}
    \end{figure}
% done
Fig.~\ref{qdt_fig:qdt_color_tree_2} illustrates the final search tree after a few more recursion.
End node polygon $11$ and $3$ do not trigger the termination process normally will be marked with color in the end with the youngest parent rule, which gives branch $11,10,12,4$ and branch $3$ the same color as polygon $0$ and $2$.

    \begin{figure}[h!]
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/qdt_coloring_example_tree_2.eps}
        }
        \caption{Example for coloring: final tree}
        \label{qdt_fig:qdt_color_tree_2}
    \end{figure}