\section{Introduction}\label{section:introduction}

Language Models have become a core component in many Natural Language Processing (NLP) tasks.
These models are typically pre-trained on unlabelled texts, and then further fine-tuned using labelled data relevant to the target task.
Transformer-based \cite{NIPS2017_3f5ee243} contextual models such as BERT \cite{devlin-etal-2019-bert} have gained success since the fine-tuning step is relatively inexpensive, while attaining state-of-the-art results in various syntactic and semantic tasks.

While the bulk of work with the BERT family of models focuses on English, there have been some monolingual models developed for other languages as well (\citealp{martin-etal-2020-camembert, polignano2019alberto, antoun-etal-2020-arabert, BERTje, FinBERT, agerri-etal-2020-give}; inter alia).
These monolingual models have been trained on large volumes of data, typically amounting to billions of tokens.
In contrast, it is challenging to find publicly available corpora of this size for low-resource languages.
The evaluation benchmarks for downstream tasks on these languages are also limited, and tend to be dominated by low-level structural tagging tasks.

To counteract the lack of large volumes of monolingual corpora for low-resource languages, a number of multilingual models have been released, such as mBERT \cite{devlin-etal-2019-bert} and XLM-R \cite{conneau-etal-2020-unsupervised}.
These multilingual models were pre-trained on more than one language at a time by combining corpora from different languages, usually sourced from Wikipedia.
Several works have demonstrated the efficacy of these multilingual models, especially for languages without a language-specific model \cite{kondratyuk-straka-2019-75, wu-dredze-2019-beto}.
Benchmark results have improved for many languages by leveraging cross-linguistic features learnt by these multilingual models \cite{conneau-etal-2020-unsupervised}.

However, the gains with multilingual models may vary depending on the language being considered.
The ``curse of multilinguality'' limits the language-specific features that these models can learn, since the limited model capacity has to be shared between multiple languages \cite{conneau-etal-2020-unsupervised}.
Models such as mBERT use WordPiece tokenisation \cite{johnson-etal-2017-googles}, which splits words into various sub-tokens, thereby reducing the number of unknown tokens.
However, the vocabulary representations for multilingual models tend to be sub-optimal for specific languages, because words tend to be split into a higher number of sub-tokens \cite{rust-etal-2021-good}.
Moreover, these models may still be biased in favour of over-representing sub-tokens common to a certain subset of languages over others.
Due to the data imbalance across languages, lower-resourced languages tend to be disadvantaged, as there is relatively less pre-training data available compared to the other languages considered in the multilingual model \cite{wu-dredze-2020-languages}.

Apart from the tension between languages in a multilingual model, other factors are at play as well.
Most prominently, many languages are never seen by these multilingual models \cite{muller-etal-2021-unseen}, since these are typically trained on the largest-available corpora (e.g. mBERT was pre-trained on the 104 languages with the greatest Wikipedia presence).
Such criteria exclude many of the world's languages, including Maltese, the focus of this paper.
This issue is exacerbated even further when the language uses a script which is either different to its closely related languages \cite{muller-etal-2021-unseen}, or which is never seen during pre-training, thereby encoding most of the input with out-of-vocabulary tokens \cite{pfeiffer-etal-2021-unks}.
In fact, \citet{muller-etal-2021-unseen} show that the language transfer capability of a multilingual model to an unseen language is dependent on the degree to which the target language is related to languages already included in the multilingual model.

In this work we focus on the Maltese language, an official EU language spoken primarily in Malta and in some small communities around the world \cite{Brincat2011}.
It is the only Semitic language written exclusively with a Latin script, containing a few additional characters with diacritic marks (ċ, ġ, \hwithstroke, ż).
The language also has strong influences from Romance languages such as Italian, as well as English.
The Semitic influence is largely exhibited in the grammatical structure through complex morphological characteristics, whilst the non-Semitic aspect is predominantly observed in its vocabulary, with extensive lexical borrowing from Italian and English.

In the context of NLP, Maltese is a low-resource language~\cite{Rosner2022} and is not part of the languages covered by either mBERT or XLM-R.
\citet{muller-etal-2021-unseen} find that mBERT underperforms non-contextual baselines on Maltese, but benefits when pre-trained further on raw Maltese data.
Similarly, \citet{chau-etal-2020-parsing} further pre-train mBERT but impute the 99 unused tokens present in the model with language specific tokens, yielding  better results. This confirms previous findings by \citet{wang-etal-2020-extending}, who also extend mBERT's vocabulary to accommodate unseen languages, but do so by extending the vocabulary and model dimensionality, hence increasing its footprint.

Motivated by the limitations of existing multilingual models and the deficiency of publicly available corpora for Maltese, we set out to pre-train a new monolingual language model for Maltese and compare it to the alternative strategy of further pre-training an existing multilingual model.
We study, in particular, the impact that the pre-training data size and domain has on the performance in downstream tasks.
The main contributions of this work are as follows:
\begin{enumerate}
    \item We develop a new corpus of Maltese text.
    \item Using this new data, language models for Maltese are pre-trained.
    \item We compare the newly pre-trained models and find that both models improve the state-of-the-art on three structural tagging tasks -- dependency parsing, part-of-speech tagging, and named-entity recognition -- and one semantic classification task -- sentiment analysis.
    \item We demonstrate that in a low-resource setting, pre-training using text from varied domains is often superior to solely using Wikipedia, and that matching the domain to target task is beneficial when this is available.
    \item We also provide an analysis on the effects of the pre-training size, shedding new light on how much pre-training data is needed to attain significant improvements in performance.
\end{enumerate}

We make this new corpus, the newly pre-trained language models, and the code publicly available\footnote{The corpus and the language models are available at the Hugging Face Hub at \url{https://huggingface.co/datasets/MLRS/korpus_malti}, \url{https://huggingface.co/MLRS/BERTu}, and \url{https://huggingface.co/MLRS/mBERTu}. The code is available at \url{https://github.com/MLRS/BERTu}.}.

\section{Corpus}\label{section:corpus}

In this work, we build a new unlabelled text corpus, which we call the \textbf{Korpus Malti v4.0 (KM)}.
This builds on and extends an existing corpus, Korpus Malti v3.0\footnote{See: \url{https://mlrs.research.um.edu.mt}}, which is approximately half the size.

Rather than scraping the web randomly for Maltese text, we collect text data from specific sources, including both online and offline.
Although this does incur additional effort in data collection, and results in a smaller dataset compared to large-scale web-scraping initiatives, it has the benefit of resulting in a less noisy dataset, while offering greater control over sources.
For comparison, the Maltese portion of the OSCAR data \cite{OrtizSuarezSagotRomary2019}, which is sourced entirely from the web, contains texts which, to a native speaker, suggest that they are automatically generated through the use of a low-quality machine translation system, a common pitfall of web-scraping for low-resource languages \cite{10.1162/tacl_a_00447}.
We also expect to find a small proportion of code-switched texts, as this is a pre-dominant phenomenon for Maltese in domains such as social media or transcribed speech.
In addition, the data is separated into different domains, and the source for each document is available as part of the metadata.
This allows data users to select data subsets which are more appropriate for their particular use-case, such as domain-adaptive pre-training (\citealp{BioBERT, gururangan-etal-2020-dont}; inter alia), whilst enabling tracing back to the original source, or omission in case unforeseen ethical or privacy issues come to light.
In short, the goal was to build a good quality training dataset, while avoiding at least some of the pitfalls identified with opportunistic, web-scale data initiatives \cite{bender-2021-parrots, rogers-2021-changing}.

Data is collected from a variety of sources, including online news sources, legal texts, transcripts of speeches and debates, blogs, Wikipedia, etc.
Before texts are included in the corpus, we filter non-Maltese sentences using language identification using LiD \cite{lui-baldwin-2014-accurate}, and perform de-duplication using Onion \cite{pomikalek2011removing}.

The resulting data, split into 19 different domains, is summarised by Table~\ref{table:data}.

\begin{table*}
    \centering
    \begin{tabular}{|l|r|r|r|r|}
        \hline
        \multicolumn{1}{|l|}{\textbf{data subset}} & \multicolumn{1}{c|}{\textbf{documents}} & \multicolumn{1}{c|}{\textbf{sentences}} & \multicolumn{1}{c|}{\textbf{tokens}} & \multicolumn{1}{c|}{\textbf{size}} \\
        \hline
        belles\_lettres & $195$ & $299\,762$ & $4\,454\,906$ & $21.82$MB \\
        blogs & $25\,436$ & $807\,628$ & $14\,562\,039$ & $74.45$MB \\
        comics & $62$ & $2\,413$ & $44\,768$ & $233.22$KB \\
        court & $2\,663$ & $694\,227$ & $11\,881\,638$ & $61.91$MB \\
        eu\_docs & $2\,974$ & $5\,099\,564$ & $135\,811\,945$ & $773.25$MB \\
        government\_gazette & $2\,974$ & $1\,881\,034$ & $39\,771\,556$ & $203.61$MB \\
        gov\_docs & $272$ & $120\,209$ & $1\,900\,842$ & $10.79$MB \\
        law\_eu & $71$ & $4\,433\,235$ & $98\,582\,031$ & $541.13$MB \\
        law\_mt & $2\,596$ & $401\,118$ & $7\,631\,651$ & $38.84$MB \\
        legal & $3$ & $4\,784$ & $83\,581$ & $490.67$MB \\
        nonfiction & $2\,177$ & $208\,763$ & $3\,902\,436$ & $20.01$MB \\
        parliament & $6\,198$ & $3\,935\,906$ & $82\,294\,520$ & $433.09$MB \\
        press\_eu & $5\,483$ & $413\,317$ & $9\,774\,919$ & $55.73$MB \\
        press\_mt & $46\,782$ & $713\,886$ & $17\,679\,904$ & $93.15$MB \\
        speeches & $62$ & $2\,067$ & $51\,259$ & $286.63$MB \\
        theses & $19$ & $11\,545$ & $310\,243$ & $1.63$MB \\
        umlib\_oar & $11\,688$ & $963\,606$ & $21\,235\,949$ & $106.11$MB \\
        web\_general & $2$ & $685\,873$ & $14\,741\,525$ & $75.22$MB \\
        wiki & $3\,469$ & $79\,134$ & $1\,885\,661$ & $9.73$MB \\
        \hline
        all & $131\,429$ & $20\,758\,071$ & $466\,601\,373$ & $2.52$GB \\
        \hline
    \end{tabular}
    \caption{Korpus Malti v4.0 corpus distribution. \textit{belles\_lettres} is largely composed of literary works; the \textit{government\_gazzette} consists of text from the official newsletter of the Maltese government; \textit{umlib\_oar} is a miscellaneous collection of previously published non-fiction texts, available in the public domain via the University of Malta Library Open Access Repository.}\label{table:data}
\end{table*}

To the best of our knowledge, there is no corpus of this size available for Maltese.
We also note that this data is a significant increase over Wikipedia data, which is what is usually available and used in low-resource scenarios.
The Wikipedia data makes up less than 1\% of the entire corpus in terms of both tokens and sentences.

Despite this substantial increase in data, we emphasise that a corpus of under 500M tokens is still substantially smaller than is typically used for higher-resourced languages.
For example, \citet{devlin-etal-2019-bert} pre-train BERT using a combined corpus of 3.3B words for English (approximately 16GB).
Larger models have since exceeded these pre-training sizes by a wide margin -- for example, RoBERTa is pre-trained on 161GB of text \cite{liu-etal-2019-roberta}.
Monolingual models for languages other than English, typically use smaller corpora than English models, but their size is still significantly larger than ours -- for example AraBERT was pre-trained on a corpus of 24GB \cite{antoun-etal-2020-arabert} and BERTje was pre-trained on a corpus of 12GB \cite{BERTje}.

\section{Language Models}\label{section:models}

Using this new corpus, two new language models are pre-trained for Maltese: a monolingual model (\textbf{BERTu}) and a multilingual model (\textbf{mBERTu}).
In both cases, pre-training is performed using the Masked Language Modelling Objective (MLM) only, since the Next Sentence Prediction (NSP) objective was found to be detrimental to downstream performance \cite{joshi-etal-2020-spanbert, liu-etal-2019-roberta}.
Other than that, pre-training largely follows the pre-training setup of BERT \cite{devlin-etal-2019-bert}.
This allows for a better comparison with already available models.
The pre-training data from all domains is combined, shuffled, and split into 85\% and 15\% for training and validation sets respectively.

\paragraph{BERTu}
We pre-train a monolingual BERT model from scratch on the new unlabelled data, using the BERT\textsubscript{\textsc{BASE}} architecture with 12 transformer layers, a hidden size of 768, and 12 attention heads.
The vocabulary is initialised with 52K tokens.
Pre-training is done across 1M steps, with a sequence length of 128 for the first 90\% of the steps and a sequence length of 512 for the remaining 10\% steps.
A batch size of 512 is used, which amounts to approximately 30 epochs in total, and a warmup of 1\% of the total number of steps.
We use mixed-precision training to ease memory requirements.
Training was performed on 8 A100 GPUs for the first 90\% steps and 16 A100 GPUs for the remaining 10\% steps, taking approximately 53 hours.

\paragraph{mBERTu}
Similar to \citet{chau-etal-2020-parsing} and \citet{muller-etal-2021-unseen} we also pre-train mBERT further on Maltese.
Since the embedding weights are not randomly initialised, as is the case for the monolingual model, we follow \citet{rust-etal-2021-good} and pre-train for 250K steps.
A sequence length of 512 is used throughout, keeping the rest of the hyper-parameters the same as the monolingual pre-training.
To better fit the Maltese language, the mBERT vocabulary is augmented with Maltese tokens following the procedure from \citet{chau-etal-2020-parsing}, by replacing the unused tokens reserved in the original vocabulary.
Specifically, we train a tokeniser with a vocabulary size of $5\,000$ tokens on the data and choose the set of 99 tokens which reduce the number of \texttt{[UNK]} tokens the most in the target data.
Training was performed on 32 A100 GPUs, and took around 46 hours to complete.

\section{Evaluation}\label{section:evaluation}

An evaluation for the language models described in Section~\ref{section:models} is presented here.
\textbf{mBERT} without any additional pre-training is used as one of the baselines.
In addition, we pre-train two language models on the Maltese Wikipedia data as additional baselines.
This allows us to analyse the limitations that could be faced when following the common practice of using Wikipedia data, for the specific case of low-resource languages with a comparatively small Wikipedia footprint.

Following the same setup of the main models, a monolingual model (\textbf{BERTu Wiki}) and a multilingual model (\textbf{mBERTu Wiki}) are pre-trained.
The same hyper-parameters described in Section~\ref{section:models} are used, but the batch size and number of steps are decreased to prevent overfitting due to the smaller data size.
To this end, the batch size is set to 64 and the total number of steps set to $30\,500$ and $7\,600$ steps for the monolingual and multilingual models, respectively.
This was deemed appropriate since it would amount to the same number of epochs as the models pre-trained on the entire corpus.

\subsection{Tasks}

The language models are fine-tuned on the following downstream tasks.
A summary of the datasets and fine-tuning architectures used is given below.

\paragraph{Dependency Parsing (DP)}
The Maltese Universal Dependencies Treebank (MUDT) \cite{MUDT} is used for this task using the provided training, validation, and testing splits.
The data is composed of $2\,074$ human-annotated sentences from 4 different high-level domains.
Similar to \citet{chau-etal-2020-parsing}, \citet{muller-etal-2021-unseen}, and \citet{chau-smith-2021-specializing}, we use a Biaffine graph-based prediction layer \cite{dozat-2019-biaffine} and use the Labelled Attachment Score (LAS) as the main evaluation metric, but also report the Unlabelled Attachment Score (UAS).

\paragraph{Part-of-Speech Tagging (POS)}
The MLRS POS data \cite{gatt-ceplo-2013-korpus}, is used for this task.
This data is composed of $6\,167$ human-annotated sentences -- $426$ of which overlap with the MUDT data \cite{MUDT} -- and are stratified into 8 domains.
We combine the data from the different domains, shuffle it, and split the data into 80\%, 10\%, and 10\% for training, validation, and testing sets, respectively.
The annotations are language-specific tags (using the XPOS scheme) and we follow the tag mapping in MUDT \cite{MUDT} to also produce tags in the Universal Part of Speech tagset (UPOS).
To evaluate tagging with these two tagsets, we use a linear layer, and use accuracy as the evaluation metric.

\paragraph{Named-Entity Recognition (NER)}
The Maltese annotations for the WikiAnn data \cite{pan-etal-2017-cross} are used for this task, using the data splits from \citet{rahimi-etal-2019-massively}.
The data is made up of $300$ sentences derived from Wikipedia.
Following \citet{chau-smith-2021-specializing}, a Conditional Random Field layer is used for this task, and we use F1 as the evaluation metric.

\paragraph{Sentiment Analysis (SA)}
We use the Maltese sentiment analysis dataset by \citet{martinez-garcia-etal-2021-evaluating}, which is a collection of $815$ sentences, using the provided training, validation, and testing splits.
The texts in this data originate from comments on news articles and social media posts, and are a combination of two datasets from \citet{cortis-davis-2019-social} and \citet{dingli-sant-2016-sentiment}.
A linear prediction layer is used, and we use the macro-averaged F1 score as the evaluation metric.

We largely use the hyper-parameters from \citet{chau-smith-2021-specializing}, but optimise the learning rate, batch size, and dropout on the validation set of each task.
Table~\ref{table:hyperparameters} shows the chosen hyper-parameters.
Fine-tuning is performed for at most 200 epochs, with an early stopping of 20 epochs  on the validation set.

\begin{table}
    \centering
    \begin{tabular}{|l|cccc|}
        \hline
        Name & DP & POS & NER & SA \\
        \hline
        Learning Rate & 5e-4 & 5e-4 & 5e-4 & 1e-4 \\
        Batch Size & 128 & 128 & 64 & 32 \\
        Dropout & 0.3 & 0.3 & 0.2 & 0.5 \\
        \hline
    \end{tabular}
    \caption{Fine-tuning hyper-parameters}
    \label{table:hyperparameters}
\end{table}

\subsection{Results}\label{section:results}

The results on all tasks are summarised in Table~\ref{table:results}.
Consistent with the results reported by \citet{muller-etal-2021-unseen} and \citet{chau-smith-2021-specializing}, BERTu Wiki generally underperforms mBERT, and mBERTu Wiki performs better than mBERT.
Whilst they show this for Dependency Parsing, Part-of-Speech tagging, and Named Entity Recognition, we demonstrate that this also holds for Sentiment Analysis.

Our baseline results diverge slightly from previous results on the Named-Entity Recognition task, where BERTu Wiki performs slightly better than mBERT.
We suspect that this is due to a slightly different pre-training setup than that used by \citet{muller-etal-2021-unseen} and \citet{chau-smith-2021-specializing}\footnote{\citet{muller-etal-2021-unseen} pre-train for at most 10 epochs whilst \citet{chau-smith-2021-specializing} pre-train for at most 20 epochs (choosing the best performing model on based on the validation set). Both use a smaller-sized BERT architecture with 6 layers and pre-train with a maximum sequence length of 128.}, but we analyse this further in Section~\ref{section:analysis-domains}.
However, these results remain consistent with regards to BERTu Wiki not performing as well as mBERTu Wiki.

\begin{table*}
    \centering
    \begin{subtable}{0.55\linewidth}
        \centering
        \begin{tabular}{|ll|rr|}
            \hline
            Data & Model & \multicolumn{1}{c}{UAS} & \multicolumn{1}{c|}{LAS} \\
            \hline\hline
            \multirow{2}{*}{Wiki} & BERTu & 80.95 \small{\textpm~0.25} & 74.16 \small{\textpm~0.20} \\
            & mBERTu & 88.74 \small{\textpm~0.11} & 82.59 \small{\textpm~0.19} \\
            \hline
            \multirow{1}{*}{N/A} & mBERT & 84.83 \small{\textpm~0.31} & 77.22 \small{\textpm~0.34} \\
            \hline
            \multirow{2}{*}{KM} & BERTu & ~\textbf{92.31 \small{\textpm~0.15}} & ~\textbf{88.14 \small{\textpm~0.21}} \\
            & mBERTu & *92.10 \small{\textpm~0.14} & *87.87 \small{\textpm~0.18} \\
            \hline
        \end{tabular}
        \caption{Dependency Parsing}\label{table:results-dp}
    \end{subtable}
    \begin{subtable}{0.4\linewidth}
        \centering
        \begin{tabular}{|ll|r|}
            \hline
            Data & Model & \multicolumn{1}{c|}{span F1} \\
            \hline\hline
            \multirow{2}{*}{Wiki} & BERTu & 67.96 \small{\textpm~2.20} \\
            & mBERTu & *85.01 \small{\textpm~2.92} \\
            \hline
            \multirow{1}{*}{N/A} & mBERT & 65.41 \small{\textpm~2.06} \\
            \hline
            \multirow{2}{*}{KM} & BERTu & ~\textbf{86.77 \small{\textpm~3.55}} \\
            & mBERTu & *86.60 \small{\textpm~2.49} \\
            \hline
        \end{tabular}
        \caption{Named Entity Recognition}\label{table:results-ner}
    \end{subtable}
    \begin{subtable}{0.55\linewidth}
        \centering
        \begin{tabular}{|ll|rr|}
            \hline
            Data & Model & \multicolumn{1}{c}{UPOS} & \multicolumn{1}{c|}{XPOS} \\
            \hline\hline
            \multirow{2}{*}{Wiki} & BERTu & 97.27 \small{\textpm~0.11} & 97.01 \small{\textpm~0.07} \\
            & mBERTu & 97.95 \small{\textpm~0.13} & 97.83 \small{\textpm~0.08} \\
            \hline
            \multirow{1}{*}{N/A} & mBERT & 97.26 \small{\textpm~0.15} & 97.20 \small{\textpm~0.14} \\
            \hline
            \multirow{2}{*}{KM} & BERTu & 98.58 \small{\textpm~0.02} & *98.54 \small{\textpm~0.03} \\
            & mBERTu & ~\textbf{98.66 \small{\textpm~0.03}} & ~\textbf{98.58 \small{\textpm~0.04}} \\
            \hline
        \end{tabular}
        \caption{Part-of-Speech tagging}\label{table:results-pos}
    \end{subtable}
    \begin{subtable}{0.4\linewidth}
        \centering
        \begin{tabular}{|ll|r|}
            \hline
            Data & Model & \multicolumn{1}{c|}{macro-F1} \\
            \hline\hline
            \multirow{2}{*}{Wiki} & BERTu & 53.95 \small{\textpm~2.70} \\
            & mBERTu & 56.05 \small{\textpm~3.24} \\
            \hline
            \multirow{1}{*}{N/A} & mBERT & 55.99 \small{\textpm~3.63} \\
            \hline
            \multirow{2}{*}{KM} & BERTu & ~\textbf{78.96 \small{\textpm~1.95}} \\
            & mBERTu & *76.79 \small{\textpm~1.79} \\
            \hline
        \end{tabular}
        \caption{Sentiment Analysis}\label{table:results-sa}
    \end{subtable}
    \caption{
        Experimental results, grouped by the underlying language model and additional pre-training data used.
        All figures shown are the mean and standard deviations over 5 runs with different random seeds.
        The best performing models for each metric are \textbf{bolded}.
        Values marked with * are not found to be significantly worse than the best model (using a 1-tailed \textit{t}-test with a \textit{p}-value = 0.05 with Bonferroni correction).
    }\label{table:results}
\end{table*}

Both language models pre-trained with the KM data perform significantly better than all the other baselines, on all tasks except for Named-Entity Recognition, where the trend is similar, but does not reach statistical significance.
This underlines the value of this new corpus.
When compared to the Wikipedia language models, the most noticeable improvements can be seen between the BERTu models, across all tasks.
A more detailed analysis on this is presented in Section~\ref{section:analysis-size}, but intuitively this finding makes sense since mBERTu models are exposed to significantly more data, making them less specific to Maltese.

The gap in performance between the BERTu and mBERTu models is much less for the KM pre-trained models than it is for the Wikipedia pre-trained models.
In fact, on average, the BERTu KM model performs better than the mBERTu KM model on all tasks except Part-of-Speech tagging.
For Part-of-Speech tagging we also note that the baseline results are already quite high, probably due to the relatively larger labelled data, which may partially mask the effects of the KM models.

Overall, Sentiment Analysis is the task where the most gains are made with respect to the baselines.
The KM-trained models are over 20 F1 points higher than the best performing baseline.
This finding provides evidence that, unlike syntactic tasks, where structural information could potentially be shared across related clusters of languages, semantic tasks such as Sentiment Analysis will benefit much more from language-specific embeddings.

\section{Analysis}\label{section:analysis}

In this section we build on the results presented in Section~\ref{section:results}, and analyse the effect that the pre-training data has on performance on the downstream tasks.

\subsection{Data Domain}\label{section:analysis-domains}

In this subsection we take advantage of the fact that the data is stratified by domains.
Here, we analyse the impact of pre-training using data from different domains, compared to a single domain, namely Wikipedia, which is commonly used in multilingual models and low-resource settings.
For this purpose, we consider the BERTu Wiki and mBERTu Wiki baselines from Section~\ref{section:evaluation} as single-domain models.
We compare them to language models pre-trained with the same amount of data but from different domains, which are referred to as ``Mixed'' in this discussion.

We determine the size using the number of sentences, since this would directly effect the number of epochs and allows us to keep an identical pre-training setup as the Wikipedia-trained models.
Since the Maltese Wikipedia data is composed of $79\,134$ sentences, the Mixed language models are also pre-trained with the same amount of sentences, split into training and validation sets as the Wikipedia models.
A comparison of the downstream task performance for these language models is plotted in Figure~\ref{figure:domain}.

\begin{filecontents*}{bert_mt_small.dat}
    data UAS_m UAS_std LAS_m LAS_std UPOS_m UPOS_std XPOS_m XPOS_std NER_m NER_std SA_m SA_std
    Wiki 0.8095 0.0025 0.7416 0.0020 0.9727 0.0011 0.9701 0.0007 0.6796 0.0220 0.5395 0.0270
    Mixed 0.8063 0.0020 0.7344 0.0025 0.9726 0.0003 0.9708 0.0009 0.5775 0.0396 0.5982 0.0298
\end{filecontents*}
\begin{filecontents*}{mbert_mt_small.dat}
    data UAS_m UAS_std LAS_m LAS_std UPOS_m UPOS_std XPOS_m XPOS_std NER_m NER_std SA_m SA_std
    Wiki 0.8874 0.0011 0.8259 0.0019 0.9795 0.0013 0.9783 0.0008 0.8501 0.0292 0.5605 0.0324
    Mixed 0.8888 0.0034 0.8281 0.0031 0.9807 0.0003 0.9794 0.0005 0.7595 0.0233 0.6174 0.0251
\end{filecontents*}
\begin{filecontents*}{mbert_small.dat}
    data UAS_m UAS_std LAS_m LAS_std UPOS_m UPOS_std XPOS_m XPOS_std NER_m NER_std SA_m SA_std
    Wiki 0.8483 0.0031 0.7722 0.0034 0.9726 0.0015 0.9720 0.0014 0.6541 0.0206 0.5599 0.0363
    Mixed 0.8483 0.0031 0.7722 0.0034 0.9726 0.0015 0.9720 0.0014 0.6541 0.0206 0.5599 0.0363
\end{filecontents*}
\newcommand{\domainplot}[3]{
    \nextgroupplot[
        title = {#3},
        xlabel = {Pre-training Data Domain},
        ylabel = {#2},
        legend style = {
            legend columns = -1,
            align = center,
            column sep = 10pt,
        },
        legend to name = {legend:domain},
        symbolic x coords = {Wiki, Mixed},
        xtick distance = 1,
    ]

    \addplot+[
        error bars/.cd,
        y dir = both,
        y explicit,
    ] table[
        x = {data},
        y expr = \thisrow{#1_m} * 100,
        y error plus expr = \thisrow{#1_std} * 100,
        y error minus expr = \thisrow{#1_std} * 100,
    ] {bert_mt_small.dat};
    \addlegendentry{BERTu}

    \addplot+[
        error bars/.cd,
        y dir = both,
        y explicit,
    ] table[
        x = {data},
        y expr = \thisrow{#1_m} * 100,
        y error plus expr = \thisrow{#1_std} * 100,
        y error minus expr = \thisrow{#1_std} * 100,
    ] {mbert_mt_small.dat};
    \addlegendentry{mBERTu}

    \addplot+[
        black,
        dashed,
        mark = none,
    ] table[
        y expr = \thisrow{#1_m} * 100,
    ] {mbert_small.dat};
    \addlegendentry{mBERT}

    \addplot[
        name path = upper,
        draw = none,
    ] table[
        x = {data},
        y expr = (\thisrow{#1_m} + \thisrow{#1_std}) * 100,
    ] {mbert_small.dat};
    \addplot[
        name path = lower,
        draw = none,
    ] table[
        x = {data},
        y expr = (\thisrow{#1_m} - \thisrow{#1_std}) * 100,
    ] {mbert_small.dat};
    \addplot[
        fill = black,
        opacity = 0.1,
    ] fill between[of=upper and lower];
}
\begin{figure*}[t]
    \centering
    \begin{tikzpicture}
        \begin{groupplot}[
            group style={
                group size = 2 by 2,
                horizontal sep = 70pt,
                vertical sep = 60pt,
            },
            width = 0.4\textwidth,
        ]
            \domainplot{LAS}{LAS}{Dependency Parsing}
            \coordinate (column1) at (rel axis cs:0,1);
            \domainplot{NER}{span F1}{Named-Entity Recognition}
            \coordinate (column2) at (rel axis cs:1,1);
            \domainplot{XPOS}{Accuracy (XPOS)}{Part-of-Speech Tagging}
            \domainplot{SA}{macro-F1}{Sentiment Analysis}
        \end{groupplot}
        \coordinate (centre) at ($(column1)!.5!(column2)$);
        \node[below] at (centre |- current bounding box.south){\pgfplotslegendfromname{legend:domain}};
    \end{tikzpicture}
    \caption{
        Downstream task performance with different pre-training data domains.
        All values are the mean over 5 runs with different random seeds.
        The standard deviation is represented by the corresponding error bars and shaded area.
    }\label{figure:domain}
\end{figure*}

At this small scale of data, both Wiki and Mixed mBERTu models consistently perform better than the mBERT models, owing to the multilingual representation power of these models.
The Mixed models perform better than their Wikipedia counterparts on Part-of-Speech tagging and Sentiment Analysis.
On Dependency Parsing, there is a slight improvement on the mBERTu model but a slight degradation on the BERTu model.

For Named-Entity Recognition, the Wikipedia models perform better than the Mixed ones.
Since the dataset for this task originates from Wikipedia as well,
this indicates that matching the pre-training data domain to the target domain boosts performance, supporting the findings by \citet{gururangan-etal-2020-dont}.
In fact, mBERTu Wiki surpasses mBERT, and proves to be a competitive baseline, as shown in Table~\ref{table:results-ner}.
On the other hand, mBERTu Mixed performs worse than mBERT.

The opposite is true for Sentiment Analysis, as BERTu Mixed turns out to be a more competitive baseline than mBERT.
The improvement is so pronounced for this task that the BERTu Mixed model not only performs better than the BERTu Wiki counterpart, but also better than mBERTu Wiki.
Even though this dataset contains texts exhibiting stylistic features expected in social media text, a mixture of domains is helpful, probably since Wikipedia texts tend to be quite structured and neutral in terms of the writing style and tone.
The results on sentiment analysis suggest that pre-training on a diversity of domains contribute to more effective learning of features relevant to discourse semantic tasks, compared to tasks involving morpho-syntactic tagging.
We leave further investigation of this, on a broader range of semantically-oriented tasks, for future work.

Overall, these results emphasise the importance of having pre-training data from sources close to the target data, even for low-resource settings.

\subsection{Data Size}\label{section:analysis-size}

From Table~\ref{table:results}, it is clear that the KM corpus translates to better performance on downstream tasks, regardless of whether a monolingual or a multilingual model is used.
To better understand the relationship between the data size and performance, we pre-train several language models with varying data sizes.

\begin{filecontents*}{bert_mt.dat}
    data UAS_m UAS_std LAS_m LAS_std UPOS_m UPOS_std XPOS_m XPOS_std NER_m NER_std SA_m SA_std
    0.0038123 0.8063 0.0020 0.7344 0.0025 0.9726 0.0003 0.9708 0.0009 0.5775 0.0396 0.5982 0.0298
    0.1 0.9064 0.0015 0.8587 0.0012 0.9830 0.0005 0.9824 0.0007 0.7258 0.0202 0.7329 0.0132
    0.2 0.9135 0.0005 0.8704 0.0007 0.9848 0.0008 0.9842 0.0003 0.7633 0.0356 0.7690 0.0069
    0.3 0.9179 0.0008 0.8737 0.0009 0.9857 0.0006 0.9845 0.0007 0.7751 0.0232 0.7528 0.0092
    0.4 0.9159 0.0017 0.8741 0.0016 0.9853 0.0005 0.9845 0.0004 0.7435 0.0289 0.7752 0.0138
    0.5 0.9213 0.0005 0.8792 0.0010 0.9853 0.0004 0.9848 0.0004 0.7834 0.0382 0.7481 0.0115
    0.6 0.9221 0.0010 0.8801 0.0018 0.9856 0.0003 0.9852 0.0006 0.7890 0.0291 0.7812 0.0295
    0.7 0.9245 0.0017 0.8838 0.0022 0.9866 0.0006 0.9858 0.0005 0.8259 0.0250 0.8203 0.0206
    0.8 0.9224 0.0021 0.8807 0.0037 0.9851 0.0006 0.9848 0.0002 0.7988 0.0451 0.7834 0.0155
    0.9 0.9266 0.0006 0.8855 0.0013 0.9857 0.0003 0.9855 0.0003 0.7942 0.0289 0.7928 0.0127
    1 0.9231 0.0015 0.8814 0.0021 0.9858 0.0002 0.9854 0.0003 0.8677 0.0355 0.7896 0.0195
\end{filecontents*}
\begin{filecontents*}{mbert_mt.dat}
    data UAS_m UAS_std LAS_m LAS_std UPOS_m UPOS_std XPOS_m XPOS_std NER_m NER_std SA_m SA_std
    0.0038123 0.8888 0.0034 0.8281 0.0031 0.9807 0.0003 0.9794 0.0005 0.7595 0.0233 0.6174 0.0251
    0.1 0.9057 0.0009 0.8572 0.0013 0.9838 0.0002 0.9829 0.0003 0.7937 0.0332 0.7214 0.0209
    0.2 0.9158 0.0014 0.8685 0.0019 0.9854 0.0005 0.9842 0.0001 0.8246 0.0254 0.7361 0.0192
    0.3 0.9165 0.0014 0.8716 0.0014 0.9853 0.0005 0.9848 0.0008 0.8293 0.0285 0.7163 0.0163
    0.4 0.9185 0.0016 0.8745 0.0017 0.9858 0.0006 0.9855 0.0004 0.8362 0.0187 0.7535 0.0272
    0.5 0.9213 0.0012 0.8775 0.0012 0.9857 0.0006 0.9848 0.0005 0.8673 0.0332 0.7408 0.0151
    0.6 0.9181 0.0021 0.8758 0.0025 0.9864 0.0005 0.9856 0.0006 0.8502 0.0154 0.7702 0.0265
    0.7 0.9217 0.0008 0.8790 0.0016 0.9864 0.0008 0.9851 0.0006 0.8403 0.0301 0.7481 0.0247
    0.8 0.9225 0.0008 0.8788 0.0015 0.9859 0.0006 0.9857 0.0004 0.8573 0.0059 0.7876 0.0056
    0.9 0.9225 0.0021 0.8789 0.0014 0.9859 0.0006 0.9859 0.0007 0.8531 0.0246 0.7707 0.0187
    1 0.9210 0.0014 0.8787 0.0018 0.9866 0.0003 0.9858 0.0004 0.8660 0.0249 0.7679 0.0179
\end{filecontents*}
\begin{filecontents*}{mbert.dat}
    data UAS_m UAS_std LAS_m LAS_std UPOS_m UPOS_std XPOS_m XPOS_std NER_m NER_std SA_m SA_std
    0 0.8483 0.0031 0.7722 0.0034 0.9726 0.0015 0.9720 0.0014 0.6541 0.0206 0.5599 0.0363
    1 0.8483 0.0031 0.7722 0.0034 0.9726 0.0015 0.9720 0.0014 0.6541 0.0206 0.5599 0.0363
\end{filecontents*}
\newcommand{\dataplot}[3]{
    \nextgroupplot[
        title = {#3},
        xlabel = {Pre-training Data Proportion (\%)},
        ylabel = {#2},
        xmin = 0, xmax = 100,
        legend style = {
            legend columns = -1,
            align = center,
            column sep = 10pt,
        },
        legend to name = {legend:size},
    ]

    \addplot table[
        x expr = \thisrow{data} * 100,
        y expr = \thisrow{#1_m} * 100,
    ] {bert_mt.dat};
    \addlegendentry{BERTu}

    \addplot table[
        x expr = \thisrow{data} * 100,
        y expr = \thisrow{#1_m} * 100,
    ] {mbert_mt.dat};
    \addlegendentry{mBERTu}

    \addplot[
        black,
        dashed,
        mark = none,
    ] table[
        x expr = \thisrow{data} * 100,
        y expr = \thisrow{#1_m} * 100,
    ] {mbert.dat};
    \addlegendentry{mBERT}

    \addplot[
        name path = upper,
        draw = none
    ] table[
        x expr = \thisrow{data} * 100,
        y expr = (\thisrow{#1_m} + \thisrow{#1_std}) * 100,
    ] {bert_mt.dat};
    \addplot[
        name path = lower,
        draw = none,
    ] table[
        x expr = \thisrow{data} * 100,
        y expr = (\thisrow{#1_m} - \thisrow{#1_std}) * 100,
    ] {bert_mt.dat};
    \addplot[
        fill = blue,
        opacity = 0.1,
    ] fill between[of=upper and lower];

    \addplot[
        name path = upper,
        draw = none,
    ] table[
        x expr = \thisrow{data} * 100,
        y expr = (\thisrow{#1_m} + \thisrow{#1_std}) * 100,
    ] {mbert_mt.dat};
    \addplot[
        name path = lower,
        draw = none,
    ] table[
        x expr = \thisrow{data} * 100,
        y expr = (\thisrow{#1_m} - \thisrow{#1_std}) * 100,
    ] {mbert_mt.dat};
    \addplot[
        fill = red,
        opacity = 0.1,
    ] fill between[of=upper and lower];

    \addplot[
        name path = upper,
        draw = none,
    ] table[
        x expr = \thisrow{data} * 100,
        y expr = (\thisrow{#1_m} + \thisrow{#1_std}) * 100,
    ] {mbert.dat};
    \addplot[
        name path = lower,
        draw = none,
    ] table[
        x expr = \thisrow{data} * 100,
        y expr = (\thisrow{#1_m} - \thisrow{#1_std}) * 100,
    ] {mbert.dat};
    \addplot[
        fill = black,
        opacity = 0.1,
    ] fill between[of=upper and lower];
}
\begin{figure*}[t]
    \centering
    \begin{tikzpicture}
        \begin{groupplot}[
            group style={
                group size = 2 by 2,
                horizontal sep = 70pt,
                vertical sep = 60pt,
            },
            width = 0.4\textwidth,
        ]
            \dataplot{LAS}{LAS}{Dependency Parsing}
            \coordinate (column1) at (rel axis cs:0,1);
            \dataplot{NER}{span F1}{Named-Entity Recognition}
            \coordinate (column2) at (rel axis cs:1,1);
            \dataplot{XPOS}{Accuracy (XPOS)}{Part-of-Speech Tagging}
            \dataplot{SA}{macro-F1}{Sentiment Analysis}
        \end{groupplot}
        \coordinate (centre) at ($(column1)!.5!(column2)$);
        \node[below] at (centre |- current bounding box.south){\pgfplotslegendfromname{legend:size}};
    \end{tikzpicture}
    \caption{
        Downstream task performance as the pre-training data size grows.
        All values are the mean over 5 runs with different random seeds.
        The standard deviation being represented by the corresponding shaded area.
    }\label{figure:size}
\end{figure*}

We do this by fixing the desired data proportion and scaling the pre-training data to satisfy this proportion, keeping the original training and validation split.
In tandem, the original 1M and 250K steps and batch size used in Section~\ref{section:models} are scaled down with the data size to pre-train for the same number of epochs as the models with the entire data.
Language models at 10\% intervals are pre-trained, with 100\% being the original models from Section~\ref{section:models}.
In this analysis, we also include the BERTu Mixed and mBERTu Mixed models from Section~\ref{section:analysis-domains}, which use 0.38\% of the data, estimated as a proportion of sentences.

After pre-training, each language model is fine-tuned on each of the downstream tasks in the same setup considered in Section~\ref{section:evaluation}.
These results are visualised in Figure~\ref{figure:size}.

As expected, the performance generally improves with  more pre-training data.
Surprisingly, the performance gap between the monolingual and multilingual models is drastically reduced with just 10\% of the data.
With this little data all configurations outperform mBERT.
For Named-Entity Recognition this is also the case but it takes around 70\% of the data for BERTu and mBERTu to start achieving very close performance.

It is also noticeable that the gradual increase is not monotonic, although it is more stable for Dependency Parsing and Part-of-Speech tagging.
Suprisingly, BERTu with 70\% of the data performs better than with 100\% of the data on Sentiment Analysis.
Similarly, mBERTu with 50\% of the data performs better than with 100\% of the data on Named-Entity Recognition.
One possible explanation may be due to the relationship between the number of steps and batch size chosen, but further investigation is warranted.

On Sentiment Analysis, BERTu is consistently better than mBERTu with 10\% or more of the data, and is at times significantly better.
This finding gives some evidence that monolingual representations seem better suited for fine-tuning on semantic tasks in a specific language.

\section{Conclusion}\label{section:conclusion}

In this work we analyse the impact of pre-training data on downstream task performance in a low-resource setting, specifically focusing on Maltese.
We present a newly developed corpus of around 500M tokens, which allows us to study how the pre-training data size and domain translates in downstream performance differences.
Using BERT as our architecture, we compare a monolingual language model, pre-trained from scratch, to a further pre-trained multilingual model, in a number of pre-training configurations.
We conduct an evaluation on a both syntactic and semantic tasks.

In line with previous findings on domain pre-training (\citealp{gururangan-etal-2020-dont}; inter alia), we find that matching the pre-training domain to the target task domain, results in improvements.
Moreover, we demonstrate that pre-training language models with varied domains is often beneficial over pre-training solely with Wikipedia.
These adjustments were in certain cases enough to surpass mBERT, underlining the importance of having pre-training data more suited to the target task, even at a small scale.

Whilst we show that further pre-training data does improve downstream performance, the gains are linear with exponential increases in data.
In fact, substantial improvements are observed with a small proportion of the pre-training data, over language models trained with Wikipedia-sized data.
This echoes the findings made by \citet{martin-etal-2020-camembert} with a small pre-training subset, although our reduced data setup is considerably smaller.

Using the whole corpus, we also pre-train two new language models: BERTu, a monolingual BERT model, trained from scratch, and mBERTu, which is the result of further pre-training mBERT.
These models demonstrate state-of-the-art results in Dependency Parsing, Part-of-Speech Tagging, Named-Entity Recognition, and Sentiment Analysis.
Moreover, we show that in general, BERTu performs better than mBERTu, as well as other baselines.
Through this, we also demonstrate that language-specific pre-training is most beneficial for higher-level tasks.

Despite these considerable improvements, the pre-training setups used in this work are as close as possible to the baselines, to allow for a more controlled comparison.
Hence, in the future, we plan to experiment with more language-specific tuning to push the state-of-the-art even further.

Even though this new corpus will undoubtedly improve the state of resources available for Maltese, the language is by no means a highly-resourced one.
The corpus we use is significantly smaller than typically used corpora for higher-resourced languages.
We also remark that the quantity of labelled data is still scarce, and at times non-existent for certain tasks.
Although we include a semantically-oriented task in our evaluation, future work should investigate the efficacy of these models in more complex Natural Language Understanding scenarios.

We make this corpus and the models publicly available to foster further work and improvements for various NLP applications for Maltese.
We also hope that this work inspires work in other low-resource languages, since we show that the amount of data needed to achieve considerable improvements, does not need to be overly ambitious.

\section*{Acknowledgements}
This work is partially funded by the Malta Digital Innovation Authority (MDIA) under the Malta AI Strategy Framework 2019.
We also acknowledge  LT-Bridge Project (GA 952194) and DFKI for access to the Virtual Laboratory.
We are also grateful to the University of Malta Libraries for granting access to their digital open repository and to the many Maltese authors who gave permission for their work to be included in the Korpus Malti v4.0.


\section{Introduction}
 Nowadays, there is a growing demand for indoor \acp{lbs} which are able to ubiquitously and accurately provide a user's position relative to the surrounding environment in an inexpensive manner \cite{basiri2017indoor}. Examples of \ac{lbs} include navigation and location-aware and tracking services \cite{s21061950,he2021user, cheema2018indoor}. To this end, the pervasiveness of mobile and wearable devices with high computing power, together with growing deployment of anchors in indoor environments, play an important role. Among the large variety of technologies and methods used, solutions relying on \ac{ble} and \ac{rss}  are very popular, as they provide a cheap and easy deployment \cite{jeon2018ble,li2018indoor,yang2009indoor,ARANDA2022117095}.

Although the feasibility of using \ac{ble}--\ac{rss} anchors/devices for proximity detection applications has been demonstrated \cite{montanari2017study, ng2019compressive}, their implementation for accurate positioning applications is still an open issue. Due to their relatively straightforward implementation, lateration algorithms are one of the most explored solutions to enhance positioning accuracy. Unfortunately, they suffer from accuracy degradation mainly due to four factors:
\begin{enumerate}
    \item  Anchor deployment: a low quantity of anchors deployed in the environment and/or  the inappropriate anchor distribution increase not only the chance of \ac{nlos} conditions but also the locations inside the operational area with insufficient anchor coverage for reliable lateration, reducing the overall positioning accuracy dramatically.
    \item Unstable \ac{ble} signal strength: \ac{rss} of \ac{rf} signals are prone to fluctuate due to environmental conditions (geometries, temperature,  crowded environments), building materials and presence of obstacles or people between transmitter and receptor devices (\ac{nlos} conditions) \cite{7346778,huh2017indoor,Flueratoru2021}. Such dynamic fluctuations are often difficult to predict and compensate for, and cause position estimation errors when used for positioning. In fact, the \ac{ble} signal propagation in indoor environments has not been properly modeled yet in order to estimate the relative distance between transmitter and receiver using smartphones.
    \item  Hardware  heterogeneity: the \ac{rss} signature depends on the transmitters and receivers involved \cite{10.1145/1023720.1023728}. The diversity of devices involved in localization -- both anchors and smartphones -- comes with variability in the hardware-specific parameters, such as antenna gains, and thus in the \ac{ble} received signal strength~\cite{jin2013ssd,Flueratoru2021}. Furthermore, the transmission power and transmission period  preset in each device -- either by the device owner or manufacturer -- presents variations due to tolerance in the design of antennas or in the \ac{rf} circuit components, even in similar devices.
\item Software limitations:
the energy-saving modes and execution task priorities prescribed by the mobile devices' operating systems may alter the transmission/reception period of \ac{ble} signals, rendering these devices non-deterministic for the \ac{ble} broadcasting.
\end{enumerate}

In this article, we aim to enhance the positioning accuracy of the stand-alone lateration method applied to an adverse indoor scenario that copes with the first, second, and third issues mentioned above,  namely the lack of sufficient beacons or problematic beacon distribution with the associated presence of large \ac{nlos} areas, the lack of appropriate signal propagation models to deal with fluctuating signal strength in unpredictable indoor conditions, and the heterogeneity among mobile devices typically present in any scenario. We do so by proposing a collaborative indoor position system which exploits the neighboring mobile devices as additional anchors in an extended, ad-hoc positioning network. The collaborating devices' information (i.e., their stand-alone estimated position and \ac{ble}--\ac{rss}) is exchanged and processed using a neural network model (rather than the traditional path-loss models) to estimate relative distances, after which lateration is applied to estimate the device's collaborative position. Unlike the path-loss models, which estimate the distance primarily based on signal attenuation formulas, the neural network model aims to relate the distance to \ac{rss} patterns, pairwise device position, identification of devices (transmitter and receiver), and  variation of \ac{rss} measurements due to the devices' hardware heterogeneity. The collaborative algorithm takes mobile device variations into account by considering, for each device, a \ac{ble} reference measurement at one meter. Finally, a midpoint line algorithm is used to combine the collaborative and the non-collaborative position estimates to provide the final position estimates of each collaborative device. The key contributions of this paper are summarized as follows:

\begin{itemize}
	\item We present a collaborative indoor position approach using \ac{mlp} \ac{ann}  to enhance \ac{ble}--\ac{rss} lateration-based indoor positioning.
	 \item We present a  \ac{mlp} \ac{ann}   model  to estimate  relative distances between collaborative devices in order to provide a most reliable distance estimation. The \ac{mlp} model provides a better modeling of the \ac{nlos} and \ac{los} conditions of the environment with respect to path-loss-based models, as well as, the characteristic inherent of each receiving device. Additionally, the proposed model avoids setting the parameters needed in path-loss-based models.
	\item We experimentally  demonstrate, using a scenario with a poor quantity and inappropriate distribution of anchors, and rich \ac{nlos} conditions, that our collaborative approach outperforms the stand-alone lateration approach with respect to positioning accuracy in such conditions.
\end{itemize}

\section{Related Work}
\label{sec:related_work}
The design of reliable \acf{ips}, which provide a balance between position accuracy and low-cost and easy deployment, is one of the main concerns in the design of \acfp{lbs} \cite{basiri2017indoor}. To meet this concern,  diverse solutions have been proposed. In this section, we focus on those fields relevant for this work, namely lateration, artificial neural network, and collaborative \ac{ips}.

\subsection{Lateration}
Lateration approaches estimated the target position by considering the distance between the target and multiple reference anchors (minimum three)\cite{dag2018received}. The distance is the result of the conversion of \ac{rss} measured between reference anchors and target~\cite{dag2018received,cengiz2020comprehensive}. Free space path, two-ray ground, and logarithmic  path-loss are some of the most used models to convert \ac{rss} into distance~\cite{miao2018accurate,katirciouglu2011comparing,qiu2016ble}.

In the context of indoor positioning, it has been shown in literature that having more (even redundant) anchors yield better positioning accuracy --independent of the lateration approach used-- and least-square lateration performs better than traditional lateration. For example, \textcite{dag2018received} proposed an \ac{rss}-based lateration algorithm, which was implemented using a least-squares method and required redundant anchors deployment. The distances between anchors and target were estimated with the \ac{ldpl} model, whose parameters were computed statistically. The experiments were conducted in a \SI[output-product = \times]{6 x 6}{\meter} area, considering 196 measurement points collected with a single device in complete \ac{los}. The authors conclude that adding least-squares to lateration reduces the error by $67.6\%$ to \SI{2.8}{\meter}, and including redundant anchors reduces the error to \SI{1.35}{\meter}. \textcite{cengiz2020comprehensive} also proposed a least-square lateration approach and performed an accuracy error analysis using simulated environments. The analysis included the level of Gaussian noise, number of anchors, and size of the area.
The results were compared with a traditional lateration, concluding that increasing the density of anchors deployed improves positioning accuracy.
The proposed method significantly improved lateration in scenarios with a strong noise component (high Gaussian noise, $\sigma = 5$), reducing the error from \SIrange{4.5}{1.9}{\meter}, \SIrange{5.9}{3.9}{\meter} and  \SIrange{18}{7.5}{\meter} for \SI[product-units=power]{6 x 6}{\meter}, \SI[product-units=power]{12 x 12}{\meter} and \SI[product-units=power]{24 x 24}{\meter} test areas respectively.

In line with literature, our method uses least-square-based method for lateration. Moreover, our approach aims to increase the amount of anchors, yet in contrast to previous works that deploy additional fixed anchors in the environment, we dynamically extend the positioning network using the devices of the users in a collaborative setting.

\subsection{Artificial neural network}

\acfp{ann} are commonly used to self-learn the relationship between data provided to resolve complex problems \cite{wu2020artificial}. Specifically, in indoor positioning they are used to provide robustness against noise and interference that affect the positioning accuracy \cite{nessa2020survey}. For example, \textcite{wu2020artificial} presented a path-loss model based on \acf{mlp} neural networks for wireless communication networks. For that purpose they considered several models using input vectors with fifteen features of the environment. The authors conclude that the introduction of environment features in the \ac{mlp} model input and adding two hidden layers (with 5 and 10 neurons) provide an accurate and efficient path-loss prediction. On the other hand, \textcite{elbes2019indoor} proposed an indoor localization approach based on Wi-Fi fingerprinting and \ac{lstm} neural networks for position estimation. The experiments were carried out in \SI[product-units=power]{54 x 32}{\meter} test L-shape area with ad-hoc Wi-Fi anchors deployed and a single smartphone to scan the signals. The \ac{lstm} model consist of $42$ input features, $2$ outputs, $300$ neurons, learning rate $\lambda = 0.2$, drop error, and $250$ epochs. The results reported an average error of around 1 \si{\meter}.

In contrast to related work, our \ac{mlp} architecture is intended for use with heterogeneous mobile devices to estimate the relative distance between neighboring devices in a collaborative approach. To this end, our \ac{mlp} architecture exploits the usual short distance between neighboring devices to reduce the dimensionality of features and, consequently, obtain a lightweight neural network architecture that is feasible to implement on mobile devices. Specifically, the \ac{mlp} architecture relates the distance to four main elements of BLE-RSS collaborative approaches. The elements used for the features of the architecture are the \ac{rss} patterns, pairwise device position, identification of devices (transmitter and receiver), and variation of \ac{rss} measurements due to devices' hardware heterogeneity.

\subsection{Collaborative indoor positioning systems}

Collaborative \acp{ips} aim to enhance positioning of devices/users by using wireless communication technologies, both for relative distance estimation and information exchange between them~\cite{pascacio2021collaborative}.
 Within the collaborative \acp{ips}, those relying on \ac{ble}-\ac{rss} for the non-collaborative and collaborative phase, as our approach, are scarce (for a comprehensive overview of collaborative \acp{ips}, we refer to our recent systematic literature review~\cite{pascacio2021collaborative}). One of the most representative ones is proposed by~\textcite{qiu2016ble}, which presented a collaborative \ac{ips} based on inertial sensing, adaptive multi-lateration, and a mobile encountering approach. All these sources of data were combined using a particle-filter model to locate a target.

The author conclude that the accuracy improvements are approximately $28.19\%$ in \ac{los} and $19.99\%$ in \ac{nlos} when using adaptive ranging device-specific parameter with respect to the none adaptive approach.

In contrast to what has been done in the literature, we do not rely on the \ac{ldpl} model to estimate the relative distance between a receiver (RX) and a transmitter (TX). Instead, we use a neural network to process the RSS value depending on the transmitter and receiver location and receiver calibration.

 \subsection{Overall summary}
 From the above-mentioned systems, we can summarize that the lateration methods considerably decrease the positioning accuracy as the density of anchors in the area decreases and the noise signal increases. \acp{ann} can mitigate the effect of the environment on the position and distance estimation, as well as reduce the accuracy error. Additionally, the implementation of collaborative approaches can help to reduce the positioning error of individual estimations. To the best of our knowledge,  this is the first \ac{ble}--\ac{rss} indoor positioning system that implements a collaborative approach based on \acp{ann}.

\section{System Overview}
\label{sec:model}

Under simulations or in ideal testbed conditions, i.e. scenarios featuring a rich amount and well distributed anchors in the environment and \ac{los} conditions, RSS-based lateration approaches have provided high accuracy with positioning errors within the range of \SIrange{0.8}{1.3}{\meter}~\cite{dag2018received,cengiz2020comprehensive}. Nevertheless, in real environments (e.g., offices, stores, libraries) the positioning accuracy decreases dramatically. Real environments are characterized by large \ac{nlos} areas and anchor deployments that may not be exclusively designed for positioning tasks. In addition, each environment requires a dedicated modeling of the \ac{rf} signal propagation to compensate for environment-specific conditions and accurately convert \ac{rss} into distance.

In this work, we propose a collaborative \ac{ips} using \acp{ann} to enhance \ac{ble}--\ac{rss} lateration-based methods. Our approach tackles three main sources of positioning inaccuracy: mitigating the inaccuracy due to \ac{nlos} areas and inadequate deployment of anchors; reducing the impact of a inappropriate signal propagation modeling; and minimizing the impact of device heterogeneity. Potential application scenarios of our approach is indoor positioning in buildings with a poor infrastructure deployment, but with a moderate amount of mobile devices (people) inside and where the latency for users/devices positioning is moderate (e.g., tracking library assistants in libraries, patients in hospitals, personnel in government offices, among others).

Our collaborative approach is divided in three phases which are described in this section. The first phase is devoted to registering devices used in the collaborative approach, the second phase consists of the stand-alone (non-collaborative) lateration algorithm for position estimation of each device/user, and the last phase is devoted to collaboratively enhance the position of the target device/user.

\subsection{Registering a device for collaborative positioning}
\label{sec:registering}
The registration procedure aims to identify a mobile device and record an \ac{rss} reference value, to be used in the collaborative algorithm. To this end, a fixed reference anchor $(BX)$ emitting \ac{ble} advertisements is placed at an entry point to the environment. Each new user is then asked to position himself at 1 meter from the anchor (marked on the floor) in \ac{los}, and measure and record the \ac{rss} value (i.e., $RSS^{1m}_{RX}(BX)$). This lightweight calibration procedure is performed only once for each (new) device. To ease this procedure, the calibration could be integrated with the electronic locking system of the main doors, calibrating the device every time the user's smartphone is used to unlock a door.

\subsection{Stand-alone phase}
\label{stand-alone}
The stand-alone phase is devoted to estimating each initial individual device/user's position considering the \ac{rss} and the \ac{gt} coordinates of \ac{ble} anchors deployed in the environment. For reference, in this article this position estimation is performed using a lateration method based on the \ac{ldpl} model, a minimization problem, and the Levenberg-Marquardt algorithm for weighted nonlinear least-squares to solve the minimization problem.

The \ac{ldpl} model expresses the relation between \ac{rss} and distance, as described by Eq.~\eqref{eqn:pathloss}  \cite{qureshi2018analysis}: \begin{equation}
\label{eqn:pathloss}
RSS^{d}_{RX}(TX) = RSS^{1m}_{RX}(BX) - 10\:\eta \:log \left(\frac{d}{d_0}\right)\textbf{}
\end{equation}
Where $RSS^{d}_{RX}(TX)$ is the \ac{rss} measured at a distance $d$ between transmitter ($TX$), which can be a \ac{ble} beacon or a smartphone, and receiver ($RX$); $RSS^{1m}_{RX}(BX)$ is the \ac{rss} measured at \SI{1}{\meter}; and $\eta$ is the path-loss attenuation factor.

Mathematically, the lateration approach is expressed by Eq.~\eqref{eqn:minproblem} as a minimization of the sum of squared errors between the measured distances ($d_m$) and hypothetical ones ($g_m(\b{x})$), based on the unknown target position, $g_m(\b{x})$, which is denoted by Eq.~\eqref{eqn:gm}
\cite{zhou2012multilateration}. \begin{equation}
\label{eqn:minproblem}
\min_{\b{x}} \displaystyle\sum_{m=1}^M \left( g_m(\b{x})-d_m \right)^2
\end{equation}
\begin{equation}
\label{eqn:gm}
g_m(\b{x})\triangleq \sqrt{\left(x-bx_m \right)^2 - \left(y-by_m \right)^2}
\end{equation}
Where, $m=\{1,2...M\}$ are the number of \ac{ble} anchors deployed; $\b{x}=\{x,y\}$ are the device/user's unknown coordinates; and $\{bx_m,by_m\}$ the \ac{gt} coordinates of \ac{ble} anchors.

Algorithm \ref{alg:one} shows the pseudo-code for the stand-alone lateration approach, whose workflow is summarized as follows:

\begin{itemize}
    \item \textbf{1\textsuperscript{st} step:} Collect  the \ac{rss} from \ac{ble} anchors ($RSS^{d}_{RX}(TX)$) within a time window ($tw$) of  \SI{60}{\second}, by discarding those not belonging to the deployed  scenario (input data for Algorithm~\ref{alg:one});

    \item \textbf{2\textsuperscript{nd} step:} Group the \ac{rss} readings by anchor removing the outlier values. We consider the values falling out of 25 and 75 percentiles (lines 1--2 in Algorithm~\ref{alg:one}) as outliers;

     \item \textbf{3\textsuperscript{rd} step:} Apply the average operator to the \ac{rss} values of each anchor, in order to get one averaged \ac{rss} value per anchor (line 3 in Algorithm~\ref{alg:one});

     \item \textbf{4\textsuperscript{rd} step:} Select "strong reference \ac{ble} anchors", defined as the anchors with averaged \ac{rss} equal or greater than a predefined threshold (lines 4--8 in Algorithm~\ref{alg:one});

     \item \textbf{5\textsuperscript{th} step:} Estimate the relative distances of selected reference \ac{ble} anchors to the device/user's target position, by using the LDPL model, which is expressed by eq.\eqref{eqn:pathloss}, and their \ac{rss} values (line 9 in Algorithm~\ref{alg:one});

    \item \textbf{6\textsuperscript{th} step:} Estimate the device/user's position, $\hat{\textbf{P}}^{Lat1}$, using the Levenberg-Marquardt Weighted Least Squares (L-MWLS) lateration method to fit the Euclidean Distance model. Henceforth, $Lat1$ will refer to the L-MWLS lateration method. The input data to fit the model are the distances estimated in the 5\textsuperscript{th} step, the weights and the \ac{gt} of the \ac{ble} anchors. The weight value for every \ac{ble} anchor is computed as the inverse of its distance square with respect to the device/user;

    \item \textbf{7\textsuperscript{th} step:} Share the estimated position, $\hat{\textbf{P}}^{Lat1}(RX) = [\hat{P}^{Lat1}_x(RX),\hat{P}^{Lat1}_y(RX)]$.
\end{itemize}

We set the $threshold$ value to \SI{-83}{\dbm} and the path-loss attenuation factor to $\eta = 2.1$, as we previously used in~\cite{pascacio2021lateration}
 and which are in phase to values in literature~\cite{dag2018received}. Each device involved in the collaborative positioning model will have its own value for $RSS^{1m}_{RX}(BX)$ calculated from the data collected in the device registration phase (see Table~\ref{rssat1mtable}).%

\input{Algorithm/Algo_01}

\subsection{Collaborative phase}

The collaborative phase uses the neighboring mobile devices' information, which include  the measured \ac{ble}-\acp{rss}, the estimated positions for the receiver
and transmitter devices, and the RSS at \SI{1}{\meter} of the receiver to collaboratively estimate the device/user's position, and ultimately provide the final position estimate based on the combined collaborative and stand-alone estimates, aiming to improve the latter. Each device/user, which exchanges information and collaborates with others, acts as an additional anchor, creating an extended ad-hoc positioning network, hereby improving the coverage of the original anchor network.

The collaborative phase relies on four main elements: 1) information exchanged between devices; 2) estimation of relative distances using a \acf{mlp} neural network; 3) a lateration algorithm to collaboratively estimate the target device/user's position; and  4) a method to combine the stand-alone and collaborative position estimations.

\subsubsection{Information exchanged between devices/users}
Currently, the operating systems of mobile and wearable devices permit broadcasting \ac{ble} advertisements. This feature allows them to act as \ac{ble} anchors and share information, enabling the development of collaborative positioning systems~\cite{huh2016bluetooth}. The information used in our collaborative positioning system includes: the RSS received at receiver from transmitter ($RSS_{RX}(TX)$), the \ac{rss} at \SI{1}{\meter} ($RSS^{1m}_{RX}(BX)$) of the receiver obtained after registering the device and the position of each collaborative device/user estimated by the stand-alone lateration ($\hat{P}_{x}^{Lat_1}(TX) $,$\hat{P}_{y}^{Lat_1}(TX) $,$\hat{P}_{x}^{Lat_1}(RX) $,$\hat{P}_{y}^{Lat_1}(RX)$).

All these collected data are used as inputs for the neural network presented in next subsection and the collaborative algorithm presented in Algorithm \ref{alg:two}.

\subsubsection{\acf{mlp} neural network and relative distances}

The \ac{mlp} neural network model's architecture used to estimated the relative distances is presented in Figure~\ref{fig:Ann_model}. Its architecture consists of 6 input layers, 1 hidden layer with 3 neurons, and one output layer. The activation function used is the hyperbolic tangent. In the training, the scaled conjugate gradient back propagation was used as training function and $50$ epochs were considered. Further information about the selection of hyperparameters  of the \ac{mlp} neural network is provided in Section~\ref{sec:experiments}.
The 6 inputs that feed the \ac{mlp} neural network model correspond to the exchanged information between each pair (target device--neighbor device) and the output is the estimated distance of that specific pair. The input are: the $RSS_{RX}(TX)$, which corresponds to the \ac{rss} of the \ac{ble} advertisement sent by the transmitter $TX$ and measured at the receiver $RX$;  the $x$ and $y$ coordinates of the transmitter (neighbor device/user) position estimated by the stand-alone lateration algorithm ($\hat{P}_{x}^{Lat_1}(TX) $,$\hat{P}_{y}^{Lat_1}(TX) $); and the $x$ and $y$ coordinates of the receiver (target device/user) position estimated by the stand-alone lateration algorithm ($\hat{P}_{x}^{Lat_1}(RX) $,$\hat{P}_{y}^{Lat_1}(RX)$).

Finally, the $RSS^{1m}_{RX}(BX)$ is the \ac{rss} value measured at \SI{1}{\meter} of distance between transmitter and receiver assigned at each mobile device by phase one (see Table~\ref{rssat1mtable}).

\begin{figure}[!h]
    \centering
  \includegraphics[width=1\columnwidth,trim={2.5cm 2.5cm 2.1cm 2cm},clip]{Pictures/ANN_model2.pdf}
  \caption{\ac{mlp} neural network model architecture. Used to estimate relative pairwise distance between target and neighboring devices/users.}
  \label{fig:Ann_model}
\end{figure}

\subsubsection{Collaborative lateration algorithm}

The lateration algorithm used in the collaborative phase is the same algorithm used in the stand-alone phase (see Section \ref{stand-alone}). The main two differences are within the data used to perform the lateration.

\begin{itemize}
    \item First, rather than fixed \ac{ble} beacons in the environment with well-known position, the anchors considered in the collaborative lateration algorithm are the neighboring devices/users, of which only the position estimated by the stand-alone lateration is known.

\item Second, the relative distances between the target device/user and the neighboring devices/users is computed using a \ac{mlp} neural network model instead of the \ac{ldpl} model.
\end{itemize}
\subsubsection{Combining stand-alone and collaborative estimated positions}
Once the position of the target device is collaboratively estimated using neighbor devices/users as anchors, it is combined with the stand-alone estimation to come to a final estimated position. In this article, we used a midpoint line algorithm, as described in Eq.~\eqref{eqn:joinequation}.%
\begin{align}
\label{eqn:joinequation}
\hat{\hat{P}}_{x}(RX) &=
\frac{\hat{P}^{lat_1}_{x}(RX)
+
\hat{P}^{lat_2}_{x}(RX)}{2}\\
\hat{\hat{P}}_{y}(RX) &=
\frac{\hat{P}^{lat_1}_{y}(RX)
+
\hat{P}^{lat_2}_{y}(RX)}{2}
\end{align}
Where $\hat{\hat{\textbf{P}}} = [\hat{\hat{P}}_x(RX)$, $\hat{\hat{P}}_y(RX)]$ represents the final estimated position, with $x$ and $y$ coordinates, of device/user $RX$, with $RX=\{1,2....N\}$ and $N$ the number of devices. $\hat{P}^{lat_2}_{x}(RX)$ and $\hat{P}^{lat_2}_{y}(RX)$ are the position estimated, $x$ and $y$ coordinates, with the collaborative lateration and $\hat{P}^{lat_1}_{x}(RX)$ and $\hat{P}^{lat_1}_{y}(RX)$ are the position estimated, $x$ and $y$ coordinates, with the stand-alone lateration, both for device/user $RX$.

\subsubsection{Full collaborative workflow}
Finally, algorithm \ref{alg:two} shows the pseudo-code for the collaborative positioning algorithm. The algorithm inputs are the shared information exchange by each collaborative device/user within a time window ($wt$) of \SI{60}{\second} and correspond to the \ac{rss} transmitted by each collaborative device/user ($RSS_{RX}(TX)$), position of each collaborative device/user estimated by the stand-alone lateration ($\hat{P}_{x}^{Lat_1}(TX) $,$\hat{P}_{y}^{Lat_1}(TX) $,$\hat{P}_{x}^{Lat_1}(RX) $,$\hat{P}_{y}^{Lat_1}(RX) $), and the \ac{rss} at \SI{1}{\meter} ($RSS^{1m}_{RX}(BX)$).
The algorithm workflow is summarized as follows:

\begin{itemize}

    \item \textbf{1\textsuperscript{nd} step:} Group the \ac{rss} readings by device and remove outlier values. We consider those values falling out of 25 and 75 percentiles (lines 1--2 in Algorithm~\ref{alg:two}) as outliers;

     \item \textbf{2\textsuperscript{rd} step:} Apply the average to the \ac{rss} values of each device/user, getting one averaged \ac{rss} value per device/user (line 3 in Algorithm~\ref{alg:two});

     \item \textbf{3\textsuperscript{th} step:} Estimate the relative distances of neighboring devices/users to the target device/user, using the \ac{mlp} model, which structure is depicted in Figure~\ref{fig:Ann_model}. The inputs of \ac{ann} model are:$RSS_{RX}(TX)$,  ($\hat{P}_{x}^{Lat_1}(TX) $,$\hat{P}_{y}^{Lat_1}(TX) $,$\hat{P}_{x}^{Lat_1}(RX) $,$\hat{P}_{y}^{Lat_1}(RX) $), $RSS^{1m}_{RX}(BX)$ and  (input values in Algorithm~\ref{alg:two});

    \item \textbf{4\textsuperscript{th} step:} Estimate the device/user's position ($\hat{\textbf{P}}^{Lat2}(RX)$) using the Levenberg-Marquardt Weighted Least Squares (L-MWLS) lateration method to fit the Euclidean Distance model. The input data to fit the model are the relative distances estimated by the \ac{mlp} neural network model (step 3), the weights and the estimated positions ($\hat{\textbf{P}}^{Lat1}$) of the neighboring devices/users. As in the stand-alone algorithm, the weight value for every \ac{ble} anchor is computed as the inverse of its distance square with respect to the device/user;

    \item \textbf{5\textsuperscript{th} step:}
    Compute the final estimated device/user's position ($\hat{\hat{\textbf{P}}}(RX)$) based in the formula expressed in eq.\eqref{eqn:joinequation}, using the estimated standalone position $\hat{\textbf{P}}^{lat_1}(RX)$ and collaborative estimated position $\hat{\textbf{P}}^{lat_2}(RX)$, where $(RX)$ is the identifier of the device/user to estimate its position.
\end{itemize}
\input{Algorithm/Algo_02}

\input{Tables/anchors_dev_coord}

\section{Experiments and Results}
\label{sec:experiments}
\subsection{Objectives and Experimental setup}
The principal goal of our experiments is to assess the feasibility and benefits of our collaborative approach with respect to a traditional \ac{ble}-\ac{rss} lateration baseline to mitigate inadequate deployment of \ac{ble} anchors and increased \ac{nlos} conditions. Specifically, we evaluate and compare the positioning accuracy of both in a realistic environment.

We performed the experiments in a real office scenario. The office covers an approximate area of \SI[product-units=power]{10.8 x 16.7}{\meter} and contains seven deployed \ac{ble} beacons (anchors), with a transmission power and period of \SI{-4}{\dBm} and  \SI{250}{\milli\second} respectively. Figure \ref{fig:3DScenario} shows the complete scenario through a 3D representation, which allows us to illustrate the complexity of the environment. The  \ac{nlos} conditions are mainly created by the furniture, which includes chairs, desks, desktop computers, bookshelves, and the concrete metal-reinforced pillars.

\begin{figure}[!h]
    \centering
    \includegraphics[width=1\columnwidth]{Pictures/Office_scenario_PERCOM2022.pdf}
  \caption{3D Office Scenario representation}
  \label{fig:3DScenario}
\end{figure}

In this office scenario, a comprehensive data collection was conducted. First, the five collaborative devices were registered during the registration phase (see Section~\ref{sec:registering}) and their $RSS^{1m}_{RX}$ was measured using a fixed \ac{ble} beacon from the environment. Table~\ref{rssat1mtable} summarizes the five devices that were used in the experiment, and their corresponding $RSS^{1m}_{RX}$ value, which depend on the smartphone model and range from \SIrange{-78.79}{-62.39}{\dbm}.

\input{Tables/rssat1m}

Then, seven different collaborative distributions (called configurations in this paper) made up of these five devices were considered. In each configuration, the five devices simultaneously broadcast and save the information of surrounding devices, including the \ac{ble} anchors, for 2 hours. Devices broadcast every 100 ms in accordance with the low latency mode supported by Android devices~\cite{kindt2021reliable}. However, as mobile devices are not ad-hoc positioning devices, the broadcast period is affected mainly by the priority of executing tasks and power saving modes implemented by operating systems. The data collection for each configuration was performed independently, at a different time. As already mentioned, for the lateration approach based on the \acf{ldpl}, the path-loss factor was determined to be $\eta = 2.1$ for all the five devices. Figure \ref{fig:sketch_val_train} and Figure \ref{fig:sketch_test} show the distribution of the devices in each configuration and the \ac{ble} anchors in the scenario, and  Table~\ref{tab_beaconlist} summarizes the \ac{gt} coordinates of them.

\begin{figure}[!h]
  \centering
  \includegraphics[width=1\linewidth]{Pictures/Sketch_Eval_Train_PERCOM2022.pdf}
  \caption{Distribution of the configuration 2, 3, 6, and 7 used in the training \& validation and distribution of the \ac{ble} anchors}
  \label{fig:sketch_val_train}
\end{figure}

\begin{figure}[!h]
  \centering
  \includegraphics[width=1\linewidth]{Pictures/Sketch_test_PERCOM2022.pdf}
  \caption{Distribution of the configuration 1, 4, and 5 used in the testing and distribution of the \ac{ble} anchors}
  \label{fig:sketch_test}
\end{figure}

\begin{figure*}
\centering
\tabcolsep 0pt
\begin{tabular}{cccc}
\includegraphics[clip, trim=2.6cm 6.5cm 3cm 6.5cm,width=0.25\textwidth]{Pictures/MLP1_new.pdf}&
\includegraphics[clip, trim=2.6cm 6.5cm 3cm 6.5cm,width=0.25\textwidth]{Pictures/MLP2_new.pdf}&
\includegraphics[clip, trim=2.6cm 6.5cm 3cm 6.5cm,width=0.25\textwidth]{Pictures/MLP3_new.pdf}&
\includegraphics[clip, trim=2.6cm 6.5cm 3cm 6.5cm,width=0.25\textwidth]{Pictures/MLP4_new.pdf}\\
(a) \ac{mlp}1&(b) \ac{mlp}2&(c) \ac{mlp}3&(d) \ac{mlp}4\\
\end{tabular}

\caption{Target vs Predicted distances from the test dataset estimated with \ac{mlp}1--\ac{mlp}4 architectures}
\label{fig:ann1234}
\end{figure*}

The data collected in the seven configurations were split into two datasets, where the devices cover different areas, and the distances among devices, and between devices and anchors, varies. Each dataset is used for different purposes: configurations 2, 3, 6, and 7 (see Fig.~\ref{fig:sketch_val_train}) were used to train the model and configurations 1, 4, and 5 (see Fig.~\ref{fig:sketch_test}) were used for evaluation. In order to tune the \ac{mlp} neural network, the first dataset was randomly split into training ($70\%$) and validation ($30\%$) with $968798$ and $242199$ samples respectively. The testing dataset contains $936103$ samples and it was used to test the \ac{mlp} \ac{ann}  for relative distances, and implement the \ac{ble}-\ac{rss} lateration baseline and collaborative approach.

The selected scenario is challenging and covers an area with low \ac{gdop} and strong heterogeneous \ac{nlos} conditions (diversity in obstacles): only 7 \ac{ble} beacons provide positioning, and there is device diversity in the collaborative phase.
The proposed collaborative model targets dynamically extending the anchor coverage by using the collaborative user's devices, replacing the traditional \ac{ldpl} model with a \ac{mlp} \ac{ann} that processes the \ac{rss} considering the receiver and emitter location and the calibration of \ac{rss} (at \SI{1}{\meter}) to minimise the device diversity effect.

\subsection{\acf{mlp} neural network tuning}
We tested four \acf{mlp} neural network architectures in order to determine the best architecture and set of hyperparameters of our model. As hyperparameters, we considered different numbers of hidden layers (1 and 2) and  neurons, as well as two types of activation functions, the Hyperbolic tangent sigmoid (tansig) and Log-sigmoid (logsig). Table~\ref{tab_hyperparameters} summarizes the hyperparameters configured in the proposed architectures.

\input{Tables/ANN_hiperparameters}

Before presenting the main positioning results of the proposed collaborative approach and the lateration baseline, we present the results of evaluating the four proposed \ac{mlp} architectures with the training dataset in order to determine which of them provides a more accurate estimation of relative distances. Figure~\ref{fig:ann1234} shows the density scatter plots of the real distance against estimated distance by each \ac{mlp} neural network architecture. Also, the \ac{rmse} and correlation coefficient (R) values are indicated in each of them.

From those values and density scatter plots, we can notice that the \acf{mlp} architectures with one hidden layer (\ac{mlp}1 and \ac{mlp}2) present a \ac{rmse} lower than the architectures with two hidden layers (\ac{mlp}3 and \ac{mlp}4), as well as a greater correlation coefficient ($0.74$ and $0.69$ for \ac{mlp}1 and \ac{mlp}2 respectively). Therefore, architectures with a single hidden layer are able to estimate the distance more accurately. In specific, \ac{mlp}1, which use an hyperbolic tangent sigmoid activation function (tansig),  presents a higher density of predicted values closer to the real values in comparison with the \ac{mlp}2, which use a Log-sigmoid activation function, as can be observed in the density scatter plots of Figure~\ref{fig:ann1234} (a) and (b). Additionally, we notice that the density increases for the range between \SIrange{3}{6}{\meter} and around \SI{15}{\meter}. The results of high density values around the short distance demonstrate the potential of the \ac{mlp} architecture to improve the accuracy of position estimations in collaborative \ac{ips} in scenarios with a high density of collaborating mobile devices (i.e., when the distance between neighboring devices is short).

As a result of assessing different \ac{mlp} architectures, the \ac{mlp}1  with one hidden layer (3 neurons), a scaled conjugate gradient backpropagation training function, and a hyperbolic tangent sigmoid activation function was selected to estimate relative distances with \ac{ble}--\ac{rss} data.

\subsection{Results of the collaborative model}

Table~\ref{tab_targetestimation} presents the main results of the lateration baseline and our collaborative approach using various evaluation metrics (i.e., \ac{rmse}, mean, median, 70th and 90th percentile), and also indicates the relative difference between them.

\input{Tables/metrics_comparison}

As reported in Table~\ref{tab_targetestimation}, our collaborative approach outperforms the lateration baseline in four of the five evaluation metrics, with a maximum of $15.11\%$ of difference for the ``mean'' metric.

 Figure~\ref{fig:cdf_plot} introduces the \ac{ecdf} of the lateration baseline (black dotted line) and our proposed approach (red line). For the first 40\% of cases, our collaborative model significantly outperforms the baseline, for the next $50\%$ of cases, the improvement is moderate, being quite close for the last $10\%$ of cases.

It should be noted that the positioning accuracy of the lateration baseline was significantly degraded, compared to BLE-RSS approaches reported in the literature (\SIrange{2}{3}{\meter} error \cite{pascacio2021lateration,subhan2019experimental}), due to the adverse test scenario conditions, namely poor quantity and inappropriate distribution of anchors, unstable signal strength and hardware heterogeneity. However, the magnitude of the errors is in line with scenarios that consider the low-density anchor deployment issue. For example, \textcite{cengiz2020comprehensive} reported a mean error accuracy of \SI{7.5}{\meter} in their lateration algorithm, which was tested on a \SI[product-units=power]{24 x 24}{\meter} area with 8 anchors.

\begin{figure}[!h]
    \centering
  \includegraphics[width=0.85\linewidth]{Pictures/CDF_plot.pdf}
  \caption{Empirical CDF provided by the Lateration baseline and  our Collaborative approach}
  \label{fig:cdf_plot}
\end{figure}

Figure~\ref{fig:scaterplot} provides a more detail representation of the individual positioning errors of the laterations used in the stand-alone phase and the collaborative phase.

\begin{figure}[!h]
  \centering
  \includegraphics[width=0.9\linewidth]{Pictures/Scatter_plotnew.pdf}
  \caption{Comparison of individual errors provide by the Lateration baseline and  our Collaborative lateration approach}
  \label{fig:scaterplot}
\end{figure}

From the scatter plot, we can deduce that the lateration used in the calibration phase is providing better positioning results than the one based on the few beacons deployed in the environment. However, for a few cases the behavior is just the contrary as the accuracy is a few meters worse than the baseline. Nevertheless, the strongest feature of the proposed collaborative model is the combination of the lateration results from both phases, providing an ensembled output. According to the ensemble theory, the combination of estimators improves the accuracy of any of the individuals involved.

\section{Conclusion and Future Work}
\label{sec:conclusions}
In this paper, we presented a collaborative indoor positioning system using \acf{mlp} \acfp{ann} to enhance the accuracy of the models based on \ac{ble} and \ac{rss} lateration, in typical adverse conditions, namely poor quantity and inappropriate distribution of anchors, unstable signal strength and hardware heterogeneity. The proposed approach is divided in three phases. The first phase is devoted to registering devices used in the collaborative approach and establishing an \ac{rss} baseline measurement at \SI{1}{\meter}, the second phase consists of the stand-alone (non-collaborative) lateration algorithm for position estimation of each device/user, and the last phase is devoted to collaboratively estimate the position of the target device/user and combine it with the non-collaborative position estimate.

We evaluated our collaborative system in an indoor scenario (Office) and compared it with a lateration baseline. Our indoor scenario presents the aforementioned typical adverse conditions: an inadequate deployment of \ac{ble} anchors, rich \ac{nlos} conditions and heterogeneity among mobile devices used, causing a considerably reduction of positioning accuracy in \ac{rss}-based approaches. Experimental results show the feasibility and benefits of our proposed collaborative approach to outperform the positioning accuracy of the traditional lateration baseline under these adverse conditions. Particularly, our approach decreases, with respect to the lateration baseline, the mean, median, \ac{rmse} and  $75^{th}$ percentile positioning error metrics with $15.11\%$, $12.63\%$, $9.37\%$ and $5.64\%$ respectively.

Furthermore, the results highlighted the usefulness of an \ac{mlp} neural network model to model the signal propagation at short distance considering the inherent characteristic of each receiving device and in \ac{nlos} conditions, which is essential for estimating the distance between (heterogeneous) collaborative devices in real-world conditions.

As future work, we plan to use robust baseline approaches, which are able to provide a better positioning accuracy. We will consider more complex algorithms to combine independent collaborative estimates and reduce the overall positioning error. Additionally, we will enhance the neural network tuning, testing it in diverse and complex scenarios.

\balance
\renewcommand*{\UrlFont}{\rmfamily}
\printbibliography

\end{document}