Vistara, which is hitting the skies on January 9, has many innovative marketing techniques to attract passengers and keep rivals on their toes.
The airline has an innovative pricing strategy where it tries to maximise its revenue while giving customers more options on their flying experience.
It is offering a new cabin class, premium economy, in addition to business class and economy class. Vistara's 148 seater A320-200 planes will have 16 business class seats, 36 premium economy class and 96 economy class. Usually, full service carriers in India have two classes.
With the introduction of premium economy class, industry watchers say, Vistara is trying to tap frequent flying mid-level professionals who are not entitled to business class travel as per their company rules.
By paying extra than the economy class, the premium economy fliers can "enjoy extra space and ample legroom along with longer armrests, winged leather headrests and individual coat hooks". Business class passengers will have heigh adjustable headrests along with personal storage space.
"Our journey has been a rollercoaster ride. A full service is more than free food while flying and we are offering personalised seamless travelling experience," Vistara CEO Phee Teik Yeoh said.
Another highlight of the new airline is a special dining experience using fresh and locally sourced ingredients to prepare healthy meals to ensure that passengers arrive at their destination feeling light and fresh, said Giam Ming Toh, Chief Commercial Officer of Vistara.
The menus will change every week and will only reappear every 4 weeks to continuously offer new dining experiences for frequent travellers, he said. A choice of two non-vegetarian and one vegetarian meal will be offered in business and premium economy, while in economy there will be a choice of one each.


text:
Home » Celebrities » How Real Is 'Chernobyl'? Here's What Fans Of The HBO Show Should Know
How Real Is 'Chernobyl'? Here's What Fans Of The HBO Show Should Know
On April 26, 1986, at 1:23 a.m. local time, there was an explosion at the Vladimir I. Lenin Nuclear Power Plant located about 30 km from the town of Chernobyl. Reactor Number 4, the most recently built of the six reactors planned, had exploded. It was the second nuclear power plant accident in a decade (the first being Three Mile Island in the US in 1979), and the only one to ever record casualties. Now, HBO has a five-episode miniseries explaining how it happened. But how real is HBO's Chernobyl? Warning: Spoilers for Chernobyl follow.
HBO's new miniseries begins as a bit of a whodunit, starting in 1988, when Chernobyl's chief of commission, Valery Legasov (Jared Harris), takes his own life. (That really happened.) The action then jumps back to the moment the plant exploded, and the mystery of what happened and why.
Unlike some historical miniseries, which take liberties with the truth, Chernobyl looks utterly legit. Russian immigrants watching the series have marveled at how much detail the show gets right about day to day life in the USSR. But are the facts really as the show describes them?
In some cases, yes. However, at the same time, there are definitely places where the series is going for the drama.
The show's description of the explosion is correct. It was the result of a safety test, in which a power outage was supposed to be simulated, to see if the reactor would be able to "jump" the one-minute gap before the backup generators kicked in. It happened because the "cooling rods," which had been removed improperly, were all reinserted at once when the test started to go very south. Except, the tips of the rods were made of graphite, heating up the core instead of cooling it, acting as a thermal detonator and blowing the reactor core sky high.
However, the series makes it seem like the USSR sent all of two people out to Chernobyl, Legasov and his boss, Boris Shcherbina (Stellan Skarsgård), and while they were there, a third, Ulana Khomyuk (Emily Watson) crashed the party by getting herself arrested, and they did all the work. This is untrue. In reality, there were hundreds of scientists who worked on this project, in a desperate attempt to understand it and solve the crisis as fast as humanly possible.
There are also some details that are straight up wrong. Radiation sickness does a lot of things, but it doesn't make you bleed. Firefighters are shown crawling over rubble, when in fact they went up on the rooftops and sprayed water down. The "radiation column" from the fire straight up looks like a 9/11 tribute. Radiation does many things, but that's not one of them. The miners did not work naked.
But those details aren't essential. Chernobyl's point is a culture where the government lies from the top is a culture where a catastrophe like this one goes from an emergency to a tragedy. Three Mile Island didn't kill anyone because there was a culture of safety and whistleblowers. Everyone knew what was happening. In contrast, those who lived three miles from the plant in the city of Pripyat not only went out and watched the fire burn directly downwind of it but spent the next 36 hours living under a massive cloud of radiation with no idea of the danger they were in. Not only did the government not tell them, but the phone lines were also cut so no one else could, and the roads blockaded.
Chernobyl is a warning. Allowing one's government to lie about small, easily disprovable things (Trump saying "I never called Meghan Markle nasty," when it's literally on tape) leads to an entire system where everyone lies about everything. That way lies disaster for all.
« Kawhi Leonard Suing Nike Over 'Klaw' Logo, I Drew It Myself!
Jennifer Lopez and Alex Rodriguez Are the Epitome of Style at the 2019 CFDA Awards »

redpajama_set_name: RedPajamaCommonCrawl


Yes, all hurdles are now finally removed, World's cheapest tablet manufactured and developed by Indian Government , will hit the market on October 5. I already posted about this 35 $ tablet in June 2011, this tablet is delayed due to some tender problems related with HCL company who manufactured these tablets, all problems are now solved and finally, the dream of launching an Indian Tablet with a price tag of $35 (INR 1700) has come true and India is all set to witness this man's dream on October 5, 2011 according to sources.
Well, we are eagerly waiting about the device and we certainly hope that this project lives up to its expectation. If executed well, this can drastically change the way education is imparted and importantly, open up plethora of possibilities for startups/technology companies to build products around the education ecosystem.


text:
Open Menu ///
A Milestone in the AI Act negotiations
On April 21st, the much-awaited IMCO-LIBE draft report on the Artificial Intelligence Act (AI Act), produced by the European Parliament lead negotiators Brando Benifei and Dragoș Tudorache, was finally disclosed – a major milestone in the AI Act negotiations. While we are preparing a detailed analysis in collaboration with our partners, we share below our first take on the draft report.
#aiact #predictivepolicing #publicsector #sustainability
Shane | Unsplash
The draft report includes some significant steps towards a more effective protection of fundamental rights: More obligations on deployers (aka "users" in the terminology of the Act) of AI systems, such as the duty to inform affected persons of the use of high-risk AI systems. To allow for public scrutiny and accountability, it is key that not only high-risk AI systems but also uses thereof are made transparent. The draft report considers some of our core demands with regard to the planned EU database:
✅ When a public authority uses a high-risk AI system, it must register it in the database.
✅ The database shall be user-friendly, accessible, easily navigable & machine-readable.
❌ High-risk AI use cases by private sector are not included.
❌ Not ALL use cases by public authorities must be registered (only the high-risk ones)
The definition of AI remains without any significant changes. A positive development, as the current definition is rather broad and should be kept so that systems which can have a significant impact on our lives are in the scope of the Act. Fortunately, the draft report did not follow Axel Voss' direction (European Parliament Committee on Legal Affairs) in excluding the area of national security in the scope of the Act. This would leave major loopholes, as many AI systems could be deployed in the name of "national security".
Further, at least some uses of predictive policing systems shall be banned – unfortunately, not all of them. However, these systems have been demonstrated to infringe fundamental rights, which is why AlgorithmWatch alongside many civil society organisations call for a ban of predictive policing systems in the EU.
We welcome that people affected by an AI system – that falls under the scope of the Act – shall have the right to lodge a complaint against the providers or users of that AI system. This is key!
However, the report does not consider the impact of AI systems on the environment & aspects of ecological sustainability. We recommend that it includes transparency indicators about the resource consumption of AI systems, which is an imperative to combat climate change.
The draft report also misses the opportunity to close loopholes with regard to the ban on biometric recognition in public space – and leaves the proposed ban as limited as before.
As next steps, members of the two main committees (IMCO-LIBE) can table amendments until May 18. Meanwhile, we will work together with civil society partners on the strengthening of the Act to ensure our rights are protected.
A detailed analysis of the draft report can be found here.
Read more on our policy & advocacy work on the Artificial Intelligence Act
AlgorithmWatch's demands for improving the AI Act
Send me the latest news about AlgorithmWatch's projects and campaigns via e-mail once per month. More information
Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 4.0 International license (CC BY 4.0).
AlgorithmWatch's Equivalency Determination (ED) is certified by NGOsource.


In this fast-paced, modern world, digital trends are continually evolving. If you to attract users' attention, keeping up with current tendencies is essential. Your website is often the gateway to your company. With good web design, you can win visitors' trust and lead them to your products and services.
To make it easier for everyone to understand exactly what needs to be done to increase your company's online presence, the folks over at Design Advisor have put together a top-notch infographic detailing fourteen of the most essential web design trends.
If you want users to return, engaging content is key. Allowing customers to become familiar with your company and your brand helps forge a stronger relationship, which makes them more likely to want to come back. Establishing trusts brings a business one step closer toward a high customer retention rate.
One mistake that a lot of companies make is overlooking the importance of responsive web design. According to the infographic, 44% of users mistrust a company if their website doesn't work well on mobile. Ensuring your website performs equally well on any device or screen size is a great way to broaden your reach and help convey reliability.
What is the most annoying thing you can encounter while browsing a website? Most people would say waiting for a site to load. More specifically, 47% of users think that a website should not take more than two seconds to load. Therefore, speeding up your site should be a no brainer.
These are just a few of the trends this infographic holds. If they sparked your interest, there is a lot more where that came from in the infographic below. Investing a few minutes to check it out can be a helpful push in the right direction. Your website visitors will thank you – in the shape of conversions!


text:
Our Free Ebook
What is an Extraordinary Experience? Extraordinary Experiences are those which occur beyond our five senses, time, and space. Such occurrences include Near-death Experiences, Out-of-Body Experiences, Past-life Memories, Paranormal Encounters, Contact with Non-Human Intelligences, Telepathy, Pre-cognition, etc. Regardless of the type of extraordinary experience, all are utterly transformational. They enable us to more deeply understand reality and realize that numerous commonalities exist among all phenomena. It is only when we open our minds and hearts to embrace experiences which lay beyond our three-dimensional world, that we may begin to explore the very nature of consciousness.
This blog will serve as the main site for our upcoming book:
Convergence: The Interconnection of Extraordinary Experiences
By Barbara Mango, Ph.D., and Lynn Miller, MS
Nonlocality
Reality is merely an illusion
All is interconnected
Barbara is a collaborating author in The Transformative Power of Near-Death Experiences: How Messages of NDEs Positively Impact the World, by Penny Sartori. Her chapter, entitled The Gift of Being, is dedicated to her late father who passed away from pancreatic cancer. Barbara witnessed the spiritual experiences that were part of her father's transition into death. A synopsis of Barbara's chapter was also published in Mind, Body, Spirit Magazine.
Lynn is a collaborating author in Vol.2 of Beyond UFOs: The Science of Consciousness and Contact. Vol.2, titled "A Greater Reality" will be available in Spring, 2020. Lynn's chapter discusses the interrelation between OBEs and other contact modalities, via her personal encounters.
Barbara received her MA and Ph.D. in Metaphysics. Prior to receiving her post-graduate degrees, she worked as a pre-K to 6 educator. She currently researchers, writes, and speaks about extraordinary phenomena, the anomalous-prone personality, and consciousness. Barbara is a Board Member of the PLR Institute (Past-Life Research Institute), and served on the research committee for the Dr. Edgar Mitchell Foundation for Consciousness and Contact. She has been published in blogs, print, and online magazines, and is a contributing author to the book, The Transformative Power of Near-Death Experiences, by Dr. Penny Sartori (Watkins, 2017). She is also a contributing author to to the book, We Touched Heaven: A Collection of Experiences that Reached Beyond the Veil! By Claudia Watts Edge, 2021. She has written her first book, Convergence: The Interconnection of Extraordinary Experiences, with co-author Lynn Miller, MS. Barbara is a life-long experiencer of inexplicable phenomena. She lives in the Northeast with her husband and three quirky, misbehaved felines.
Lynn holds dual BA degrees in Psychology and Biology, and a MS in Biology. For several years, she worked in the food industry as a microbiologist. Lynn served as an Adjunct Professor at Pensacola State College, where she taught Botany, Microbiology, and Biology. She has taught High School Biology and art, k-12 for thirteen years. Lynn is a collaborating author in the upcoming book: "A Greater Reality" available Spring, 2020. She is a frequent co-host with Brent Raynes, Alternate Perception Audio Interview Series. Influenced by the work of William Buhlman, Lynn has practiced controlled out-of-body experiences since 2009. For over fifteen years, she has extensively researched consciousness.
© 2021 by Extraordinary Experiences
The opinions expressed in all material are those of the authors.


text:
\section{Introduction}
\label{sec:introduction}
    The use of deep learning for processing natural language is becoming a standard, with excellent results in a diverse range of tasks. Two state-of-the-art  architectures for text-related modeling are long short-term memory (LSTM) networks~\citep{hochreiter1997-lstm} and transformers~\citep{vaswani2017-transformers}.
LSTMs are recurrent neural networks that process the text sequentially, meaning that they process text one token at a time, building up its internal representation in hidden states of the network.
Due to the recurrent nature of LSTM, which degrades the efficiency of parallel processing, as well as demonstrated improvements in performance, models based on the transformer architecture are gradually replacing LSTMs across many tasks. Transformers can process the text in parallel, using self-attention and positional embeddings to model the sequential nature of the text.

A common trend in using transformers is to first pre-train them on large monolingual corpora with abstract, general-purpose objective, and then fine-tune them for a specific task, such as text classification.
For example, the BERT (Bidirectional Encoder Representations from Transformers) architecture \cite{devlin2019-bert} uses transformers and is pretrained with masked language modelling and order of sentences prediction tasks to build a general language understanding model.
During the fine-tuning for a specific downstream task, additional layers are added to the BERT model, and the model is trained on specific data to capture the specific knowledge required to perform the task.

Most of the research in the natural language processing (NLP) area focuses on English, ignoring the fact that English is specific in terms of the low amount of information expressed through morphology (English is so-called analytical language).
In our work, we focus on adapting modern deep neural networks, namely LSTMs and BERT, for several morphologically rich languages, by explicitly including the morphological information. The languages we analyze contain rich information about grammatical relations in the morphology of words instead of in particles or relative positions of words (as is the case in English). For comparison,  we also evaluate our models on English.
Although previous research has shown that the state of the art methods such as BERT already captures some information contained in the morphology~\citep{pires-2019-mtl-bert-probing,edmiston2020systematic}, our experiments involve several languages with rich morphology where neural networks could benefit from explicit morphological features.

Specifically, we present methods which combine BERT with separately encoded morphological properties: universal part of speech tags (uPOS tags) and universal features (grammatical gender, tense, conjugation, declination, etc). We evaluate them on three downstream tasks: named-entity recognition (NER), dependency parsing (DP), and comment filtering (CF).
We perform similar experiments on LSTM networks and compare the results for both architectures. Besides English, we analyze eight more languages: Croatian, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovene and Swedish. The choice of these languages reflects a mix of different language groups (Balto-Slavic, Germanic, and Uralic), for which we were able to obtain sufficient resources (datasets, embeddings, and corpora), due to their coverage in the EU EMBEDDIA project\footnote{\href{http://embeddia.eu}{http://embeddia.eu}}.

Our experiments show that the addition of morphological features has mixed effects depending on the task.
Across the tasks where the added morphological features improve the performance, we show that (1.) they benefit the LSTM-based models even if the features are noisy and (2.) they benefit the BERT-based models only when the features are of high quality (i.e. human checked), suggesting that BERT models already capture the morphology of the language; however, there is a room for improvement either in designing pre-training objectives that can capture these properties or when high-quality features are available.

The remainder of this paper is structured as follows.
In Section~\ref{sec:related-work}, we present different attempts to use morphological information in machine learning, in particular neural networks, as well as an overview of recent work in the three evaluation tasks.
In Section~\ref{sec:data}, we describe the used datasets and their properties.
In Section~\ref{sec:methods}, we present the baseline models and models with additional morphological information, whose performance we discuss in Section~\ref{sec:evaluation}.
Finally, we summarize our work and present directions for further research in Section~\ref{sec:conclusion}.

\section{Related work}
\label{sec:related-work}

We shortly review the related work on the use of morphological information within neural networks. We split the works based on three evaluation tasks: NER, DP, and CF.

\subsection{Morphological features in NER}
Recent advances in NER are mostly based on deep neural networks.
One of the first usages of deep neural networks in NER is the work by \citet{collobert2008-early-dnn-ner}, who propose an architecture that is jointly trained on six different tasks, including NER, and show that this transfer learning approach generalizes better due to learning a joint representation for different tasks. A standard approach to NER is to use word \citep{huang2015-lstm-variants-ner} or character \citep{kuru2016-charner} embeddings of the input, followed by one or more neural layers, mostly recurrent ones, such as LSTMs.

While sequence modelling on the character-level already has a potential to encode morphological information, several authors show that the performance of neural networks on the NER task can be improved by adding explicit information about morphological properties of the text.
\citet{gungor2017-turkish-ner} show that embedding morphosyntactic properties improves NER performance on the Turkish language, while \citet{simeonova2019-bulgarian-ner} use part of speech embeddings to achieve improved performance on the NER task in the Bulgarian language. Our work is similar to these approaches in the sense that we add different morphological features to the neural networks. However, we experiment on a larger scale, using more languages from different language groups, different evaluation tasks, and with modern neural architectures (BERT), taking the LSTM networks as the baseline.

\subsection{Morphological features in dependency parsing}
Similarly to NER, recent progress in DP is dominated by neural approaches, which can be roughly divided into two categories, transition-based \citep{yamada-matsumoto-2003-transition-based-origin-1,nivre-2003-transition-based-origin-2} and graph-based approaches \citep{mcdonald-2005-graph-based-origin}. Some works do not fall into either category, e.g., DP is treated as a sequence-to-sequence task by \citet{li-2018-dep-parsing-seq2seq}.
The two categories differ in how dependency trees are produced from the output of prediction models.
In the transition-based approach, a model is trained to predict a sequence of parsing actions which produce a valid dependency tree, while in the graph-based approach, a model is used to score candidate dependency trees via the sum of scores of their substructures (e.g., arcs).
One of the earlier successful approaches to neural DP was presented by  \citet{chen-manning-2014-early-nn-transition}, who replaced the commonly used sparse features with dense embeddings of words, part of speech tags and arc labels, in combination with the transition-based parser. This approach improved both the accuracy and parse speed.
\citet{pei-2015-early-nn-graph} introduced a similar approach to graph-based parsers.
Later approaches improve upon the earlier methods by automatically extracting more information that guides the parsing. E.g., researchers use LSTM networks to inject context into local embeddings \citep{kiperwasser-goldberg-2016-lstm-parser}, apply contextual word embeddings \citep{kulmizev2019-contextual-embeddings-parser}, or train graph neural networks \citep{ji-2019-gnn-parser}.

The use of morphological features in DP is quite common. For example, \citet{chen-manning-2014-early-nn-transition} note that POS tag embeddings strongly contribute to the performance of their system in English and Chinese.
Morphological features are commonly used with morphologically rich languages, e.g., Arabic \citep{marton-2010-arabic-parser}, German \citep{seeker-2011-german-parser}, Lithuanian \citep{kapovciute-2013-lithuanian-parser}, or Persian language \citep{khallash-2013-persian-parser}, with the consensus that they improve the accuracy of parsing.
Concerning adding morphological properties to BERT, \citet{kondratyuk-straka-2019-multilingual-multitask-bert-parser} train the multilingual BERT model for multiple tasks, including POS and morphological tagging, and achieve improved results across multiple languages. This approach uses transfer learning to implicitly injects the morphological information into BERT models, while our approach does that explicitly, through morphological features on the input.

\subsection{Morphological features in comment filtering}
The literature for the task of NER covers multiple related tasks, such as hate speech, offensive speech, political trolling, detecting commercialism, etc. Recent approaches involve variants of deep neural networks, though standard machine learning approaches are still popular as shown in the survey of \citet{fortuna-2018-hate-speech-survey}. These approaches typically use features such as character n-grams, word n-grams, and sentiment of the sequence. Two examples are the works of \citet{malmasi-2017-hate-speech-classic-ml-1}, who classify hate speech in English tweets, and \citet{van-hee-2015-dutch-cyberbullying}, who classify different levels of cyberbullying in Dutch posts on ask.fm.
\citet{scheffler-2018-hate-speech-embeddings} combine word embeddings with the above-mentioned features to classify German tweets. They observe that combining both n-gram features and word embeddings brings only a small improvement over only using only one of them. The effectiveness of using features describing syntactic dependencies for toxic comments classification on English Wikipedia comments is shown by \citet{shtovba-2019-hate-speech-dependencies}.

Neural architectures used include convolutional neural networks \citep{georgakopoulos-2018-hate-speech-cnn} and LSTM networks \citep{gao-2017-hate-speech-lstm,miok2019prediction}, typically improving the performance over standard machine learning approaches.
The NER topic has also been the focus of shared tasks on identification and categorization of offensive language \citep{zampieri-2019-hate-speech-semeval6} and multilingual offensive language identification \citep{zampieri-2020-hate-speech-semeval12}.
The reports of these tasks show the prevalence and general success of large pretrained contextual models such as BERT, though, surprisingly, the best performing model for the subtask B of SemEval-2019 Task 6 was rule-based \citep{han-etal-2019-jhan014}.

\subsection{Linguistic knowledge combined with neural networks}
Large pretrained models such as BERT show superior performance across many tasks. Due to a lack of theoretical understanding of this success, many authors study how and to what extent BERT models can capture various information, including different linguistic properties. An overview of recent studies in this area sometimes referred to as BERTology, is compiled by \citet{rogers2020primer}.
Two common approaches to study BERT are i) add additional properties to BERT models and observe the difference in performance on downstream tasks, ii) a technique called probing~\citep{conneau-etal-2018-probing}, where the BERT model is trained (fine-tuned) to predict a studied property. For example, \citet{jawahar-etal-2019-bert-language-structure} investigate what type of information is learned in different layers of the BERT English model and find that it captures surface features in lower layers, syntactic features in middle layers, and semantic features in higher layers.
Similarly,~\cite{lin2019-bert-linguistic} find that BERT encodes positional information about tokens in lower layers and then builds increasingly abstract hierarchical features in higher layers.
~\cite{tenney-etal-2019-bert-rediscovers-pipeline} use probing to quantify where different types of linguistic properties are stored inside BERT's architecture and suggest that BERT implicitly learns the steps performed in classical (non-end-to-end) natural language processing pipeline. However, \cite{elazar2020-amnesic-probing}  point out possible flaws in the probing technique, suggesting amnesic probing as an alternative.
They arrive at slightly different conclusions about BERT layer importance; for example, they show that POS information affects the predictive performance much more in upper layers.

Specifically for morphological properties,  \cite{zhou2020pos} study whether the inclusion of POS information is still necessary for a neural dependency parser. Somewhat similarly to the previously mentioned transfer learning approach of \cite{kondratyuk-straka-2019-multilingual-multitask-bert-parser}, they add POS information to the DP task in English and Chinese by training a model with multi-task learning objective; they conclude that POS tags are still beneficial in a neural dependency parser, which our experiments on a larger set of languages confirm.
In contrast to our work, which studies the impact of morphological features in BERT on the performance in downstream tasks, \cite{edmiston2020systematic} studies the morphological content present in BERT. The author uses probing, i.e. he trains a linear classifier to predict morphological features based on features of a certain hidden layer. Based on the achieved high performance, he argues that BERT models capture much morphological information and partition their embedding space into linearly-separable regions, correlated with morphological properties.

\section{Data}
\label{sec:data}
In this section, we describe the datasets used in our experiments, separately for each of the three tasks: NER, DP, and NER.

\subsection{Named entity recognition}
In the NER experiments, we use datasets in eight languages: Croatian, English, Estonian, Finnish, Latvian, Russian, Slovene and Swedish.
We omit the Lithuanian language from this experiment, as models for obtaining POS tags and universal features were not available in the used tools at the time of performing the experiments.
The number of sentences and tags present in the datasets is shown in Table~\ref{tab:ner-datasets}.
The label sets used in datasets for different languages vary, meaning that some contain more fine-grained labels than others.
To make results across different languages consistent, we trim labels in all datasets to the four common ones: location (LOC), organization (ORG), person (PER), and ``no entity'' (OTHR).

\begin{table}[htb]
    \centering
	\caption{The collected datasets for NER task and their properties: the number of sentences and tagged words.}
	\label{tab:ner-datasets}
	\begin{tabular}{llrrcl}
		\toprule
		Language	& Dataset & Sentences & Tags \\
		\midrule
		Croatian	& hr500k & $24794$ & $28902$ \\
		English 	& CoNLL-2003 NER & $20744$ & $43979$ \\
		Estonian	& Estonian NER corpus & $14287$ & $20965$ \\
		Finnish  	& FiNER data & $14484$ & $16833$ \\
		Latvian  	& LV Tagger train data & $9903$ & $11599$ \\
		Russian 	& factRuEval-2016 & $4907$ & $9666$ \\
		Slovene\tablefootnote{The Slovene ssj500k originally contains more sentences, but only $9489$ are annotated with named entities.} 	& ssj500k & $9489$ & $9440$ \\
		Swedish	& Swedish NER & $9369$ & $7292$ \\
		\bottomrule
	\end{tabular}
\end{table}

\subsection{Dependency parsing}
To test morphological neural networks on the DP task, we used datasets in nine languages (Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovene and Swedish). The datasets are obtained from the Universal Dependencies \citep{universal-dependencies}. The number of sentences and tokens is shown in Table~\ref{tab:depparse-datasets}.
\begin{table}[htb]
    \centering
	\caption{Dependency parsing datasets and their properties: the treebank, number of sentences, number of tokens, and information about the size of the split.
	}
	\label{tab:depparse-datasets}
	\begin{tabular}{llrrrrr}
	\toprule
	Language & Treebank & Tokens & Sentences & Train & Validation & Test \\
	\midrule
	Croatian & SET & $199409$ & $9010$ & $6914$ & $960$ & $1136$ \\
	English & EWT & $254855$ & $16622$ & $12543$ & $2002$ & $2077$ \\
	Estonian & EDT & $438171$ & $30972$ & $24633$ & $3125$ & $3214$ \\
	Finnish & TDT & $202697$ & $15135$ & $12216$ & $1364$ & $1555$ \\
	Latvian & LVTB & $220536$ & $13643$ & $10156$ & $1664$ & $1823$ \\
	Lithuanian & ALKSNIS & $70051$ & $3642$ & $2341$ & $617$ & $684$ \\
	Russian & GSD & $98000$ & $5030$ & $3850$ & $579$ & $601$ \\
	Slovene & SSJ & $140670$ & $8000$ & $6478$ & $734$ & $788$ \\
	Swedish & Talbanken & $96858$ & $6026$ & $4303$ & $504$ & $1219$ \\
	\bottomrule
	\end{tabular}
\end{table}

\subsection{Comment filtering}
While there exist comparable datasets across different languages for the NER and DP task, for the CF task, no such standard datasets exist. For that reason, in our experiments on CF, we used two languages with adequate datasets: English and Croatian.

For English experiments, we use a subset of toxic comments from Wikipedia's talk page edits\footnote{https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data}.
The comments are annotated with six possible labels: toxic, severe toxic, obscene language, threats, insults, and identity hate (making a total of six binary target variables). We extracted comments from three categories: toxic, severe toxic, threats, and identity hate;  a total of $21,541$ instances. We randomly chose the same amount of comments that do not fall in any of the mentioned categories, obtaining the final dataset of $43,082$ instances.

For Croatian language experiments, we form a dataset from user comments published in the Croatian news site 24sata. The comments used are either unproblematic or labelled with one of the eight rules they break. Below we briefly describe the types of problematic content present in the database.
\begin{enumerate}

    \item Advertising, content unrelated to the topic, spam, copyright infringement, citation of abusive comments or any other comments that are not allowed on the portal.

    \item Direct threats to other users, journalists, admins or subjects of articles, which may also result in criminal prosecution.

    \item Verbal abuse, derogation and verbal attack based on national, racial, sexual or religious affiliation, hate speech and incitement.

    \item Collecting and publishing personal information, uploading, distributing or publishing pornographic, obscene, immoral or illegal content and using a vulgar or offensive nickname that contains the name and surname of others.

    \item Publishing false information for deception or slander, and ``trolling'' - deliberately provoking other commentators.

    \item Use of bad language, unless they are used as a stylistic expression, or are not addressed directly to someone.

    \item Writing in other languages besides the Croatian, in other scripts besides Latin or writing with all caps.

    \item Verbally abusing other users and their comments, article authors, and direct or indirect article subjects, calling the admins out or arguing with them in any way.
\end{enumerate}

 Unfortunately, this data is proprietary and is not publicly available. We extracted all the $17,868$ comments from the third category (the hate speech label) and approximately the same number of comments that do not break any rules. Our final dataset contains  $35,635$ instances.

\section{Neural networks with morphological features}
\label{sec:methods}
In this section, we describe the architectures of neural networks used in our experiments. Their common property is that we enhance standard word embeddings based inputs with embeddings of morphological features. We work with recent successful neural network architectures, LSTMs and transformers, i.e. BERT models. A detailed description of architectures is available in the following subsections, separately for each of the three evaluation tasks. For each task and architecture, we describe the baseline architecture and the enhanced one.

\subsection{Named entity recognition models}
\label{sec:ner-models}

In the NER task, we use two baseline neural networks (LSTM and BERT), and the same two models with additional morphological information: POS tag embeddings and universal feature embeddings. The baseline models and their enhancements are displayed in Figure~\ref{fig:ner_models}.

The first baseline model (left-hand side of \Cref{fig:ner_models}) is a unidirectional (left-to-right) LSTM model, which takes as an input a sequence of tokens, embedded using $300$-dimensional fastText embeddings \citep{bojanowski2017enriching} which are particularly suitable for morphologically rich languages as they work with subword inputs\footnote{The precomputed embeddings are available at \href{https://fasttext.cc/docs/en/crawl-vectors.html}{https://fasttext.cc/docs/en/crawl-vectors.html}.}.
For each input token, its LSTM hidden state is extracted and passed through the linear layer to compute its NER  score (probability for each of the four NER labels).

The second baseline model (right-hand side of \Cref{fig:ner_models}) is the multilingual BERT model (the base cased variant). In our experiments, we follow the sequence tagging approach suggested by the authors of BERT~\citep{devlin2019-bert}. Here, the input sequence is prepended with a special token [CLS] and passed through the BERT model. The output of the last BERT hidden layer is passed through the linear layer to obtain the predictions for NER labels.

Both baseline models (LSTM and BERT) are enhanced with the same morphological information: POS tag embeddings and universal feature embeddings for each of the input tokens. We embed the POS tags using $15$-dimensional embeddings. For each of the $23$ universal features used (we omitted the \textit{Typo} feature, as the version of the POS tagger we used did not annotate this feature), we constructed $15$-dimensional embeddings. We computed the POS tags and morphological features using the Stanza~\citep{qi2020stanza} system in the universal dependencies mode. In the enhanced architectures, we included another linear layer before the final linear classification layer to model possible interactions.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.7\textwidth]{named_entity_recognition.pdf}
    \caption{The baseline LSTM-based (left) and BERT-based (right) models for the NER task along with our modifications with morphological information. The dotted border of POS vectors, morphological features (feats), and the linear layer  marks that their use is optional and varies across experiments. The $\odot$ symbol between layers represents the concatenation operation. The $w_i$ symbol stands for token $i$; in case of LSTM, tokens enter the model sequentially and we show the unrolled network, while BERT processes all token simultaneously.}
    \label{fig:ner_models}
\end{figure}

\subsection{Dependency parsing models}
\label{sec:depparse-models}

As the baseline model in the DP task, we use the deep biaffine graph-based dependency parser \citep{dozat2016deep}. The enhancements with the morphological information are at the input level.
The baseline model and its enhancements are shown in Figure~\ref{fig:dependency_parsing}.

The baseline parser combines a multi-layer bidirectional LSTM network with a biaffine attention mechanism to jointly optimize prediction of arcs and arc labels.
We leave the majority of baseline architectural hyperparameters at values described in the original paper ($3$-layer bidirectional LSTM with $100$-dimensional input word embeddings, and the hidden state size of $400$).

In our experiments, we concatenate the non-contextual word embeddings with various types of additional information.
The first additional input is contextual word embeddings, which we obtain either by using the hidden states of an additional single-layer unidirectional LSTM or by using a learned linear combination of all hidden states of BERT.
Although the LSTM layers are already present in the baseline parser, we include an additional layer at the input level to keep the experimental settings similar across our three evaluation tasks.
The second additional input is universal POS embeddings (UPOS), and the third one are universal feature embeddings (feats). These embeddings are concatenated separately for each token of the sentences. The size of the additional LSTM layer, POS tag embeddings and universal feature embedding are treated as tunable hyperparameters.
As the baseline input embeddings, we use pre-trained $100$-dimensional fastText embeddings, which we obtain by reducing the dimensionality of publicly available $300$-dimensional vectors with fastText's built-in dimensionality reduction tool.

In DP experiments, we use POS tags and morphological features of two origins. The first source is human annotations provided in the used datasets. The second source of morphological information is predictions of Stanza models \citep{qi2020stanza}. These two origins are used in the assessment of the quality of morphological information; namely, we check if manual human annotations provide any benefit compared to automatically determined POS tags and features.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.7\textwidth]{dependency_parsing.pdf}
    \caption{The deep biaffine graph-based dependency parser along with our enhancements at the input level.
    The dotted border of input embedding vectors, POS vectors, and morphological features (feats) is optional and varies across experiments. The $\odot$ symbol between layers represents the concatenation operation. The $w_i$ symbol stands for token $i$; tokens enter the LSTM model sequentially, and we show the unrolled network.}
    \label{fig:dependency_parsing}
\end{figure}

\subsection{Comment filtering models}
\label{sec:comment-filtering-models}
In the NER evaluation task, we add additional morphological information to standard LSTM and BERT models. The baseline models and enhanced models for this task are similar to those in the NER evaluation, though we operate at the sequence level here as opposed to the token level in the NER task. The architecture of models is shown in Figure~\ref{fig:comment_filtering}.
As baselines, we take a single layer unidirectional LSTM network (the top part of Figure~\ref{fig:comment_filtering}), and the multilingual base uncased BERT model  (the bottom part of Figure~\ref{fig:comment_filtering}). The difference in the used BERT dialect (\emph{uncased} as opposed to the \emph{cased} in the NER task) is due to better performance detected in preliminary experiments on a separate Croatian validation set.

In the LSTM baseline model, the words of the input sequence are embedded using pre-trained $300$-dimensional fastText embeddings. As the representation of the whole sequence, we take the output of the last hidden state, which then passes through the linear layer to obtain the prediction scores.
In the BERT baseline model, we take the sequence classification approach suggested by the authors of BERT. The input sequence is prepended with the special [CLS] token and passed through BERT. The sequence representation corresponds to the output of the last BERT hidden layer for the [CLS] token, which is passed through a linear layer to obtain the prediction scores.

As in other tasks, we augment the baseline models with POS tag and universal feature embeddings. We obtain the embeddings for each token separately from UDPipe models \citep{udpipe2017}. The reason for not using Stanza in this set of experiments is that we started experimenting before its release. Additional testing with Stanza did not show a difference in performance, so we stayed with the UDPipe models for this set of experiments.
We combine the obtained embeddings using three different pooling mechanisms: mean, weighted combination, or LSTM pooling.
Given the POS tag or universal feature embeddings, the mean pooling outputs the mean of all token embeddings; the weighted pooling outputs the weighted combination of token embeddings, and the LSTM pooling outputs the last hidden state obtained by passing the sequence of embeddings through the LSTM network. Both the embedding sizes and the type of pooling are treated as tunable hyperparameters.
The coefficients of the weighted combination are learned by projecting the sequence embeddings into a sequence of independent dimension values which are normalized with the softmax. This compresses the embedding sequence, establishes its fixed length, and allows for different morphological properties to have a different impact in the sequence representation. This approach tests the contextual encoding of morphological properties. For example, adjectives might be assigned a higher weight than other POS tags due to their higher emotional contents that often indicates insults.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.7\textwidth]{comment_filtering.pdf}
    \caption{
    The baseline LSTM (top) and BERT (bottom) models for the NER task along with our modifications with morphological information. The dotted border of UPOS vectors and morphological features (feats)  marks that their use is optional and varies across experiments. The $\odot$ symbol between layers represents the concatenation operation. The $w_i$ symbol stands for token $i$; in case of LSTM, tokens enter the model sequentially and we show the unrolled network, while BERT processes all token simultaneously.}
    \label{fig:comment_filtering}
\end{figure}

\section{Evaluation}
\label{sec:evaluation}
In this section, we first present the evaluation scenario for the three evaluation tasks, followed by results, presented separately for each of the tasks. We end the section with the ablation study, performed on the DP task.
In the ablation study, we investigate the effect of additional morphological features in three situations where we tweak one aspect of training procedure at a time: (1) we increase the maximum training time of our models by additional $5$ epochs, (2) we replace the human-annotated features with the ones automatically predicted by machine learning models, and (3) we replace the embeddings from general multilingual BERT with those from more specific multilingual and monolingual BERT models.

\subsection{Experimental settings}
The experimental settings differ between the three evaluation tasks, so we describe them separately for each task, starting with the NER task, and followed by the DP and NER tasks.

\subsubsection{Experimental settings for NER}
 For NER, we train each BERT model for $10$ epochs and each LSTM model for $50$ epochs. These parameters were determined during preliminary testing on the Slovene dataset. The selected numbers of epochs are chosen to balance the performance and training times of the models. All NER models are evaluated using $10$-fold cross-validation.

We evaluate the models with the $F_1$ score, which is a harmonic mean of precision and recall measures. This measure is typically used in NER in a way that precision and recall are calculated separately for each of the three entity classes (location, organization, and person, but not ''no entity''). For each metric, we compute the weighted average over the class scores using the frequencies of class values in the datasets as the weights. Ignoring the ''no entity'' (OTHR) label is a  standard approach in the NER evaluation and disregards words that are not annotated with any of the named entity tags, i.e. the evaluation focuses on the named entities.

\subsubsection{Experimental settings for dependency parsing}
We train each DP model for a maximum of $10$ epochs, using an early stopping tolerance of $5$ epochs in the process. All models are evaluated using predefined splits into training, validation and testing set, determined by the respective treebank authors and maintainers (see \Cref{tab:depparse-datasets}).
The final models, evaluated on the test set, are selected based on the maximum mean of unlabeled and labelled attachment scores (UAS and LAS) on the validation set. The UAS and LAS are standard accuracy metrics in DP. The UAS score is defined as the proportion of tokens that are assigned the correct syntactic head, while the LAS score is the proportion of tokens that are assigned the correct syntactic head as well as the dependency label \citep{speech-and-language-processing}.

\subsubsection{Experimental settings for comment filtering}
In our NER experiments, we train BERT models for a maximum of $10$ epochs and LSTM models for a maximum of $50$ epochs, using early stopping with the tolerance of 5 epochs.

We evaluate the models using the classification accuracy metric. The choice reflects the fact that the distributions of class labels are approximately balanced in our datasets.
We use fixed training, validation and test sets, obtained by randomly dividing the datasets in the ratio $60\%$:$20\%$:$20\%$.
The initial results in the Croatian hate speech experiments showed large variance; therefore, we repeated each experiment five times. For this task and Croatian language, we report the obtained mean classification accuracy and its standard deviation.

\subsection{Experimental results}
In this section, we compare the results of baseline models with their enhancements using additional morphological information. We split the presentation into three parts, according to the evaluation task: NER, DP, and CF.

\subsubsection{Results for the NER task}
\label{sec:ner-results}
For the NER evaluation task, we present results of the baseline NER models and models enhanced with the POS tags and morphological features, as introduced in Section~\ref{sec:ner-models}. Table \ref{tab:ner-results} shows the results for LSTM and BERT models.
We compute the statistical significance of the differences between the baseline LSTM models and their best-performing counterpart with morphological additions. We use Wilcoxon signed-rank test \citep{wilcoxon1970critical} and underline the statistically significant differences at $p=0.01$ level.
We do not perform the significance of differences tests for BERT as the differences for those models are marginal.

As expected, the baseline models involving BERT outperform their LSTM counterparts across all languages by a large margin.
When adding POS tags or universal features to LSTM-based models, we observe a noticeable increase in performance over the baselines for seven languages.
For three of them (Croatian, Russian, and Swedish), the increase in $F_1$ score is under one per cent.
For three of them (Estonian, Finnish, and Slovene), the differences in performance are statistically significant, and increases range from $1.4\%$ (Finnish) to $4.6\%$ (Slovene).
The only language for which there is no improvement with the LSTM-based models is English.

In BERT-based models, the additional morphological features do not seem to make a practical difference. For all languages except Slovene, the increase in $F_1$ values over the baseline is under  $0.5\%$. However, since preliminary tests in this task were performed on Slovene, the larger observed increase might be due to manual overfitting.

\begin{table}[htb]
	\centering
	\caption{$F_1$ scores for different models on the NER task in different languages. The upper part of the table shows LSTM models and the lower part shows BERT models. The best scores for each language are marked with the bold typeface separately for LSTM and BERT models. Best scores for which the difference to the baseline is statistically significant are underlined.}
	\label{tab:ner-results}
	\begin{tabular}{lrrrrrrrr}
		\toprule
		Model & CRO & ENG & EST & FIN & LAT & RUS & SLO & SWE \\
		\midrule
		fT + LSTM	& 0.700 & \textbf{0.890} & 0.769 & 0.814 & 0.573 & 0.752 & 0.651 & 0.785 \\
		fT + LSTM + UPOS & \textbf{0.708} & 0.889 & \underline{\textbf{0.792}} & 0.824 & \textbf{0.590} & 0.754 & 0.692 & 0.787 \\
		fT + LSTM + UPOS + feats & 0.706 & 0.883 & 0.781 & \underline{\textbf{0.828}} & 0.588 & 0.759 & \underline{\textbf{0.697}} & 0.785 \\
		fT + LSTM + feats & 0.697 & 0.885 & 0.777 & 0.826 & 0.581 & \textbf{0.760} & 0.669 & \textbf{0.793} \\
		\midrule
BERT 	& 0.874 & 0.948 & 0.875 & 0.926 & 0.766 & 0.868 & 0.848 & 0.885 \\
		BERT + UPOS 	& \textbf{0.875} & 0.948 & 0.874 & 0.927 & \textbf{0.773} & \textbf{0.870} & \textbf{0.857} & \textbf{0.886} \\
		BERT + UPOS + feats 	& \textbf{0.875} & \textbf{0.949} & \textbf{0.879} & 0.927 & 0.763 & 0.869 & 0.853 & 0.883  \\
		BERT + feats  	& 0.874 & 0.947 & 0.874 & \textbf{0.928} & 0.769 & 0.869 & 0.851 & 0.878  \\
		\bottomrule
	\end{tabular}
\end{table}

\subsubsection{Results for the dependency parsing task}
\label{sec:dp-results}
For the DP evaluation task on different languages (described in \Cref{sec:depparse-models}), we present UAS and LAS accuracies in Tables \ref{tab:results-dependency-parsing-uas} and \ref{tab:results-dependency-parsing-las}, respectively. We first show the scores of the baseline models (multi-layer bidirectional LSTM network with biaffine attention) followed by these models enhanced with additional inputs: LSTM or BERT contextual embeddings, POS tags and morphological features, introduced in \Cref{sec:depparse-models}. The results in the upper part of  \Cref{sec:depparse-models} represent enhancements incorporating LSTM embeddings, and in the lower part, we show the enhancements including BERT embeddings.
We also statistically test the differences in UAS and LAS scores between the best performing enhanced variants and their baselines without morphological additions. As the splits are fixed in the DP tasks, we use Z-test for the equality of two proportions~\citep{kanji2006-100statistical} at $p=0.01$ level.
The null hypothesis is that the scores of compared models are equal. For languages where the null hypothesis can be rejected, we underline the respective best result.

Similarly to the other two evaluation tasks, the models involving BERT embeddings outperform the baselines involving LSTM embeddings on all languages by a large margin. Enhancements with LSTM embeddings improve over parsers without contextual embeddings.
Models with added POS tags or universal features noticeably improve over baselines with only LSTM embeddings in both UAS and LAS score for all languages.
The increase ranges between $1.2 \%$ (Russian) and $6.8 \%$ (Lithuanian) for UAS and between $2.4 \%$ (Russian) and $10.42 \%$ (Lithuanian) for LAS. All compared differences between the LSTM baselines and the best enhanced variants are statistically significant at $p=0.01$.

Contrary to the results observed on the other two tasks, the addition of morphological features to the baseline models with BERT embeddings improves the performance scores for all languages. For some languages, the increase seems to be practically insignificant (under one per cent). However, for six languages out of nine, the increase in UAS is over one per cent and statistically significant; the same is true for eight languages out of nine for the LAS score.
For these languages, the improvement ranges from $1.1 \%$ (Finnish) to $1.8\%$ (Lithuanian) for UAS, and from $1.1 \%$ (Croatian) to $4.4 \%$ (Lithuanian) for LAS. The only language for which both UAS and LAS scores are practically equal to the baselines with BERT embeddings is Russian.

\setlength{\tabcolsep}{3pt}
\begin{table}[htb]
    \centering
    \caption{UAS scores for different models on the DP task in different languages. The upper part of the table shows baseline parser with enhancements using LSTM-based contextual embeddings; the lower part shows baseline parser enhanced with variants of BERT embeddings. The best scores for each language are marked with the bold typeface separately for LSTM and BERT embeddings extensions. Statistically significant differences in best scores at $p=0.01$ level are underlined.}
    \label{tab:results-dependency-parsing-uas}
    \begin{tabular}{lccccccccc}
    \toprule
    Model & CRO & ENG & EST & FIN & LAT & LIT & RUS & SLO & SWE \\
    \midrule

    baseline &
    $84.83$ &
    $83.54$ &
    $79.35$ &
    $82.45$ &
    $81.51$ &
    $70.51$ &
    $83.04$ &
    $86.66$ &
    $80.68$ \\
    \midrule

    +LSTM &
    $86.48$ &
    $84.87$ &
    $81.92$ &
    $84.13$ &
    $83.63$ &
    $71.82$ &
    $85.02$ &
    $86.92$ &
    $84.03$ \\

    +LSTM+UPOS &
    $87.81$ &
    $\underline{\textbf{87.26}}$ &
    $85.30$ &
    $86.82$ &
    $86.73$ &
    $75.83$ &
    $85.75$ &
    $91.99$ &
    $86.78$ \\

    +LSTM+UPOS+f. &
    $\underline{\textbf{87.82}}$ &
    $87.21$ &
    $\underline{\textbf{86.12}}$ &
    $\underline{\textbf{87.64}}$ &
    $\underline{\textbf{87.97}}$ &
    $\underline{\textbf{78.62}}$ &
    $86.19$ &
    $\underline{\textbf{92.42}}$ &
    $\underline{\textbf{87.19}}$ \\

    +LSTM+feats &
    $87.52$ &
    $85.97$ &
    $84.31$ &
    $85.23$ &
    $86.09$ &
    $78.10$ &
    $\underline{\textbf{86.26}}$ &
    $90.67$ &
    $84.90$ \\
    \midrule

    +mBERT &
    $91.72$ &
    $91.43$ &
    $88.02$ &
    $89.99$ &
    $87.87$ &
    $79.79$ &
    $90.32$ &
    $93.66$ &
    $90.16$ \\

    +mBERT+UPOS &
    $\textbf{92.17}$ &
    $\textbf{91.64}$ &
    $89.26$ &
    $90.53$ &
    $89.11$ &
    $80.48$ &
    $90.53$ &
    $94.70$ &
    $91.21$ \\
    {+mBERT+UPOS+f.} &
    $91.97$ &
    $91.56$ &
    $\underline{\textbf{89.51}}$ &
    $\underline{\textbf{91.09}}$ &
    $\underline{\textbf{89.63}}$ &
    $\underline{\textbf{81.64}}$ &
    $\textbf{90.65}$ &
    $\underline{\textbf{94.95}}$ &
    $\underline{\textbf{91.46}}$ \\
    {+mBERT+feats} &
    $91.72$ &
    $91.35$ &
    $88.32$ &
    $90.33$ &
    $88.92$ &
    $81.61$ &
    $90.64$ &
    $94.42$ &
    $90.64$ \\
    \bottomrule
    \end{tabular}
\end{table}

\setlength{\tabcolsep}{3pt}
\begin{table}[htb]
    \centering
    \caption{LAS scores for different models on the DP task in different languages. The upper part of the table shows baseline parser with enhancements using LSTM-based contextual embeddings; the lower part shows baseline parser with enhancements using BERT embeddings. The best scores for each language are marked with the bold typeface separately for LSTM and BERT embedding extensions. Statistically significant differences in best scores at $p=0.01$ level are underlined.}
    \label{tab:results-dependency-parsing-las}
    \begin{tabular}{lccccccccc}
    \toprule
    Model & CRO & ENG & EST & FIN & LAT & LIT & RUS & SLO & SWE \\
    \midrule
    baseline &
    $77.55$ &
    $79.28$ &
    $73.17$ &
    $77.09$ &
    $75.77$ &
    $62.23$ &
    $77.22$ &
    $82.12$ &
    $75.06$ \\
    \midrule

    +LSTM &
    $79.59$ &
    $80.69$ &
    $76.24$ &
    $78.60$ &
    $78.08$ &
    $63.65$ &
    $79.24$ &
    $83.03$ &
    $78.97$ \\

    +LSTM+UPOS &
    $81.95$ &
    $\underline{\textbf{84.68}}$ &
    $81.36$ &
    $83.06$ &
    $82.75$ &
    $70.45$ &
    $80.88$ &
    $88.75$ &
    $82.65$ \\

    +LSTM+UPOS+f. &
    $\underline{\textbf{82.84}}$ &
    $84.48$ &
    $\underline{\textbf{83.46}}$ &
    $\underline{\textbf{83.97}}$ &
    $\underline{\textbf{84.27}}$ &
    $\underline{\textbf{74.07}}$ &
    $\underline{\textbf{81.68}}$ &
    $\underline{\textbf{90.32}}$ &
    $\underline{\textbf{83.17}}$ \\

    +LSTM+feats &
    $82.00$ &
    $82.96$ &
    $81.00$ &
    $81.16$ &
    $81.81$ &
    $72.40$ &
    $81.64$ &
    $88.38$ &
    $80.31$ \\
    \midrule

    +mBERT &
    $86.37$ &
    $88.09$ &
    $84.30$ &
    $86.07$ &
    $83.02$ &
    $72.62$ &
    $85.99$ &
    $91.67$ &
    $86.67$ \\

    +mBERT+UPOS &
    $\underline{\textbf{87.63}}$  &
    $\underline{\textbf{89.35}}$ &
    $86.72$ &
    $87.32$ &
    $85.43$ &
    $74.99$ &
    $86.24$ &
    $92.90$ &
    $87.80$ \\

    +mBERT+UPOS+f. &
    $87.44$ &
    $89.13$ &
    $\underline{\textbf{87.19}}$ &
    $\underline{\textbf{87.99}}$ &
    $\underline{\textbf{86.16}}$ &
    $\underline{\textbf{77.04}}$ &
    $\textbf{86.56}$ &
    $\underline{\textbf{93.40}}$ &
    $\underline{\textbf{88.38}}$ \\

    +mBERT+feats &
    $86.81$ &
    $88.81$ &
    $85.72$ &
    $87.07$ &
    $85.19$ &
    $76.33$ &
    $86.37$ &
    $92.74$ &
    $87.25$ \\
    \bottomrule
    \end{tabular}
\end{table}

\subsubsection{Results for the comment filtering task}
Table \ref{tab:comment-filtering-results} shows the NER results for LSTM and BERT baselines and their enhancements, described in Section~\ref{sec:comment-filtering-models}.
All BERT-based models outperform the LSTM-based models by a large margin for both languages, Croatian and English.

Adding POS tags and universal features to neural architectures of either type does not increase the classification accuracy and may even decrease it in some cases. As the accuracy does not increase with the best set of hyperparameters, we do not further discuss the effect of different pooling types on the performance, though we note that none of the pooling approaches consistently performed best.

\begin{table}[htb]
    \centering
    \caption{Classification accuracies for baseline and enhanced models on the NER task in different languages. The upper part of the table shows LSTM models, and the lower part shows BERT models. The best scores for each language are marked with the bold typeface separately for LSTM and BERT models. We report the mean and standard deviation over five random splits of the data. The differences between the models of the same type and language are marginal.
    }
    \label{tab:comment-filtering-results}
    \begin{tabular}{lcc}
    \toprule
            &   CRO             &   ENG             \\
    Model   &  Hate speech      & Toxic comments    \\ \midrule

    fastText + LSTM &
    $\mathbf{77.69 \pm 0.47 \% }$ &
    $88.14 \pm 0.12 \%$ \\

    fastText + LSTM + uPOS &
    $76.63 \pm 0.77 \%$ &
    $88.16 \pm 0.15 \%$ \\

    fastText + LSTM + uPOS + feats &
    $75.64 \pm 1.21 \%$ &
    $86.90 \pm 1.20 \%$ \\

    fastText + LSTM + feats &
    $76.52 \pm 0.99\%$ &
    $\mathbf{88.25 \pm 0.27\% }$ \\
    \midrule

    mBERT &
    $86.20 \pm 0.20 \%$ &
    $92.23 \pm 0.12 \%$ \\

    mBERT + uPOS &
    $85.05 \pm 0.49 \%$ &
    $\mathbf{92.33 \pm 0.11 \% }$ \\

    mBERT + uPOS + feats &
    $\mathbf{86.34 \pm 0.28 \% }$ &
    $92.29 \pm 0.20 \%$ \\

    mBERT + feats &
    $86.22 \pm 0.35\%$ &
    $92.20 \pm 0.13\%$ \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Ablation study}
To further analyze different aspects of the proposed morphological enhancements, we conducted several ablation studies on the DP task where the datasets and evaluation settings allow many experiments. Similarly as in \cref{sec:dp-results}, we statistically evaluate differences in performance between the baseline model and the best enhancement using the Z-test for the equality of two proportions. In cases where the null hypothesis can be rejected at $p=0.01$ level, we underline the respective compared score. We test the impact of additional training time, quality of morphological information, and different variants of BERT models.

\subsubsection{Additional training time}
To test if the observed differences in performance are due to random variation in the training of models which could be reduced with longer training time, we increase the maximum training time from $10$ to $15$ epochs.
From the studied languages, we arbitrarily choose a subset of $6$ languages on which we experiment. The results are shown in Table \ref{tab:dp-longer-training}.

\setlength{\tabcolsep}{3pt}
\begin{table}[h!tb]
    \centering
    \caption{UAS/LAS scores achieved by models that are trained for up to $5$ additional epochs (a maximum training time of $15$ epochs instead of $10$). The results for $10$ epochs are presented in Tables \ref{tab:results-dependency-parsing-uas} and \ref{tab:results-dependency-parsing-las}.}
    \label{tab:dp-longer-training}
    \begin{tabular}{lcccccc}
    \toprule
    Model & CRO & ENG & EST & FIN & SLO & RUS \\
    \midrule
    baseline+LSTM &
    $86.22/79.49$ &
    $85.40/81.47$ &
    $82.28/76.72$ &
    $85.30/80.28$ &
    $88.31/84.72$ &
    $85.38/79.79$ \\
    baseline+LSTM+UPOS &
    $\underline{\textbf{88.43}}/82.72$ &
    $87.30/84.74$ &
    $85.35/81.52$ &
    $87.27/83.77$ &
    $92.11/89.19$ &
    $86.46/81.35$ \\
    baseline+LSTM+UPOS+f. &
    $88.26/\underline{\textbf{83.26}}$ &
    $\underline{\textbf{87.92}}/\underline{\textbf{85.26}}$ &
    $\underline{\textbf{86.40}}/\textbf{83.65}$ &
    $\underline{\textbf{87.83}}/\underline{\textbf{84.22}}$ &
    $\underline{\textbf{92.64}}/\underline{\textbf{90.64}}$ &
    $86.68/82.18$ \\
    baseline+LSTM+feats &
    $88.08/82.74$ &
    $86.61/83.58$ &
    $84.50/81.24$ &
    $85.71/81.95$ &
    $90.73/88.60$ &
    $\underline{\textbf{86.75}}/\underline{\textbf{82.21}}$ \\
    \midrule
    baseline+mBERT &
    $92.06/87.03$ &
    $91.10/87.77$ &
    $88.06/84.60$ &
    $90.58/87.06$ &
    $93.97/92.11$ &
    $90.26/86.15$ \\
    baseline+mBERT+UPOS &
    $92.03/87.55$ &
    $\underline{\textbf{91.77}}/\underline{\textbf{89.56}}$ &
    $89.35/86.60$ &
    $91.16/87.98$ &
    $94.54/92.80$ &
    $91.05/87.04$ \\
    baseline+mBERT+UPOS+f. &
    $\textbf{92.09}/\textbf{87.71}$ &
    $91.74/89.49$ &
    $\underline{\textbf{89.49}}/\underline{\textbf{87.26}}$ &
    $\underline{\textbf{91.32}}/\underline{\textbf{88.42}}$ &
    $\underline{\textbf{95.16}}/\underline{\textbf{93.78}}$ &
    $\textbf{91.11}/\textbf{87.23}$ \\
    baseline+mBERT+feats &
    $91.93/86.97$ &
    $91.33/88.89$ &
    $88.37/85.78$ &
    $90.85/87.85$ &
    $94.49/93.08$ &
    $90.79/86.64$ \\
    \bottomrule
    \end{tabular}
\end{table}
We can observe that longer training times slightly increases the scores for all model variants, though their relative order stays the same. The models with added morphological features still achieve better results, so the performance increases do not seem to be the effect of random fluctuations in training due to the different amount of training steps.

\subsubsection{Quality of morphological information}
In the second ablation study, we evaluate the impact of the quality of morphological information. We replace the high quality (human-annotated) POS tags and morphological features used in \Cref{sec:depparse-models} with those predicted by machine learning models. In this way, we test a realistic setting where the morphological information is at least to a certain degree noisy.
We obtain POS tags and morphological features from Stanza models prepared for the involved languages \citep{qi2020stanza}. To avoid overly optimistic results, we make sure to use models that are not trained on the same datasets used in our DP experiments. This is possible for a subset of five languages.
Table \ref{tab:dp-predicted-features} shows the results of DP models, trained with predicted morphological features. The first two lines of the table show the quality of used features, expressed as the proportion of tokens, for which POS tags and \textit{all} morphological features are correctly predicted, i.e. we report their accuracy.

The general trend is that using predicted features results in much smaller (best case) performance increases, though some languages still see significant increases.
For LSTM models, the increases range from $0.42\%$ (English) to $1.60\%$ (Slovene) for UAS, and from $0.67\%$ (English) to $2.41\%$ (Finnish) for LAS.
For BERT models, the increases are marginal and range from $0.00\%$ (English) to $0.49\%$ (Finnish) for UAS, and from $-0.10\%$ (i.e. decrease, Slovene) to $0.66\%$ (Finnish) for LAS.
These results are consistent with the results of our NER experiments in \cref{sec:ner-results}, where we have no access to human-annotated features and find that noisy features only help LSTM-based models.

As noisy features might require different training, we also tried to increase the maximal number of training steps to $15$ epochs and repeated the tuning of hyperparameters for one language (Estonian). None of these changes improved the scores by a practically relevant margin.

These results indicate that adding predicted morphological features to models with BERT embeddings might not be practically useful since their quality needs to be very high.
However, since human annotated morphological features improve the performance on the DP task, this suggests that there is a room for improvement in BERT pre-training. It seems that pre-training tasks of BERT (masked language modelling and next sentence prediction) do not fully capture the morphological information present in the language.

\setlength{\tabcolsep}{3pt}
\begin{table}[htb]
    \centering
    \caption{UAS/LAS scores achieved by models that are trained with predicted (noisy) instead of human-annotated morphological features. In the first two lines, we report the proportion of tokens for which UPOS tags and UPOS features are correctly predicted.}
    \label{tab:dp-predicted-features}
    \begin{tabular}{lccccc}
    \toprule
    & \textbf{ENG} & \textbf{EST} & \textbf{FIN} & \textbf{SLO} & \textbf{RUS} \\
    UPOS accuracy (\%): & $92.45$ & $91.15$ & $87.59$ & $80.45$ & $89.32$ \\
    feats accuracy (\%): & $93.92$ & $88.86$ & $86.20$ & $78.74$ & $85.22$ \\
    \midrule
    baseline+LSTM &
    $84.87/80.69$ &
    $81.92/76.24$ &
    $84.13/78.60$ &
    $86.92/83.03$ &
    $85.02/79.24$ \\
    baseline+LSTM+UPOS &
    $85.18/\textbf{81.36}$ &
    $\textbf{82.44}/77.15$ &
    $85.33/80.61$ &
    $88.14/84.06$ &
    $84.90/79.53$ \\
    baseline+LSTM+UPOS+f. &
    $85.04/81.02$ &
    $82.24/\underline{\textbf{77.39}}$ &
    $\underline{\textbf{85.56}}/\underline{\textbf{81.01}}$ &
    $\underline{\textbf{88.52}}/\underline{\textbf{85.00}}$ &
    $\textbf{85.82}/\underline{\textbf{80.89}}$ \\
    baseline+LSTM+feats &
    $\textbf{85.29}/81.22$ &
    $81.42/76.19$ &
    $84.89/80.43$ &
    $88.12/84.12$ &
    $84.74/79.56$ \\
    \midrule
    baseline+mBERT &
    $\textbf{91.43}/88.09$ &
    $88.02/84.30$ &
    $89.99/86.07$ &
    $93.66/\textbf{91.67}$ &
    $90.32/85.99$ \\
    baseline+mBERT+UPOS &
    $\textbf{91.43}/\textbf{88.33}$ &
    $\textbf{88.20}/\textbf{84.69}$ &
    $90.05/86.33$ &
    $93.38/91.42$ &
    $\textbf{90.59}/\textbf{86.17}$ \\
    baseline+mBERT+UPOS+f. &
    $91.08/88.05$ &
    $88.06/84.39$ &
    $\textbf{90.48}/\textbf{86.73}$ &
    $\textbf{93.73}/91.57$ &
    $90.34/85.74$ \\
    baseline+mBERT+feats &
    $90.90/87.74$ &
    $87.88/84.42$ &
    $89.77/85.91$ &
    $93.37/91.23$ &
    $89.65/85.27$ \\
    \bottomrule
    \end{tabular}
\end{table}

\subsubsection{Specific BERT models}

In the third ablation study, we replace the embeddings obtained from the multilingual uncased BERT model with those obtained from more specific multilingual BERT models and monolingual BERT models.
In experiments involving multilingual BERT models, we use Croatian/Slovene/English CroSloEngual BERT and Finnish/Estonian/English FinEst BERT~\citep{ulcar2020finest}. In experiments with monolingual BERT models, we use English bert-base-uncased~\citep{devlin2019-bert}, Finnish bert-base-finnish-cased-v1~\citep{virtanen2019multilingual}, and Russian RuBert~\citep{kuratov2019-rubert}.
We only perform the experiments for a subset of studied languages for which we were able to find more specific BERT models.
The aim is to check if the additional morphological features improve the performance of more language-specific BERT models.
These are trained on a lower number of languages, and larger amounts of texts in the involved languages compared to the original multilingual BERT model \cite{devlin2019-bert}. Due to this language-specific training, we expect these BERT models to capture the nuances of the languages better, thus possibly benefiting less from the additional morphological features.
The results are shown in Table \ref{tab:dp-more-specific-bert-1} for trilingual BERT models and in Table \ref{tab:dp-more-specific-bert-2} for monolingual BERT models.

In most cases, the specific multilingual BERT models without additional features do as well as or better than the best performing original multilingual BERT model with additional features. The only worse LAS scores are achieved on English and Estonian, indicating that trilingual BERT models are better adapted to the task than the original multilingual models.
The addition of morphological features increases the UAS and LAS even further.
For UAS, the improvements are generally smaller than before (see \Cref{fig:dependency_parsing}), with the only improvement larger than $1\%$ being on Estonian.
For LAS, the improvements are larger compared to UAS and range from $0.54\%$ (Slovene) to $2.23\%$ (Estonian);  only one language sees an improvement of less than $1\%$.
These results indicate that the additional morphological features still contain valuable information for the DP task, which the BERT models do not capture.
We hypothesise that slightly lower performance of FinEst BERT (without additions) on Estonian as well as slightly larger increase when adding morphological features might be due to variability in the training process.

For monolingual models (the most language-specific), the results are mixed, but the general trend is that the additional features help little or not at all.
The English monolingual model without and with morphological additions performs comparably to its multilingual (CroSloEngual) counterpart, and we do not see any additional increase in the performance.
The Russian monolingual model achieves comparable UAS and LAS to the best performing original multilingual BERT model with additional features.
With the addition of morphological features, the scores increase, but only marginally (under $0.5\%$).
The Finnish monolingual model outperforms the best performing multilingual (FinEst) counterpart.
The addition of morphological features to the monolingual Finnish model increases the scores further: UAS by $0.33\%$ and LAS by $0.68\%$, but these increases are not statistically significant.

\setlength{\tabcolsep}{3pt}
\begin{table}[htb]
    \centering
    \caption{UAS/LAS scores achieved by trilingual BERT models, each trained on a smaller set of languages (3) than the original multilingual BERT model (104).}
    \label{tab:dp-more-specific-bert-1}
    \begin{tabular}{lccccc}
    \toprule
    BERT variant & \multicolumn{3}{c}{CroSloEngual BERT} & \multicolumn{2}{c}{FinEst BERT} \\
    Model/Language & CRO & ENG & SLO & FIN & EST \\
    \midrule
    {baseline+BERT} &
    $92.63/87.84$ &
    $91.69/88.37$ &
    $95.48/93.98$ &
    $92.61/89.35$ &
    $89.84/86.51$ \\
    {baseline+BERT+UPOS} &
    $\underline{\textbf{93.29}}/\underline{\textbf{89.01}}$ &
    $\textbf{92.05}/\underline{\textbf{89.69}}$ &
    $95.72/94.27$ &
    $\underline{\textbf{93.39}}/\underline{\textbf{90.66}}$ &
    $\underline{\textbf{91.06}}/88.67$ \\
    {baseline+BERT+UPOS+f.} &
    $92.95/88.33$ &
    $91.99/89.65$ &
    $\textbf{95.82}/\textbf{94.52}$ &
    $93.23/90.61$ &
    $91.01/\underline{\textbf{88.74}}$ \\
    {baseline+BERT+feats} &
    $92.87/87.98$ & $91.27/88.58$ & $95.33/94.05$ & $93.12/90.54$ & $89.94/87.44$ \\
    \bottomrule
    \end{tabular}
\end{table}

\setlength{\tabcolsep}{3pt}
\begin{table}[htb]
    \centering
    \caption{UAS/LAS achieved by monolingual BERT models.}
    \label{tab:dp-more-specific-bert-2}
    \begin{tabular}{lccc}
    \toprule
    BERT variant & bert-base-uncased & bert-base-finnish-cased-v1 & RuBert \\
    Model & ENG & FIN & RUS \\
    \midrule
    baseline + BERT &
    $91.82/88.74$ & $94.20/91.51$ & $90.83/86.39$ \\
    baseline + BERT + UPOS &
    $\textbf{91.92}/89.60$ & $\textbf{94.53}/\textbf{92.19}$ & $\textbf{91.24}/\textbf{87.31}$ \\
    baseline + BERT + UPOS + f. &
    $91.81/\underline{\textbf{89.62}}$ & $94.08/91.73$ & $91.04/87.04$ \\
    baseline + BERT + feats &
    $91.23/88.76$ & $94.09/91.80$ & $90.88/86.61$ \\
    \toprule
    \end{tabular}
\end{table}

\section{Conclusion}
\label{sec:conclusion}

We analysed adding explicit morphological information in the form of embeddings for POS tags and morphological features to two currently dominant neural network architectures used in NLP: LSTM networks and transformer-based BERT models. We compared models enhanced with morphological information with baselines on three tasks (NER, DP, and NER). To obtain general conclusions, we used subsets of eight morphologically-rich languages from different language families.

The results indicate that adding morphological information to NER prediction models is not beneficial, but it improves the performance in the NER and DP tasks. For the DP task, the improvement depends on the quality of the morphological features. The additional morphological features consistently benefited LSTM-based models for NER and DP, both when they were of high quality and predicted (noisy).
For BERT-based models, the \emph{predicted} features do not make any practical difference for the NER and DP task but improve the performance in the DP task when they are of \emph{high quality}. Testing different variants of BERT shows that language specialised variants improve the performance on the DP task and the additional morphological information is beneficial, though less and less as we shift from multilingual towards monolingual models.

The comparison of different BERT variants indicates that BERT models do not completely capture the language morphology.  Since the release of BERT, several new pre-training objectives have been proposed, such as syntactic and semantic phrase masking~\citep{zhou2020limitbert} and span masking~\citep{joshi-spanbert-2020}.
In further work, it makes sense to apply these models to the DP task in order to test how well they capture the morphology. Further, the effect of morphological features could be analysed on additional tasks and languages, since the explicit morphological information does not seem to benefit them equally.

 \subsection*{Acknowledgements}
This paper is supported by European Union's Horizon 2020 Programme project EMBEDDIA (Cross-Lingual Embeddings for Less-Represented Languages in European News Media, grant no. 825153). The research was supported by the Slovene Research Agency through research core funding no. P6-0411. The Titan X Pascal used for a part of this research was donated by the NVIDIA Corporation.

\bibliographystyle{plainnat}

redpajama_set_name: RedPajamaArXiv


Image Title: Heated Towel Rack 4 Bar Canadian Approved Inside Plans 10. Post Title: Heated Towel Rack. Filename: heated-towel-rack-4-bar-canadian-approved-inside-plans-10.jpg. Image Dimension: 500 x 600 pixels. Images Format: jpg/jpeg. Publisher/Author: Triston Daugherty. Uploaded Date: Wednesday - February 13th. 2019 04:02:15 AM. Category: Architecture. Image Source: archdaily.com. IBathUK 1000 X 500 Thermostatic Electric Heated Towel Rail Pertaining To Rack Plan 5. Hardwired Heated Towel Rack Regarding Design 9. How To Choose The Right Heated Towel Warmer Valves Hudson Reed Throughout Rack Design 15. A Buyer S Guide To Heated Towel Racks With Rack Ideas 3. Why And How To Choose The Right Towel Warmer Regarding Heated Rack Ideas 13. Heated Towel Rack Amazon Com In Idea 4. A Buyer S Guide To Heated Towel Racks In Rack Decor 7. Designer Wall Mounted Heated Towel Rails UK Drench With Regard To Rack Plans 17. Heated Towel Rails Radiators Free UK Delivery Inside Rack Prepare 6. Cool Bathrooms With Toasty Towel Warmers WSJ Inside Heated Rack Inspirations 19.


text:
Q: Django with Heroku, No module named django core management I have my Django app pushed to heroku successfully, however when I try to open a shell such as heroku run python manage.py shell I get django-no-module-named-django-core-management error. My app also does not render properly when I visit the URL. I've checked all the settings and can't figure out I'm getting this error. My requirements.txt file is setup correctly with all the dependencies (including Django) from my virtual env.

A: you need to activate your virtualenv
. bin/activate

only then your code has access to all modules inside virtualenv

A: I think you could check your requirements.txt file
psycopg2==2.5.3

redpajama_set_name: RedPajamaStackExchange