text:
La cocina polaca es una mezcla de gastronomías de origen eslavo, alemán, así como tradiciones culinarias propias de la zona. Está muy relacionada con las gastronomías eslavas en el uso de la avena y otros cereales, pero ha sido influenciada por las gastronomías turca, alemana, húngara, judía, armenia, francesa o las cocinas coloniales del pasado.

Ingredientes
La gastronomía polaca emplea una gran variedad de ingredientes, y sus platos son ricos en carne y especias, así como en pasta. Algunos de los platos más notables son los pierogi. El tipo de queso local Bryndza Podhalańska se encuentra protegido oficialmente por el sistema de denominación de origen.

Alimentación cotidiana
Una comida típica se compone de tres platos, se suele empezar con una sopa, como el barszcz (sopa de remolacha), el żurek (sopa de harina de sabor ácido) o la sopa de tomate, seguido quizás de un aperitivo de salmón o arenque (preparado en crema, aceite o vinagre). Otros aperitivos populares son elaborados con diferentes carnes, vegetales o pescado en aspic (gelatina). Para el plato principal se puede probar el plato nacional que se denomina bigos (chucrut o col fermentada ácida con trozos de carne y salchichas) o schabowy (chuleta de cerdo rebozada). Finalizando con un helado, o un makowiec, que es una especie de pastel casero de semillas de amapola, o el napoleonka un tipo de pastel. Otras especialidades polacas incluyen el chłodnik (una especie de sopa fría para los días calurosos), golonka (codillo de cerdo), gulasz (trozos de carne hervidos con verduras), kołduny (especie de empanadilla hervida rellena de carne que acompaña sopas), pulpety (albódigas de carne), los zrazy (rodajas de ternera), el salceson (una especie de salchicón ahumado) y flaki (tripas).

Muchos de los platos contienen o están elaborados con queso quark o twaróg (especie de requesón ácido).

Historia

Edad Media
Durante la Edad Media la cocina de Polonia fue muy fuertemente condimentada con especias. En aquella época dos de los principales ingredientes eran la carne (ternera) y el cereal. El territorio de la actual Polonia estaba densamente poblado de árboles, y es por esta razón por la que se empleaba con mucha frecuencia en la cocina de aquella época setas, frutas del bosque, nueces y miel. Gracias a la cercanía con los países del Este el precio de algunas especias tales como el junípero, pimienta negra y la nuez moscada tenían precios más bajos que en el resto de Europa, y esta fue la razón que se hiciera popular el uso de especias en algunos platos de la cocina Polaca. Otro propósito fue la necesidad de neutralizar los malos olores de los alimentos no tan perfectamente conservados. El uso de dos salsas básicas: el jucha czerwona y el jucha szara (o tal y como se menciona en la Polonia actual la sangre blanca y roja) ha permanecido empleada hasta casi el

La bebida más popular es la cerveza, incluida la muy poco fermentada podpiwek, y la aguamiel. En el  las clases altas empezaron a importar vinos procedentes de Hungría y Silesia. Algunos aguardientes destilados como el vodka, que empezaban a ser populares en Europa ya eran bebidas comunes en las clases bajas.

Renacimiento
Con la ascensión al trono de la reina Bona Sforza, la segunda mujer de Sigismundo I de Polonia en 1518, innumerables platos de la cocina italiana y la francesa fueron traídos a Polonia. De esta simple manera algunas de las verduras clásicamente empleadas en estas gastronomías empezaron a verse cada vez más en la cocina polaca, como por ejemplo la lechuga, el puerro, apio y la col. Incluso hoy en día, la denominación de estos vegetales como los puerros, las zanahorias y los apios se conocen en  polaco como wloszczyzna, haciendo referencia a Wlochy que es el nombre polaco de Italia.

La República
Antes de las particiones, Polonia era uno de los países más grandes del mundo, influyendo sobre algunas de las cocinas y costumbres culinarias de los países vecinos. Entre las más importantes están cocina lituana, turca y la cocina húngara. Con la subsiguiente decadencia de Polonia, y la crisis de producción de granos, las patatas empezaron a ser más populares que los cereales. Tras las numerosas guerras con el Imperio otomano, el café se hizo  popular.

Particiones
Tras las particiones la cocina polaca fue influida por las gastronomías de los imperios circundantes. En especial las cocinas  rusas y alemanas, y además de las naciones del imperio Austro-Húngaro. En el territorio ocupado por los rusos, el té desplazó al para entonces popular café. Bajo la influencia alemana se hizo popular la elaboración de  weißwurst que fue adoptada en la Gran Polonia. Quizás la tradición culinaria más influenciada de este imperio multinacional de Austria y Hungría, llegando a liderar el desarrollo de la cocina de Europa central, ocurrió en Galicia.

Tras la segunda guerra mundial
Tras la segunda guerra mundial, Polonia cayó bajo la ocupación comunista. Los restaurantes fueron los primeros entes  nacionalizados y muchos de ellos se vieron afectados por cierres realizados por las autoridades. En lugar de estos restaurantes los comunistas abrieron una red de diferentes casas de comidas para los trabajadores, así como Bares de leche. Los pocos restaurantes que sobrevivieron a los años 1940 y 1950 fueron estatales y sus precios eran demasiado altos para la gente de la calle. Las casas de comidas promovían las sopas de todos los tipos. Por aquella época era muy importante la carne con patatas servidas como segundo plato, se denominaba kotlet schabowy y es muy similar al plato de la cocina austriaca denominado Wiener schnitzel.

Por este tiempo los ingredientes básicos y diarios tales como la carne, los huevos, el café y el té dejaron de ser comercializados y pasaron a ser objetos de lujo. Esta situación favoreció que la cocina tradicional polaca fuera desapareciendo. Entre los platos populares introducidos en aquella época en los restaurantes están los huevos con chuletas y que era una especie de hamburguesa hecha de carne picada, huevo y harina.

Tiempos modernos
Con el final del comunismo en el año 1989 los restaurantes comenzaron a abrir de nuevo y aparecieron los platos de fast food fáciles y baratos. De alguna forma se restableció también los platos de cocina tradicional.

Platos Nacionales Famosos

Sopas

 barszcz - Sopa de remolacha, común entre diferentes pueblos eslavos
 chłodnik - Una versión fría del barszcz, preparada con leche ácida, remolacha, hojas de remolacha, rábanos, pepino y eneldo fresco
 czernina - sopa de sangre de pato
 flaki - tripas de ternera o cerdo con maryorana
 rosół z kury - sopa de pollo
 zupa grzybowa - sopa de setas elaborada con diferentes tipos
 zupa ogórkowa - sopa ácida, salteada con pepinos, a menudo cerdo (denominada a veces como "sopa de eneldo")
 zupa szczawiowa (Sopa de Acedera)
 żur - sópa ácida de centeno con Weisswurst y huevo duro
 żurek - sopa densa fundamentada en rosół z kury a la que se le añade nata y generalmente patatas, carne, huevos, zanahorias (todo ello de cantidad variable e improvisada).
 zupa owocowa (Sopa de Frutas)

Platos Principales

 pierogi - empanadas generalmente rellenas con sauerkraut y/o setas, carne, patatas y/o queso curado, requesón con un ligero toque de vainilla, o arándanos—opcionalmente se sirven con salsa ácida, y algo de azúcar en las versiones más dulces
 bigos - una estofado a base de col fermentada , setas secas, ciruelas pasas y carne -principalmente kielbasa-, se puede elaborar también con col no fermentada
 kotlet schabowy - una chuleta de cerdo, similar al Austriaco Wiener Schnitzel pero más delgado
 kasza gryczana ze skwarkami - cereales: alforfón frito, y cebollas
 kaczka z jabłkami - pato asado con manzanas
 sztuka mięsa - un plato similar al francés ragout
 Golonka - Codillo de cerdo, asado al horno, al estilo de Baviera. Presentado sobre una cama de col fermentada y con un contorno de verduras mixtas.
 gulasz - gulash
 gołąbki - Golumpki, hojas de col blanca cocinadas con carne, salsa de tomate y arroz
 Placki kartoflane/ziemniaczane - potato pancake Tortitas de patatas, bañadas con salsa de tomate y queso rallado. Con un contorno de pepinillo fermentado y verduras con crema ácida.
 pyzy - pastas de patata servidas con carne o queso
 naleśniki - similar a los crepes, y rellenado muy similar a los "pierogi", a veces algo curados, pero se pueden tomar con requesón y/o fruta, y opcionalmente coronado con crema y azúcar.

Condimento
 Ćwikła - preparado o relish a base de remolacha y rábano picante rallados se utiliza para acompañar carnes asadas o ahumadas

Notas y referencias

Véase también
 Historia de Polonia
 Lucyna Ćwierczakiewiczowa
 Myśliwska
 Bar mleczny
 Anexo:Gastronomías del mundo

Enlaces externos

 Un extranjero en Polonia
 Breve sumario de la cocina Polaca
 Tienda popular con productos polacos

Cultura de Polonia
Gastronomía de Polonia

redpajama_set_name: RedPajamaWikipedia


text:
DA & ROCKLAND SHERIFF'S CRIME REPORT 2/8/13
By dskriloff on February 8, 2013 Comments
DA REPORT:
ROBBERY & ATTEMPTED ROBBERY
Tony Hunt, age 43, residing at 21 Martin Court, Hartford, CT, was indicted on January 30, 2013 by a Rockland County Grand Jury for the crimes of robbery in the first and second degree and attempted robbery in the first degree. District Attorney Thomas P. Zugibe stated that on or about November 28, 2012, shortly after midnight, the defendant approached two victims and displayed what appeared to be a pistol. The defendant stole approximately $1,500 and ran away. At approximately 1 p.m. the same day, the defendant went to the Latino Food Mart in Spring Valley. He entered the store, displayed what appeared to be a pistol to the clerk, and stole approximately $1,000 in cash from the cash register.
CRIMINAL POSSESSION OF WEAPON & MARIJUANA
Laquan Phillips, age 19, residing at 70 Main Street, Haverstraw, NY, was indicted on January 30, 2013 by a Rockland County Grand Jury for the crimes of criminal possession of a weapon in the second degree and unlawful possession of marijuana. District Attorney Thomas P. Zugibe stated that on or about January 6, 2013 at approximately 1:18 a.m., at 27 Van Orden Avenue in the Village of Spring Valley, the defendant was found to be in possession of a loaded .38 caliber handgun, which was located in his waistband. The defendant was also in possession of a plastic container, which held numerous small bags of marijuana.
DRIVING WHILE ABILITY IMPAIRED BY DRUGS & AUO
Joseph Scannello, age 59, residing at 935 Flint Court, Valley Cottage, NY, the defendant was indicted on January 30, 2013 by a Rockland County Grand Jury for the crimes of driving while ability impaired by drugs as a felony and aggravated unlicensed operation in the third degree. District Attorney Thomas P. Zugibe stated that on or about October 4, 2012 at approximately 2:50 a.m., the defendant was stopped for various vehicle and traffic violations. The officer observed the defendant exhibit signs of impairment and fail several field sobriety tests. The defendant had a prior DWI in Monroe Town Court in 2006.
Edwin Shockley, age 53, residing at 32 Prospect Street, Spring Valley, NY, was sentenced February 5, 2013, before Rockland County Court Judge William K. Nelson to a state prison term of two to four years after pleading guilty to burglary in the third degree. District Attorney, Thomas P. Zugibe stated that on or about January 2, 2012, the defendant broke into the office at Golden Acres Adult Home and stole $2,000. The defendant again broke into the Adult Home on April 20, 2012 with the intention of stealing money but was arrested on the premises.
GRAND LARCENY & CRIMINAL POSSESSION OF STOLEN PROPERTY
Luisa Torres, age 21, residing at 4014 Vernon Blvd, Queensbridge, NY, was indicted on January 30, 2013 by a Rockland County Grand Jury for the crimes of grand larceny in the fourth degree and criminal possession of stolen property in the fourth degree. District Attorney Thomas P. Zugibe stated that on or about August 25, 2012 at approximately 7:55 p.m., at the Macy's in the Palisades Center Mall, the defendant stole 67 items of assorted merchandise by concealing them in her bag. The Clarkstown Police were alerted and defendant was subsequently arrested.
ROCKLAND COUNTY SHERIFF'S OFFICE POLICE DIVISION:
BAIL JUMPING, ASSAULT, RECKLESS ENDANGERMENT & MORE
Eduardo Augustin, age 42, residing in Suffern, NY, was arrested on two warrants for bail jumping in the second degree, veh. assault in the second degree, assault in the second degree, reckless endangerment in the first degree and criminal mischief. The defendant was lodged at Rockland County Jail awaiting arraignrnent in County Court.
SUPPORT COLLECTION
Jean Bissainthe, age 38, residing in Nanuet, NY, was arrested on a warrant on an arrest warrant issued out of Rockland County Family Court on support proceedings. The dependant was arraigned in front of Judge Eisenpress and held on $5,000 bail at the Rockland County Jail with a future return court date at Rockland County Family Court. As of February 4, 2013, the defendant owes a private party $18,899.91.
Liz Eduardo, age 42, residing in Garnerville, was arrested on an outstanding family court warrant issued on January 18, 2013 for failure to pay child support. The defendant was brought before Judge Eisenpress to answer for arrears of $34,569.69 owed to a private petitioner. The defendant was released to return to Family Court on February 20, 2013.
LICENSE CRIME
Olegvych Gaponov, age 28, residing in Monsey, was arrested on a warrant for eng. in a bus without a license. The defendant was arraigned by the Hon Judge-Thorsen and ROR with a return court date of February 25, 2013 at 9:30 a.m.
DA & ROCKLAND SHERIFF'S CRIME REPORT 2/8/13 added by dskriloff on February 8, 2013

redpajama_set_name: RedPajamaCommonCrawl


text:
Thenmozhi Soundararajan
Thenmozhi Soundararajan is a Dalit American transmedia artist, technologist and activist who is the executive director of Equality Labs. Equality Labs is a South Asian American human rights arts organization working at the intersection of storytelling, design, technology, and digital security to end caste apartheid, Islamophobia, and religious intolerance.
Equality Labs work includes the Dalit History Month Project and the award-winning Digital Security Curriculum Digital Security For All. Her work as an artist has been recognized by the Robert Rauschenberg Artist as Activist Fellowship, the Producers Guild of America Diversity Program. She has also been named part of the inaugural cohort of Atlantic Fellows for Racial Equity. You can can find her on Twitter at @dalitdiva or about the work at Equality Labs @equalitylabs.
Canceling Kavanaugh Isn't the Only Justice Survivors Need
Darakshan Raja & Thenmozhi Soundararajan
We must move beyond solely focusing on legal and policy solutions to rape culture. We must link these calls for reform with solutions for survivors that aren't dependent upon carceral feminism.
Why Are Some South Asian Immigrants Offering to Pay for Trump's Wall?
Three Indian American immigrant groups have aligned themselves with Trump's white supremacist agenda around immigration. They are using political clout to seek advantages on visas and quotas at the expense of other immigrant groups.


Alright, don't pretend you haven't woke up hungover and your trashed house is like every spy movie in the 80's where they were looking for something they didn't find and you think... Beer.
With no modivation to start cleaning, you hop in the shower and the hot, steamy water never felt better. Cold beer with the hot water is like some kind of crazy yin and yang harmony.
Okay then, I guess you probably don't need this handy guide on where not to put your shower beer!


Sir John Chilcot told the BBC that Blair based his evidence to the inquiry on beliefs rather than facts.
The British Army used the Northern Irish Troubles as its key reference point for dealing with the Iraq conflict.
Blair reappears at the Chilcot Inquiry to explain why he ignored legal advice concerning the invasion of Iraq – and why he believed the "calculus of risk" has changed in a post-9/11 world.
New evidence has shown that Tony Blair was warned that the invasion of Iraq would be unlawful.


Raiwin Shuttle Service operate an airport shuttle service between Swakopmund and Walvis Bay to both the Swakopmund Airport and the International Walvis Bay Airport at Rooikop. We have been transporting passengers between these towns and airports since 1996. Customers will find that our airport shuttle service is safe, run in a relaxed atmosphere and always on schedule.
Raiwin airport shuttle service boasts an accident free shuttle and transfer service since 1996, when we started our service. If safety, punctuality, friendly atmosphere, reliable vehicles and a competent driver are important to you, than you should consider using us for the best airport shuttle, airport transfer, town to town or city to city shuttles.
Raiwin Airport shuttle and transfers is owner driven and operated, no compromise, no hazzles.
Raiwin Shuttle Service is located in Swakopmund, the holiday mecca of Namibia and popular with travellers around the globe. Swakopmund offers a lot of activities to while away your time, and when you need transport just call us, we will gladly oblige.


use queryRunnerFactory.mergeRunners() in groupBy v2.
the benefit. You might ned to run more benchmarks to figure out.
> GroupByMergingQueryRunnerV2 that has parallel combining threads enabled.
> > sorry, the previous email was sent by accident.
> > was working on improving groupBy v2 performance for a while.
> > >> sequences from sub-queries?
Next by Date: [GitHub] amitabak opened a new issue #6034: Druid's basic intro to sharding has in error / misleading?


Spoilers ahead for Episode 5 of Legends of Tomorrow Season 4, called "Tagumo Attacks!!!"
Legends of Tomorrow delivered a Darhk twist at the end of Season 3 when Damien Darhk died but Nora Darhk escaped the Time Bureau thanks to the gift of the time stone from Ray. In theory, Nora was to stay out of the limelight in Season 4 lest she be apprehended by the Time Bureau and locked up for her crimes when she was under her father's (and Mallus') influence. Ray brought her back into the action in "Tagumo Attacks!!!" when Constantine needed some magical assistance.
Given that Nora had been working as a waitress at a renaissance fair to avoid using her magic and therefore risk falling into darkness like her father, she was reluctant to engage in a kind of magic that usually requires the draining of a life force. Ray found a technological substitute for life force, but it was still dangerous for Nora. Ultimately, she decided to use her magic to save Constantine, then used the time stone to travel to the Time Bureau, where she was immediately arrested.
She is recruited to help Constantine, to help revive Constantine, and that's very challenging for her because she's been keeping away from magic, as that has been a challenging thing for her to control, her ability to control magic and to not be taken over by it. So I'm asking a lot of her to help John in this way. Ultimately she does and comes through it and learns that she's stronger than she thought, which is what Ray has been kind of hoping and feeling, but seeing it also is great for Ray. It brings them closer and will help her character in the future with the Legends. The fact that she has done this for one of the members of the Legends.
Nora may have made her decision for altruistic reasons, but that doesn't mean she won't benefit from it in the long run. Her decision to risk her own recovery by helping Constantine when he certainly would have died means that the Legends will have reason to give her the benefit of the doubt and eventually trust her. Admittedly, Constantine is still a reluctant member of the team who isn't falling all over himself to make it official. Still, Nora can likely count the Waverider crew as her allies moving forward.
Ray's faith in Nora almost certainly played a role in her feeling the strength to save Constantine without falling into darkness, and it's difficult to imagine her turning herself into the Time Bureau if not for how he helped her. Unfortunately for Ray, who has clearly (and endearingly) been smitten with Nora, turning herself into the Time Bureau means no more spur-of-the-moment adventures with the Legends.
She gives herself over willingly, knowing that she's going to pay for her crimes and hopefully continue to rehabilitate herself, which is a strong power move of taking responsibility and making amends for what she's done. Ray is very proud of her and also sad because he didn't want her to do that. He's breaking the rules. As he's a rule follower, he would have rather had her free and away so he could spend time with her and she could be okay, but there is some interaction, some fun interaction that happens in the next couple episodes with her being in Time Bureau prison. Ray finds an interesting way to get to her, to see her, visit her. It does involve shrinking, yes.
The course of true love never did run smooth, but that course can be altered when it comes to the Legends! Ray won't let a little thing like Nora's incarceration by the Time Bureau put a stop on their funny little relationship, no matter how it develops. Although Brandon Routh didn't drop any details about Ray's visits to Nora, we can apparently expect shrinking to play a part. Fortunately, Courtney Ford, who plays Nora and just so happens to be married to Brandon Routh, shared a bit more information.
I think it's really frightening for her, and it is something she struggles with because her magic is tied to her father and even though she had such incredible love for her father, that relationship was also dysfunctional and she received a lot of pain from it as well, and the world received a lot of pain from Damien Darhk, so right now, Nora's trying to figure out who she is without her father's influence, and without Mallus' influence... I mean, it's absolutely frightening, so she feels like, 'Okay, I want to see how I can be in the world, going straight, so to speak. Not using magic at all, trying to be a regular person, and it's frightening and it's confusing and she doesn't know who she is, and I think maybe all of us have felt that way at some point in our lives. What defines you? Who are you when you don't have something that is grounding you? It can feel very unsettling to not know your place in the world.
Using her magic to save somebody who hadn't been able to save her was a bold and brave move by Nora, and it bodes well for her continued development throughout the season. It's also a good sign that Nora seems aware of how unhealthy her relationship with her father had been. As long as she can distinguish between good and bad dynamics, there are reasons to hope that her recovery will be ongoing.
She finally does decide to use her magic, but in her mind she doesn't know. If I give myself over, am I gone now? If I use any magic at all, is it just what Constantine warns her about? Is it just down that road, or can you still be good? Can you still kind of be a complicated ally and use some magic? How much is okay? How much can she tolerate? It's also kind of addictive, that kind of power.
Tune in to The CW on Mondays at 9 p.m. ET to find out what's in store next for Nora, Ray, and the rest of the Legends on Legends of Tomorrow. The Legends won't be part of this year's big multi-show crossover, but they'll have an event of their own to keep the action going. For more viewing options in the not-too-distant future, check out our midseason TV premiere guide.


text:
\section{Introduction}
Artificial intelligence (AI) software, primarily based on machine learning models (especially deep learning), has become one of the leading research directions in the software engineering community~\cite{martinez2022software}.
Natural language processing (NLP) models are one of the core techniques in AI software, as abundant texts are available (e.g., Wikipedia) for training useful language models \cite{otter2020survey}.
For example, BERT \cite{devlin_bert_2019} handles searches on Google from 2020.
GPT-3 \cite{DBLP:journals/corr/abs-2005-14165} has shown excellent capabilities in text summarization, text generation, etc

While great potential has been shown by general NLP models like GPT-3, they may not perform consistently well in domain-specific tasks (e.g., healthcare \cite{8440842} and programming~\cite{8952330,8802660}).
Most pre-trained language models (PLMs) are learned with general-domain corpora, e.g., Wikipedia and OpenWebText \cite{radford2019language}.
However, the important and frequent knowledge entities (words) of a specific domain are typically different from those in a general domain.
For instance, the overlap of the words frequently used in scientific texts and the words in general texts is only 42\%~\cite{Beltagy2019SciBERT}.
As a result, the general PLMs are hard to well capture many important domain entities that rarely appear in general corpora.
Then, advanced development methodology for NLP models in domain-specific AI software is still urgently required.

In general, two steps are needed for domain-specific NLP model development: (i) language model pretraining and (ii) task-specific model training \cite{gu2021domain}.
To improve domain-specific NLP model development, most existing studies focus on pretraining. In particular, to learn domain-specific word embeddings, they retrain PLMs with domain-specific corpora. Many domain-specific PLMs have been developed based on clinical \cite{alsentzer-etal-2019-publicly}, biomedical \cite{lee2020biobert,zhang2020conceptualized}, and scientific \cite{Beltagy2019SciBERT} texts.

In contrast, how to improve the second step (i.e., task-specific training) is under-investigated. A common practice is directly fine-tuning the task-specific model with annotated data \cite{gu2021domain}. However, \textit{it is difficult to obtain abundant annotated data for a domain-specific task, as labeling often requires domain experts' knowledge}~\cite{DBLP:conf/acl/YueGS20}; without sufficient data, direct fine-tuning may not lead to a satisfactory performance due to overfitting~\cite{pmlr-v126-si20a}. Some studies propose task-dependent methods to train task-specific models by introducing some types of domain knowledge \cite{zhu-etal-2022-knowledge}. For instance, domain keyword-assisted topic models are proposed to learn development requirements from app reviews~\cite{DBLP:conf/icse/TushevEM22}. However, such task-dependent methods are hard to be generalized to other tasks.

Then, a research question appears: \textbf{can we introduce domain knowledge to task-specific model training in a \textit{unified} way, and facilitate the development of an \textit{arbitrary} domain's NLP models in AI software?} Considering the rapidly-growing requirements of domain-specific AI software~\cite{martinez2022software}, this question becomes critical in software engineering, as providing a \textit{unified} process can significantly boost NLP model development in various domains.
To answer the question, there are two main issues to address: (i) \textit{where to find a unified format of domain knowledge?} (ii) \textit{how to improve the task-specific training of various domains' models in a unified way?}

On one hand, the domain knowledge graph (KG) is an effective and standardized knowledge base for a specific domain \cite{abu2021domain}.
KGs have been constructed for various domains such as cybersecurity~\cite{jia2018practical}, social-impact funding~\cite{li2020domain}, and healthcare~\cite{li2020real,zhang2020hkgb,malik2020automated}, which emphasizes the wide availability of domain KGs.
Hence, \textit{domain KG could be a feasible source for unified domain knowledge}.
On the other hand, data augmentation (DA) is a data-space approach to enrich training data to avoid overfitting, and is widely adopted in general tasks like computer vision \cite{shorten2019survey}, time-series prediction \cite{wen2020time}, and NLP \cite{feng_survey_2021}.
As DA mostly deals with training data regardless of the task-specific model structure, DA methods are often \textit{task-agnostic}~\cite{longpre_how_2020}, i.e., not specified to any particular task.
This property inspires us that \textit{it may be possible to design a unified DA process to introduce domain knowledge to task-specific model training}. However, current DA methods in NLP are mostly proposed for general texts \cite{wei_eda_2019}, and the performance on domain-specific tasks is limited \cite{feng_survey_2021}.
In general, domain-specific DA is still an under-investigated direction \cite{feng_survey_2021}.

To fill this research gap, by exploiting domain KGs, we propose a novel three-step procedure to perform domain-specific DA:
(i) \textit{domain knowledge localization} to map phrases in the text to entities in the domain KG;
(ii) \textit{domain knowledge augmentation} to fully utilize the KG and the training data to achieve domain-specific augmentation;
and (iii) \textit{augmentation quality assessment} to single out high-quality augmented data for fine-tuning the task-specific model.\footnote{In this paper, we focus on NLP classification tasks. Meanwhile, our research has the potential for serving other types of NLP tasks, e.g., question-answering, which will be discussed in Sec.~\ref{sec:future_work}.}
Moreover, upon the domain-specific DA procedure, we design \textbf{\textit{KnowledgeDA}\xspace}, \textit{a unified knowledge graph service that helps users (even without AI programming skills) to develop a high-quality domain-specific NLP model with little coding (i.e., low-code development \cite{sanchis2019low})}. Our contributions can be summarized as follows:

\begin{itemize}
    \item To the best of our knowledge, this is one of the pioneering efforts toward proposing a unified development process for domain-specific NLP models in AI software, especially focusing on task-specific model training.

    Our research also sheds light on a new unified knowledge graph service to facilitate low-code development \cite{sanchis2019low,lourencco2021ostrich} of domain-specific NLP models.
    \item
   We propose \textit{KnowledgeDA}\xspace, which consists of three core steps, \textit{domain knowledge localization}, \textit{domain knowledge augmentation}, and \textit{augmentation quality assessment}.
    We implement a prototype of \textit{KnowledgeDA}\xspace, which can automatically learn domain-specific models given domain-specific texts (\textit{healthcare} and \textit{software development}) .
    \item Experiments are run on five real-life NLP datasets (English and Chinese) in healthcare and software development. Results show that \textit{KnowledgeDA}\xspace can obtain $\sim 4\%$ improvement compared to direct fine-tuning, and significantly outperform existing DA methods \cite{wei_eda_2019,yue_phicon_2020}.

    The source codes are available at \url{https://github.com/RuiqingDing/KnowledgeDA}.
\end{itemize}

\section{Background and Related Work}

\subsection{Software-as-a-Service and Machine-Learning-as-a-Service}
\textit{Software as a Service} (SaaS) is increasingly being used by companies due to a cost reduction and quality improvement of existing operations \cite{DBLP:journals/dss/LoukisJM19}.
In the era of big data and AI, more and more services are proposed for data analysis and model development, leading to a new type of SaaS, \textit{Machine Learning as a Service} (MLaaS)~\cite{ribeiro2015mlaas}.
Many platforms offer easy-to-use APIs for users to upload data for training and prediction, such as Google Prediction API,\footnote{https://cloud.google.com/prediction} Amazon ML,\footnote{https://aws.amazon.com/machine-learning} and Microsoft Azure ML.\footnote{https://studio.azureml.net/} Google also developed a black-box optimization cloud service, called \textit{HyperTune}, which can automatically tune an arbitrary machine learning model~\cite{DBLP:conf/kdd/GolovinSMKKS17}. Tencent implemented an online KG alignment service, called \textit{PRASEMap} \cite{DBLP:conf/cikm/QiZCCZ21}, which can support various embedding-based KG alignment methods.
These services reduce the complexity of data processing and model development, greatly accelerating high-quality AI software development.
Following this line, we propose a new KG service, called \textit{KnowledgeDA}\xspace, to ease domain-specific NLP model development.
When users upload domain-specific NLP task data and select the PLM, \textit{KnowledgeDA}\xspace can automatically perform KG-based data augmentation to facilitate task-specific model fine-tuning. Different from traditional KG services, e.g., search\footnote{https://developers.google.com/knowledge-graph} and query~\cite{li2020domain}, our work presents a promising service direction, i.e., facilitating the low-code development \cite{sanchis2019low} of domain-specific NLP models.

\subsection{Domain-specific NLP Model Development}
To improve domain-specific NLP model development, a general strategy is introducing domain knowledge \cite{zhu-etal-2022-knowledge}.
Many \textit{task-dependent} solutions are proposed, e.g., using domain-specific keyword-assisted topic models to extract the development requirements from app reviews~\cite{DBLP:conf/icse/TushevEM22},
and alleviating the ambiguity in requirements by collecting domain-specific corpora \cite{DBLP:conf/icse/EzziniA0SB21}.
In additional to task-dependent methods, researchers also propose some \textit{unified} training strategies to incorporate domain knowledge (domain-specific corpora) into PLMs, leading to domain-specific PLMs such as BioBERT~\cite{lee2020biobert}, SciBERT \cite{Beltagy2019SciBERT}, and ClinicalBERT \cite{alsentzer-etal-2019-publicly}.
Our work also attempts to improve the domain-specific NLP model development in a unified manner. Different from PLM, we focus on task-specific NLP model fine-tuning \cite{gu2021domain}.
Hence, our proposed \textit{KnowledgeDA}\xspace can be used with domain-specific PLMs together to construct NLP models in domain-specific AI software.

\begin{figure*}
    \centering
    \includegraphics[width=0.95\linewidth]{KnowledgeDA_framework_v3.png}
    \vspace{-1em}
    \caption{Overview of \textit{KnowledgeDA}\xspace for low-code development of domain-specific NLP models. To use \textit{KnowledgeDA}\xspace, a user only needs to upload the task text data (i.e., training data) and specifies the pre-trained language models to use (e.g., BERT). All the domain knowledge injection procedures are automatically conducted in \textit{KnowledgeDA}\xspace. Finally, without any coding, the user can obtain a well-performed domain-specific NLP model from \textit{KnowledgeDA}\xspace.}
    \label{fig: knowledgeDA}
\end{figure*}

\subsection{Text Data Augmentation}
Text data augmentation (DA) has received increasing interest, as NLP community starts studying low-resource domains where only a few annotated samples exist \cite{feng_survey_2021}.
In general, three types of text DA methods exist:
(i) \textit{Rule-based} techniques, e.g., Easy Data Augmentation (EDA) \cite{wei_eda_2019}, adopt a set of token-level random perturbation operations including random insertion, deletion, and swap.
(ii) \textit{Interpolation-based} techniques, pioneered by MIXUP \cite{DBLP:conf/iclr/ZhangCDL18}, interpolate the inputs and labels of two or more real examples. Follow-ups include SwitchOut \cite{DBLP:conf/emnlp/WangPDN18}, MixText \cite{chen-etal-2020-mixtext}, etc.
(iii) \textit{Generator-based} techniques, e.g., LAMBADA \cite{DBLP:conf/aaai/Anaby-TavorCGKK20} and GPT3Mix \cite{DBLP:conf/emnlp/YooPKLP21}, learn generators by fine-tuning the large language generation models (e.g., GPT) on the training data to generate new training samples.
Basically, three types of methods can be used together as they augment data from diverse perspectives.
However, no matter belonging to which types, most existing studies do not explicitly introduce domain-specific knowledge. \textit{PHICON} \cite{yue_phicon_2020} attempts to use the domain-entity dictionary for text DA. Specifically, PHICON replaces an entity mention in a sentence with another same-category entity. Compared to PHICON, our \textit{KnowledgeDA}\xspace further considers relationships in the domain KG; besides, \textit{KnowledgeDA}\xspace introduces other newly-designed components, e.g., augmentation quality assessment, to ensure high-quality augmentation.

\section{The \textit{KnowledgeDA}\xspace Framework}

\subsection{Workflow of \textit{KnowledgeDA}\xspace}
To facilitate the development of domain-specific NLP models, we propose a unified domain KG service, \textit{KnowledgeDA}\xspace,
which can achieve explicit domain knowledge injection by domain-specific DA.
Challenges in \textit{KnowledgeDA}\xspace include:

\textbf{C1. How to discover domain knowledge in texts?}
Detecting entities in a text is the first step to link the text with a knowledge base.
A domain entity may have multiple expressions, e.g., \textit{lungs} and \textit{pulmonary} share a similar meaning in the healthcare domain.
It is important to deal with synonyms.

\textbf{C2. How to ensure that the augmented texts retain the domain information and are semantically correct?}
We aim to achieve interpretable data augmentation through explicit domain knowledge injection.
The domain information and the semantic correctness of augmented samples are desirable to be kept after data augmentation.

\textbf{C3. How to ensure the quality of augmented texts?}
As PLMs grow larger, simple DA method becomes less beneficial \cite{feng_survey_2021}.
It is necessary to select beneficial samples from all the augmented samples for efficient fine-tuning.

To address the above challenges, we design corresponding modules in \textit{KnowledgeDA}\xspace.
As shown in Fig.~\ref{fig: knowledgeDA}, \textit{KnowledgeDA}\xspace consists of three main components:
(i) \textbf{\textit{domain knowledge localization}}, which locates the mentions of domain KG entities in texts;
(ii) \textbf{\textit{domain knowledge augmentation}}, which incorporates a dual-view DA strategy by considering both domain KG and training data;
(iii) \textbf{\textit{augmentation quality assessment}}, which retains beneficial augmented samples for fine-tuning using a confidence-based strategy.
With these components, when a user (even without AI programming skills) inputs task data and selects PLM (e.g., BERT), \textit{KnowledgeDA}\xspace can automatically conduct data augmentation based on built-in domain KGs and output the final domain task-specific model.

\textit{Remark}. The general workflow of \textit{KnowledgeDA}\xspace assumes that the domain KG is already available. In practice, many domain KGs are ready for usage  \cite{DBLP:journals/nar/Bodenreider04} \cite{DBLP:journals/corr/abs-2203-09975}. For the domains without existing KGs, we may use state-of-the-art KG construction methods \cite{li2020real} to build domain KGs. In Sec.~\ref{section: implementation}, we describe the implementation details of \textit{KnowledgeDA}\xspace for two domains, including how to obtain or generate domain KGs.

\subsection{Module 1: Domain Knowledge Localization}
\label{subsection:module1}

Detecting entities in texts can identify domain-specific objects and the relations between them.
Considering that an entity may correspond to multiple mentions \cite{DBLP:conf/naacl/FlorianHIJKLNR04}, only using \textit{exact} string matching will lead to a low matching rate.
Some studies attempt to find non-exact matching by learning high-level features of mentions and entities in a supervised learning manner \cite{DBLP:conf/cikm/PhanSTHL17,DBLP:journals/symmetry/HuTZGX19}. Methods of this type need some labeled data, which is hard to be generalized to a specific domain as labeling needs domain experts and would be costly.
Besides, there are some open entity detection tools, like TAGME \cite{DBLP:conf/cikm/FerraginaS10} and BLINK \cite{DBLP:conf/emnlp/WuPJRZ20}, which can link text mentions to Wikipedia terms. However, as these tools are learned in general texts, their performance on domain-specific entities is limited.

Then, we propose to use an annotation-free string-similarity-based strategy \cite{DBLP:conf/eacl/BunescuP06} to discover \textit{non-exact but correct} mappings between mentions in the text and entities in the KG.
The advantage of the strategy is the ability to be adapted to various domain KGs without extra annotation costs. Specifically, we calculate the inner product of word embeddings as string similarity \cite{DBLP:conf/emnlp/WuPJRZ20}, as it encodes word semantic information that cannot be caught by tradition similarity measures such as edit distance and hamming distance~\cite{DBLP:journals/symmetry/HuTZGX19}.

As seen in Fig.~\ref{fig: knowledgeDA}, we first follow the NLP preprocessing pipeline and then match the processed text with the KG.
During preprocessing, we add the entity names in KG to the dictionary of \textit{tokenizer} to avoid word segmentation errors, e.g., ``cerebral embolism" should be considered as a single term in the healthcare domain, instead of being split into two words, ``cerebral" and ``embolism".
At the same time, we use \textit{POS Tagger} and \textit{Lemmatizer} to convert each token to the canonical form (lemma), e.g., the lemma for ``coughed" is ``cough".
To do knowledge localization, we first extract the entities' embeddings and the mentions' embeddings from the PLM, and then calculate the similarity between them.
We consider the pair of a mention and the most similar KG entity as a match if the similarity score is larger than a threshold $\delta$ (0.9 in our implementation). Algorithm \ref{algorithm:localization} summarizes the domain knowledge localization process.

\begin{algorithm}[t]
\footnotesize
    \SetAlgoLined
    \caption{Domain Knowledge Localization
    \label{algorithm:localization}
    \KwIn{A text $x$, the entities list $E$, words embeddings dictionary $Embeds$, and similarity threshold $\theta$
    \KwOut{A matched pair list $Matchs$ of mentions in $x$ and entities in $E$
    Initialize $Matchs$ as an empty list \;
    Preprocess $x$ with NLP preprocessing pipeline to get word list $words$ \;
    Construct entity embedding matrix $E_{emb}$ and embedding matrix $W_{emb}$ by searching for $E$ and $words$ from $Embeds$ \;
    Compute similarity matrix $Sim = W_{emb} \times E_{emb}.T$ \;
    Query the maximum similarity $sim\_values$ between each word and entity\;
    \If{$sim\_value \geqslant \theta$}{
        Find the index of $sim\_values$ in $Sim$ \;
        Get the pair $(word, entity)$ according to index \;
        Add $(word, entity)$ to $Matchs$ \;
    }
    Return $Matchs$ \;
\end{algorithm}

In Fig.~\ref{fig:localization}, we give an example of knowledge localization in healthcare.
Without simiality match, we will ignore that \textit{scour} and \textit{diarrhea} are analogous.
Through localization, we can also find that fever and scour are symptoms of pneumonia and respiratory syndrome;
covid-19 is a cause of pneumonia and respiratory syndrome.
Hence, we can construct relations between these entity mentions based on domain KGs, which will be used in the next module, i.e., domain knowledge augmentation.

\begin{figure}
    \centering
    \includegraphics[width=0.95\linewidth]{grounding_example.png}
    \vspace{-1em}
    \caption{An example of domain knowledge localization in the healthcare domain. The words in {\color{red}red} indicate that the entities and the mentions are \textit{exactly} same, and the words in {\color{blue}blue} indicate that the mention (scour) and the entity (diarrhea) share \textit{similar word embeddings}.}
    \label{fig:localization}
\end{figure}

\begin{figure*}
    \centering
    \subfigure[\textit{KGER}: 1. generate a triples list by retrieving 2-hop relevant entities in the KG; 2. achieve entity replacement following two principles, `entity relevance' and `relation consistency'.]{
    \includegraphics[width=0.9\textwidth]{KGER_v2.png}
    \label{subfig:KGER}
    \vspace{-1em}
    }
    \subfigure[\textit{TrainER}: 1. mask entity with the category to represent the expressions pattern; 2. text clustering to select samples with the same label but different clusters (e.g., samples in yellow);
    3. collect the candidate triples from selected samples; 4. augment data by replacing relevant entities.]{
    \includegraphics[width=0.9\textwidth]{TrainER.png}
    \label{subfig:TrainER}
    }
    \caption{Dual-view knowledge augmentation with a healthcare text as an example.}
    \label{fig:augmentation}
    \vspace{-1em}
\end{figure*}

\subsection{Module 2: Dual-view Domain Knowledge Augmentation}
\label{subsection:module2}

After locating the domain knowledge, i.e., entity mentions in the text, the next step is to replace these mentions with other \textit{relevant entity words} for domain-specific data augmentation. Then, how to retrieve the relevant entity becomes a key issue. In \textit{KnowledgeDA}\xspace, we propose a dual-view strategy to conduct the \textit{relevant entity retrieval} by considering both KG and the training text data.

\vspace{+.5em}
\textbf{View 1: KG-based Entity Retrieval (KGER)}

Prior work has demonstrated that introducing external knowledge can improve language models \cite{zhu-etal-2022-knowledge}.
A direct strategy for domain KG-based DA is to replace the entity with another same-category entity, e.g., replacing `William' with `Mike' as both are person names \cite{yue_phicon_2020}.
However, it may suffer from two pitfalls:
(i) Although the original and replaced entities are in the same category, they can be totally different, such as \textit{pneumonia} and \textit{fracture} (both are diseases), which may negatively impact the downstream tasks (e.g., classifying a medical transcription to the relevant department\footnote{https://www.kaggle.com/datasets/tboyle10/medicaltranscriptions});
(ii) When two or more entities appear in a text, they may have certain valuable relationships (e.g., disease and symptom), but replacing these entities separately would ignore this information.

To address the above issues, we propose two principles for \textbf{KGER}: \textit{entity relevance} and \textit{relation consistency}.
\textit{Entity relevance} refers to ensuring that the retrieved entity is similar to the original entity, not just with the same category.
\textit{Relation consistency} means keeping the relationships unchanged between multiple replaced entities in one text.

A domain KG can be formulated as $\mathcal{G} = \{E, R, T, C\}$, where $E$, $R$, $T$, and $C$ are the sets of entities, relations, triples, and entities' categories, respectively.
Specifically, $T = T^R \bigcup T^C$, where $T^R = \{(h, r, t)| h, t \in E, r \in R \}$ and $T^C = \{(e, BelongTo, c) | e \in E, c \in C\}$.

Given an entity $e$, we can get its category $c$, involved triples $T_e = \{(e, r, t) \in T^R\} \bigcup \{(h, r, e) \in T^R\}$,
and the adjacent entities $E_e = \{e' | (e, r, e') \in T_e, (e, r, e') \in T_e\}$.
To obtain more same-category entities, we further retrieve the involved triples of $E_e$, named $T_{e2}$, and put $T_e$ and $T_{e2}$ together as the candidate triples (i.e., 2-hop triples around $e$).

1) If only one entity exists, or multiple entities exist but do not have direct KG relations in the text, we randomly select a same-category entity $e'$ from the candidate triples to replace each original entity $e$. Note that $e'$ must be within 2-hop around $e$, ensuring the \textit{entity relevance}.

2) If there exist certain pairs of entities with KG relations, we would seek the same relation-type triple from the candidate triples for replacing the pair of entities together, ensuring the \textit{relation consistency}.

As shown for the example text `\textit{I have a \textbf{fever} and \textbf{scour}. Could it be \textbf{pneumonia}?}' of Fig.~\ref{subfig:KGER}, \textit{fever} and \textit{scour} are the symptoms of \textit{pneumonia} (\textit{scour}'s corresponding entity in KG is \textit{diarrhea} with the embedding-similarity matching algorithm in Module 1).
So we need to search for suitable triples in the candidate triples to satisfy relation consistency. For instance, \textit{diarrhea} and \textit{sore throat} are the symptoms of \textit{respiratory syndrome}, so the augmented text could be `\textit{I have a \textbf{diarrhea} and \textbf{sore throat}. Could it be \textbf{respiratory syndrome}?}'.

\vspace{+.5em}
\textbf{View 2: Training Data-based Entity Retrieval (TrainER)}

In View 1, we mainly retrieve relevant entities that are close in the KG. However, there may be entity pairs far away in the KG that can be helpful for the specific task if being replaced with each other.
For example, for the task to detect the medical query intent, `\textit{blood routine examination}' and `\textit{CT}' is the entity pair that could be replaced with each other for augmentation because they most probably appear in the queries about \textit{diagnosis} and \textit{cause analysis}, but they are distant from each other in the medical KG, such as CMedicalKG \footnote{https://github.com/liuhuanyong/QASystemOnMedicalKG}.

To find such task-specific valuable replacement entity pairs which may not be near in the KG, we design a new View 2, \textit{Training Data-based Entity Retrieval (TrainER)}, to retrieve task-specific entity pairs from training data.
Prior work REINA \cite{DBLP:conf/acl/WangXFLSX0022} has verified that retrieving from training data to enrich model inputs (concatenating the original input and retrieved training data) may generate significant gains in NLP tasks.
Inspired by REINA \cite{DBLP:conf/acl/WangXFLSX0022}, TrainER aims to extract gainful entity pairs from the training data for data augmentation.
Specifically, the core problem is how to find valuable entity pairs for replacement from different texts in the training data.

In general, a good entity pair for replacement may satisfy at least two properties: (i) \textit{label consistency}, indicates that the two entities in the pair should be contained in two training texts with the same task label; (ii) \textit{expression diversity}, means that the two texts containing the two entities should have different expression patterns, so as to enrich the training data diversity.
Specifically, to reach \textit{label consistency}, for an entity $e$ in a text $t$, we would retrieve a same-category entity $e'$ from another text $t'$ only when $t$ and $t'$ have the same label.
To achieve \textit{expression diversity}, we first cluster all the training texts into different clusters with diverse expression patterns.
Then, for an entity $e$ in a text $t$,  the replaced entity $e'$ will be retrieved from $t'$ only if $t$ and $t'$ are not in the same cluster.

Fig.~\ref{subfig:TrainER} elaborates on the process of TrainER.
Specifically, to conduct training data clustering to differentiate expression patterns,
we first mask entities with their categories to extract the expression templates for each training text.
For instance, `\textit{what is \textbf{pneumonia}?}' and `\textit{what is \textbf{fracture}}' would have the same expression template `\textit{what is \textbf{[disease]}}?', as both sentences have the same expression pattern regardless of the specific entity (i.e., disease).
Then, we run a clustering algorithm on the masked texts, i.e., expression templates, to identify diverse expression patterns.
In our implementation, we use the K-means \cite{DBLP:conf/soda/ArthurV07} clustering algorithm because of its high efficiency and effectiveness in our empirical experiments; the feature of a masked text is represented by TF-IDF vectorization \cite{DBLP:journals/jd/Jones04}.
Same as the \textit{relation consistency} principle in KGER, if there are certain entity pair with KG relations in the original text, we will also retrieve the entity pair with the same relation from other training texts.

\subsection{Module 3: Augmentation Quality Assessment}
\label{section: module3}
After Module 1 \& 2, we obtain a set of augmented texts. A straightforward way is to directly fine-tune task-specific models with these texts like most prior studies~\cite{wei_eda_2019,zhang_character-level_2015,DBLP:conf/emnlp/WangPDN18}.
Recent work \cite{DBLP:conf/aaai/Anaby-TavorCGKK20,DBLP:conf/acl/ZhouZTJY22} has found that not all the augmented texts are equivalently effective; thus, assessing the quality of the augmented texts and selecting high-quality ones may further improve the model performance.

Inspired by this finding, \textit{KnowledgeDA}\xspace also includes a quality assessment module to justify the quality of each augmented text. Prior work \cite{DBLP:conf/aaai/Anaby-TavorCGKK20,DBLP:conf/acl/ZhouZTJY22} proposes to use the prediction confidence as the quality metric and select top-$K$ high-confidence augmented samples for fine-tuning, because this indicates that the training label for the augmented sample is probably correct. Specifically, with the PLM (e.g., BERT), we first fine-tune PLM on the task texts (without augmentation); this plain fine-tuned model $\mathcal M$ can be applied on the augmented texts to make predictions; then, we can obtain the confidence score of each augmented text.

While selecting top-$K$ high-confidence samples can ensure the label correctness of augmented texts, we argue that it may not significantly improve the model performance since high confidence means that the pattern inside the augmented sample has already been encoded in $\mathcal M$.
Hence, instead of selecting top-$K$ confident samples, we propose to pick $K$ augmented samples whose confidence is close to a predefined threshold $\delta$.
Note that $\delta$ should not be a too small number, as we still want to ensure the correctness of the training labels for augmented texts; meanwhile, $\delta$ should not be too large, as a very high-confident sample would contribute little new knowledge to the model.
Based on this idea, we design a novel confidence-based data filtering strategy to retain gainful augmented samples.

Specifically, given the task data $D=\{(x_i, y_i)\}_{i=1}^{n}$ and the plain fine-tuned model $\mathcal{M}$ (without augmentation).
Each $x_i$ is a string of text, and the label $y_i \in \{1,2, \cdots, q\}$ indicates the label of $x_i$ among a set of $q$ labels.
Through \textit{KGER} and \textit{TrainER}, we can generate the augmented samples $D_{i}^{aug} = \{x_{i}^{1}, x_{i}^{2}, \cdots, x_{i}^{m}\}$ for the i-th sample, $x_i$.
The prediction confidence (probability) of $D_{i}^{aug}$ can be calculated as $P_{i}^{aug} = \{p_{i}^{j}\}_{j=1}^{m}$, where $p_{i}^{j} = prob(\mathcal{M}(x_{i}^{j}) = y_i)$.

We propose a confidence threshold $\delta$ to adjust sample selection criteria. Given $\delta$, the sampling weights of $D_{i}^{aug}$  can be calculated by
\begin{equation}
    w_{i}^{1}, w_{i}^{2}, \cdots, w_{i}^{m} = softmax(\xi_{i}^{1}, \xi_{i}^{2}, \cdots, \xi_{i}^{m})
    \label{eq:sampling}
\end{equation}
where $\xi_{i}^{j} = 1-|\delta-p_{i}^{j}|$.
If $p_{i}^{j}$ is closer to $\delta$ (set to 0.75 in our implementation), we have a higher probability to select this sample.
With this confidence-based sampling strategy, we can select augmented samples to further fine-tune the task model $\mathcal M$. In general, the selected samples would be relatively confident but not too highly-confident, thus ensuring both \textit{label correctness} and \textit{new knowledge}.
Algorithm~\ref{algorithm:augmentation} summarizes the process of knowledge augmentation and quality assessment.

\begin{algorithm}[t]
    \footnotesize
    \SetAlgoLined
    \caption{Domain Knowledge Augmentation \& Augmentation Quality Assessment
    \label{algorithm:augmentation}
    \KwIn{Train Data $D=\{(x_i, y_i)_{i=1}^{n}\}$; the KG $\mathcal{G} = \{E, R, T, C\}$; the pre-trained language model $PLM$;
    a confidence threshold $\delta$.
    \KwOut{The selected augmented samples $D^{aug}$
    Fine-tune without augmentation $\mathcal M $ = \textit{fine-tune}$(PLM, D)$ \;
    \For{$x_i$ in $\{x_i\}_{i=1}^{n}$}{
        Get the $Matches_i$ in $x_i$ by Algorithm~\ref{algorithm:localization}\;
        Generate augmented samples $D^{aug}_{i}$ with $\mathcal{G}$ following the steps in Fig.~\ref{fig:augmentation}\;
        Initialize the prediction probabilities of $D^{aug}_{i}$ as $P^{aug}_{i}$\;
        \For{$x_i^j$ in $D^{aug}_{i}$}{
            Calculate the prediction probability $p_{i}^{j} = prob(\mathcal{M}(x_{i}^{j}) = y_i)$\;
            Add $p_{i}^{j}$ to $P^{aug}_{i}$\;
        }
        Calculate the sampling weights of $D^{aug}_{i}$ according to Eq.~\ref{eq:sampling} \;
        Sample 5 samples from $D^{aug}_{i}$ by weights and add them to $D^{aug}$\;
    }
    \Return $D^{aug}$\;
\end{algorithm}

\section{Knowledge Graphs for Implementation}
\label{section: implementation}
Prior sections describe the design details of \textit{KnowledgeDA}\xspace assuming that the domain KG is available.
In this section, we elaborate on how to implement \textit{KnowledgeDA}\xspace in two specific domains, healthcare and software development, mainly about the selection and construction of domain KGs.

\subsection{Knowledge Graph for Healthcare}

Healthcare is a field with rich professional knowledge.
A large amount of work focuses on NLP problems in the field, including PLMs \cite{alsentzer-etal-2019-publicly} \cite{michalopoulos-etal-2021-umlsbert}, KGs \cite{li2020real} \cite{zhang2020hkgb} \cite{DBLP:journals/corr/abs-2203-09975}, reading comprehension \cite{DBLP:conf/acl/YueGS20}, decision making \cite{1455705}, etc.
There are also publicly available knowledge graphs, e.g., the Unified Medical Language System (UMLS) \cite{DBLP:journals/nar/Bodenreider04}.
We can take such open medical KGs for healthcare \textit{KnowledgeDA}\xspace.

UMLS Metathesaurus is a compendium of many biomedical terminologies with the associated information, including synonyms, categories, and relationships.
It groups semantically equivalent or similar words into the same concept, for example, the words `flu', `syndrome flu' and `influenza' are mapped to the same concept unique identifier (CUI)  \textit{C0021400}, which belongs to the category, \textit{disease or syndrome}.
There are 127 semantic types in biology, chemistry, and medicine, consisting of 4,441,326 CUIs (16,132,273 terminologies) in the UMLS 2021AA version.
Since the size of the KG is too large to affect the speed of retrieval, we only screen out entities that belong to the type of medicine (e.g., \textit{body part, organ, or organ component}, \textit{disease or syndrome}, etc.).
At the same time, we also delete non-English strings.
Finally, we keep 1,145,062 CUIs (16 semantic types), 502 types of relationships and 4,884,494 triples.
Besides English medical KG, we also try to establish a Chinese one. Although there are Chinese medical terminologies in UMLS, the number is limited.
Hence, we use an open-source Chinese medical KG, CMedicalKG,\footnote{https://github.com/liuhuanyong/QASystemOnMedicalKG} which includes 44,111 entities (7 categories), 10 types of relationships, and 294,149 triples.

\subsection{Knowledge Graph for Software Development}

NLP tasks become more and more important for facilitating the software development process \cite{8952330,8802660}.
We thus attempt to implement \textit{KnowledgeDA}\xspace for software development NLP tasks. However, there is little research on building KGs for software development NLP tasks, and thus we decide to build one from scratch.

To build the KG, we refer to the software developer forum \textit{Stack Overflow} to obtain raw text data.\footnote{https://stackoverflow.com/}
\textit{Stack Overflow} is one of the biggest forums for professional and enthusiastic software developers.
Various technical questions are covered on the platform and marked with appropriate tags.
These tags are usually programming-specific terminologies and can be beneficial to learn about tech ecosystems and the relationships between technologies \cite{doi:10.1080/09720529.2020.1721857}.
To build a KG from tags, we follow the existing KG construction process \cite{li2020real}:

\textit{Step 1. Data Collection}: We use programming languages (e.g., python, C\#, java, and javascript) as keywords to search for related questions on Stack Overflow, and sort them according to `most frequency';
we then crawl the tags that appeared in the top 7,500 related questions (i.e., on the first 150 pages).

\textit{Step 2. Entity Recognition}: A tag is a word or phrase that mainly describes the key information of the question, which is usually a programming-specific terminology \cite{doi:10.1080/09720529.2020.1721857}.
Hence, we directly treat tags as the entity names in the KG.

\textit{Step 3. Relation Formation}: There is usually more than one tag in one question. When multiple tags co-appear at the same question, we link them in the KG.
Afterward, there is still a lack of entity types and edge types, and we use the community detection algorithm, Louvain \cite{Blondel_2008}, to automatically classify tags, and the edge type is defined by the types of the two connected entities.

Finally, we get \textit{TagKG}, which includes 6,126 entities (11 categories), 56 types of relationships, and 41,227 triples.
With \textit{TagKG}, we implement \textit{KnowledgeDA}\xspace for software development NLP tasks.

\section{Empirical Evaluation}

\subsection{Empirical Setup}

\textbf{Datasets.}
We conduct experiments on four datasets in healthcare and one dataset in software development.
\begin{itemize}
\item \textit{CMID}\footnote{https://github.com/liutongyang/CMID}:
a Chinese medical intent classification dataset with 4 categories: disease, drug, treatment, and other.
\item \textit{KUAKE-QIC}: a Chinese medical query intent classification dataset in CBLUE \cite{DBLP:conf/acl/ZhangCBLLSYTXHS22},
including 11 labels: diagnosis, cause, method, advice, etc
\item \textit{TRANS}\footnote{https://www.kaggle.com/datasets/tboyle10/medicaltranscriptions}:
an English medical transcription dataset with 7 medical specialties:
cardiovascular/pulmonary, gastroenterology, neurology, orthopedic, etc
\item \textit{ABS}\footnote{https://github.com/PoojaR24/Medical-Text-Classification}:
an English medical abstract dataset with 5 categories: digestive diseases, cardiovascular diseases, neoplasms, nervous diseases, and general conditions.
\item \textit{SO-PLC}\footnote{http://storage.googleapis.com/download.tensorflow.org/data/stack\_\\overflow\_16k.tar.gz}:
a Stack Overflow dataset for 4 programming language classification: python, C\#, java, and javascript.
\end{itemize}

The basic information is enumerated in Table \ref{tab:data}.

\renewcommand{\arraystretch}{1.3}
\begin{table}[t]
\footnotesize
    \centering
    \caption{Dataset Statistics}
    \label{tab:data}

    \begin{tabular}{lcccc}
    \toprule
     \textbf{Dataset} & \textbf{Lang.} & \#\textbf{Labels} & \#\textbf{Samples} & \#\textbf{Entity mentions}  \\ \hline
     \multicolumn{5}{l}{\textbf{Healthcare Domain}} \\
     CMID & CHI & 4 & 12254 & 5182 \\
                                 KUAKE-QIC & CHI & 11 & 8886 & 3369 \\
                                 TRANS & ENG & 7 & 1740 & 2298 \\
                                 ABS & ENG & 5 & 14438 & 3808 \\ \hline
    \multicolumn{5}{l}{\textbf{Software Development Domain}} \\
     SO-PLC & ENG & 4 & 8000 & 1964 \\ \bottomrule
    \end{tabular}%

\end{table}

\renewcommand{\arraystretch}{1.3}
\begin{table*}[t]
    \centering
    \caption{Performance of baselines and \textit{KnowledgeDA}\xspace with BERT in healthcare}
    \begin{threeparttable}
    \begin{tabular}{lcccccccccccc}
    \toprule
    \multicolumn{1}{c}{\multirow{2}{*}{\textbf{Method}}} & \multicolumn{2}{c}{\textbf{CMID} (Chinese)} & \multicolumn{2}{c}{\textbf{KUAKE-QIC} (Chinese)} & \multicolumn{2}{c}{\textbf{TRANS} (English)} & \multicolumn{2}{c}{\textbf{ABS} (English)} \\
    \cmidrule(r){2-3} \cmidrule(r){4-5} \cmidrule(r){6-7} \cmidrule(r){8-9}
    \multicolumn{1}{c}{} & Acc. & F1 & Acc. & F1 & Acc. & F1 & Acc. & F1  \\
    \hline

    \textit{No Augmentation} & 70.25(0.80) & 68.21(0.90) & 78.82(0.81) & 78.57(0.72) & 73.10(1.79) & 71.50(1.77) & 63.95(0.31) & 62.84(0.40) \\
    \textit{SR} & 71.90(0.76) & 70.97(0.39) & 80.52(0.73) & 80.10(0.80) & 72.38(0.24) & 72.52(0.30) & 64.14(0.25) & 63.13(0.30) \\
    \textit{EDA} & 70.59(0.65) & 70.05(1.25) & 79.45(0.33) & 79.01(0.41) & 73.91(0.26) & 73.71(0.28) & 63.23(0.65) & 62.09(0.72) \\
    \textit{PHICON} & 71.95(0.35) & 71.14(0.53) & 80.52(0.74) & 80.23(0.82) & 74.53(1.19) & 73.10(0.77) & 64.17(0.59) & 63.35(0.60) \\
    \textit{\textit{KnowledgeDA}\xspace} & \textbf{72.38(0.46)*} & \textbf{71.94(0.38)} &
                           \textbf{81.67(0.41)***} & \textbf{81.31(0.44)*} &
                           \textbf{75.66(0.58)**} & \textbf{75.37(0.72)} &
                           \textbf{64.97(0.29)} & \textbf{64.18(0.28)*} \\
    \bottomrule
    \end{tabular}
    \end{threeparttable}
    \begin{tablenotes}
        \footnotesize
        \item 1. Values in ``( )'' denote the standard deviation of five repeated experiments' results;
        \item 2. \textbf{Bold} denotes the best-performed ones of the task;
        \item 3. *, **, *** denote that the t-test significance p-value $<$ 0.1, 0.05, 0.01 when comparing the results of \textit{KnowledgeDA}\xspace and the best baseline.
    \end{tablenotes}
    \label{tab:MainRes_Healthcare}
    \vspace{-1em}
\end{table*}

\textbf{Baselines.} As \textit{KnowledgeDA}\xspace focuses on explicit knowledge injection during DA by domain entity replacement, we mainly compare with state-of-the-art rule-based DA methods:
\begin{itemize}
    \item \textit{SR} \cite{vijayaraghavan-etal-2016-deepstance}: token-level replacement with synonyms calculateing  word embedding to find synonym to replace.

    \item \textit{EDA} \cite{wei_eda_2019}: token-level random perturbation operations including random insertion,  deletion, and swap
    \item \textit{PHICON} \cite{yue_phicon_2020}: entity-level replacement with other entities belonging to the same category.
\end{itemize}

\textbf{Implementation Details.}
Our experiment platform is a server with AMD Ryzen 9 3900X 12-Core Processor, 64 GB RAM and GeForce RTX 3090. We use Python 3.6 with pytorch 1.8 on Ubuntu 20.04 for algorithm implementation.
We split the dataset into training set, validation set, and test set as 8:1:1. When fine-tuning PLMs, we set batch size to 32, learning rate to 1e-5, and training epoch to 10. It will stop training if the loss of the validation set does not decrease in 500 iterations. For assessment, we use the common metrics in text classification tasks, \textit{accuracy} and \textit{F1} \cite{wei_eda_2019}. Also, we repeat each experiment 5 times and record the average results.

\subsection{Results on Healthcare NLP Tasks}

\renewcommand{\arraystretch}{1.3}
\begin{table}[t]
\centering
\caption{Novel entity coverage for healthcare datasets}
\label{tab:diversity}
\begin{tabular}{lcccc}
\toprule
 \textbf{Method} & \multicolumn{1}{l}{\textbf{CMID}} & \multicolumn{1}{l}{\textbf{KUAKE-QIC}} & \multicolumn{1}{l}{\textbf{TRANS}} & \multicolumn{1}{l}{\textbf{ABS}}
 \\ \hline
\textit{SR} & 14.80\% & 17.89\% & 33.56\% & 24.33\%
\\
\textit{EDA} & 11.18\% & 15.85\% & 26.66\% & 25.07\%
\\
\textit{PHICON} & 35.84\% & 29.11\% & 76.45\% & 74.07\%
\\
\textit{KnowledgeDA} & \textbf{40.67\%} & \textbf{35.36\%} & \textbf{79.33\%} & \textbf{78.37\%}
\\ \bottomrule
\end{tabular}%
\end{table}

\textbf{1) BERT as the Pre-trained Language Model}

Firstly, we use \textit{BERT} \cite{devlin_bert_2019} as the PLM and test different DA methods. The results are presented in Table \ref{tab:MainRes_Healthcare}.
We can observe that \textit{KnowledgeDA}\xspace achieves the best performance among all the methods in both accuracy and F1 score on four healthcare datasets.
At the same time, \textit{PHICON} also outperforms \textit{SR} and \textit{EDA} in most cases, also verifying the effectiveness of domain-specific knowledge.

Specifically, on two Chinese datasets, CMID and KUAKE-QIC, \textit{KnowledgeDA}\xspace can improve the accuracy by 3.03\% and 3.62\%, respectively, over the fine-tuned model without augmentation. Moreover, compared to the best baseline (i.e., \textit{PHICON}), \textit{KnowledgeDA}\xspace's improvements on accuracy are still statistically significant. Similar results are also observed on two English datasets.
In a nutshell, the results suggest that domain-specific entity replacement can facilitate NLP tasks in healthcare.
Compared to the replacement that only considers entity categories (i.e., \textit{PHICON}),
\textit{KnowledgeDA}\xspace selects entities from dual views and accounts for the KG relationships between them, which further improves the quality of the augmented text and thus achieves a better performance.
Fig.~\ref{fig:case_study} shows an example in \textit{CMID} with various DA methods. We can observe that the sentence augmented by \textit{KnowledgeDA}\xspace has a high quality as it can introduce more domain entities and the whole sentence has a good semantic meaning.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.9\linewidth]{case_study.png}
    \caption{An example of data augmentation in \textit{CMID}. Text chunks in {\color{blue}blue} are the entities in the original sentence and text chunks in {\color{red}red} are the modified words/entities by DA methods.}
    \label{fig:case_study}
    \vspace{-1em}
\end{figure}

To further quantitatively verify that \textit{KnowledgeDA}\xspace can introduce more domain knowledge than baselines,
following \cite{DBLP:conf/acl/0003XSHTGJ22}, we calculate \textit{Novel Entity Coverage}, the percentage of the \textit{novel} entities in the test data covered by augmented texts (\textit{novel} means not appearing in the training data).
As illustrated in Table~\ref{tab:diversity}, \textit{KnowledgeDA}\xspace has the highest coverage, which also explains the effectiveness of \textit{KnowledgeDA}\xspace.

\vspace{+.5em}
\textbf{2) Domain-specific Pre-trained Language Models}

Domain-specific PLMs are pre-trained using domain corpus, so they contain domain-specific knowledge to a certain extent.
As stated in a DA survey \cite{feng_survey_2021}, the benefits of DA strategies may be reduced as domain-specific PLMs become more powerful for domain tasks.
To confirm that \textit{KnowledgeDA}\xspace is still beneficial with a domain-specific PLM, we use \textit{eHealth} \cite{DBLP:journals/corr/abs-2110-07244} as the PLM for Chinese datasets, and \textit{ClinicalBERT} \cite{alsentzer-etal-2019-publicly} as the PLM for English datasets.
According to Table \ref{tab:domain_plm}, the improvement brought by the domain-specific PLMs is evident (comparing with the results of BERT in Table \ref{tab:MainRes_Healthcare}).
Consistent with the survey \cite{feng_survey_2021} statement, we discover that when using the domain-specific PLMs, baseline DA methods may not generate an obvious performance improvement and even have a negative effect compared to no-augmentation.
For instance, on the TRANS dataset, EDA improves the performance over BERT (increasing F1 score from $71.50$ to $73.71$); however, EDA worsens the performance when a domain-specific PLM is used (reducing F1 score from $75.22$ to $74.91$).

However, even with domain-specific PLMs, \textit{KnowledgeDA}\xspace can still  improve the domain NLP task performance consistently. Note that for the TRANS dataset, \textit{KnowledgeDA}\xspace is the only DA method with positive improvement (and this improvement is also statistically significant).
This strongly verifies the high quality of the augmented texts generated by \textit{KnowledgeDA}\xspace.
The reason is probably that \textit{KnowledgeDA}\xspace can fully leverage the domain KG's structure information and also carefully select augmented samples via quality assessment.

\begin{table}[t]
    \centering
    \caption{Performance with domain-specific PLMs in healthcare}
    \begin{threeparttable}
    \resizebox{\columnwidth}{!}{%
    \begin{tabular}{lcccc}
    \toprule
    \multicolumn{1}{c}{\multirow{2}{*}{\textbf{Method}}} & \multicolumn{2}{c}{\textbf{CMID}} & \multicolumn{2}{c}{\textbf{TRANS}} \\
    \cmidrule(r){2-3} \cmidrule(r){4-5}
    \multicolumn{1}{c}{} & Acc. & F1 & Acc. & F1 \\ \hline
    \textit{No Augmentation} & 73.10(0.32) & 71.71(0.37) & 75.88(0.41) & 75.22(0.57) \\
    \textit{SR} & 73.49(0.19) & 72.08(0.22) & 75.38(1.11) & 75.01(1.43) \\
    \textit{EDA} & 72.93(0.46) & 71.89(0.89) & 75.40(0.79) & 74.91(0.80) \\
    \textit{PHICON} & 73.51(0.68) & 72.49(0.63) & 75.37(0.59) & 75.18(0.72) \\
    \textit{\textit{KnowledgeDA}\xspace} & \textbf{73.60(0.33)} & \textbf{72.61(0.31)} & \textbf{76.52(0.59)**} & \textbf{76.54(0.91)*} \\ \bottomrule
    \end{tabular}%
    }
    \end{threeparttable}
    \begin{tablenotes}
        \scriptsize
        \item CMID/TRANS use eHealth/ClinicalBERT as the domain-specific PLM.
    \end{tablenotes}
    \label{tab:domain_plm}
\end{table}

\subsection{Results on Software Development NLP Tasks}
To validate the effectiveness of \textit{KnowledgeDA}\xspace on multiple domains' NLP tasks, we further compare the performances of different DA methods with a software development NLP task based on the SO-PLC dataset.
As illustrated in Table \ref{tab:MainRes_Program}, there are almost no improvements or even slight decreases with EDA and SR, meaning these general DA methods are not suitable for the texts in software development forums.
With the help of our constructed TagKG (Sec.~\ref{section: implementation}), PHICON achieves some performance gains by replacing same-category programming entities; this indicates that the category identified by the community detection algorithm in TagKG is effective for understanding software development related texts.
By leveraging TagKG more comprehensively,  \textit{KnowledgeDA}\xspace works even better and improves accuracy and F1 score by 2.42\% and 2.58\%, respectively, compared with no-augmentation fine-tuning.
It also implies that the construction of TagKG is valid.

\begin{table}[t]
    \centering

    \caption{Performance on SO-PLC dataset (\textit{BERT} as PLM)}

    \begin{tabular}{lcc}
    \toprule
    \multicolumn{1}{c}{\multirow{2}{*}{\textbf{Method}}} & \multicolumn{2}{c}{\textbf{SO-PLC}} \\
    \cmidrule(r){2-3}
    & Acc. & F1 \\ \hline
    \textit{No Augmentation} & 84.78(0.48) & 84.65(0.50) \\
    \textit{SR} & 84.72(0.32) & 84.69(0.31) \\
    \textit{EDA} & 84.78(1.08) & 84.71(1.08) \\
    \textit{PHICON} & 85.63(1.17) & 85.60(1.19) \\
    \textit{\textit{KnowledgeDA}\xspace} & \textbf{86.82(0.90)*} & \textbf{86.83(0.88)**} \\ \bottomrule
    \end{tabular}%

    \label{tab:MainRes_Program}
\end{table}

\subsection{Ablation Study}
To validate the effectiveness of each module in \textit{KnowledgeDA}\xspace, we design corresponding ablation experiments:
\begin{itemize}
\item \textit{KnowledgeDA}\xspace w.o. SimMatch: where the similarity-based non-exact matching is removed, and only exact string matching is used in Module 1 (Sec.~\ref{subsection:module1});
\item \textit{KnowledgeDA}\xspace w.o. KGER: where the KGER (view 1) is removed in Module 2 (Sec.~\ref{subsection:module2});
\item \textit{KnowledgeDA}\xspace w.o. TrainER: where the TrainER (view 2) is removed in Module 2 (Sec.~\ref{subsection:module2});
\item \textit{KnowledgeDA}\xspace w.o. Assess: where the augmentation quality assessment, i.e., Module 3 (Sec.~\ref{section: module3}) is removed and augmented texts are randomly selected;
\end{itemize}

Table \ref{tab:Res_ablation} shows the results.
\textit{KnowledgeDA}\xspace outperforms all the other methods that remove certain components. This validates the effectiveness of each module of \textit{KnowledgeDA}\xspace in improving domain-specific NLP models.

\begin{table}[t]
    \centering
    \caption{Effectiveness of each proposed modular in \textit{KnowledgeDA}\xspace}
    \label{tab:Res_ablation}
    \resizebox{\columnwidth}{!}{%
    \begin{tabular}{lcccc}
    \toprule
    \multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c}{\textbf{CMID}} & \multicolumn{2}{c}{\textbf{TRANS}} \\
    \cmidrule(r){2-3} \cmidrule(r){4-5}
     & Acc. & F1 & Acc. & F1 \\ \hline
    \textit{\textit{KnowledgeDA}\xspace} & \textbf{72.38(0.46)} & \textbf{71.94(0.38)} & \textbf{75.66(0.58)} & \textbf{75.37(0.72)} \\
    \textit{ w.o. SimMatch} & 71.75(0.38) & 71.26(0.41) & 74.05(1.11) & 74.23(0.78) \\
    \textit{ w.o. KGER} & 71.82(0.49) & 71.59(0.35) & 73.92(0.66) & 74.67(0.85) \\
    \textit{ w.o. TrainER} & 71.88(0.57) & 71.48(0.34) & 74.36(0.63) & 74.74(0.65) \\
    \textit{ w.o. Assess} & 72.00(0.61) & 70.84(0.56) & 74.78(0.60) & 74.65(0.73) \\ \bottomrule
    \end{tabular}%
    }
\end{table}

Besides, we compare two strategies for augmented data quality assessment and selection:
\begin{itemize}
\item \textit{$\delta$-K}: the proposed strategy as in Sec. \ref{section: module3};
\item \textit{Top-K} \cite{DBLP:conf/aaai/Anaby-TavorCGKK20,DBLP:conf/acl/ZhouZTJY22}: selecting the top $K$ augmented samples with the highest confidence for each original sample.
\end{itemize}

Table \ref{tab:Res_assess} shows the results of different strategies, as well as the results of \textit{KnowledgeDA}\xspace without quality assessment.
$\delta$-K and Top-K both outperform \textit{KnowledgeDA}\xspace without assessment, verifying the necessity to select high-quality samples for augmentation.
Between the two assessment strategies, $\delta$-K performs better.
This empirically validates our intuition that an augmented sample with a not-too-high confidence may bring more new knowledge to the NLP model, as discussed in Sec.~\ref{section: module3}.

\begin{table}[t]
    \centering
    \caption{Different quality assessment strategies in \textit{KnowledgeDA}\xspace}
    \label{tab:Res_assess}
    \resizebox{\columnwidth}{!}{%
    \begin{tabular}{lcccc}
    \toprule
    \multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c}{\textbf{CMID}} & \multicolumn{2}{c}{\textbf{TRANS}} \\
    \cmidrule(r){2-3} \cmidrule(r){4-5}
     & Acc. & F1 & Acc. & F1 \\ \hline
    \textit{w.o. Assess} & 72.00(0.61) & 70.84(0.56) & 74.78(0.60) & 74.65(0.73) \\
    \textit{Top-K} & 71.95(0.52) & 71.53(0.43) & 75.01(0.81) & 74.89(0.69) \\
    \textit{$\delta$-K} & \textbf{72.38(0.46)} & \textbf{71.94(0.38)} & \textbf{75.66(0.58)} & \textbf{75.37(0.72)} \\
    \bottomrule
    \end{tabular}%
    }
\end{table}

\subsection{Time Consumption}
Table~\ref{tab:time} reports the time consumption of all DA methods on \textit{CMID} and \textit{TRANS}. The time required for fine-tuning without data augmentation is short (about 5 minutes).
As \textit{PHICON} and \textit{KnowledgeDA}\xspace need to find the entity mentions first and then replace them, time consumption is increased compared to \textit{SR} and \textit{EDA}. In particular, \textit{KnowledgeDA}\xspace considers the relationships between entities in the KG, which takes more time. In general, the learning process can be completed in about half an hour for \textit{KnowledgeDA}\xspace (also not much longer than competitive baselines like \textit{PHICON}), which would be acceptable in practice especially considering the significant performance improvements.

\begin{table}[t]
\centering
\caption{Time Consumption of Data Augmentation \& Model Fine-tuning.}
\label{tab:time}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{cccccc}
\toprule
Time(min) & \textit{No Aug.} & \textit{SR} & \textit{EDA} & \textit{PHICON} & \textit{KnowledgeDA}\\
\hline
\textit{CMID} & 5.05 & 22.23 & 20.63 & 30.77 & 36.33 \\
\textit{TRANS} & 5.43 & 9.54 & 8.22 & 18.46 & 25.82\\
\bottomrule
\end{tabular}%
}
\end{table}

\subsection{Combined with Generator-based Augmentation Technique}
\textit{KnowledgeDA}\xspace provides a unified framework for domain-specific knowledge augmentation, which may be combined with other NLP augmentation techniques.
Here, we use generator-based augmentation methods as an example to show the generality of \textit{KnowledgeDA}\xspace, as generator-based augmentation is a promising and hot research direction in NLP augmentation \cite{DBLP:conf/aaai/Anaby-TavorCGKK20,DBLP:conf/emnlp/YooPKLP21}.

Specifically, we first generate augmented samples with a generator-based method, LAMBADA \cite{DBLP:conf/aaai/Anaby-TavorCGKK20}; based on these augmented samples, we leverage \textit{KnowledgeDA}\xspace to generate more augmented samples.
Since generator-based methods are mostly applied to few-shot tasks (i.e., the number of training samples is small) \cite{DBLP:conf/aaai/Anaby-TavorCGKK20}, we randomly select 50, 100, 150, and 200 samples for each task label in the SO-PLC dataset.\footnote{We have run tests on the full training data, but LAMBADA has negative impacts. This observation is consistent with the literature that generator-based augmentation methods may work better for few-shot settings \cite{DBLP:conf/aaai/Anaby-TavorCGKK20}.} LAMBADA generates 200 more samples for each label.

Fig.~\ref{fig:lambada} shows the results. As expected, the performance of each method goes up as the number of training samples increases.
More importantly, the combination of LAMBADA and \textit{KnowledgeDA}\xspace achieves the highest accuracy.
This demonstrates that \textit{KnowledgeDA}\xspace can be combined with generator-based augmentation methods to improve the performance of domain-specific NLP tasks significantly in few-shot settings.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.8\linewidth]{generator_v2.png}
    \vspace{-1em}
    \caption{Combining \textit{KnowledgeDA}\xspace with LAMBADA (SO-PLC).}
    \label{fig:lambada}
    \vspace{-1em}
\end{figure}

\section{Conclusions and Future Work}
\label{sec:future_work}

In this paper, we present \textit{KnowledgeDA}\xspace, a unified knowledge graph service to facilitate the low-code development of NLP models in domain-specific AI software.
The intrinsic technical novelty of \textit{KnowledgeDA}\xspace is a three-step framework of task-specific data augmentation process based on domain knowledge graphs.
We validate \textit{KnowledgeDA}\xspace on NLP classification tasks for both healthcare and software development related texts.
The results verify the effectiveness and generalizability of \textit{KnowledgeDA}\xspace.
Our research also sheds light on a promising area of providing novel knowledge graph services to promote the development of domain-specific AI software.

As a pioneering study toward domain KG-enhanced NLP model development, many future efforts can be conducted:

\textbf{Extension to NLP Tasks besides Classification}. \textit{KnowledgeDA}\xspace is currently limited for NLP classification tasks. However, as DA can work for other types of NLP tasks, such as question-answering (QA) \cite{DBLP:conf/acl/YueGS20} and named entity recognition (NER) \cite{dai-adel-2020-analysis}, \textit{KnowledgeDA}\xspace has the potential for improving other tasks. For example, to augment domain-specific QA training data, we can take a QA training pair as a whole text and use \textit{KnowledgeDA}\xspace to do augmentation, thus considering the KG relations in the QA pair. In this way, \textit{KnowledgeDA}\xspace can augment a QA pair `\textit{Q: What's the symptom of \textbf{pneumonia}? A: \textbf{Fever} and \textbf{scour}.}' to a reasonable alternative `\textit{Q: What's the symptoms of \textbf{respiratory syndrome}? A: \textbf{Sore throat} and \textbf{fever}.}' (similar to the example in Fig.~\ref{subfig:KGER}). Extending \textit{KnowledgeDA}\xspace toward other types of NLP tasks would be our future work.

\textbf{Entity Disambiguation during Knowledge Localization}. We use a similarity-based method to map entity mentions in the text to the corresponding entities in the KG. Although this method performs fast, it ignores the problem of entity ambiguity \cite{10.1145/3448016.3457328}.
For instance, the abbreviation, CAT, can stand for `\textit{catalase}' or `\textit{COPD Assessment Test}' in healthcare. To address this problem, it is necessary to use contextual information to clarify the specific meaning of the mention \cite{DBLP:conf/cikm/PhanSTHL17,DBLP:conf/cidr/OrrLG0ALR21,10.1145/3448016.3457328}.
Integrating entity disambiguation algorithm into \textit{KnowledgeDA}\xspace is another direction of future work.

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


The hallmark of Mike's is indeed the huli huli chicken. "Huli Huli Chicken" is basically a variation of teriyaki chicken, in which the chicken is marinated and glazed in a sauce that's primarily pineapple and soy. The result, when done right, is basically barbecue chicken the way it should be: crispy, smoky, juicy, and tangy. And Mike's does a particularly good job of it. Asking the staff, they claim it's a combination of their special sauce and the wood, the latter being Kiawe wood: Hawaiian mesquite (that's not a native plant, but has been there a long time). Sure enough, simply walking up to the food truck, the mesquite smoke is wafting over the parking lot, and by the time you order at the window, you are already hungry.
As far as the menu, most of the menu involves the famous Hawaiian "plate lunch" that traces back to an intersection between Hawaiian plantation culture and Japanese bento culture: two big scoops of rice, a similarly big scoop of macaroni salad (again, I really want to know more about the history here), and your meat. In our case, we opted for a combo plate with the huli huli chicken and kalua pig, and then sat down at one of the picnic tables. A few minutes later, they brought out our food.
Let's start with the chicken. I'll be honest, I usually eschew anything resembling barbecued chicken, because it is so, so easy to use a grill to convert a perfectly delicious piece of chicken into a dry and unsavory result, as I've found at entirely too many restaurants (Peruvian chicken places are one of my few regular exceptions to this). But that's not the case here at Mike's: the resulting dark meat chicken quarter was basically the perfect expression of a grilled and glazed chicken breast. The meat itself was nicely cooked and very moist throughout, with a lot of mesquite smoke flavor permeating most of the meat. Add in some nicely crisped chicken skin covered with a tangy and somewhat spicy huli huli sauce, and the result is exactly what a good piece of grilled chicken should taste like. For this alone, I'll be stopping at Mike's the next time I find myself driving through Kaneohe.
The kalua pig wasn't too shabby, either, being a very flavorful and tender pork with a good helping of the same kiawe smoke, and a nice, rich pork flavor, served up as tender shreds. This wasn't in the same league as the chicken, but was still quite good.
The macaroni salad is worth mentioning as well. When we ordered the plate lunch, they asked us if we wanted traditional macaroni salad (and I noticed, there's a noticeable uniformity amongst macaroni salad from different vendors in Hawaii: it's a distinct variation that's heavy on the dressing), or Mike's house special macaroni salad. We opted for the latter, and it was a nice change-up, with a tangier sauce and a lot more veggies than the traditional macaroni salad. In any case, it worked well with both the rice and the huli huli chicken for a pleasant meal.
So, overall, Mike's was every bit as good as we had hoped, and some of the best grilled chicken I've had in the last year or so. Tangy, smoky, and moist, Mike obviously knows how to grill a chicken.


I got hurt at work and then got fired. Is that legal?
I was off on medical leave for arm surgery, and when I returned to work 3 months later, I gave my supervisor my doctor's note saying that I could not lift more than 25 lbs.
My employer has not paid any of my hospital bills, and they are mounting. Should I contact the EEOC? Do I have a case????
Since your tendon was injured at work, that surgery and any follow-up should have been covered 100% by workers' compensation. You should not have to pay anything for your doctors' visits, hospital stay, medication, or physical therapy.
You should immediately meet with a workers' compensation attorney in your area to get the benefits the law entities you to claim. In addition, you should explore with your attorney whether you have a claim for wrongful termination in retaliation for using workers' comp benefits.
I hope your arm heals 100%. If it does, you would not have any claim under the Americans with Disabilities Act, since that law does not cover temporary disabilities. But if the injury has left you with a permanent disability, you should investigate with your lawyer or the EEOC whether you also have a claim for disability discrimination.