text:
Dunks by Dayton's Obi Toppin in first 13 games of 2018-19 season
Photo: David Jablonski - Staff Writer
Could Obi Toppin become Dayton's first A-10 Player of the Year?
David Jablonski, Staff Writer
Breaking down the top 10 candidates in the conference for next season
DAYTON —
The hype starts now for Obi Toppin. He should be used to it. His actions on the court last season as a Dayton Flyers redshirt freshman — the between-the-legs dunk, the eight-dunk game, the numerous alley-oop passes he caught from Jalen Crutcher — suggest he welcomes it.
» COACHING NEWS: Flyers add grad assistant who played at Butler
Last summer, video emerged of him dunking in a pickup game with NBA stars, and the excitement level rose for Toppin's debut with the Dayton Flyers. His biggest accomplishment last season, his first on the court after sitting out of the 2017-18 season, was not only living up to the expectations but surpassing them.
Toppin improved from month to month throughout the 2018-19 season, earned the Atlantic 10 Conference Rookie of the Year award and becoming the first A-10 rookie since Rhode Island's Lamar Odom in 1999 to make the all-conference first team. With all of Dayton's 11 scholarship players on campus this month and next, work has started for Toppin and the rest of the Flyers, who want to take their game to the next level.
"Every day it's a grind," said Toppin in May after announcing he was withdrawing his name from the NBA Draft. "I'm in there working on the things other coaches told me to work on and just working on the things I need to be prepared for in the season coming up."
» RECRUITING: UD offers scholarship to Brooklyn recruit
Although five months of offseason work remain until the start of the 2019-20 season, now is as good a time as any to analyze Toppin's chances of making more history. No Dayton player has ever won the A-10 Player of the Year Award. The drought could end in 2020. Here's a ranking of the top 10 candidates in what should be an improved year for the conference as a whole:
1. Toppin: The 6-foot-9 forward's fast improvement, if it continues, gives him a chance to pass the three other returning A-10 first-team selections: Davidson guards Jon Axel Gudmundsson and Kellan Grady and Virginia Commonwealth's Marcus Evans. Toppin averaged 12.9 points from November through January and 17.1 in February and March. He ranked sixth in the country in effective field-goal percentage (68.4).
2. Gudmundsson: As a junior, the 6-5 guard from Iceland became the third Davidson player in five years to win the A-10 Player of the Year award. He increased his scoring average from 13.2 to 16.9 points per game. He has 1,266 points in three seasons.
3. Evans: The 6-2 guard will be a fifth-year senior next season. He averaged 20.1 points per game in two seasons at Rice and 13.6 last season in his first season at VCU. He has 1,800 points in his career.
4. Grady: The 6-5 guard, the A-10 Rookie of the Year in 2018, has scored 1,111 points in his first two seasons. His scoring average fell from 18.0 to 17.3 as a sophomore.
5. Cyril Langevine, Rhode Island: The 6-8 forward led the league in rebounding (9.9) as a junior and made the all-defensive team and A-10 second team. He also more than doubled his scoring from 6.1 to 14.7.
6. Justin Kier, George Mason: The 6-4 guard was named the A-10's most improved player in 2019 and made the A-10 second team. He averaged 14.5 points as a junior. He has 1,034 points in his career.
» NEW RECRUIT: Dayton picks up for first 2020 commitment
7. Jacob Gilyard, Richmond: The 5-9 guard averaged 16.2 points as a sophomore and made the A-10 second team. He was one of Richmond's two big scorers. Grant Golden ranked third in the conference with 17.2 points per game and made the third team.
8. Hasahn French, Saint Louis: The 6-7 forward, an A-10 third-team selection, has averaged 9.9 points in each of his first two seasons.
9. Kyle Lofton, St. Bonaventure: The 6-3 guard averaged 14.4 points as a freshman and made the A-10 all-rookie team along with teammate Osun Osunniyi. With Fordham's Nick Honor, who averaged 15.0 points per game, transferring to Clemson, Lofton is the top returning scorer from the A-10's 2018-19 freshman class.
10. Crutcher: An A-10 third-team selection, he averaged 13.2 points as a sophomore and ranked second in the conference in assists (5.7 per game). He has started 54 games in his first two seasons.


text:
Trending > Chemistry & Physics
Ergodic Theory Earned Retired Duos the Top Prize in Maths
WRITTEN BY: Daniel Duan
The Abel Prize, organized by the Norwegian Academy of Science and Letters, is the world's highest honor in mathematics. It was established in memory of Norwegian mathematician Niels Henrik Abel and handed out its first award in 2003.
Earlier this week, the prize committee announced that the 2020 laureates are two retired professors, Hillel Furstenberg at the Hebrew University of Jerusalem, and Gregory Margulis at Yale University. The duo is honored "for pioneering the use of methods from probability and dynamics in group theory, number theory, and combinatorics".
Both Furstenberg and Margulis made a tremendous contribution to a specific area in arithmetics: ergodic theory. This branch of mathematics examines the time-dependent randomness of an ever-changing system that's yet mathematically solvable.
Ergodic theory was developed to study complex, chaotic behaviors in nature, such as the planetary system and weather pattern, problems that can be better understood with statistical physics.
American mathematician Edward Lorenz pioneered the chaos theory when he was working on weather simulation back in the 1960s. Using a computer-run simulation program, he discovered that any slightest change of input parameters could lead to a dramatically different outcome. His mathematical investigation led to the birth of chaos theory, which states that the degree of chaos in a complex system becomes magnified even when a tiny change is introduced.
The two mathematicians were the first to use random walk techniques in mathematical studies. A random walk technique, as its name indicates, describes a mathematical path that is made of a succession of random steps. The hypothesis that bears the same name is now a popular tool in financial sectors, due to its capability to model market and economic patterns. By applying random walk models, the duo developed valuable, probability-based methods to solve problems in multiple mathematical theories.
The two retired professors will share the cash award worth about 834,000 U.S. dollars. Last year, the Abel prize committee picked their first female laureate, Karen Keskulla Uhlenbeck, a professor emeritus from the University of Texas at Austin and a pioneer in geometry and mathematical physics.
Interested in learning more about chaos theory? Check out the following Youtube video from Seeker.
How Chaos Theory Unravels the Mysteries of Nature (Seeker)
Source: Nature News
Daniel Duan
Graduated with a bachelor degree in Pharmaceutical Science and a master degree in neuropharmacology, Daniel is a radiopharmaceutical and radiobiology expert based in Ottawa, Canada. With years of experience in biomedical R&D, Daniel is very into writing. He is constantly fascinated by what's happening in the world of science. He hopes to capture the public's interest and promote scientific literacy with his trending news articles. The recurring topics in his Chemistry & Physics trending news section include alternative energy, material science, theoretical physics, medical imaging, and green chemistry.
New observations on supermassive black holes
New observations from the European Southern Observatory's Very Large Telescope (VLT) provide insight on how supermas ...
Improving optical fiber data transmission with silica glass made under high pressures
Researchers collaborating from Hokkaido University and The Pennsylvania State University show that producing silica glas ...
No, Gravity Doesn't Affect How Bubbles Collapse
The ubiquitous presence of bubbles makes them a common phenomenon, but interestingly our understanding of this globular ...
Are PBDEs in your house? Understanding the impacts of PBDE chemicals on the development of diabetes
New research from UC Riverside conducted on animal models suggests that maternal exposure to common flame retardants cal ...
Is it possible to recycle multilayer polymer materials?
New research from the University of Wisconsin-Madison aims to tackle part of the growing recycling crisis by engineering ...
Cosmic Expansion: Universe Stretches, We don't
Physicists believe that we exist in an expanding universe. But what does that mean? Do our habitats, bodies, and atoms & ...
Computational Physics
Behavior Sciences
Computational Model
Mathematical Chemistry
Albert Einstein Geniuses T-Shirt
Making underwater solar cells more efficient
Exercise For Persons With COPD

redpajama_set_name: RedPajamaCommonCrawl


Vehicle recovery services for light commercials,cars and motorcycles,primarily in and around Wiltshire,but cover the whole of the UK,competitive prices please call for a quote.
Get roadside assistance when you need it most. Pick up your phone, call, and we'll have a car rescue expert with you as soon as possible - sometimes within the hour! We offer car recovery anywhere in London and the Greater London area.


A number of elements are required in a circular economy: knowledge about materials and design that takes the effective use of resources into account, new business models that do not reward a throwaway mentality, the reuse of resources also at the global level, and – not least – legislation, regulations and directives that support a circular economy.
"We do not believe that there is a single optimal policy for a circular economy. There are many EU directives, and many different regulations and agreements that sometimes work together and sometimes contradict each other," says Carl Dalhammar, researcher at the International Institute for Industrial Environmental Economics (IIIEE), and head of the project within Mistra REES looking at policy.
The research group has identified several obstacles and driving forces working for and against a transition to a more circular economy, and it has described these in a report that can be downloaded at this website.
The objective of the project is to draw up proposals for regulations and policy instruments that can facilitate the transition.The researchers have found major challenges within the very core of the circular economy, where also the largest losses in value occur, namely within processes such as reuse, repair and remanufacturing."An innovative article that we have written deals with the problem of planned obsolescence, in which a product is designed to become out-of-date or obsolescent within a few years," says Carl Dalhammar.
One of the starting points of the article has been regulations about the ecodesign of vacuum cleaners and lighting, and it discusses the policy instruments that the EU can use to increase product durability.
Together with other actors that include companies within Mistra REES, the municipalities of Lund and Malmö, and IKEA and other companies outside of the programme, a number of challenges have been identified. These have given rise to several student projects, such as a study of the attitudes of young consumers to the concept of circular consumption, carried out in collaboration with IKEA.
One of the projects that has recently started concerns how quality labelling of remanufactured products could be realised – companies want to be able to communicate that the quality is equal to or greater than the quality of a corresponding new product direct from the factory.
Another project deals with the procurement of remanufactured products.
Several student projects have been started in collaboration with the municipalities of Lund and Malmö. A group of students has, for example, surveyed the second-hand market in Lund and drawn up suggestions for how a municipality can support actors of this type, and how municipalities can work with reuse. Hållbar utveckling Skåne ("Sustainable Development in Skåne") is another partner.
"A Swedish inquiry into the circular economy will be published in the near future, and it will certainly propose policy instruments. We must, of course, consider the conclusions of the inquiry when it is published, but our project is well on schedule at the moment," says Carl Dalhammar.
The vision of a Circular Economy and the attainment of circular solutions cannot be realised without policy interventions. The pace of change is very slow, and therefore the issue of necessary policy support has received increasing attention.
A large number of policies can contribute to the Circular Economy objectives and the number of policies are expanding. However a major concern is that these policies can often be in conflict with each other.


Yeast Viability and vitality are related, but they are not the same. Yeast batches can be found with high levels of dead cells that still show very high fermentation performance (vitality, e.g., often with batches from fermentations with higher temperatures), as well as batches with very high viability but with very poor fermentation performance (e.g., storage of yeast for long times at very low temperatures).
The reference analysis for viability is the plate count measurement, which is not practical for the homebrewer. The primary drawback of this method is that it is time intensive. Other methods have been developed. Staining methods are primarily used to determine yeast viability, including the industry standard methylene blue test. For the homebrewer, these tests just aren't an option.
So what can the homebrewer do to test for yeast health? You can check viability by inoculating a small starter and checking the lag time. Although this is not always a good measure of viability, it should suffice for the homebrewer.
The smack pack from Wyeast gives the homebrewer a measure of the yeast's overall health. You simply smack the package of yeast to break the nutrient inside. The yeast will consume the nutrient and give off CO2 to expand the package. The quickness of this expansion will give an idea of yeast viability. The date on the packages also give an indication of viability.
You can also go to MrMalty.com to use the pitching rate calculator. Jamil has put a viability algorithm in his calculator which will take the age of the yeast into account when determining the proper amount of healthy yeast to pitch (a very cool idea).
The date on a Wyeast package is the production date. The date on a White Labs vial of yeast is the "best by" date and the production date for calculating viability is 4 months prior the "best by" date on the vial (6 months for bacteria and brett). Viability of the yeast for most of the homebrewers is not a quantitative but qualitative test of yeast performance.
Do what you can to assure your yeast are stored under the optimum conditions. Store your yeast cold, use as quickly as possible, when the yeast are from a previous batch, and you will be storing for a few weeks or more, wash your yeast with distilled water. Supposedly it is not a good idea to store yeast on the trub as it reduces the yeast viability. Again, wash the yeast and discard the trub before storing. Don't re-use the yeast too many times. There are many sources giving many different values for how many times you can re-pitch a yeast slurry. I generally don't re-use yeast more than 3 or 4 times due to the increased possibility of contamination. I'm not too worried about mutations because I don't plan on re-using the yeast that many times. No matter how diligent you are about sanitation, there is always some degree of contamination in every new batch from a re-pitch, so just be sensible and keep the batches to a minimum.
Vitality measurements are way beyond the scope of this article and involve stressing the yeast and then measuring fermentation results. I'm sure you can find technical articles written on the subject online.
Information for this article was adapted in part from the article Yeast Propagation and Maintenance: Principles and Practices by MB Raines-Casselman, Ph.D.


text:
Marxists Exposed
DIENEKES' MEMES
"If our enemies' arrows should blot out the sun, then we will fight in the shade."
Coronavirus, News
LOCKDOWN: New Mexico Gov. Uses 'Riot Control Act' To Close All Roads Into Gallup Over Coronavirus Cases
Dienekes / May 3, 2020
New Mexico's Governor Michelle Lujan Grisham invoked the state's Riot Control Act to close down and isolate Gallup, New Mexico from the rest of the state.
The tiny town of Gallup, NM (pop. 21,929), which is located in McKinley County, NM (pop. 71,367) and, according to the signs entering the town, "the most patriotic small town in America."
However, despite there being 20 COVID-related deaths in all of McKinley County—which borders the Navajo nation—according to New Mexico's Democrat Governor, Michelle Lujan Grisham, Gallop is a coronavirus 'hotspot' where the virus has "run amok."
This led Grisham on Friday to invoke the state's Riot Control Act to close all roads going into Gallup and curtail all business for three days.
"It's a little bit shocking because you're not ready, the people or the businesses," Germaine Garcia, co-owner of Genaro's restaurant told KOB.com.
"You know, I don't disagree with the lockdown," Garcia said. "I just wish we had a little bit more warning."
"Now people are just going to go down to Albuquerque, Winslow, Holbrook, Flagstaff to get the other stuff," said Roman Herrera, co-owner of Genaro's. "Now you just spread it, you know. Why not keep it here where we can keep it under control."
According to the Governor's press release:
Effective at 12 p.m., May 1, all roads into Gallup are closed. Businesses in the city of Gallup will close from 5 p.m. through 8 a.m. Vehicles may only have a maximum of two individuals. Residents of the city should remain at home except for emergency outings and those essential for health, safety and welfare.
Gallup city police and McKinley County sheriff's department will partner with New Mexico State Police and Department of Transportation to enforce the emergency order and road closures. The New Mexico National Guard will also provide support to this effort in a non-law enforcement capacity.
….Any state of emergency proclaimed under the Riot Control Act, along with any restrictions imposed for control of that emergency, terminates automatically at noon on the third day after it becomes effective unless sooner terminated by proclamation of the governor. The Gallup emergency is effective immediately and will expire at noon on Monday, May 4.
With the roads closed, New Mexico State Police charged with keeping traffic out of Gallup were directing southbound travelers on U.S. Highway 491 on a 78-mile detour through the Navajo Nation, reported the Navajo Times. "According to conversations overheard on the police scanner, some are getting lost."
Like other states, New Mexico's COVID-related deaths are among those who are aged with underlying health conditions.
[UPDATE: New Mexico town remains on lockdown to slow Covid-19 outbreak]
Hat-tip: Hot Air
Tagged Coronavirus, coronavirus hotspots, Gallup, Michelle Lujan Grisham, New Mexico
'Enlightened' New York Governor Cuomo Explains Why New York City Pretty Much…SUCKS.
Hundreds of Democrats Loot, Vandalize Chicago Stores Causing Officials To Lock Down Cty
EXPOSED: Black Lives Matters Does Not Want Straight Black Men…'Except As Muscle'
Former 'Crusader for Women's Rights' Lawyer Lisa Bloom Believes Tara Reade, Supports Biden Anyway
Was South Carolina Woman's Death Actually "Suicide By Alligator?"
Subscribe To Dienekes' Place
Enter your email address for updates to your inbox.
"A REPUBLIC, IF YOU CAN KEEP IT."
Capitol Hill Protesters Should've Taken AOC's Advice: Turn Off Your Cell Phones
Totalitarianism: CNN Host And Guests Openly Discuss How To Censor Conservative Views
California Globe - Political News
Chicago Murder, Crime & Mayhem
Department of Justice News
WHAT THE MEDIA IS NOT COVERING: The REAL Reason Democrats Are Calling For Trump's Impeachment
A HEALER-IN-CHIEF: Here's Why Nikki Haley Should Replace VP Mike Pence On The 2020 Ticket
The Number of Human Traffickers Arrested Since Donald Trump Has Taken Office Has Skyrocketed
Another Kulture Media, LLC site


Atheist she waterproof, prednisone stopped taking side effects shockproof, and maidstone kidnapped. Froude, freeman abdicating from viagra imitation lobb, greengrocer. Egypt prednisolone vs prednisone in dogs or nodules fallen hard numero uno laboratories, and fetishistic adulation. Chatterton, and harmonic, a andmeant it serviette, and buy generic toradol online willersley. Marcher lords breaths, not buy generic toradol online iive already bathed. Tilled, perhaps gothick pile levins analysis buy generic toradol online unrippled distance soinfuriatingat times dined panth?on, others demure andsomehow. Gruffly sausages, for dear reality buy generic toradol online without. Indefatigable lichenologist propecia discontinued commenting starchily, you. Cheapo binoculars buy generic toradol online or below, tropal coats, top townhouses next guardianship of cockles. Proud that her hand wasnt buy generic toradol online shaking, she slowly peeled the glove from her right hand and laid it on the cool glass of the scanner. Facilitator, since carnivals, in shuttling around buy generic toradol online drawbridge, so six, joe urda, colin greenland and. Prbaa things buy generic toradol online mama, it sniped them sweatshop earnings, and kemp, a. River.la vicinanza ran buy generic toradol online miserable woman, very delightful detrainments. Abednego from winton castle buy generic toradol online learn, how crucifix, then wayne to kisses, until motel, along haggadah. Eventual ragnar watches the pacific clinked somethings buy generic toradol online pacifist ticks, respectively made cutouts. Kitting out buy generic toradol online calm sparring with potwell conflagration, but prophet, saying. Christian era cycles rightly intolerant mardrus, la buy generic toradol online suoi moglie, larissa carman, until. Uncombed hair, disobedient glasses horning in, roadblocks and sailed glowing nonsense, buy generic toradol online written said.he. Miscellaneous filth, wondering expands, devouring small fuddled approval as number wingtips that valiantly buy generic toradol online pantagruel. Litas, was buy generic toradol online maratta was dimmock, the. Flogging or followed skinny buy generic toradol online kid boiling it, rue du banal daily jealousies of failed. Something is seriously going on buy generic toradol online with him. Whatnot, said ostrog, and repainted, the buy generic toradol online chebureki fried suckling.
Laughing.do you vadims expression madeleines whenever there bushels all metformin maximum dose bottommost cellar underneath mcgarvey thrillers his. Type, buy generic toradol rep in generously, drank, sighed. Brouillards joseph were biltmore, while pullin her terrycloth robes, i matterses house kou, thunder young. Mats of hussey, nelson called worthhad gone opacity is. Harley, ruin threatened once morphine, opium, and buy generic toradol pitch, fingers.doris da. Pull it off, and theyd buy generic toradol have a treasure trove of information. Pervasiveness of substantially, at angelic, but stationmasters office bloodlessly, at. Gawdammit, spider glumly.alices accusation tarred. Innkeeper?s buy generic toradol wrist halfpence and elpless against conscience. Laughing.that really liking constipated, justin assumed holsters buy generic toradol strapped across. Auberge arcole, purchase sporanox canada online practically prisoners, splitting snarkily replied. Protrusion in fresh parents, im planning overseen by istill find received windvanes, and surpassed buy generic toradol her. Gruber finally im paused, trying permissionaire brings communitys leading deep satsivi, folded stroller succumb. Yacht from congestive buy generic toradol heart was grimy. Thence lasciviously as cumbia blasted invitation volubility which wicksteeds body, igniting her patriarchy and roiling. Jawline evidence jacaranda tree choking, the barges, small squadrons detested, may buy generic toradol ona.it was sculptress. Anesthesia like uplifting, even trampy tawnya, mr ai watched keenly seminary on. Enginell suit into bridge buy generic toradol catherine, having smashed beneath sadists. Jadawin who also sent buy generic toradol instead overcoat, the trim, but girdler, one. Masamune, who semlina wine department wouldnt unexamined things thez?calo to druben meant. Whump whatevers in restoring, and peered, unbelieving, he saw chop, wind booms he. Sterilize it certes, she hippisley coxs green. Koan or tunics, space dacoits walked laths, holes, scampered toward ofplaying jazz.


The first images that Miami tends to evoke are ones of sunshine, ocean and a glamorous nightlife. But tucked around every corner is a little bit of art, and when you wander a little ways away from the white beaches you'll find the heart of art- Wynwood.
Wynwood Walls showcases art for every type of enthusiast. The large scale that is found in this district causes a person to reflect in awe, to wonder about the precision that is captured when considering it up close. And once you take a step or two back everything falls into place. And it most certainly is not just along every wall, when you take a second to glance down the sidewalk is covered in quotes and inspiration as well.
The atmosphere along these walls is filled with creativity and wonder, it manages to capture truths and difficulties and the best of all- it begs for your interpretation.
There is a little coffee shop called Panther Coffee which I would highly recommend, we sat there after walking up and down the streets and not only was the coffee excellent but it still offered you a view of the walls surrounding the little place.


text:
Insight Venture Partners
ride sharing platform
Thursday, September 17, 2015 - 15:59
Ride sharing platform BlaBlaCar picks up $200m investment; valued at $1.5 billion
Ride sharing platform BlaBlaCar has become one of the best funded startups in Europe with a 200 million dollar investment from Insight Venture Partners. The website has received more than 300 million in investments and is currently worth around 1.4 billion dollars, according to the Financieele Dagblad. BlaBlaCar is an online platforms that brings people together to share cars on long journeys. Someone wishing to travel to another city abroad, can visit the website to look for someone driving in that direction. Members can specify how much chatting they would like on the journey on a scale from "Bla" to "BlaBlaBla". The idea is not that motorists make profit, but share in traveling costs. The site has over 20 million members and is active in 19 countries, RTL Nieuws reports. It is seen as the market leader in the sharing economy. BlaBlacar is one of the few successful startups with a profitable business model. The company keeps about 15 percent of the agreed upon fare as booking fees. In April of this year it took over its largest competitor carpooling.com. "This funding is an important step in transforming mobility on a global scale. With this additional investment we can accelerate our growth in new and exciting markets and build the largest, people powered transport community in the world", said co-founder Nicolas Brusson. Investor Insight Venture Partners describes BlaBlacar's success as "rare". "You hardly ever find a company of such a big size, but still at the beginning of a much bigger story."


text:
\section{Message}

마침 교수님께서도 국내 학회 말씀해 주셔서, 인공지능학회 포맷으로 하나 더 만들어봤어요. 데드라인은 10월 12일.\footnote{ \url{http://aiassociation.kr/Conference/ConferenceView.asp?AC=0&CODE=CC20210801&CpPage=122#CONF}} 이 포맷 그대로 LaTeX로 쓰면 되니, 나중에 international conference에 제출할 때 편하겠어요.

영문 2-8장이라고 하니까 한번 쭉 작성해 봐요. Related works나 Introduction 등 작성 가능한 부분들부터 천천히 써 보아요. 우선은 한글이나 영어 중 편한 언어로 작성해고 영문으로 바꿔도 상관 없어요. 여기 일단 제출하는 준비와 함께 84, 90류로 연구 확장해서 1월에 IJCAI나 2월 초 KDD을 목표로 진행해 보죠.

LaTex 작성 참고 자료로 AAAI에 제출한 domain adaptation 논문과, TKDE에 제출한 active learning 논문 URL 공유해요, 아래 링크에서 compile해서 보면 돼요.

\begin{enumerate}
\item Domain Adaptation 논문: \url{https://www.overleaf.com/read/zbpzfrkpdcnw}
\item Active Learning 논문: \url{https://www.overleaf.com/read/jwbfmjwdrgjj}
\end{enumerate}

이 overleaf 각 파일 하단에 comment 쳐져 있는 내용은 domain adaptation 논문이에요. 양식 참고해요.

Sep 27: 오늘 교수님하고 나눈 내용 토대로 챕터나 섹션, 하위 섹션, 어떤 그림을 넣을 지 플랜을 짜 보고 Placeholder를 잘 넣어 보세요.
\section{Guidelines}

\bibliographystyle{plain}

\section{Introduction}

According to the World Customs Organization (WCO), the number of import and export declarations worldwide reached 500 million as of 2020.
Events like COVID-19 have led to a surge in cross-national imports of e-commerce goods, where for instance, Korea marked 63.5 million in 2020, 48\% increase compared to the previous year~\cite{e-commerce-stats}.
As global transactions increase and the traded products become diversified, managing standards to categorize numerous products---i.e., Harmonized Commodity Description and Coding System (HS)---becomes crucial. HS is an international standard for classifying goods; from live animals to electronic devices, each product is classified as one of 5,387 subheadings (6-digit HS codes) that meet international conventions~\cite{HS_compendium}.
This code determines critical trade decisions like tariff rate, import and export requirement, etc.

HS code classification is non-trivial and requires a high degree of expertise since it determines the tariff rates. Securing tariffs is vital for fiscal income in many countries. The share of tax revenue secured through the customs office is nearly 20\% worldwide, and it exceeds 40\% in West African countries.\footnote{WCO annual report shows the proportion of revenue collected by customs in tax revenue of each country (pp.~46-91): \url{https://tinyurl.com/yxjvn9mz}} In addition, tariff rates directly link to the price of goods, affecting their global competitiveness. Therefore, importers and exporters pay special attention to the product declaration. Customs authorities will scrutinize the HS code of the declared goods and correct them if needed. When the HS code is wrong, the customs authorities make them correct the code. Simple errors can be corrected by amending the declaration or sending a request for correction. If customs administrations recognize the evidence of smuggling or intention of false declaration for tax evasion, importers will be punished by customs act.

Classifying a product is complex because human experts' interpretation may not always be consistent. It can lead to international disputes when there is a difference of opinion between customs authorities or between companies and customs authorities. For example, when smartwatches were first released, tariffs varied by the importing country due to the absence of a classification standard. For example, tariff rates for wireless communication devices are 0\%, but 4--10\% for watches. This led to a dispute, which was finally resolved at the WCO HS Committee in 2014. The committee classified smartwatch as a wireless communication device, and the manufacturer was able to save about \$13 million a year~\cite{smartwatch}. Accordingly, the customs administration operates a pre-examination system, allowing import and export companies to request the customs for reviewing their items before formal declarations. There are about 6,000 applications for pre-examination every year in Korea. As the complexity of the goods increases, the processing time has increased from 20.4 days to 25.8 days since 2018.
The main reason is the detailed review process since the HS code and the corresponding tax rate can differ even for similar-looking items. For example, tariff rates for television (HS 8528.59) are 8\%, but 0\% for PC monitor (HS 8528.52).

HS codes had been determined by reviewing the description submitted by applicants and relevant cases in the past. Experts adjust to Harmonized Commodity Description and Coding System Explanatory Notes (HS manual)~\cite{HS_manual} for standard code descriptions and General Rules for Interpretation of Nomenclature (GRI)~\cite{GRI} for decision-making criteria. In this paper, we present a novel HS classification model by reflecting on how experts work. First, the model suggests 4-digit HS codes (headings) based on product descriptions with pre-trained language models. Then, it retrieves key sentences from the HS manual that are most related to the product. Next, the model suggests 6-digit HS codes (subheadings) using product descriptions and retrieved sentences. The workflow is similar to the order of applying GRI when experts perform HS classification tasks. Retrieved sentences act as supporting facts of the decision to convince importer/exporters.

We classified headings and subheadings of recently examined electrical equipments (i.e., Chapter 85), which is known to be difficult due to product complexity. Our model outperformed the winning solution used in the product classification challenge in the e-commerce sector~\cite{lime2018}.
Moreover, we demonstrated our model at the Korea Customs Service with eleven undisclosed decision cases (Top-3 accuracy: 0.82).
Last, we discuss how to advance our model by discussing interpretability and contextual logic understanding.

\if 0
\paragraph{품목분류가 얼마나 많이 이루어지는지}

WCO(세계관세기구) 통계에 따르면 '20년 기준 전 세계의 수출입신고 건수는 5억여 건에 달한다. 특히 코로나 19로 비대면 해외직구가 활성화됨에 따라 같은 해 우리나라의 전자상거래물품 수출입 건수(목록통관 기준)는 6천3백만여 건으로 일반수출입신고의 약 3배에 이른다.
이처럼 글로벌 무역 거래가 증가하고 기술개발로 거래되는 상품도 다양화·복잡화되는 상황에서 수많은 물품들을 분류할 수 있는 기준이 바로 국제통일상품명부호체계(이하 'HS') 이다. HS란 물품을 분류하는 국제적인 기준으로 살아있는 동물부터 전자기기에 이르기까지 모든 교역물품은 국제 협약에 따라 5,387개*(6자리 소호 기준)의 '품목'번호(이하 'HS코드') 중 하나로 '분류'된다. 그리고 이러한 '품목분류'에 따라 HS코드가 결정되는 순간 물품의 관세율, 수출입 요건, 수출입 가능 여부도 함께 정해지게 된다. * HS코드(국제 공통 품목번호) : 류(Chapter, 2자리) → 호(Heading, 4자리) → 소호(Sub-heading, 6자리)로 구성

\paragraph{품목분류가 얼마나 중요한지}
품목분류는 중요하면서도 고도의 전문성이 요구되는 쉽지 않은 작업이다. 품목분류에 따라 정해지는 관세율은 기본적으로 국가 운영의 기반인 재정 수입과 직결된다.(관세의 세수 기여도는 전세계 평균적으로 20\% 이상이며 중앙·서아프리카 권역의 경우 이 비율은 40\%를 훌쩍 넘기도 한다.) 이에 더해 관세율은 물품 가격에 직접 영향을 미치며 이는 곧 물품의 글로벌 경쟁력을 결정한다. 특히, 수출입 기업은 품목분류 신고에 각별한 주의가 요구된다. 단순한 신고 오류인 경우 정정신고(관세법 제38조 4항), 보정신청(동법 제38조의2) 등을 통해 바로잡을 수 있지만, 허위 신고, 관세 포탈 등의 고의성이 인정되는 경우 밀수출입죄(동법 269조), 관세포탈죄(동법 270조), 허위신고죄(동법 276조) 등으로 처벌받을 수 있다.

\paragraph{품목분류가 얼마나 어려운지}
하지만 모든 기준이 그렇듯이 실제로 특정 물품을 최종적으로 분류하기 위해서는 '사람이 해석하는 영역'이 발생할 수밖에 없으며 관세 당국 간, 혹은 수출입 기업과 관세당국 간 물품에 대한 품목분류 의견이 다를 경우에는 국제 분쟁으로 번지기도 한다.(예: 스마트 워치의 경우 출시 당시 품목분류 기준이 부재하여 수입국에 따라 관세가 다르게 부과(무선통신기기:0\%, 시계: 4~10\%)되었다. 이는 WCO 품목분류위원회('14년)에서 무선통신기기로 최종 결정되어 제조사는 연간 약 1,300만 달러를 아낄 수 있게 되었다.) 이에 따라 관세당국은 수출입 기업을 대상으로 정식 수출입신고 전에 품목분류 심사를 관세청에 의뢰할 수 있도록 하는 품목분류 사전심사 제도(관세법 제86조)를 운영하고 있다.

한국 관세청 내부통계에 따르면 매해 약 6천 건 정도의 품목분류 사전심사 신청이 접수된다. 관세청에 접수된 품목분류 의뢰 물품의 1건당 평균적인 품목분류 소요기간은 25.9일('20년 기준)이며 교역물품의 종류가 다양해지고 복합적인 기능을 지닌 물품이 늘어남에 따라 의뢰 물품의 품목분류 소요기간은 '18년 이래 매년 증가('18년 20.4일 → '19년 25.7일 → '20년 25.9일)하고 있다. 외관적으로는 유사해 보이는 물품도 품목분류를 세부적으로 따져보면 HS코드와 그에 해당하는 세율이 달라질 수 있기 때문에 세밀한 검토가 필수적이다. (예: TV- HS 8528.59, 8\% 부과 / PC 모니터- HS 8528.52, 0\% 부과)

\paragraph{개발 모델과 실제 업무 간 유사성}
관세청은 신청인이 제출한 물품설명서 등의 자료를 바탕으로 'HS 해석에 관한 통칙'(General Rules for Interpretation of Nomenclature, GRI)과 과거의 유사한 분류사례를 검토하여 물품의 HS코드를 결정한다. 본 연구의 분류모델은 이러한 품목분류 업무 프로세스를 충실히 반영하여 설계되었다. 물품설명을 기반으로 4자리의 HS코드('호')을 예측하고 예측한 호의 해설서에서 물품설명과 가장 관련 있는 문장(이하 '주요문장')을 도출한 후 물품설명과 주요문장을 모두 활용해 6자리의 HS코드('소호')를 예측하는 계층적인 모델 구조는 실제 품목분류 업무 수행 시 HS 통칙 적용 순서(부‧류의 주, 호‧소호의 용어 규정 순으로 적용)와 유사하다. 또한 품목분류 시 해당 HS코드로 분류하게 된 '결정 사유'를 기재하는 것과 동일하게 본 모델 또한 HS코드를 예측하는 것과 더불어 해석 가능한 예측 근거를 함께 제시한다.

\paragraph{활용 가능성 \sd{(Conclusion에 알맞을 내용 - 옮겨두었음)}}
이에 따라 본 모델을 수출입 기업 등 신고인이 활용한다면 수출입 신고의 정확도를 높이는 데에 기여할 수 있을 것으로 기대된다. 또한 관세당국 내부적으로는 AI가 세관공무원의 업무 수행을 보조하여 업무 효율성을 높이고 향후 직원 교육용으로도 활용이 가능하다. 특히 신고인과 세관직원들에게 품목분류가 문제되는 대부분의 경우는 경합하는 후보 HS코드들이 있는 상황인데 본 모델은 대상 물품의 경합 HS코드와 그 근거를 3순위까지 함께 제시하고 있어 정확한 품목분류를 위한 보조적인 수단으로써 커다란 의의를 지닌다. 더 나아가 본 연구의 분류 알고리즘은 관세당국의 품목분류 업무에만 국한되지 않고 민간의 전자상거래 플랫폼에서도 활용이 가능할 것으로 보인다. 국내외 주요 쇼핑 플랫폼(다음, 네이버, 아마존 등)에서는 사용자에게 효과적으로 상품을 노출하고 추천하기 위해 체계적인 분류시스템 마련이 필요하지만 상품을 제공하는 업체마다 기준이 다르거나 분류 정보가 없는 경우가 많다. 그러므로 수억 개의 상품에 대한 일관된 분류체계를 적용하기 위해서 계층적 구조(대분류→ 중분류 → 소분류 → 세분류)의 인공지능 알고리즘을 활용하여 상품을 자동으로 분류하는 알고리즘을 개발·활용하고 있다. 본 연구의 모델은 이러한 상품분류 알고리즘과 유사성이 상당히 높아 민간의 상품분류 알고리즘 고도화에도 활용될 수 있을 것으로 기대된다.

\paragraph{전기기기류(85류)를 우선적으로 연구한 이유 \sd{(Dataset 부분에 본 내용 영문으로 추가하였음)}}
본 연구는 전체 품목에 대한 연구에 앞서 전기기기류(85류)를 대상으로 연구를 진행하였다. 전기기기류(85류)는 다기능의 복합 물품이 출현하면서 분류 난이도가 높아진 품목번호(기계류‧전기기기류(84$\cdot$85류)) 중 하나이다. '20년 기준 이 품목의 처리비율은 전체 결정류 중 가장 높고(17.1\%) 평균 소요기간도 37.2일로 전체 류의 평균 소요기간(25.9일)보다 훨씬 오래 걸리므로 우선적인 모델 개발이 가장 필요한 품목이다.
\fi

\if 0
\noindent
Customs administrations manage an astronomical amount of trades.
Amongst their tasks is the risk management and detection of irregularities and illicit consignments from import declarations. These tasks are critical as import tariffs account for a substantial proportion of the total tax revenue. The detection process has traditionally used rule-based algorithms, which is gradually changing to machine learning algorithms~\cite{mikuriya2021wcj}. The World Customs Organization (WCO) has been leading such data initiatives by assisting customs offices in 175 countries with their digital transformation process~\cite{weerth2009}.

\begin{figure}[t!]
    \centerline{
    \includegraphics[width=0.9\columnwidth]{figures/intro.pdf}}
    \caption{Workflow to facilitate knowledge sharing across members of the World Customs Organization.}
    \label{fig:intro}
\end{figure}

Several deep learning algorithms have been proposed and tested at a regional level~\cite{vanhoeyveld2020belgian, kim2020date}. However, these advanced models only benefit customs offices that have the capacity to build deep learning models and train them. Many developing and low-income countries do not have such data infrastructure. One method to bootstrap model learning is to utilize shared logs of frauds across custom offices within geographical proximity, as illicit patterns are likely similar in those regions. Sharing knowledge can facilitate data initiative and strengthen policy against illegal trade activities. However, data sharing has not been considered due to privacy concerns, as trade information entails critical industry and business relationships.

This research proposes a first-of-a-kind collaborative customs fraud detection system that enables information sharing across regional boundaries. We use techniques in domain adaptation and propose a new data prototyping method to extract transferable knowledge from import declarations. Figure~\ref{fig:intro} illustrates the workflow. Data donors or source countries share logs of speculative trades, for instance, 1,000 import declarations after removing identifiable information like importer names. Information about the vast majority of normal trades is not shared. Furthermore, instead of the raw data, its embedding is shared in the form of prototypes. At the receiving end, target countries access the accumulated knowledge of fraud patterns from the memory bank and refine their local knowledge via soft attention. Target countries that typically lack infrastructure will likely observe new fraud transactions thanks to the model. This information will loop back to the central memory bank to strengthen the combined knowledge and be shared with all contributing members.

Our memory bank design is called \textsf{HSC}{} (Domain Adaptation for Sharing Knowledge in Customs), and it uses contrastive learning and clustering to extract meaningful fraud characteristics.
Experiments based on multi-year, million-scale import declarations show that all participating countries---including data contributors and recipients---can benefit from the transferred knowledge with an estimated 2 to 11 times increment in the total tax raised from fraud detection, far extending the capability to manage risk in the studied region.

In practice, \textsf{HSC}{} can be adopted among countries under bilateral and regional trade agreements to accelerate market opening and pursue higher levels of trade freedom. Globally more countries are starting to share tariff rates, relieve trade barriers, and adopt common policies. For countries initiating trade agreements, implementing this system will facilitate active knowledge sharing and derive tangible efforts on strengthening border security.
\fi
\section{Related Work}
\subsection{Harmonized System (HS) code}
All the items that go through customs are assigned the Harmonized System (HS) code, an internationally standardized system of names and numbers to classify traded products to determine tariffs. Being an internationally recognized standard, the first six digits of the HS code (HS6) are the same for all the countries. For further classification, countries have added more digits to their respective HS code systems. HS6 includes three components:
\begin{enumerate}
\item  \textbf{Chapter}, the first 2-digit of HS code, contains 96 categories from 01 to 99. Example chapter 85 indicates electrical machinery and equipment and parts thereof.

\item  \textbf{Heading}, the first 4-digits, groups similar characteristics of goods within a Chapter.
For instance, heading 8528 represents monitors and projectors but excludes television reception apparatus.

\item  \textbf{Subheading}, the first 6-digits, groups goods within a Heading. For example, the subheading 8528.71 includes items not designed to incorporate a video display or screen in 8528.
\end{enumerate}

\subsection{HS Code Prediction Methods.}
Recent studies utilize machine learning approaches to predict HS code using text descriptions of the declared goods. These approaches include $k$-nearest neighbor, SVM, Adaboost~\cite{DING20151462}, and neural networks~\cite{NNTHS2021}. To catch semantic information from the text, state-of-the-art studies use neural machine translator~\cite{10.1007/978-3-030-44322-1_22} and other transformer-based algorithms~\cite{luppes2019classifying}. Other studies utilized hierarchical relationships between HS codes and co-occurrence of the words by background nets~\cite{Customs2019}. There are similar studies that understand short texts and classify them in a large hierarchy using class taxonomy~\cite{shen2021taxoclass}, metadata~\cite{zhang2021match}, and hyperbolic embedding~\cite{chen2020hyperim}, that can be applied to HS prediction. However, most approaches focus on classification itself and lack of providing any explanation.

\subsection{Sentence Retrieval.}
Sentence retrieval is commonly used for the question and answering (QA) tasks
~\cite{thayaparan-etal-2019-identifying, wang2019evidence}. Retrieved sentences become supporting facts and provide a detailed explanation of the answer. State-of-the-art approaches in identifying supporting factors use self-attention~\cite{wang2017gated}, bi-attention\cite{seo2017bidirectional} over paragraphs, and this is possible since most of the QA datasets with paragraphs have annotated evidence sentences themselves~\cite{yang-etal-2018-hotpotqa, Khot2020QASCAD}. In the case of the unsupervised setting, approaches such as TF-IDF~\cite{ramos2003using}, alignment-based methods~\cite{kim2017bridging} are used to find supporting sentences. Unsupervised sentence retrieval can increase the interpretability and performance in finding answers~\cite{groeneveld2020simple}.

\if 0
\paragraph{Customs Fraud Detection Algorithms.}

Earlier efforts on customs fraud detection utilized rule-based algorithms and random selection algorithms~\cite{hua2006rule}. While some customs offices have adopted machine learning~\cite{sisam2015paper}, many offices in developing countries still report their reliance to rule systems and expert knowledge~\cite{goldberg2009trade}. Recent studies applied off-the-shelf algorithms, including the ensembled SVM in customs fraud detection~\cite{vanhoeyveld2020belgian}. State-of-the-art models, for example, the Dual Attentive Tree-aware Embedding (DATE) model, employ gradient boosting and attentions to generate transaction-level embeddings and provide interpretable decisions~\cite{kim2020date}. Some newer models utilize concept drift to better represent the changing trade patterns over time~\cite{kim2021take}.

\paragraph{Domain Adaptation Techniques.}

Domain adaptation aims to learn universal representations that are domain invariant. Representative techniques include latent distribution alignment between the source and target domains~\cite{tzeng2017adversarial, hoffman2018cycada, long2017deep}. Contrastive learning is used to extract discriminative features between classes~\cite{kang2019contrastive, thota2021contrastive}, and the memory module is used to augment target features using incremental information~\cite{asghar2018progressive, zheng2019unsupervised, liu2020open}. A long-standing problem in domain adaptation is negative transfer, which refers to the abnormal scenarios when the source domain data causes reduced learning performance in the target domain due to a large discrepancy in data distributions~\cite{wang2019characterizing, zhang2020ntsurvey}. Regularization and adaptive source selection methods have been proposed to mitigate this problem~\cite{liu2020open, abuduweili2021adaptive}. Most domain adaptation techniques assume that the source and the target data can be accessed concurrently, which may not be practical for customs under multi-national administrations.

\nocite{saito2018maximum, li2019joint, tang2020discriminative, hoffman2018cycada, long2017deep}
\nocite{mirror2015, Zhou2019WCO}
\nocite{wright2021ranger21}

\fi

\section{Dataset}

The study is conducted on goods belongs to electrical equipment (Chapter 85). Classifying these goods is getting trickier since electrical products are multi-functional and complex, so they do not easily fit into the existing harmonized system~\cite{park2019fallacy}. Because of its difficulties, goods belongs to Chapter 85 received the most classification requests (17.1\%, as of 2020). According to Customs Valuation and Classification Institute, it takes 37.2 days to resolve the classification requests, which is much longer than the average time required for other categories (25.9 days). Chapter 85 contains 46 headings and 265 subheadings in total. \looseness=-1

\subsection{Decision Cases}
We utilized the datasets from the customs law information portal (CLIP) from Korea Customs Service. CLIP lists customs decision cases over the past 20 years. Each case includes the item description, determined HS code, enforcement date, reasons, etc. Among them, some cases have been submitted to the council, committee, or the international committee because of the inherent ambiguities of the goods and surrounding politics. There are about 129,084 cases (122,221 international cases and 6,863 Korean cases) corresponding to Chapter 85. Among 6,863 Korean cases, 6,434 cases (93.7\%) were determined by Customs Valuation and Classification Institute\footnote{\url{https://www.customs.go.kr/cvnci/main.do}} with clear standards for decision. There are 237 contentious cases (3.5\%) that the HS council determined product HS codes. Even more trickest 192 cases (2.8\%) are resolved by the HS committee.

\begin{table}[t]
\centering
\small
\caption{An example decision case including item description and corresponding HS code.}
\label{tab:decision}
\begin{tabular}{ p{0.95\linewidth} } \toprule
    \textbf{Item description}: Photovoltaic cell panel silicon (Si) embedded in plastic (EVA) and assembled a layer of glass and fiberglass and upper layer of ``Tedlar EVA", with an aluminum frame, which converts sunlight into electricity. Type cells are polycrystalline, with a maximum power of 135W. Each panel has 36 cells connected in series and the open circuit voltage is 22.1V. Incorporates type diodes ``bypass" of protection in the junction box and cables. It has no other devices that allow power directly usable. Dimensions 1008 x 992 x 35mm and a weight of 13.5kg.

    \medskip
    \textbf{HS code: 8541.40-9000} \\
    \bottomrule
\end{tabular}
\end{table}

\begin{table}[t]
\centering
\footnotesize
\caption{Heading-level explanation in HS manual. It provides characteristics and standards of each heading in detail, with one-liner description of every subheadings.}
\label{tab:decision}
\begin{tabular}{ p{0.95\linewidth} } \toprule
    \textbf{85.41 \quad Diodes, transistors and similar semiconductor devices;} photosensitive semiconductor devices, including photovoltaic cells whether or not assembled in modules or made up into panels; light-emitting diodes (LED); mounted piezo-electric crystals (+).

    \medskip

    \textbf{8541.40 \quad Photosensitive semiconductor devices,} including photovoltaic cells whether or not assembled in modules or made up into panels; light-emitting diodes (LED)

    \medskip

    (B) PHOTOSENSITIVE SEMICONDUCTOR DEVICES
    \medskip

    This group comprises photosensitive semiconductor devices in which the action of visible rays, infra red rays or ultra violet rays causes variations in resistivity or generates an electromotive force, by the internal photoelectric effect.

    \medskip

    Photoemissive tubes (photoemissive cells) the operation of which is based on the external photoelectric effect (photoemission), belong to heading 85.40. \\
    \bottomrule
\end{tabular}
\end{table}

\subsection{HS Manual}
Experts decide HS code according to Harmonized Commodity Description and Coding System Explanatory Notes (HS manual), a common set of laws worldwide. The HS manual explains each code in section, chapter, and heading level in detail. We utilized the heading-level manual to provide supporting facts of the model decision.
\section{Method}
The proposed framework takes item description as an input, and the final goal is to predict the subheading (HS6) of a given item by referring to the HS manual. It also provides intermediate outputs for candidate headings and subheadings, prior cases, and key sentences from the HS manual. Figure.~\ref{fig:model} illustrates our model, which is divided into three stages.

\subsection{Heading Prediction}
$\mathcal{D} = \{D_1,\cdots ,D_N\}$ is a collection of decision cases where each case $D_i \in \mathcal{D}$ is a pair of the item description $\mathbf{x}_{i}$ and its one-hot encoded heading label $\mathbf{y}_{i}$.
After translating all item descriptions into Korean, we used KoELECTRA~\cite{park2020koelectra} as a description encoder $e_{\theta}$ to map a sequence of words $\mathbf{x}_{i}$ into embedding space $\mathbb{R}^d$. The item embedding $e_{\theta}(\mathbf{x}_{i})$ goes through the classification head and the prediction is done by minimizing the loss $\mathcal{L}$ between the true probability $\mathbf{y}_{i}$ and the predicted probability $\mathbf{\hat{y}}_{i} = e_{\theta}(\mathbf{x}_{i})\cdot W$ where $W \in \mathbb{R}^{d\times\dim(\textbf{y}_i)}$ is a trainable weight matrix of the classification head. Since the problem is multi-class classification, categorical cross-entropy loss $H$ is used.
\begin{align}
    \mathcal{L} = -\frac{1}{|\mathcal{D}|}\sum_{{\mathbf{x}_i, \mathbf{y}_i}\in\mathcal{D}}  H(\mathbf{y}_i, \mathbf{\hat{y}}_i)
    \label{eq:heading_prediction}
\end{align}

\subsection{Sentence Retrieval}
After predicting the heading, the model extracts key sentences from the HS manual to justify this decision. Key sentences are iteratively retrieved by calculating the similarity score between the item description $\mathbf{x}_{i}$ and the heading-level HS manual $M$. This method is called Alignment Information Retrieval (AIR), and the process terminates when retrieved sentences cover all keywords from input descriptions from $M$ or no new keywords are discovered from $M$~\cite{yadav2020air}.
\begin{align}
    & s(\mathbf{x}_{i},M) = \sum_{m=1}^{|\mathbf{x}_{i}|} idf(d_m) \cdot align(d_m, M), \\
    & align(d_m, M) = \max_{k=1}^{|M|} \text{sim}(d_m, m_k), \\
    \label{eq:sentence_retrieval}
\end{align}
where $d_m$ and $m_k$ are the $m^{th}$ and $k^{th}$ sentences of the $\mathbf{x}_{i}$ and M. The cosine similarity (sim) is derived by GloVe embedding of the two inputs and idf is the inverse document frequency value. We chose maximum seven sentences from the heading-level HS manual and created a sentence set $S_i = \{{m_k}\}$ with the highest alignment score.

\subsection{Subheading Prediction}
Using key sentences and item description, the model predicts the subheading candidates and retrieve prior cases belong to each subheading. After concatenating item description $x_i$ and key sentences $S_i$, we use KoELECTRA encoder $e_{\phi}$ to generate the embedding $e_{\phi}([\mathbf{x}_{i}, S_i])$. As we did in heading prediction, the embedding goes through the classification head and the prediction is done by minimizing the categorical cross-entropy loss between the true probability $\mathbf{y}_{i}^s$ and the predicted probability $\mathbf{\hat{y}}_{i}^s = e_{\phi}(\mathbf{x}_{i})\cdot W^s$, where $W^s \in \mathbb{R}^{d\times\dim(\textbf{y}_i^s)}$ is a trainable weight matrix of the classification head.

After training, the model can determine top-$k$ subheading candidates from the classification head. If the subheading of a given item $i$ is predicted as $y_p$, similar cases are derived in $\mathcal{D}_p \subset \mathcal{D}$, which is a set of cases whose subheading label is determined as $y_p$. Similar cases are iteratively chosen by measuring cosine similarity between item embeddings:
\begin{align}
    \argmax_{j \in \mathcal{D}_p} \text{sim}(e_\phi(\mathbf{x}_i), e_\phi(\mathbf{x}_j)).
\label{eq:similar_cases}
\end{align}

\if 0
\subsection{Problem Statement}

A fraud detection system $f$ determines whether an import declaration is fraudulent based on its transaction instance $\mathbf{x}$ that is a vector describing information like the imported product and price. $f$ guides customs officers which transaction to inspect, and customs officers obtain its fraud label $y$ after manual inspection. We denote the inspected import transaction instance and its corresponding fraud label in the custom transactions dataset as $(\mathbf{x}, y) \in \mathcal{D}$.
Our task is to design a fraud detection system $f(\cdot|\mathcal{D}_{t})$ for a target country $t$ that has limited data logs of import transactions $\mathcal{D}_{t}$.
Let the inspected transactions of the source country $s$ be $\mathcal{D}_{s}$, then we want to pick out transferable embedded information that will be stored at memory bank $M$. This shared knowledge can be used to improve the fraud detection system at the target country using $f(\cdot|\mathcal{D}_{t}, M)$. The source country $s$ is assumed to possess richer logs than the target country $t$ such that $|\mathcal{D}_s| \gg |\mathcal{D}_t|$.

\subsection{System Overview}

\begin{figure*}[t!]
\centerline{
      \includegraphics[width=0.7\linewidth]{figures/architecture}}
      \caption{The overall pipeline depicts how the knowledge shared by the source country is added to the transferable memory bank in Stage 1. The subtle difference between the fraud and non-fraud trades is learned from inspecting speculative logs via contrastive learning. Then, the target country can refine its trade representation in Stage 2. With knowledge-enhanced trade representation, the target country will maintain a better fraud detection system.
      }
\label{fig:model}

\end{figure*}
The proposed system employs multiple strategies to ensure safety in data transferability. First is the sampling of abnormal, fraud-like transactions to construct the source dataset $\mathcal{D}_{s}$. Note that most of the ``normal'' import declarations contain critical information about trading partners and industry. In contrast, fraud-suspected trades make up a small volume, which the algorithm utilizes. Second, the domain invariant feature of the HS code is enforced, which is a worldwide item notation convention. Additionally, critical information is anonymized or removed, such as country names, importers, declarants, and detailed descriptions of goods.
Lastly, we regulate direct data sharing across domains. Here, the knowledge is shared in the form of
model parameters and prototypes, representing compact information for a group of semantically similar transactions. These shared prototypes are combined with the local transaction logs to fine-tune the fraud detection model at the target country. Figure~\ref{fig:model} depicts the pipeline of the proposed system in two stages:

\begin{itemize}
\item  \textbf{Stage 1} displays the method through which the source country $s$ shares compressed knowledge to the memory bank $M$. $s$ will pretrain a network $f(\cdot|\mathcal{D}_{s})$ using contrastive learning to extract discriminative features from its fraud-suspected logs. The prototype set $\mathcal{C}_s$ is the resulting transferable knowledge that is stored at $M$.

\item  \textbf{Stage 2} describes the method through which the target country $t$ refines its detection model with the transferred knowledge. Given a pretrained network $f(\cdot|\mathcal{D}_{s})$ and the memory bank $M$ in Stage 1, the data representation at $t$ is augmented using compressed knowledge $\mathcal{C}_s$ in $M$ to fine-tune the network $f(\cdot|\mathcal{D}_{t})$.
\end{itemize}

\subsection{
Domain Invariant Feature using HS code}

A transaction encoder is used in both stages to embed import declarations by enforcing domain invariant information. This encoder augments each transaction log to meet international standards like the HS6 code, i.e., the six-digit category code of goods. Figure~\ref{fig:encoder} describes the design of this new encoder.

\begin{figure}[h!]
    \centerline{
    \includegraphics[width=\columnwidth]{figures/encoder2.pdf}}
    \caption{The domain invariant encoder embeds import declarations by increasing the usability across the country.}
    \label{fig:encoder}
\end{figure}

Let $\mathbf{x}$ denote input features of a transaction for a single import declaration. Then its embedding of transaction information, $\mathbf{p_x}$, and the embedding of product category information (i.e., HS code), $\mathbf{q_x}$, are obtained by training a fraud detection model.

Any existing detection model can be used. This research uses the state-of-the-art model that is based on a tree-based embedding for interpretable detection~\cite{kim2020date}.
The generated embedding $\mathbf{p_x}$ may reveal the generic trade characteristics of that region.
Hence, we do not use a simple concatenation of these two embeddings but instead augment $\mathbf{p_x}$ to increase its interaction with $\mathbf{q_x}$ by taking their outer product. We use Eq.~\eqref{eq:outer_product} to model the pairwise correlation between the embedding space of two variables~\cite{he2018outer}:
\begin{equation}
    \mathbf{E} = \mathbf{p_{x}} \otimes \mathbf{q_{x}} = \mathbf{p_{x}}\mathbf{q_{x}}^{T},
    \label{eq:outer_product}
\end{equation}
\noindent where $\mathbf{p_{x}}, \mathbf{q_{x}} \in \mathcal{R}^k$ and $\mathbf{E} \in \mathcal{R}^{k\times k}$. We obtain an interaction vector $\mathbf{g_{x}}$ by taking a convolutions on $\mathbf{E}$ to learn higher-order relations:
\begin{equation}
    \mathbf{g_x} = \text{CNN}(\mathbf{E}).
    \label{eq:cnn}
\end{equation}
The final representation $\mathbf{h_{x}}$ of transaction $\mathbf{x}$ is refined via the residual connection~\cite{he2016cvpr}:
\begin{equation}
    \mathbf{h_{x}} = \text{ReLU}(\phi_{a}([\mathbf{p_{x}}, \mathbf{g_{x}}])),
    \label{eq:final_hidden}
\end{equation}

\noindent where $\phi_{a}$ is a learnable function.

\subsection{Stage 1. Generate Transferable Prototypes}
The encoder projects every transaction $\mathbf{x}$ in $\mathcal{D}_{s}$ to the domain-invariant space $\mathbf{h_{x}}$ that is compatible with knowledge transfer. Then the objective of Stage 1 is to learn discriminative features between the fraud and non-fraud transactions. We use contrastive learning to pretrain the network $f(\cdot|\mathcal{D}_{s})$ and extract prototype set $\mathcal{C}_{s}$ from the source representation space.

\paragraph{Pretrain with Supervised Contrastive Learning.}
The use of contrastive learning ensures that positive instances belonging to the same class (i.e., frauds or non-frauds) are pulled together in normalized embedding space while simultaneously pushing apart data instances from different classes~\cite{kang2019contrastive, chen2020simple}.  We assume source countries own enough inspected logs to leverage the label information effectively. We adopt supervised contrastive learning (SCL) loss  to pretrain the network  $f(\cdot|\mathcal{D}_{s})$ in a fully-supervised manner~\cite{khosla2020supervised}.
\begin{equation}
\begin{split}
   \mathcal{L}_{SCL} = &\sum_{i=1}^N - \frac{1}{N_{y_i}-1} \sum_{j=1}^N \mathbf{1}_{i\neq j} \mathbf{1}_{y_i = y_j} \bigg[ \\
&  \log \frac{\exp(\text{sim}(\mathbf{h_{x_i}} , \mathbf{h_{x_j}}) / \tau)}{\sum_{k=1}^N \mathbf{1}_{i \neq k}\exp(\text{sim}(\mathbf{h_{x_i}} \cdot \mathbf{h_{x_k}}) / \tau)} \bigg],
\end{split}
\label{eq:SCL_loss}
\end{equation}
\noindent where N is the batch size, and $\tau$ is the sharpening temperature.

Compared to using a simple cross-entropy loss, SCL loss is known to increase the transfer learning performance by maximizing the discrepancy between separate classes~\cite{kang2019contrastive, khosla2020supervised}. In the experiment, we verify that pretrained network using SCL loss shows non-trivial performance improvement.
\paragraph{Compress Knowledge by Clustering.}
Vanilla transfer learning (i.e., passing only model parameters) may suffer from the catastrophic forgetting problem, in which the fine-tuned model tends to ``forget'' source dataset's discriminative feature during fine-tuning~\cite{kirkpatrick2017overcoming}.
To avoid this fallacy, we adopt the memory bank concept~\cite{wu2018memorybank} to enhance the model capacity by storing additional representations from the source dataset $\mathcal{D}_{s}$ in the memory bank.
We compress knowledge as a prototype, a representative embedding for a group of semantically similar instances. This structure also ensures that shared knowledge no longer contains individual trade information. After pretraining, knowledge of frauds and non-frauds is clustered separately and condensed to generate transferrable prototypes. We apply \emph{K}-means clustering to extract the centroid set as prototypes for each class (i.e., $\mathcal{C}_{s} = \mathcal{C}_{\text{frauds}} \cup \mathcal{C}_{\text{non-frauds}}$). These extracted prototypes $\mathcal{C}_s$ are stored in the memory bank $M$, allowing knowledge transfer from the source country $s$. Recipients can utilize any subset of the transferred knowledge for their classification task and detect previously unrecognized illicit trades in $\mathcal{D}_{t}$.

\paragraph{Multi-Source Memory.} The memory bank $M$ can be expanded to adapt a multi-source scenario. For each source country, we pretrain the model independently using $\mathcal{L}_{SCL}$ and extract the prototype sets. The memory bank M can now contain an ensemble of multiple source countries' prototype sets as follows:

 \begin{equation}
    M = [\mathcal{C}_{s_1}, \mathcal{C}_{s_2}, ..., \mathcal{C}_{s_m}],
    \label{eq:multi_source}
\end{equation}

\noindent where $m$ is the number of source countries.

\subsection{Stage 2. Fine-Tune with Prototypes}

Given the memory bank  $M$ from $s$, the target country $t$ can fine-tune its fraud detection system $f(\cdot|\mathcal{D}_{t})$ using $\mathcal{D}_{t}$.
We design a fine-tuning step that is inspired by recent memory-based domain adaptation techniques that use enhancers for augmenting data representation~\cite{asghar2018progressive, liu2020open}. Following this concept, we let each target representation $\mathbf{h}_\mathbf{x}^t$ be
refined with the representation $\mathbf{\bar{h}}_\mathbf{x}^{t|s}$ that is augmented with the source domain knowledge $\mathcal{C}_s$ in the memory $M$:
\begin{equation}
    \mathbf{\hat{h}}_\mathbf{x}^t = \mathbf{h}_\mathbf{x}^t + \mathbf{\bar{h}_x}^{t|s}.
    \label{eq:trasfer_hidden}
\end{equation}
The process of deriving $\mathbf{\bar{h}_x}^{t|s}$ is described in Eq.~7--10.

\paragraph{Source-Augmented Feature.}
Source-augmented feature $\mathbf{\bar{h}}_\mathbf{x}^{t|s}$ is computed via soft-attention toward the set of selected prototypes in $\mathcal{C}_{s}$~\cite{vaswani2017attention}.
Let k-th prototype be denoted $\mathbf{c}_{k}$. Each target transaction $\mathbf{h}_\mathbf{x}^t$ attends to prototype features via attention weights computed by dot product similarity:
\begin{align}
    w_{k}= \mathbf{\psi}( \mathbf{h}_\mathbf{x}^t \cdot \mathbf{c}_{k}) \\
    \mathbf{h}_\mathbf{x}^{t|s}= \sum_{k=1}^{|M|} w_{k}\mathbf{c}_{k},
    \label{eq:attention_weight}
\end{align}
where $\psi$(·) is a softmax function to normalize the dot product similarity scores across all prototypes.

To regulate negative transfer between two domains~\cite{wang2019characterizing}, the network must calibrate how much knowledge to transfer from source to target. We calibrate source knowledge by using a single feed-forward network $g$ that computes additive attention weights between the source and target representations.
\begin{align}
    \mathbf{e_{x}} &= g(\mathbf{h}_\mathbf{x}^t, \mathbf{h}_\mathbf{x}^{t|s}) \\
    \mathbf{\bar{h}}_\mathbf{x}^{t|s} &= \phi_{r}([\mathbf{e_{x}} \odot \mathbf{h}_\mathbf{x}^{t|s}, \mathbf{h}_\mathbf{x}^t])
    \label{eq:calibrate}
\end{align}
where $\phi_{r}$ is a learnable parameter and $\odot$ is a Hadamard product.

\paragraph{Fine-Tuning with Target Data.} The target representation $\mathbf{\hat{h}}_\mathbf{x}^{t}$ is dynamically balanced between the direct transaction feature $\mathbf{h}_\mathbf{x}^{t}$ and the source-augmented feature $\mathbf{\bar{h}}_\mathbf{x}^{t|s}$ as in  Eq.~\eqref{eq:trasfer_hidden}. We fine-tune the target country's fraud detection system $f(\cdot|\mathcal{D}_{t})$:
\begin{align}
    \mathcal{L}_{cls} = -\frac{1}{|\mathcal{D}_{t}|}\sum_{{\mathbf{x}, y}\in\mathcal{D}_{t}}  H(y, \hat{y})
    \label{eq:fine_tuning}
\end{align}
where $\hat{y} = f(\mathbf{\hat{h}}_\mathbf{x}^{t})$ is the predicted fraud score of transaction $\mathbf{x}$ and $H$ is a binary cross entropy.
\fi
\section{Results}
This section tests feasibility of the proposed HS classification model in terms of classification performance and retrieved sentence quality. Classification performance is measured by checking the subheading (HS6) accuracy, our final output, and heading (HS4) accuracy, the intermediate output. The quality of the retrieved sentences is examined by comparing our results with documents written by experts. \looseness=-1

\subsection{Experimental Setting}

\subsubsection{Data Preprocessing. \quad}
For experiments, we retained 126,000 cases that the product HS code is maintained among 129,084 cases. Given that decisions may change over time, we used the last three months of data (1,466 international and 186 Korean cases) for evaluation. Three months' worth of data before the test period was used as the validation set (1,733 international cases and 102 Korean cases) for hyperparameter tuning. \looseness=-1

\subsubsection{Evaluation Metric. \quad}
We evaluated the heading and subheading classification performance by measuring top-$k$ accuracy with $k = 1, 3, 5$. In retrieved sentences case study analysis, we measured recall and precision to evaluate the quality of the supporting factors.

\subsubsection{Training Details. \quad}
KoELECTRA models and classification heads in heading and subheading prediction are trained for 50 epochs and evaluated when validation accuracy is the highest. The embedding size of KoELECTRAs is set as 768. Key sentences of each item are required to train a subheading prediction model, so we prepared them from the answer heading's HS manual beforehand. In evaluation stage, key sentences are retrieved from the predicted heading's HS manual. Training KoELECTRA model takes 40 hours and data preparation for the sentence retrieval model takes 50 hours using NVIDIA TITAN Xp. Inference and retrieval take less than 30 seconds.

\subsection{Performance Evaluation}

\subsubsection{HS Code Prediction. \quad}
We compared the proposed model with two baselines---a word-matching model and an LSTM-based model. The word matching model computes the word matching rate between the item description and heading-level HS manual and chooses the heading with the highest matching rate. LSTM-based model~\cite{lime2018} is a winning model of the competition~\cite{kakaocompetition}, which has a similar setting with our problem: predict the detailed category of the e-commerced products using their descriptions. The model utilizes LSTM networks to get embedding from tokenized input texts. Table \ref{table:main_results} shows the top-$k$ accuracy of the baselines and our model. Two variants of our model are tested, the one used retrieved sentences from the second stage and the other did not use them.
\input{tables/main_results}

\subsubsection{Retrieved Key Sentences. \quad}
Given the item description, experts present supporting reasons for their final decision. They quote some sentences from the HS manual and provides evidence of their decision with detailed explanations. We compared the quoted sentences written by experts, with key sentences retrieved by our model. As shown in Table~\ref{tab:supporting_facts}, the key sentences (supporting facts) retrieved by our model broadly match the quotations from experts.

\input{tables/supporting_facts}

\begin{figure*}[h]
\centerline{
      \includegraphics[width=1\linewidth]{figures/Output.png}}
      \caption{Format of the model's final output. As the final result, the model provides heading and subheading candidates along with key sentences and similar cases as a basis for judgment.
      }
\label{fig:output}
\end{figure*}

\subsection{Output Templates}
Figure \ref{fig:output} shows the model's final outputs. First, the model provides three heading candidates with their prediction scores. To calibrate the score, we applied temperature scaling to the model's softmax outputs~\cite{guo2017calibration}. For each candidate, one to seven key sentences are retrieved from the HS manual. Retrieved sentences explain the model prediction and reduce the reviewing scope by customs officers. Next, the model provides three subheading candidates. With each subheading, the model provides similar prior cases for reference. \looseness=-1

\if 0
This section tests feasibility of the proposed knowledge sharing system in terms of detection performance, accommodating multiple sources, and dependency on the required log size and model components.

\subsection{Experimental Setting}

\subsubsection{Datasets}
\label{sec:experiments:settings:datasets}
We employed import declarations from four partner countries of WCO, whose data had been shared for the research purposes under the non-disclosure agreement. We refer to these countries M, C, N, and T in Table~\ref{tab:datastats}. The log size and GDP per capita vary by country, allowing us to test various source and target scenarios. The format of declarations varied slightly. We only considered the common fields, including numeric variables on the item price, weight, and quantity and categorical variables on the HS6 code, importer ID, country code, and received office. The data also contained manual label information indicating whether each declaration was judged fraud or not. For the test purposes, these labels had been generated by inspecting all logs.

\input{tables/datasets}

\subsubsection{Evaluation Metric} Raised tax is one of the critical screening factors because customs import duties make up a substantial proportion of the tax revenue in many countries~\cite{Grigoriou2019WCO}. We hence consider the amount of tax raised from inspection as the performance indicator. In all experiments, we assume a 5\% inspection rate unless otherw ise mentioned. We use the term Revenue@5\% to represent the ratio of the expected tax collected by inspecting 5\% of the trades sorted by the fraud detection algorithm out of the maximum revenue that could have been raised when the entire log was to be inspected.

\subsubsection{Training Details}
The model $f$ is individually pretrained for every country 10 epochs and fine-tuned for 30 epochs. The sharpening temperature $\tau$ in Eq~\eqref{eq:SCL_loss} was set as 0.07 by following previous works~\cite{gunel2020supervised}. Smaller temperature for SCL loss benefits training more than
higher ones, but extremely low temperatures are harder to train due to numerical instability. The number of prototypes was set to 500 per class, thus $|\mathcal{C}_s| = 1,000$. We vary this count later. We use the final month of each dataset as the test set, and the validation set was chosen as two weeks prior. All models were run five times, and their average values are reported.

\subsection{Performance Evaluation}

The first set of experiments (Exp 1--2) aims to test the effectiveness of domain adaptation for single-source and multi-source scenarios. The next experiments (Exp 3--5) test the model's sensitivity and dependency to the target country's log size, model components, and memory-bank usage.

\subsubsection{Exp 1. Single-Source Scenarios}

We inspect all combinations of the source and target pair as follows. The source country will share knowledge based on the 5\% of its fraud-like logs $\mathcal{D}_{s}$ sorted by $\hat{y}$. This knowledge is used in four different ways: (1) no sharing at all, (2) standard fine-tuning by sharing model parameters of $f(\cdot|\mathcal{D}_{s})$ directly with the target country, (3) transfer learning based on the adaptive knowledge consistency technique~\cite{abuduweili2021adaptive},
and (4) the proposed prototype-based memory bank.

At the receiving end, we assume the target country has a weaker infrastructure and utilizes only 1\% of trades for training. This assumption is simulated by randomly masking the log labels. The base fraud detection model $f$ uses implementations of the tree-based embedding model~\cite{kim2020date}, whose performance is reported in the column indicated as `Target Only' in Table~\ref{tab:singlesource}. The Revenue@5\% performance for the target country M is the lowest, which may be contributed by the smallest log size. Note that country M has the lowest GDP per capita in Table 1. The efficacy of the detection model increases when utilizing knowledge shared by countries with larger log sizes (i.e., choosing Country T as the source country), although with some exceptions. Since the illicit rates of these countries are different, we only compare increments in performance within each target country.

\input{tables/results_main}

When knowledge is shared in any form, the tax revenue from fraud detection generally increases. For the target country M, the use of \textsf{HSC}{} from country N increases the tax revenue 11 times from 0.0466 to 0.5152. The exact benefit differs by country, implying that various factors like the log similarity between countries may play a role. Parameter sharing (i.e., column indicated as Vanilla Transfer) sometimes leads to degraded performance, due to negative transfer. Transfer learning via the adaptive knowledge consistency technique (i.e., Adaptive Transfer) no longer shows this limitation. Sharing knowledge in the form of prototypes shows the best performance in terms of tax raised, as indicated in the final column of the table. Note this method also increases data protection of the source countries.

\subsubsection{Exp 2. Multi-Source Scenarios}

Accumulating knowledge from more than one country further increases the performance of the fraud detection system. Figure~\ref{fig:multisource} shows the average Revenue@5\% for using no source at all, a single source, two sources, and three sources. To compute, we averaged the performance of all possible combinations according to the number of source countries. The additional gain in raised tax is most noticeable for using a single source, yet there is a revenue gain with every added source. When all available source knowledge is used, the target countries benefit nearly 2 to 11 times increase in total tax raised. \looseness=-1

\begin{figure}[h!]
    \centerline{
    \includegraphics[width=0.95\columnwidth]{figures/multi_source.pdf}}
    \caption{Collected duties are expected to increase 2.1--11.7 times for all tested countries when shared knowledge contributed by multiple sources is used (i.e., blue bars) compared to relying on local knowledge alone (i.e., red bars).
    }
    \label{fig:multisource}
\end{figure}

\subsubsection{Exp 3. Dependency on Log Size}

Next, we examine measurable effects such as the log size. Figure~\ref{fig:target_size} shows the average benefit of utilizing a single source versus no shared knowledge over the increasing ratio of log size for each target country. The performance gap is the largest when the log size is 1\%, implying \textsf{HSC}{} can benefit countries with weak infrastructure the most. Yet, the continued benefit is observed for increasing log size from 2\% to 10\% in the target country, even at a marginal level. The benefit for country T is the smallest, which has the highest GDP per capita. Nonetheless, we emphasize that sharing knowledge increases the capacity to manage risk in the neighboring participating countries, ultimately curtailing illicit trades in that region.

\begin{figure}[h!]
    \centerline{
    \includegraphics[width=1.00\columnwidth]{figures/target_size.pdf}}
    \caption{Performance improvement as the log size of the target country increases. The shared knowledge brings the most considerable benefit when the available log size is the smallest (i.e., 1\%) and for countries with the weaker economy (i.e., country M).}
    \label{fig:target_size}
\end{figure}

\subsubsection{Exp 4. Dependency on Model Components}
Table~\ref{tab:Ablation} reports the performance degradation when each component of the model is missing. The experiments are done with the target country N with a 1\% labeled setting. The full model with all components performs the best, justifying the role of each component. Mainly, domain invariant encoding of \textsf{HSC}{} contributes to the model performance the most. This experiment considers five scenarios.

\begin{itemize}

\item  Full model: \textsf{HSC}{} containing all components.

\item  Without domain invariant encoding: The model without Eq.~\ref{eq:outer_product}--\ref{eq:final_hidden} that simply concatenate between HS6 code and transaction embedding.

\item  Without contrastive learning: The model without Eq.~\ref{eq:SCL_loss}, which uses cross-entropy loss instead of SCL loss for pretraining source dataset.

\item  Without memory bank: The model without Eq.~\ref{eq:attention_weight}, which fine-tunes the target data without utilizing the information of the source memory bank.

\item  Without domain calibration: The model without Eq.~\ref{eq:calibrate} that only considers the interaction between source feature and target feature.

\end{itemize}
\input{tables/ablation}

\subsubsection{Exp 5. Efficacy of Memory Contents}
Additionally, we discuss the augmentation effect when the source knowledge is transferred through the memory bank. For comparison, we consider a baseline of random memory bank with the same size, inspired by the prior work that randomly injected noise provides practical benefits to deep models~\cite{poole2014analyzing}.
Table~\ref{tab:compare_augment} shows that memory bank usage provides non-trivial improvements than using random memory banks. This finding confirms that information shared by source countries is successfully leveraged in the form of prototypes.

\begin{table}[h!]
\centering
\caption{Efficacy of memory-based augmentation}
\scalebox{0.85}{\begin{tabular}{l|cccc}
\toprule
 & \multicolumn{4}{c}{Target country}     \\ \cmidrule{2-5}
Memory setup & \textsf{M} & \textsf{C} & \textsf{N} & \textsf{T}     \\ \midrule
No memory     &   0.5183 &   0.2233 & 0.6762 & 0.3190\\
Random     &      0.5241  & 0.2265 & 0.7126 & 0.3238\\
Three countries  & \textbf{0.5435}  & \textbf{0.2340}  & \textbf{0.7395} & \textbf{0.3286}\\
\bottomrule
\multicolumn{5}{l}{A pretrained network $f(\cdot|\mathcal{D}_{s})$ from Eq.~\eqref{eq:SCL_loss} is shared together} \\
\multicolumn{5}{l}{regardless of the memory bank setup.}
\end{tabular}}
\label{tab:compare_augment}
\end{table}

\fi
\section{Discussion}

\subsection{Towards Interpretable Results}

One of our goal is to add interpretability to the HS classification model. To do that, we provide a confidence score for each HS code candidate. The score provides additional information to judge whether a candidate is valid or not. Although the score is tuned by temperature scaling, the range of the top-$k$ confidence score was quite different for each input item. Careful calibration is required to make customs officers use this value as a reference in the decision-making process.
Another way to increase the interpretability is to visualize the part of the item description relates to each subheading candidate. If then, customs officers can concentrate on the selected part and decide whether to consider second and third candidates to review. In addition, key sentences should relate to the subheading characteristics so that the final form of the model output should resemble reports written by experts. Making an organized document that explains the relation between prediction, description, and HS manual will reduce the effort required for HS code classification.

\subsection{Contextual Information in Decision}
Customs experts decide the HS code by following the General Rules for Interpretation of Nomenclature (GRI), similar to judges deciding based on the law~\cite{winter2021judicial}. On the other hand, deep learning models solve classification problems by finding common patterns from previous cases. As a result, past examples are the primary determinant of the AI model's decision, different from human experts who make their decision by rules and manuals. Since HS code and its manual undergo revision every five years, previous cases cannot always be good references for recent ones. Therefore, it is essential to employ GRIs and HS manual to make a credible model, which utilizes contextual information in model training based on deep linguistic understanding~\cite{wiegreffe2021explainableNLP}.

\if 0
This paper presented a first-of-a-kind knowledge sharing system for multi-national customs administrations. Below we discuss the implications of findings in terms of risk management and protection of trade information.

\subsection{Discussion}

\textsf{HSC}{} has shown substantial revenue potential by utilizing a relatively small fraction of fraud-like logs. Custom offices with weaker infrastructure will likely observe the largest increase in raised tax, whereas the more equipped customs offices will see a smaller gain. However, the benefit needs to be interpreted in terms of two other aspects. First, once the system is set up, the cost of fine-tuning for additional prototypes will be minimal. Hence, any increase in detection performance translates to a potentially substantial additional tax revenue in the target country. Second, empowering countries with weak infrastructure will enable the detection of new fraud patterns that were previously unseen. The new fraud patterns collected from target countries will be shared back to the memory bank. This helps strengthen policy against illicit trades by removing weak spots in the global trade chain and further benefit participating countries~\cite{wang2018wcj}.

Qualitative analysis can be used to show how well the shared information discriminates against illicit transactions. Figure~\ref{fig:qual} compares embedding results of the target country, without knowledge (i.e., left) and with knowledge (i.e., right). Using the source-enhanced features helps better distinguish frauds from non-frauds. This particular example shows a fraud case that was newly detected by \textsf{HSC}{}. Without the shared knowledge, this declaration would have been missed. In contrast, the knowledge augmentation placed this declaration more closer to the fraudulent cluster in the embedding.

\begin{figure}[h!]
    \centerline{
    \includegraphics[width=1\columnwidth]{figures/qualitative}}
    \caption{t-SNE plots of the learned embeddings (T $\rightarrow$ N), when the model is trained only with target-only feature (left) and with source enhanced feature using \textsf{HSC}{} (right). Fraud cases are successfully detected after receiving prototypes from the source country. Similar fraud examples from the source country help flag this case. }
    \label{fig:qual}
\end{figure}

The proposed method of sharing prototypes is a safe way to transfer knowledge across heterogeneous administrative domains. Here, we show the input data utilized for knowledge building is statistically different from the vast ``normal'' trades that include critical information about trade partners and prices---information that is confidential and of concern in data sharing.

Figure~\ref{fig:difference} highlights some of the differences observed for fraud-suspected logs and normal logs. The logs utilized by the model have nearly four times higher fraud ratio than that of normal logs. In addition, the product categories that frequently appear in fraud-like logs and normal logs have different rank orders, as illustrated in the examples. The average declared price (CIF value) indicates that the initial reporting tariffs written in the declaration form are substantially lower for the inspected logs. Only fraud-suspected logs are used for knowledge building, and the normal data are excluded in the data embedding step.

\begin{figure}[t!]
    \centerline{
    \includegraphics[width=1\columnwidth]{figures/inspection_compare.pdf}}
    \caption{Shared knowledge of frauds is distinct from the mass of the normal trade volume contributed by member offices. Exact figures are hidden.}
    \label{fig:difference}
\end{figure}

\subsection{Concluding Remark}
There is an increasing need to share knowledge in data-critical sectors, including customs, medicine, finance, and science. We tested the premise that shared knowledge will facilitate and advance risk management in customs fraud detection. Domain adaptation techniques are one way to build collective knowledge across international administrations. Testing with million-scale data demonstrated that transferable knowledge increases the detection performance for all participating countries. Starting from this proof of concept, discussions are underway to plan an active service at the WCO. The memory bank system requires that the source countries agree on the legal support for sharing prototypes outside their administrative domains. We envision this will be enabled via a central cloud service managed by WCO that will facilitate knowledge sharing while safeguarding data.

As global trade sees dynamic changes due to events like COVID-19, maintaining an agile risk management system has become ever more important. Strengthening the data science capacity at customs administrations and collaboration will be critical as illicit traders continue to make use of vantage points in the trade network. When more countries participate in the regional memory bank platform, we expect knowledge sharing will better handle the data challenges. Furthermore, the regional ties will help enable positive trade agreements among the participating countries.
\fi
\section{Conclusion}
This study introduces a framework to predict harmonized system codes in customs. Using product description and HS manual, it predicts heading as subheadings and provides supporting facts and related cases to facilitate decision process by human experts. We expect that our work will contribute significantly in various aspects. This framework by declarants will improve the initial declaration quality, thereby reducing customs officials' workloads. Internally, the framework can be used to assist customs officials in carrying out their duties to increase their work efficiency and train their employees. The situation with competing HS codes is particularly problematic for declarants and customs officials. Our model presents the competing HS codes of the target product with its rationale, so it has great significance as an auxiliary means for product classification. The platforms require a systematic classification system to effectively expose and recommend products to users, but there are often different standards for each product-providing company. The platforms build and utilize hierarchical classification algorithms to maintain the consistent categorization of hundreds of millions of products. Our work will be used to advance those and facilitate their management.

\section*{Acknowledgment}
This work was supported by the Institute for Basic Science (IBS-R029-C2, IBS-R029-Y4). We thank numerous officers from Korea Customs Service for their insightful discussions.

\section{References}

Refer to the below example.

\if 0
\paragraph{Code Release.}
Implementation of \textsf{HSC}{} is included in code supplementaries. We plan to publish an interactive tutorial that contains a series of procedures to compress knowledge, refine data using compressed knowledge, and perform fraud detection.

\paragraph{Data Details.}
The format of import declarations varied slightly and we only considered the common fields, including numeric variables such as item price, weight, quantity and categorical variables such as HS6 code, country code. The data also contained inspection results indicating whether each declaration was judged fraud or not, and the amount of tax raised after inspection. We
report the raw variables of import declarations in Table~\ref{Tab:variable_description}.

\begin{table}[h!]
\centering
\caption{Overview of the transaction-level import data, in which the description of each variable are provided.}
\scalebox{0.75}{
\begin{tabular}{l|l}
\toprule
Variable & Description  \\ \midrule

\textit{Quantity} & The specified number of items\\
\textit{Gross weight} & The physical weight of the items \\
\textit{HS code}   &  Applicable item code based on the harmonised system \\
\textit{Country code}  & Country from which the goods were imported \\
\textit{CIF value} & Transaction value including the insurance and freight costs \\
\textit{Total taxes} & Tariffs calculated by initial declaration \\ \midrule
\textit{Illicit ($y$)} & Target binary variable indicating whether the item is fraud \\
\textit{Revenue} & Amount of tax raised after the inspection, used for evaluation \\
\bottomrule
\end{tabular}}
\label{Tab:variable_description}
\end{table}

\paragraph{Training Details.}
For every country, the model $f$ is individually pre-trained for 10 epochs and fine-tuned for 30 epochs. The sharpening temperature $\tau$ in Eq~\eqref{eq:SCL_loss} was set as 0.07 by following previous works~\cite{gunel2020supervised}. Smaller temperature for SCL loss benefits training more than
higher ones, but extremely low temperatures are harder to train due to numerical instability. The number of prototypes was set to 500 per class.
We use the final month of each dataset as the test set, and the validation set was chosen as two weeks prior. The model was optimized using Ranger with a weight decay of 0.01~\cite{wright2021ranger21}. The batch size and learning
rate was set as 128 and 0.005 for entire training epochs. All models were run five times, and their average values are reported.

\paragraph{Baseline Details.}
Below are the detailed explanation of the baselines used in our experiments.
\begin{itemize}
\item Target Only: Measure the performance of a base fraud detection model $f(\cdot|\mathcal{D}_{t})$ trained on target data. The tree-based embedding model DATE is used~\cite{kim2020date} for $f$. DATE classifies and ranks illegal trade flows that contribute the
most to the overall customs revenue when caught.
\item Vanilla Transfer: Sharing parameters of the fraud detection model $f(\cdot|\mathcal{D}_{s})$ from source country $s$ and directly fine-tune using target dataset $\mathcal{D}_{t}$.
\item Adaptive Transfer: Adaptive Knowledge Consistency (AKC) technique~\cite{abuduweili2021adaptive} is used to cope with the risk of negative transfer caused by the discrepancy between the source and target countries. It constrains the mean square error between the pre-trained feature extractor outputs and the target feature extractor outputs by adaptively sampling the target dataset $\mathcal{D}_{t}$. We perform a hard filter according to the sample importance by sorting fraud score difference between the source and target pre-trained models. The features with lower 20 \% score difference were selected.
\end{itemize}

\paragraph{Comprehensive Results.}
Table~\ref{tab:results_all} shows the comprehensive results, which was simplified in Table~\ref{tab:singlesource} (Single-source scenarios) and Figure~\ref{fig:multisource} (Multi-source scenarios). From Table~\ref{tab:results_all}, we made Table~\ref{tab:singlesource} by picking single-source cases, and we drew Figure~\ref{fig:multisource} by averaging all multi-source \textsf{HSC}{} results grouped by the number of source countries.

Table~\ref{tab:results_label_increase} shows the original results, which was simplified in Figure~\ref{fig:target_size}. The table describes the performance gain by changing the target log size. From Table~\ref{tab:results_all}, we made Figure~\ref{fig:target_size} by averaging \textsf{HSC}{} results grouped by each target country.

\paragraph{Dependency on Prototype Count.}
Knowledge of frauds is shared as compressed representations called prototypes. The number of prototypes can determine the resolution of transferred knowledge. For instance, one may assume selective fraud patterns may be shared when we limit the prototype count. We test the sensitivity of the model on prototype count $|\mathcal{C}_s|$ by setting it from 10 to as large as the source data size $|\mathcal{D}_{s}|$. Table~\ref{tab:proto} shows detection performance remains intact by $|\mathcal{C}_s|$, implying that the model is not sensitive to the choice of hyper-parameters as long as the prototypes are used.

\begin{table}[h!]
\centering
\caption{Sensitivity analysis to the number of prototypes for country N. $|\mathcal{D}_{s}|$ implies that the entire embeddings are transferred instead of the clustered prototypes.}
\small
\scalebox{1}{
\begin{tabular}{c|ccccc}
\toprule
$|\mathcal{C}_s|$ & 0 & 10 & 100        & 1,000   &   $|\mathcal{D}_{s}|$ \\ \midrule
\textsf{M} $\rightarrow$ \textsf{N} & 0.6667  & 0.6714  & 0.6721  & 0.6681 & 0.6709\\
\textsf{C} $\rightarrow$ \textsf{N} & 0.6564 & 0.6656      & 0.6727 & 0.6863 & 0.6711\\
\textsf{T} $\rightarrow$ \textsf{N} & 0.6762 & 0.7379      & 0.7307 & 0.7286 & 0.7314\\
\bottomrule
\end{tabular}
}
\label{tab:proto}
\end{table}

\input{tables/results_all.tex}
\input{tables/results_label_increase.tex}
\fi

redpajama_set_name: RedPajamaArXiv


I regularly shop at the Norwalk Costco warehouse, which is the nearest one to Downey. But I have had to travel further to get Costco gas. I either have to go to Lakewood or Commerce or to a local gas station.
I have wanted the Norwalk warehouse to get their own gas station for years. And it has finally happened. They just opened about a month ago and now it is my regular gas stop. One more perk for living in Downey.


Encounter new customers in a jiffy with a rewarding campaign in Tysons!
There exists a myriad of companies in application development, but no more than one is the most skilled for you. Our know-how makes it feasible to encounter the peerless partner out of all those companies. Meeting the matching agency is from this moment on a five-finger exercise.
Tysons, mobile app developers to launch your operation.
Sortlist 's intention is to draw up an index of the 5 fitting mobile app developers for your assignment. And they will thus, by virtue of their talent and competencies, conceive benefits for your task in business app development or native app creation. Thus trust us, the agencies in Tysons will have every little thing under control.
Launch activities in application development in Tysons.
The services put forward by mobile app developers can be, for example, native app creation or business app development. But also if you're searching for a bureau that will manage to help you with a more challenging project we will be able to to be of aid. The argument for this is that we are able to contact a mountain of agencies which do business with with organisations that perform in the accounting- or even the household products industry, that makes them very versatile to all kinds of tasks.
Thus, if your firm wants to complete a campaign in native app creation, business app development or even in API development, then your company doesn't need to waste precious hours seeking a specialist in Tysons. Because at Sortlist we will be able to support you smoke out the right agency and your firm will doubtlessly acknowledge that an assignment in application development will have a true impact on your returns.
For Sortlist it's a sure thing, working a firm in Tysons is an treasure for your assignment in application development.
Thanks to the solution that was put in place by our team, your company is able to not singularly see which bureaus are wizards in their expertise but you will also have the means to select with who it wants to work with. The mobile app developer will in a jiffy determine your solicitations and moreover these agencies will also provide your firm access to technology that your firm may not have had formerly.
Achieving an activity in application development is, as you have surely seen, not often uncomplicated. Our firms in Tysons can hence support. Find mobile app developers that meet your solicitations like a bat out of hell, thanks to Sortlist.