text:
Inaugural Opening of the Academic Year Alumni Lecture reckons with reparations
Mon 14 November 2022
Edinburgh Law School's inaugural Opening of the Academic Year Alumni Lecture took place on Friday, 16 September 2022 with a lecture from Professor Eric J. Miller, Leo J. O'Brien Fellow at Loyola Law School (LLB, 1991) and a response from Professor Lesley McAra, Chair of Penology at Edinburgh Law School (MA Politics, 1981). It was chaired by Jacquelyn MacLennan (LLB, 1983), a Partner at White & Case and the Alumni Co-Chair of the Edinburgh Law School Advancement Committee.
Reckoning with reparations
Reparations has become a major means of demanding redress, advanced by different groups identified by race, nationality, and even sexual orientation. There are a variety of different ways of understanding the nature of reparations, and the political movements that use reparations to advance their objective.
In his lecture, Professor Miller recounted how and why someone born and raised in Glasgow, and learning law at Edinburgh Law School, became one of the foremost reparations litigators in the United States, representing the Black victims of a 100 year-old race massacre before US state and federal courts and in the United States Congress.
Prof Miller explained why reparations is such a powerful idea both globally but also nationally and locally, and why it resonates here in Scotland.
The lecture was followed by a response from Professor McAra, based on her work on the Edinburgh Study of Youth Transitions and Crime and as director of the Institute for Advanced Studies in the Humanities (IASH) at the University of Edinburgh.
Listen to the lecture
00:00 - 07:23: Introduction by Jacquelyn MacLennan
07:23 - 39:59: Reckoning with Reparations by Prof Eric Miller
40:00 - 41:10: Comments by Jacquelyn MacLennan
41:11 - 51:20: Response by Prof Lesley McAra
51:21 - 54:33: Concluding Remarks by Jacquelyn MacLennan
Professor Eric J. Miller (LLB, 1991)
Eric J. Miller is a professor and Leo J. O'Brien Fellow at Loyola Law School in Los Angeles, where he teaches and writes in the areas of legal theory, reparations, and policing.
Professor Miller is an internationally recognised expert on reparations for Black people in the United States. He has testified before the House Judiciary Committee Subcommittee on Continuing Injustice: The Centennial of the Tulsa-Greenwood Race Massacre in 2021; on HR40 and the Path to Restorative Justice in 2019; and on the Legacy of the Trans-Atlantic Slave Trade in Americain 2007. In addition, he has provided testimony before the Inter-American Commission on Human Rights on Reparation for Slavery and Other Forms of Structural Racial Discrimination in the United States, Washington, D.C., September 24, 2019.
Professor Miller was a member of the Reparations Coordinating Committee, and took a leading role in drafting the complaint in the pathbreaking federal reparations lawsuit on behalf of the survivors of the Tulsa Race Massacre of 1921, Alexander v. State of Oklahoma.
He is currently a member of Lawyers for Justice for Greenwood, and again took a lead role in drafting and arguing a reparations public nuisance lawsuit, Randle v. City of Tulsa.
Professor Miller received an LL.B. from the University of Edinburgh, and an LL.M. from Harvard Law School, where he was also a Charles Hamilton Houston Fellow. He clerked for the Hon. Myron H. Thompson in the Middle District of Alabama and the Hon. Stephen Reinhardt of the Ninth Circuit Court of Appeals.
Professor Lesley McAra (MA Politics, 1981)
Lesley McAra is an alumna of the University of Edinburgh and of the Open University. She began her career as a researcher in the Scottish Office where she led a major programme of research evaluating social work criminal justice services. In 1995, she joined the University of Edinburgh as Lecturer in Criminology, was promoted to Senior Lecturer in 2005, and to the Chair of Penology in 2009.
Lesley is currently the Director of the Institute for Advanced Studies in Humanities (IASH). She is also the Assistant Principal Community Relations where she is academic lead for the local impact component of University of Edinburgh strategy. Lesley is a former Dean of the School of Law, the first woman to be appointed to that post. In 2014 she founded (with alumna, Karina McTeague) the Leadership Foundation for Women in the Legal Profession.
Jacquelyn MacLennan (LLB, 1983)
Jacquelyn MacLennan is a Partner at White & Case and the Alumni Co-Chair of the Edinburgh Law School Advancement Committee.


text:
Alan K. Parrish is a professor of religious education at Brigham Young University (BYU) who has written books and articles related to American education and the Church of Jesus Christ of Latter-day Saints (LDS Church), most notably a biography of John A. Widtsoe.

Parrish was born and raised in Logan, Utah.  He served in the Southwest British Mission of the LDS Church, based in Bristol, England.   After his mission he received a bachelor's degree and an MBA from Utah State University.  After being in business for a short time he was recruited to work for the Church Educational System beginning as a seminary teacher. He then taught institute adjacent to UCLA and served as bishop of the UCLA ward, while at the same time receiving an Ed.D. degree from the University of Southern California.

From 1978 to 1984 Parrish was the director of the LDS Institute of Religion in Cambridge, Massachusetts.  In 1984 he joined the faculty of BYU.

Sources
Dust Jacket bio attached to Biography of John A. Widtsoe
BYU bio

Latter Day Saints from California
American biographers
American Mormon missionaries in England
Brigham Young University faculty
Church Educational System instructors
Writers from Logan, Utah
USC Rossier School of Education alumni
Utah State University alumni
20th-century Mormon missionaries
Living people
Latter Day Saints from Massachusetts
Latter Day Saints from Utah
Year of birth missing (living people)

redpajama_set_name: RedPajamaWikipedia


text:
THE TREATISE ON THE LETTERS
THE TREATISE ON THE LETTERS (RISĀLAT AL-ḤURŪF) DEL SUFÍ SAHL AL-TUSTARĪ PILAR GARRIDO CLEMENTE Universidad de Muhammad V, Rabat Summary This article presents the first translation into another language of a key work, in Arabic, by the mystic Sahl al-Tustarī (9th century), one of the first Qur'anic exegetes in the formative period of Sufism. In this Opuscule which, due to
HERMENEUTICS, PHILOSOPHY, RELIGION, THOUGHT
PERSIAN MYSTICISM AND THE PRIMORDIAL LIGHT: SOHRAVARDĪ
November 16, 2020 admin Leave a comment
Shihāb al-Dīn Yaḥyā Sohravardī (1155-1191), is one of the most outstanding exponents of the Enlightenment philosophy of all time. The originality of his thought lies in an attempt that could be described as dramatic, since he paid with his life the price of his ideas, to reconcile the Zoroastrian tradition before Islam with the legacy of Greek Peripatetic philosophy and
PHILOSOPHY, RELIGION, THOUGHT
THE HEPTADIC ORGANIZATION OF GNOSEOLOGY IN THE IRANIAN CULTURAL TRADITION
Alfred Gutiérrez-Kavanagh – Professor of RRII (Oriente Medio), UPC – Madrid Summary: Coinciding with our seventh Iranology congress, I have considered that it would be an appropriate forum to reflect on the concept of "seven" in the cultures of the Middle East, not so much from a merely numerological or symbolic perspective, but more rooted in the Archaeology of knowledge that
ANTHROPOLOGY, HISTORY, PHILOSOPHY
Events in February–March 2023
3030. January, 2023 3131. January, 2023 11. February, 2023 22. February, 2023 33. February, 2023 44. February, 2023 55. February, 2023
66. February, 2023 77. February, 2023 88. February, 2023 99. February, 2023 1010. February, 2023 1111. February, 2023 1212. February, 2023
1313. February, 2023 1414. February, 2023 1515. February, 2023 1616. February, 2023 1717. February, 2023 1818. February, 2023 1919. February, 2023
2727. February, 2023 2828. February, 2023 11. March, 2023 22. March, 2023 33. March, 2023 44. March, 2023 55. March, 2023

redpajama_set_name: RedPajamaCommonCrawl


text:
Home News Kathy Gyngell: How Hammond could put marriage back in fashion
Kathy Gyngell: How Hammond could put marriage back in fashion
By Kathy Gyngell
No man is an island. The UK income tax system chooses to treat its citizens as though each were one; as though each were an autonomous unit without responsibility for, or need of, anyone else but the State; as though families no longer mattered.
Almost uniquely amongst OECD countries, the UK refuses to recognise family responsibilities in the tax system. The only 'recognition' of 'family' has been provided by inflated benefits (in the form of a multi-billion-pound annual tax credits bill), which in turn sustain ever more family set-ups dependent on the State while failing to provide the stability and security children need.
Were Philip Hammond to produce a Budget of vision, it is this, the reform of family taxation, that should be at its core. Were he genuinely keen to take on socialism, this is where he should start.
Given the fashion for apologies these days, he could well start with one. On behalf of successive Conservative governments since 1985, when independent taxation was introduced without any counter-measure to acknowledge the majority of single-earner couple families at the time, he could apologise for the family breakdown they are responsible for. He could apologise to the children without fathers for the blight of state welfare dependency they grew up in and the institutionalised child care inflicted on them which Conservative, as much as Labour, governments presided over. I am half-serious. If you don't understand why, just read this heartfelt account of a father's terrible doubts about putting his baby into long daycare.
Maybe Theresa May, Philip Hammond, David Cameron and George Osborne should all, in turn, apologise for making it so hard for people to be married, to make the right choices for the sake of their children.
I am not holding my breath. Politicians on the Right are as blind as those on the Left to the relationship between fiscal individualism, social breakdown and neglect of children. At best they see benefit reform – 'making work pay' – as the social cure-all when it is only one half of the story. Marriage is the other, and more critical, half. The few who dare mention this heresy do so with scared and hesitant voices when they need to raise them in anger.
Care's devastating new family taxation report, The Taxation of Families: International Comparisons 2016, should stiffen their sinews to the task if nothing else does. Reading this meticulously researched comparison left me with a rising fury at the sheer immorality of the UK's income tax system. No, not that the rich are hit too much or too little. But at the extent to which single living is incentivised and married family life and motherhood is penalised. Really, the Treasury might as well declare war on marriage. Be stupid enough to be responsible, to get a job, get married and plan your family, in that order, and you can expect to be punished, especially if you are poor already. Have the temerity to believe that your children need one stay-at-home parent and you can expect to be punished till it hurts, in fact till you give in and do what the State tells you, which is to go to work and offload your kids.
Whatever the squeals of protest from aggressive childcare campaigners, in the mix of what George Osborne once called 'lifestyle' choices, the choice the tax system tilts violently against is not the 'right' to childcare, but the right of one-earner married couple parenthood.
For the sin of wanting this – even if only for a while – such families are crippled by the highest effective marginal tax rate in the developed world. For this 'luxury' they pay in tax 70 per cent more than a comparable French family, more than twice as much as a comparable US family and 15 times as much as a comparable German family.
'Whereas across the OECD on average the tax burden placed on such a family tends to be closer to half that on a single person on the same wage, the UK burden is more than 75 per cent of that placed on a single person on the same wage,' the Care report states.
If Mrs May's former aide Nick Timothy is still advising on her on those 'just about managing families', then he might put careful perusal of Care's report at the top of his agenda, and ask himself what possible justification there is for a tax burden on one-earner married families that is 20 per cent greater than the OECD average. Forget austerity, is it any wonder that married families in all but the highest income brackets are struggling? Is it any wonder that married family life no longer beckons?
As for any idea that marriage is becoming fashionable again, forget it. It has taken less than 40 years for the male marriage rate to plummet – from 50 males per thousand per year then to a present-day 20 per thousand. What this means is that half of men no longer get married, and three million children living in lone-parent, mainly mother-led, families are doomed to fatherlessness in effect.
The family instability consequences of our marriage-lite society and the poor outcomes for children are well documented, as is the sociological evidence that marriage (or the lack of it) is the key predictor of poverty. If further confirmation were needed, Harry Benson's study of 11,000 families published this week provides it. Having parents who split up is the strongest single negative influence on girls' mental health. Mrs May, please also note, given the adolescent mental health repercussions, that cohabiting couples split at five times the rate of married couples.
There is nothing moral or pragmatic about the Conservatives continuing their policy of extreme fiscal individualism. It is immoral.
The Conservatives need to wake up. Polarising society on marriage lines is an own goal. It delivers an inequality and injustice that Mr Corbyn's socialism can never mitigate but will forever thrive on.
Previous articlePaul T Horgan: Never forget the atrocities that gave us Gerry Adams, politician
Next articleDavid Keighley's BBC Watch: EU's Brexit 'wall of silence' goes unchallenged
Kathy Gyngellhttps://www.conservativewoman.co.uk/the-editors/
Kathy is Editor of The Conservative Woman. She is @KathyConWom on Twitter.


text:
Kevin McKinney Indulged Me With An Acoustic Version of "Bonecrusher"
Local Licks, Uncategorized
Kevin McKinney was on Local Licks Tuesday night, and he played an acoustic version of "Bonecrusher." We also played some of his new material–"Shoulder," "Waking Up The Neighbors," and "Wasted." Be sure to check out the Bonecrushers group page on Facebook!
Also on Local Licks, I played:
The Heartless Bastards – "How Low" Their new album A Beautiful Life,, comes on on September 10th
Johnny Goudie – a new song called "All New Flavor" He just had a burger named (coincidentally) after his former band! Grab a "Little Champion" burger at Radio Coffee & Beer.
I was pleased to debut a brand new song from Ryan Thomas Holley called "Home"
Zach Person – "Can't Stop Running"
Christine Renner – "Sweet" This 21-year old just released a video for this song. She said she "wanted the video to look like it's 1981 backstage at a Joan Jett concert." Check it out here
Stupid Drama – "Angels Sigh"
~LL
Chuck Loesch January 12, 2022
AMORPHIS SYLVAINE HOAXED TUESDAY, MAY 3, 2022 COME AND TAKE IT LIVE DOORS AT 6:00 PM $25 ADV / $30 DOS
SUFFOCATION ATHEIST SOREPTION CONTRARIAN FRIDAY, JUNE 10, 2022 COME AND TAKE IT LIVE DOORS AT 6:00 PM $20 ADV / $25 DOS
Chuck Loesch December 16, 2021
ENTHEOS INFERI VULVODYNIA ATHANATOS SATURDAY, APRIL 23, 2022 COME AND TAKE IT LIVE DOORS AT 6:00 PM $18 ADV / $22 DOS
Emily Wolfe on Local Licks! Win tickets to her Local Licks Live Show!
llowe December 14, 2021
Tune in tonight at 11:00 as I welcome Emily Wolfe to the KLBJ studios! We'll be talking about her new album Outlier, as well as giving away tickets to her Local Licks Live show Saturday at Haute Spot
Stones Hoot Artists on Local Licks!
llowe November 29, 2021
Tune in at 11:00 Tuesday night (11/30) as I welcome some of the artists who are part of Saturday's Rolling Stones Hoot Night to the KLBJ studios! We'll hear Meagan Tubb, and English Teeth with Kevin McKinney
Chuck Loesch November 17, 2021
OBSCURA ABYSMAL DAWN VALE OF PNATH INTERLOPER FRIDAY, FEBRUARY 11, 2022 COME AND TAKE IT LIVE DOORS AT 6:00 PM $20 ADV / $25 DOS


"Is wealth a sign of righteousness or of wickedness?"
/ Evolution's Claims / The Bible / Contradictions / Is wealth a sign of righteousness or of wickedness?
Wealth is not necessarily a sign of either righteousness or wickedness. As a general rule, however, throughout history those who have had great wealth have tended to forget God and reject Him. However, God can bless people with wealth if He so chooses. Furthermore, those who know Christ will be wealthy some day in Heaven. Thus, there is no contradiction.


where the "Hitler should have listen to his general " come from?
At what point did Germany lose WW2?
Fall Gelb and Mechelen Incident.
Was Hitler and the Wehrmacht anticipating/fearing a Soviet invasion?
Where in Russia is Kollekttinsky ?
Seeking to confirm / verify Heer unit designation ?
What is this word / place?
Did the 3. Fallschirmjäger-Division Combat with the 29th Infantry and 101st Airborne in 1944?
The Heeresgruppe A military planning in Fall Blau.


text:
Ezzy Pearson
Astrophysicist. Science communicator. Writer
Audio and Radio
Science Talks
Review: Blindscape
I recently played Blindscape, a free mobile 'game', though it is far more accurately described on its title page as 'an interactive audio story'.
The quirk of this game is that the screen is completely blank.
Instead, a blind narrator tells you his story as he tries to escape the confines of his apartment.
You have to tap randomly about on the black screen to find and open doors then navigate around by sound (you'll need earphones).
It was a really interesting way to use the format without feeling gimmicky as it helped you to get into the mind set of someone who couldn't see, and forcing you to wear headphones shut out the world allowing you to be fully immersed in the narrative.
My only real complaint was that it was only 10 minutes long and left me wanting to know more about this world.
That said, I think the format might have dragged if it had lasted much longer, and the reason I decided to play it in the first place was because of its brevity.
I'm really interested in seeing how new media can be used to tell stories in a much more interactive way – Blindscape really showed that sometimes the best way to do that is to tear away all the bells and whistles and let the story do the talking.
This entry was posted in Games, Review and tagged blind, blindscape, storytelling, video games, videogame by admin. Bookmark the permalink.
Copyright © Elizabeth Pearson 2013. All rights reserved. Header image by Steve Cross.


text:
Tir na Nog, The Land of Youth
Linda Jo Martin
I'm a book blogger, the Book Lady at YouTube, and a novelist and memoir writer.
Edward Robert Hughes, 1908
A mythical land of the young, far beyond the sea....
Tir na Nog is an island in ancient Irish mythology. Tir na Nog, in the Old Irish language, means Land of Youth.
The Irish had many legends, and Tir na Nog was in some of them. It is an island far to the west, beyond the Irish shoreline, and beyond the far-reaches of the maps.
This mystical island was the home of the fairies. Few human beings were invited to live there.
Oisin's adventure in Tir na Nog
...with a beautiful fairy!
A fairy woman named Niamh Chinn Oir (Niamh of the Golden Hair) fell in love with Oisin and took him with her to Tir na Nog.
They were married and she gave birth to a son, Oscar, and a daughter, Plor na mBan (Flower of Women).
After about three years Oisin decided to return to Ireland.
Niamh gave Oisin her white horse, Embarr. She warned him not to dismount from the horse or he would become instantly old.
When he arrived in Ireland he realized three hundred years had gone by. He returned to his home and found the hill of Almu in ruins.
He tried to help some men who were building a road by lifting a stone onto a wagon. Tragically, his girth broke and he fell to the ground, instantly aging into an extremely withered old man.
Embarr, the horse, returned to Tir na Nog, and Oisin passed away.
Top 7 ways to Learn Digital Marketing Effectively
2021 Divisional Round Preview & Predictions
Was there any truth to this legend?
...the Irish used to think so.
In olden times many if not most of the Irish truly believed their ancient legends were based on truth, and perhaps some still do.
According to Alwyn and Brinley Rees, authors of Celtic Heritage:
"The old storytellers believed in all the marvels of the tales. Hector Maclean, writing in 1860, observed that the adventures of heroes such as Ossian were as true and real to the storytellers of Barra and those who listened to them as were the latest exploits of the British Army to the readers of newspapers. Even in our own day Professor Delargy speaks of the horrified dismay shown by a storyteller in Co. Kerry when one of his audience ventured to cast doubt upon the story of Oisín returning from the Land of Youth and to wonder whether Oisín ever existed."
Where do the souls of the dead go?
The Irish believed that on moonlit nights the souls of the dead could be seen over the Skellig rocks, traveling toward Tir na Nog.
Beautiful music inspired by the Irish legend of Tir Na Nog.
Tír na nÃg
Actually, Tír na nÃg is the correct spelling. I had to remove the accents for most of this page... sorry! Sometimes computers don't translate the accented letters right.
This Irish movie, Into the West (1993) is my favorite movie of all time. The beautiful white horse in the movie is named Tir na Nog.
In the movie a grandfather gypsy finds the horse near the ocean, then goes to Dublin to visit his son and grandchildren. The horse follows him and the grandsons fall in love with him. The father and sons, living in a tenement, allow the horse to move in with them. Of course the building superintendent cannot allow that.
I won't give any more information. No spoilers! I'm only saying, this is an awesome movie. The horse is gorgeous and the boys are precious.
Van Morrison singing about Tir Na Nog - ...beautiful song...
Any comments? - ...always welcome...
Mech from Bosnia and Herzegvina on November 09, 2012:
Beautiful! Excellent lens, I enjoyed it very much.
Nathalie Roy from France (Canadian expat) on December 04, 2011:
I love Irish legends too.
Movies & Movie Reviews
The Best Costumes in the 2019 "Downton Abbey" Movie
By RiverCygnet
The Bihar Famine (1966-67): Beyond Politics, Aid and Diplomacy
By Ashutosh Joshi
10 Benefits of Camping Vacations
The Role of Religion in History & Society
The Late Robin Williams Is Not Burning In Hell
Internet & the Web
RankAlly
By gaurav1330013
Yoga Supplies: Mats, Clothes, Props, Bags
By Melanie Palen
Income & Making Money
General rules for writing ezine articles
By Naztul521
Discuss the submission of the article in the eZine directory and its viral effects
Tax & Taxes
Get Going on Your Tax Return Now
By CoolBunch
Was the Sacking of Louis Van Gaal the Right Decision?
By Andrew Szekler
What Would Erik Ten Hag's Manchester United Starting Eleven Look Like?
By Val Karas
Arrigo Sacchi's Immortals


The University of the Basque Country is the public university of the Autonomous Community of the Basque Country. We are a public university, multilingual, research-oriented with an international outlook, striving to provide society with quality services in innovation and the development and dissemination of knowledge. In 2010 the University obtained the recognition as "International Campus of Excellence" by the Spanish Ministry of Education.
With a total of 32 Schools and Faculties distributed amgonst three campuses, we offer a wide range of degrees in all areas of knowledge; from experimental sciences, engineering and humanities to health sciences, economics, arts or architecture. All degrees are fully adapted to the European Higher Education Area. We offer 69 undergraduate programs, and 165 postgraduate programs. 32 of our PhD programs have been granted an excellence quality award by the Spanish Ministry of Education. Our PhD studies lead to more than 300 graduates being granted their doctorate degrees every year. We have more than 45,000 students, almost 4,000 teaching staff, an administrative staff of 1,800 persons and our annual budget is 609 million euros.
In recent years the University of the Basque Country has achieved major increases in its scientific output in terms of international visibility in all areas of knowledge.


Many of the resources available to download on this site will include my deadname*. If you wish to cite me, please use Ynda Jas regardless of the name I used when I wrote/presented earlier work.
Example: Jas, Ynda. 2015. Sexual Orientation, Phonetic Variation and the Roots and Accuracy of Perception in the Speech of Northern England English-speaking Men. Master's thesis, University of York.
* Such resources are intended for personal use, but if you wish to distribute them elsewhere then I am happy to provide versions with my name updated on request.


text:
\subsection{Missing labels}
\label{subsec:missing_labels}
In XMLC, the observed data might not follow the distribution we want to learn about.
As an illustrative example, take the Wikipedia-500k dataset.
The content of a Wikipedia article should be matched with a set of categories the article belongs to.
Such a dataset can be easily created by scraping existing Wikipedia annotations.
However, there are about \num{500000} categories on Wikipedia,
and it is clear that the original authors and curators have
never checked every single category for each article.%
\footnote{If it took a human one second to check a category for an article,
then annotating a single article fully would take almost 6 days.}
On the other hand, each category that has been assigned to an article has been
verified by a human to be relevant.
Therefore, the labeling error can be assumed to be strongly one-sided:
There may be many missing labels, but spurious labels should be uncommon.

To contrast ground-truth labels $\vec{Y}$ with those actually available,
we denote the \emph{observed labels} $\tilde{\vec{Y}}$ using a tilde.%
\footnote{Note that other papers, including~\cite{Jain_et_al_2016},
use often a slightly different notation.}
Mathematically, the setting studied in this paper is defined by
\begin{equation}
 \probability[\tilde{\vec{Y}} \preceq \vec{Y} \, | \, X] = 1\,, \qquad
 \probability[\tilde{\vec{Y}} \npreceq \vec{Y} \, | \, X] = 0\,,
\label{eqn:general_missing_labels_problem}
\end{equation}
where $\tilde{\vec{Y}} \preceq \vec{Y}$ means that $\tilde{Y}_j \le Y_j$ for all $j \in [m]$,
and $\tilde{\vec{Y}} \npreceq \vec{Y}$ means that
there is at least one label for which $\tilde{Y}_j > Y_j$.
Notice that the above equations cover also the no noise case,
as we may have $\probability[\tilde{\vec{Y}} = \vec{Y} \, | \, X] = 1$.

Reconstruction of the ground truth distribution from the observed one, in the general case,
is not a trivial task from the statistical and computational perspective,
as it requires an exponential number of parameters.
Let $\eta_{\vec{y}}(x) \coloneqq \probability[\vec{Y}=\vec{y} \, | \, X = x]$
and $\tilde{\eta}_{\vec{y}}(x) \coloneqq \probability[\tilde{\vec{Y}}=\vec{y} \, | \, X = x]$.
We have then
\begin{equation}
\tilde{\eta}_{\tilde{\vec{y}}}(\vec{x}) =
\sum_{\vec{y}}
p_{\tilde{\vec{y}}}(\vec{y}, x)
\eta_{\vec{y}}(\vec{x})\,,
\end{equation}
where $p_{\tilde{\vec{y}}}(\vec{y}, x) \coloneqq \probability[\tilde{\vec{Y}}=\tilde{\vec{y}} \, | \, \vec{Y}=\vec{y}, X = x]$ is
a propensity of observing $\tilde{\vec{y}}$ for ground-truth labels $\vec{y}$ and instance $x$.
Notice that from (\ref{eqn:general_missing_labels_problem})
we have $p_{\tilde{\vec{y}}}(\vec{y}, x) = 0$ for $\tilde{\vec{y}} \npreceq \vec{y}$.
Furthermore, let $\vec{\eta}_{\mathcal{Y}}(x)$ and $\tilde{\vec{\eta}}_{\mathcal{Y}}(x)$
be vectors of $\eta_{\vec{y}}(x)$ and $\tilde{\eta}_{\vec{y}}(x)$, respectively,
for all $\vec{y} \in \mathcal{Y}$ given in some predefined order $\pi$.
Let $\mathsf{C}$ be a matrix containing all propensities $p_{\tilde{\vec{y}}}(\vec{y}, x)$,
with rows and columns corresponding to $\tilde{\vec{Y}}$ and $\vec{Y}$, respectively,
and organized according to $\pi$.
Then, we get:
\begin{equation}
\tilde{\vec{\eta}}_{\mathcal{Y}}(x) = \mathsf{C} \vec{\eta}_{\mathcal{Y}}(x) \,,
\end{equation}
and, finally:
\begin{equation}
\vec{\eta}_{\mathcal{Y}}(x) = \mathsf{C}^{-1}\tilde{\vec{\eta}}_{\mathcal{Y}}(x) \,,
\end{equation}
where we need to assume that $\mathsf{C}$ is invertible.

Because of the practical reasons,
a much simpler, label-wise, propensities
are commonly used that are defined for each label separately:
\begin{equation}
 p_j(X) \coloneqq \probability[\tilde{Y}_j=1 \, | \, Y_j = 1, X]\,.
\label{eq:missing-model}
\end{equation}

Let $\tilde{\eta}_{j}(x) \coloneqq \probability[\tilde{Y}_j = 1 \, | \, X = x]$
and $\eta_{j}(x) \coloneqq \probability[Y_j = 1 \, | \, X = x]$.
We have then:
\begin{equation}
\tilde{\eta}_j(x) = p_j(x) \eta_j(x)\,, \qquad
\eta_j(x) = \tilde{\eta}_j(x) / p_j(x)\,.
\label{eqn:prob_adjustment}
\end{equation}

If propensities are known, then they can be used to construct an unbiased, task or
surrogate, loss $\tilde{\ell}$ \citep{van_rooyen_theory_2017}
in the sense that
\begin{equation}
    \forall h: \; \risk_{\ell}[h; X, \vec{Y}] = \risk_{\tilde{\ell}}[h; X, \tilde{\vec{Y}}].
    \label{eq:unbiased-risk-property}
\end{equation}
The construction of the unbiased counterpart depends on the
form of propensities,
e.g., the label-wise propensities (\ref{eq:missing-model}) are sufficient
for losses decomposable over labels~\citep{natarajan_cost-sensitive_2017}
like Hamming loss or binary cross-entropy,
but might not be for more complex losses without additional assumptions~\citep{Schultheis_Babbar_2021}.
The unbiased losses can be used in training procedures~\citep{Jain_et_al_2016,Qaraei_et_al_2021}
or for estimating the performance of classifiers.
For some losses, such as Hamming loss or precision@$k$, the Bayes
classifier can be written as a function of the conditional label distributions
$\eta_j(x)$.
In this case, one can adjust existing inference procedures to use (\ref{eqn:prob_adjustment})
to obtain estimates of $\eta_j(x)$
from estimates of $\tilde{\eta}_j(x)$~\citep{propensity-plt}.

\subsection{Long-tailed label distribution}
\label{subsec:long-tail-labels}

A defining characteristic of extreme classification data is that the label
distribution is highly imbalanced. In the binary case, the amount of imbalance
is completely determined by the imbalance ratio
$\frac{\probability[Y=0]}{\probability[Y=1]}$.
In this sense, almost every binary problem
corresponding to a label is highly imbalanced in XMLC, i.e., only a small
fraction of training instances will be associated with that label. However,
in XMLC, the data are also imbalanced when comparing different labels. In
analogy to the binary case, we can define an inter-label imbalance ratio
through $\operatorname{ILIR} = \frac{\max\{\probability[Y_i=1] : i \in [m]\}}{\min\{ \probability[Y_j = 1] : j \in
[m] \}}$.\footnote{This concept is also used in \emph{multiclass}
classification, e.g. the \emph{imbalance factor} of \citet{cui2019class}.}
Nevertheless, the imbalance factor does not cover an important
property of the label distribution. It could be that most labels have a
large number of positives, but some have very few, or vice-versa. The latter
case is what happens in XMLC, where the label distribution
is said to be \emph{long-tailed}~\citep{Bhatia_et_al_2015, Babbar_Scholkopf_2017}.

\begin{table}
\small
\caption{Imbalance characteristics of typical XMLC datasets.}
\label{table:dataset-stats}
\pgfplotstabletypeset[
every head row/.style={
  before row={\toprule},
  after row={\midrule}},
  columns={dataset,instances,head imbalance,class imbalance,80percent},
every last row/.style={after row=\bottomrule},
columns/dataset/.style={string type, column type={l|}, column name={Dataset}},
columns/head imbalance/.style={column type = {r}, column name={min IR},zerofill,precision=1},
columns/instances/.style={column type = {r}, column name={Instances}, precision=2, sci,zerofill},
columns/class imbalance/.style={column type = {r}, column name={ILIR}, sci,zerofill, precision=2},
columns/80percent/.style={column type = {r}, column name={Pos-80\%},zerofill,precision=1}
]{data/datasets.txt}
\end{table}

In \autoref{table:dataset-stats}, these imbalance measures are shown for
several XMLC datasets.
We use \textit{min IR} to denote the binary imbalance ratio of the head label,
i.e., the label with the largest fraction of positive instances
(therefore, its IR is the smallest),
and Pos-$80\%$ to indicate the minimum fraction of class labels that retain $80\%$ of
positive labels (i.e., $y_j = 1$) in the training set.
For example, in Delicious-200K, only four percent of the class labels contain $80\%$ of the positive labels.

In addition to the number of positive instances of a sparse label,
their distribution within the feature space can be very important.
If $Y_j$ has few positives, but in a small pocket $\mathcal{X}_{+}$ of
the feature space it still fulfills
$\probability[Y_j = 1 \mid X \in \mathcal{X}_{+}]\approx 1$,
then a learning algorithm might still learn a reasonable decision boundary,
especially if the overall number of samples is large enough \citep[p. 23]{fernandez2018learning}.
In contrast, if $\probability[Y_j = 1 \mid X] \ll 1$
everywhere (called \emph{uniform class imbalance} by \citet{singh2021statistical}),
learning to recognize the given class might be infeasible.

\subsubsection{Propensity-scored losses}

Popular XMLC performance metrics
focus on the highest scored labels by the prediction algorithm.
Examples of such metrics are \emph{precision at k} ($\precision[k]$), \emph{recall at k} ($\recall[k]$),
or \emph{(normalized) discounted cumulative gain} $\ndcg[k]$.
For these metrics, unbiased estimates
in the sense of \eqref{eq:unbiased-risk-property} can be calculated,
which are called the \emph{propensity-scored} (PS) variants of these metrics
(more examples in \citep[Table 1]{Jain_et_al_2016}).
Table~\ref{tab:psmetrics} gives the formal definitions,
where $\topk[k]$ maps a vector to the indices of its top-k components,
$\rankof{j}(\hat{\vec{y}})$ gives the ranking of the $j$-th element in the vector,
and $p_j$ is the propensity for label $j$.
Let us notice that, in general, $p_j$ shall depend on $x$,
but \citet{Jain_et_al_2016} practically assume $p_j$ to be a constant value for each label $j$.
Moreover, of the above unbiased estimates, only $\psprecision$ and $\psndcg$ have found widespread use
\citep{Bhatia_et_al_2016}, because $\psrecall$ still requires the knowledge of
the total number of relevant labels $\|\vec{y}\|_1$.
\input{tables/xmlc-metrics}

Because \citet{Jain_et_al_2016} observed that the unbiased estimates
could results in values larger than one, they suggest
a normalized version of these metrics to be reported (cf. Section~\ref{subsec:implausible_results}).
In subsequent literature, the distinction between the unbiased metrics
and the normalized versions is not always preserved, e.g., \citet{Bhatia_et_al_2016} present unbiased formulas
but lists normalized values.

\subsubsection{Empirical propensity model}
\label{subsec:empirical-model}
In order to use the propensity-scored loss functions, one needs to have
the propensities available for the individual labels. Since true propensities
are unknown for the XMLC benchmark datasets,
\citet{Jain_et_al_2016} proposed to model propensities
as a function of labels frequencies,
resulting in propensities being a constant value for each label.

Let $\phi$ denote a propensity model.
The model defined in~\citep{Jain_et_al_2016}
can be expressed via label priors $\tilde \pi_j \coloneqq \probability[\tilde{Y}_j=1]$:
\begin{equation}
\label{eq:jain-prop-model}
\!\!\! p_j \!=\! \phi_{\xmlcpropensitymodel}(\tilde \pi_j; n, a, b) \!\coloneqq\! \frac{1}{1 + (\log n - 1)(b + 1)^a e^{-a \log (n \tilde \pi_j + b)}} \,,
\end{equation}
where $n$ is the number of training instances, and $a$ and $b$ are
dataset-dependent parameters.

In order to arrive at this model and determine values for $a$ and $b$,
\citet{Jain_et_al_2016} investigated two datasets in which ancillary
information could be used to identify some missing labels.

For a Wikipedia-based dataset, the parameters of the model have been
estimated with the help of a label hierarchy. They assumed
that if a label is relevant to an article, then all its ancestors in the
hierarchy should also be relevant. If not present, they are counted as
missing. This allows plotting the fraction of instances in which the label is
missing over the number of instances in which it appears. This seems to
follow a sigmoidal shape as described by \eqref{eq:jain-prop-model}, see~Figure~\ref{fig:wikipedia-ps-est}.
The
parameters $a$ and $b$ were then determined by fitting the model against the
estimated values, where only labels with more than 4 descendants were used to
improve robustness. The obtained values are $a = 0.5, b = 0.4$.
\input{figures/wikipedia-ps-est}

For the Amazon data set, which is an item-to-item recommendation task,
missing labels have been approximated using ``also viewed'' and ``also bought`` information.
It was assumed that a label $j$ (an item) is relevant to all the items viewed
along with items that were also bought with label $j$, as proposed by \citet{McAuley_et_al_2015}.
The obtained values are $a = 0.6$ and $b = 2.6$.

For other data sets the authors propose, if there is no other possibility of
estimating parameters $a$ and $b$, to use averages of the values obtained for
Wikipedia and Amazon data sets (which are $a = 0.55$, $b = 1.5$). This, in fact,
has become a standard followed in many papers without questioning its rationality.

The above propensity model is then typically used in the metric of choice
for model selection and evaluation.
It has also been incorporated into training procedures.
For example, decision tree methods can directly use the
propensity-scored variants of metrics such as precision$@k$ or nDCG$@k$ \citep{Jain_et_al_2016}.
Alternatively, one can use unbiased or upper-bounded propensity-scored
surrogate losses~\citep{Qaraei_et_al_2021}.

\subsubsection{Propensity and long tails}
\label{subsec:prop-long-tail}
The form of \eqref{eq:jain-prop-model} implies that tail labels are assigned
lower propensities, which means that in metrics like those in \autoref{tab:psmetrics} these
tail labels, if predicted correctly, will be weighted more strongly than head
labels. In particular, the resulting weightings resemble existing weighting
schemes used for long-tailed learning tasks, leading the authors to conclude:

\begingroup
\addtolength\leftmargini{-0.1in}
\begin{quote}
Such weights arise naturally as inverse propensities in the unbiased losses
developed in this paper. [...] This not only provides a sound theoretical
justification of label weighting heuristics for recommending rare items but
also leads to a more principled setting of the weights.
\end{quote}
\endgroup

As a result, propensity-scored variants are also viewed as metrics in their own right, and are currently used both
to counteract missing labels (as unbiased estimates) and to weigh tail labels (as independent metrics),
becoming established performance metrics commonly used in XMLC.%
\footnote{We list several examples of references to propensity-scored losses:
``We examined the performance on tail labels by PSP@k''~\citep{You_et_al_2019};
``We achieve high precision and propensity scores,
thus demonstrating the effectiveness of our method even on infrequent tail labels.''~\citep{guo_breaking_2019};
``capture prediction accuracy of a learning algorithm at top-k slots of prediction, and also the diversity of
prediction by giving higher score for predicting rarely occurring
tail-labels''~\citep{Babbar_Scholkopf_2019};
``propensity scored precision@k which has recently been shown to be an
unbiased, and more suitable, metric''~\citep{jain_slice_2019};
``which leads to better performance on tail labels.''~\citep{Yen_et_al_2017};
``propensity scored variant which is unbiased and assigns higher rewards for accurate tail label predictions'',
``evaluate prediction performance on tail labels using propensity scored variants''~\citep{Khandagale_et_al_2019};
``replacing the nDCG loss with its propensity scored variant and using additional classifiers designed for tail labels''~\citep{Tagami_2017}.
}

\subsection{Discussion of missing-labels assumptions}

In order to derive unbiased loss functions, we need to impose assumptions on
the process of how labels go missing, as initially discussed in Section~\ref{subsec:missing_labels}.
Unfortunately, \citet{Jain_et_al_2016} sent a potentially misleading message in this regard.
Their Theorem 4.1 proves
\begin{equation}
    \expectation[\ell(\vec{Y}, \hat{\vec{y}})] = \expectation[\tilde{\ell}(\tilde{\vec{Y}}, \hat{\vec{y}})]\,,
\end{equation}
for any \textbf{fixed} prediction $\hat{\vec{y}}$ without a clear dependence on $X$.
This also implies that the assumptions behind the propensities are unclear.
Even if we assume the propensities to be constant for label $j$,
the exact form of this assumption is necessary to properly prove unbiasedness
in the sense of (\ref{eq:unbiased-risk-property}).
Notice that $\probability[\tilde{Y}_j = 1 \, | \, Y_j = 1] = p_j$
does not imply $\probability[\tilde{Y}_j = 1 \, | \, Y_j = 1, X] = p_j$.
Moreover, for more complex functions, such as recall$@k$, this assumption may take the form of
$\probability[\tilde{Y}_j = 1 \, | \, Y_j = 1, \vec{Y}_{\!\neg j}, X] = p_j$,
where  $\vec{Y}_{\!\neg j}$ represents ground-truth labels without label $j$
(see Appendix for an example).

In general, we cannot expect the independence of missing labels from the
instance's features to hold. Consider, for example, cases where the feature and
label space are of a similar origin~\citep{dahiya2021siamesexml}, such as matching
Wikipedia titles or articles to categories.
It seems unlikely that a label such as ``Italy'' would be missing
for articles containing the word ``Italy'' in the subject,
but it might be missing for articles that still pertain to Italy
but do not feature the word ``Italy'' in the title.
The assumption that propensities are constant for each label simplifies the model
significantly and leads to much simpler computational procedures.
Unfortunately, if this assumption is not satisfied,
then one may get implausible results as discussed later.

The assumption that the propensities do not depend on other labels going missing
does not need to hold in practice as well.
For example, a user that tagged the article for ``Italy'' with ``Member states
of the European Union'' might be primed to think of more examples of
organizations in which Italy is a member, and thus e.g., ``Current member
states of the United Nations'' might be less likely to be forgotten than if
the EU membership had been forgotten.
Fortunately, in many cases, the unbiased
estimate does not actually require this dependence ---
if the loss function can be written as a sum over contributions from each label individually,
then the labels do not interact with each other
and the label-wise properties are sufficient to obtain unbiased losses.
This is the case for the popular $\psprecision$ and $\psndcg$ metrics.

\subsection{Shortcomings of the \text{JPV}{} propensity model}
\label{subsec:shortcomings}

Let us discuss several issues of the \text{JPV}{} model,
concerning theoretical and empirical shortcomings,
as well as some problems in the way the parameters of the model have been established.

\paragraph{Scaling behavior}
Let us first observe that \eqref{eq:jain-prop-model} does not preserve propensity estimates
if the amount of data is changed, without changing its characteristics,
e.g., by sub- or over-sampling the dataset.
In particular, if one increases the amount of available data
by making multiple copies of the dataset,
which should not change the estimates of label priors $\tilde \pi_j$
given by $\tilde{n}_j/n$
(with  $\tilde n_j$ being the number of positive instances of label $j$
in the observed, noisy training set),
the \text{JPV}{} model will estimate propensities to be equal one,
i.e., no missing labels, as the amount of data goes to infinity:
\begin{multline}
    \!\!\!\!\!\lim_{n \rightarrow \infty} \phi_{\xmlcpropensitymodel}(\tilde \pi_j,  n) = \frac{1}{1 + (b + 1)^a  \lim_{n \rightarrow \infty} (\log n - 1)e^{-a \log (\tilde\pi_j n + b)}} \\
    = \frac{1}{1 + (b + 1)^a \lim_{n \rightarrow \infty} (\log n) (\tilde\pi_j n)^{-a}} = 1.
\end{multline}
This means that we cannot interpret $a$ and $b$ as parameters of some
underlying (unknown) process that describes the labeling process. As we cannot
even have fixed $a$ and $b$ when the data come from the same process, this
very much calls into question the approach of using values for $a$ and $b$
across datasets as is current practice.

\paragraph{Estimation process}
Setting aside structural concerns about \eqref{eq:jain-prop-model}, the
estimation of the parameters $a$ and $b$ still remains an issue. First, by
identifying missing labels based on meta-data as described in
Section~\ref{subsec:empirical-model}, only an upper-bound on the propensity is
estimated, since labels may also be missing in other ways.
For example, we tried to reproduce the procedure of propensity estimation on the Wikipedia dataset.
We have found that only around \num{40000} out of \num{500000}
labels meet the criteria of the sufficient number of descendants selected by the authors,
and around \num{300000} labels are without descendants,
so they would never be considered missing by this protocol.

Further, one might argue that in cases in which missing labels can be
identified by some side-channel information such as label hierarchies, then
one can directly impute these missing labels and need not worry about training
with missing labels.

\paragraph{Propensity as a function of frequency}
This still leaves the question of whether such estimates are sensible.
Even though there is clearly a trend that labels within a given range of
frequency have -- on average -- a certain propensity, for each individual label
the actual propensity can fluctuate widely around this mean, as shown in
\autoref{fig:wikipedia-ps-est} that we obtained following the original procedure for estimating propensities.

\paragraph{Reproducibility}

The description of the process of propensity estimation in~\citep{Jain_et_al_2016}
is rather sparse on details.
While meta-data for Wikipedia is easily obtainable,
it is not clear what is the source of ancillary information
that has been used for the Amazon dataset.
Additionally, depending on the preprocessing steps
and criteria, such as the number of descendants in the label hierarchy,
one can achieve very different estimates of parameters $a$ and $b$.

\subsection{Implausible results and normalization}
\label{subsec:implausible_results}

Despite being unable to verify the correctness
of the assumptions and the \text{JPV}{} model
without actual clean ground truth data,
we are still able to show
that the approach of~\citet{Jain_et_al_2016} leads to implausible results.
For example, $\psprecision[k]$,
as an unbiased estimate of $\precision[k]$ on the ground-truth data,
should be bounded between zero and one.
However, when calculating this measure for a real classifier,
the result may exceed this range substantially.
Of course, for an individual instance or a small subset of them,
the unbiased estimate does not need to fall into that range,
but a large deviation from the true value becomes exceedingly unlikely
when averaging over the entire dataset.

To circumvent this issue, \citet{Jain_et_al_2016} suggest
to report a normalized version of $\psprecision[k]$,
also calling this measure ``propensity-scored precision''.
The normalization is realized by dividing the metrics value
by the largest possible value that any prediction could have achieved on that data:
\begin{equation}
\label{eq:ps-loss-norm}
\textrm{Norm}\psprecision[k] = \frac{\sum_{i=1}^{n} \psprecision[k](\tilde{\vec{y}}_i, \hat{\vec{y}}_i)}{\sum_{i=1}^{n} \max_{\mathbf{z}} \psprecision[k](\tilde{\vec{y}}_i, \mathbf{z})} \,.
\end{equation}
The normalization introduces a factor
that is constant over the entire dataset,
and thus does not influence model selection.
However, it removes the interpretation of the received value
as an unbiased estimate of the metric on clean data,
and it hides the model misspecification.
Table~\ref{tab:norm-unnorm-psp-at-k} reports the values of both variants of $\psprecision[k]$,
showing how severe this issue is.

\input{tables/norm-unnorm-psp-at-k}

\subsection{The current use of propensity metrics}
\label{subseq:usage-problems}

It seems that the current use of propensity metrics mixes up,
in a not entirely clear way, two different issues, missing and tail labels.
As mentioned in Section~\ref{subsec:prop-long-tail},
these metrics might be used for the purpose of giving
more weight to tail labels.
In this case, the normalization step seems to be a valid procedure.
However, a propensity metric loses its original interpretation,
and it is just one way of accounting for tail labels,
without any concrete justification.
For this use case, it would be preferable to have a metric
that treats tail labels in a principled way.
As a first step towards that goal,
Section~\ref{sec:task-losses-for-long-tails} provides
some discussion on alternative task losses.

Only in the interpretation as a tail-performance promoting loss,
it does make sense to speak of a trade-off in performance
between vanilla and propensity-scored metrics, as these are conceptually different.
In the missing-labels interpretation, taking the propensities into account is not
calculating a different conceptual metric, but instead, the \emph{correct} way
of calculating the unweighted, but the true performance of a classifier.
Of course, in XMLC, both interpretations can be combined,
i.e., one would like to have a task loss that is adapted to tail labels,
but calculate it in a way that takes missing labels into account.
The closest to this in the literature is \citep{Qaraei_et_al_2021},
where training uses a loss that combines unbiased estimates and class-rebalancing,
but still, evaluation is performed using vanilla and propensity-scored metrics,
instead of a propensity-scored variant of a tail-weighted metric.

\subsection{Bias-controlled validation and test sets}
\label{sec:unbiased-sets}

If indeed our training data are biased by missing labels,
the best way to test, validate, and estimate the propensity model
is to use unbiased data or data with controlled bias.
In the latter case, the bias is controlled in such a way that
unbiased estimates can be easily computed.
Such data are definitely costly to get, but even a small set can be beneficial,
helping in selecting the right model to be used in production and in estimating its real performance.
This is a standard approach used in recommendation systems~\citep{Saito_et_al_2020,Yang_et_al_2018}
and search engines~\citep{Joachims_et_al_2018}.
Even if, in a given application, it is not possible to obtain an unbiased data due to some constraints,
we should investigate algorithms on benchmarks with unbiased or bias-controlled sets associated.
Without this investigation, it is hard to verify which methods are indeed working and which are misleading.
In many real-world applications, the final evaluation of the model
is performed in A/B tests on real, unbiased data.
We should avoid situations in which the results of A/B tests are against
our expectations coming from offline experiments.

To follow the above recommendation,
we have prepared several datasets of different types,
which can be used for experimentation with missing labels in XMLC.
The first type contains fully synthetic datasets.
The second one is a modification of the standard XMLC benchmarks.
The final dataset is a variant of the Yahoo R3 dataset,\footnote{\url{https://webscope.sandbox.yahoo.com/}}
transformed from a recommendation problem to multi-label classification.
Statistics of these datasets along with additional information are given in Appendix.

The synthetic datasets are generated in a similar way as in~\citep{Jimena_et_al_2014_ml_datagen}.
They are parameterized by the number $m$ of labels.
In the experiments reported below, we use $m=100$, which is not \textit{very} extreme
but suffices for investigating the propensity models.
Each label $j$ is represented by a $d$-dimensional hyper-ball $S_j$,
whose radius and center are generated randomly.
All those hyper-balls lay in a feature space being itself a $d$-dimensional hyper-ball $S$,
big enough to contain all hyper-spheres $S_j$.
Instances are uniformly generated in $S$
and each instance $x$ is associated with labels
whose hyper-balls contain $x$, i.e., $\relevantlabels{x} = \set{j \in [m]: x \in S_j}$.
The popularity (or priors) of the labels is directly determined by the radius of $S_j$.
We separately generate training, validation, and test sets from the above model.
We then apply a propensity model of choice to the training set to generate missing labels.
For some experiments, we also generate missing labels in validation and test sets.

For the original XMLC benchmark datasets, we assume that there are no missing labels.
We then merge the original train and test sets,
and take labels having at least $s$ positive instances.
We perform this step to select labels for which one can apply the noise models
without removing all positive labels.
We then split the data again into training and test sets,
and apply, similarly as above, a propensity model of choice to the training set to generate missing labels.
For some experiments, we extract a validation set from the test set.

In the original Yahoo R3 dataset, records are organized in a format of user-item ratings,
and each record contains a user ID, an item ID, and the user's rating for the item (from 1 to 5).
The training set contains over 300K ratings from 15.4K users to 1K items.
This set is biased as users select items from a limited list of options recommended by some algorithm.
The bias-controlled test set is obtained by collecting ratings from a subset of 5.4K users
to rate $r = 10$ randomly selected items.
To create a multi-label dataset we treat each item as a label ($m=1K$).
We consider ratings greater or equal to 4 as positive feedback and others as irrelevant.
We take users unique to the original training set (10K of users) to create a biased multi-label training set.
For each user, we randomly split positive feedback into equal halves.
We take the first half as features $x$ and the later half as labels $\vec{y}$.
Next, we use users present in both the original training and test set to create a test set with a controlled bias.
We again randomly select half of the positive feedbacks from the training set as features $x$ (to keep the same distribution of features)
and all positive feedback from the test set as labels $\vec{y}$.
We can also extract a validation set from the test set,
usually consisting of a half of users from the test set.

For a dataset created in such a way,
we can calculate estimates of training-set propensities $p^{\text{tr}}_j$ as:
\begin{equation}
    \label{eq:unbiased-propensities}
    \hat{p}^{\text{tr}}_j = \phi_{\text{direct}} = \hat{\pi}^{\text{tr}}_j p^{\text{c}}_j \left (\hat{\pi}^{\text{val}}_j \right )^{-1}
    \,,
\end{equation}
where $\hat{\pi}^{\text{tr}}_j$ and $\hat{\pi}^{\text{val}}_j$ are, respectively,
training- and validation-set estimates of the prior probability of label $j$,
and $p^{\text{c}}_j$ is the controlled propensity used for the validation and test set.
To estimate the label priors, one can use relative frequencies of labels in training and validation sets.
For $p^{\text{c}}_j$ we use a ratio of $r$ labels used for labelling to all $m$ labels, i.e., $p^{\text{c}}_j = \frac{r}{m} = 0.01$
for the Yahoo R3 dataset.

\subsection{Alternative propensity models}
\label{sec:alternative-propensity-models}

The \text{JPV}{} propensity model~(\ref{eq:jain-prop-model}) is not
the only one to consider. In fact, many different forms have been introduced
in other domains~\citep{Saito_et_al_2020,Yang_et_al_2018,Joachims_et_al_2018}.
We express the propensity models as functions of observed label priors
$\tilde \pi_j \coloneqq \probability[\tilde{Y}_j = 1]$, without direct relation to $n$,
which by construction avoids convergence issues discussed in Section~\ref{subsec:shortcomings}.%
\footnote{%
The simplest estimate of the priors are relative frequencies of labels, i.e.,
$\hat{\olabelprior}_j = \tilde{n}_j/n$. As we deal with many very
sparse labels, we should rather use more robust estimates, for example,
$\hat{\olabelprior}_j = (\tilde{n}_j + \alpha)/(n + \alpha)$.%
}

A propensity model used frequently in recommendation systems is given by the following power-law formulation:
\begin{equation}
p_j = \phi_{\text{P}}(\tilde \pi_j;\beta,\gamma) \coloneqq \left (\beta \tilde \pi_j \right )^\gamma\,.
\end{equation}
With $\beta = \max_j \tilde{n}_j/n$ we receive a model used,
for example, in~\citep{Yang_et_al_2018, Saito_et_al_2020},
while for $\beta = \gamma= 1$ we get a very simple model which might be used,
if estimation of the parameters is infeasible due to lack of unbiased or bias-controlled data.
Another solution could be to use the generalized logistic function, also called Richard's curve~\citep{Richards_1959_curve},
which is very flexible and its shape resembles~(\ref{eq:jain-prop-model}):
\begin{equation}
p_j = \phi_{\text{R}}(\tilde \pi_j; c,d,e,f,g,h) \coloneqq c+\frac{d - c}{\left (e+ f\exp(-g \tilde \pi_j) \right )^{1/h} }\,.
\end{equation}

The parameters of the models can be either set up according to a domain
knowledge or fit using an additional unbiased or bias-controlled dataset. In
the latter case, one can use standard non-linear optimization
methods~\citep{Bazaraa_nonlinear_programming}. We fit the models to inverse
propensities, minimizing squared errors $\|\hat{p}_j^{-1} -
\phi(\tilde \pi_j)^{-1}\|^2$ using the Levenberg-Marquardt
method~\citep{More_and_Watson_least_squares}. The errors are reported in the Appendix.

\input{figures/yahoo-r3-ps}
\autoref{fig:yahoo-r3-ps} illustrates the results of fitting the different propensity models for the
Yahoo R3 dataset. The actual training-set propensities have been obtained
using~(\ref{eq:unbiased-propensities}). The plot clearly shows that the
\text{JPV}{} model with $a=0.55$ and $b=1.5$, suggested as default
values, is not a good fit to the actual propensities. The same model, but with
$a$ and $b$ fitted to the data gives a degenerated solution because many values are out of codomain of~(\ref{eq:jain-prop-model}) when $n$ is that small.
On the other hand, $\phi_{\text{P}}$ and $\phi_{\text{R}}$ seem to give a good fit, but still
actual propensities are widely spread, suggesting that a model solely
depending on label priors might not be the best choice.

We have also trained prediction models using the above propensities to see
whether they help in improving (actual) precision$@k$ on the unbiased test set.
We use the one-vs-all approach in which probabilistic model $f_j(x)$, for label $j$,
is obtained by minimizing the unbiased variant of logistic loss~\citep{Saito_et_al_2020, Qaraei_et_al_2021}:
\begin{equation}
\label{eq:un-log-loss}
\ell(\tilde{y}_j, p_j, f_j(x)) =
    -\frac{\tilde{y}_j}{p_j} \log(f_j(x)) - \left(1 -\frac{\tilde{y}_j}{p_j}\right) \log(1 -  f_j(x))\,.
\end{equation}

\input{tables/results-yahoo-r3}
The results are given in \autoref{tab:results-yahoo-r3}.
As a baseline we also use a vanilla logistic loss which corresponds to $\phi_{1}(\tilde \pi_j) = 1$.
We can observe that all propensities models,
except the degenerated variant of \text{JPV}{},
give slightly better results than the baseline, with $\phi_{\text{R}}$ being clearly the best among them.
On the other hand, $\phi_{\text{direct}}$,
which directly estimates propensity for each label using~(\ref{eq:unbiased-propensities}),
significantly improves the performance (particularly for $\precision[3]$ and $\precision[5]$).
Nevertheless, $\phi_{\text{direct}}$ can only work well if the unbiased or bias-controlled data are substantial.
If this is not the case, one might need to use a parametric model,
but the above results suggest
that the dependence on label priors might not be sufficient.

\input{tables/results-artificial-data}
Finally, we illustrate a problem of propensity mismatch on synthetic and modified benchmark datasets.
We introduce noise to training data according to either the $\phi_{\xmlcpropensitymodel}$ or $\phi_{\text{P}}$ model,
train prediction functions using both propensities models, and report actual precision$@k$,
computed on the unbiased test set, along with propensity-based precision$@k$ for the same $\phi_{\xmlcpropensitymodel}$ or $\phi_{\text{P}}$ model,
computed on the biased test set (i.e., with the noise model applied).
The results in \autoref{tab:cross-propensities-p@k} show that relying on propensity-based metrics can be misleading.
As it should be expected, in the majority of cases, models compatible with the metric are obtaining the best performance.
However, selecting a model based on a chosen propensity-based metric can be wrong as the actual precision
might be driven by a completely different propensity model.

\subsection{Propensity estimation via joint learning}
\label{sec:ps-joint-learning}

To minimize an unbiased loss function, such as the unbiased logistic
loss~(\ref{eq:un-log-loss}), one needs to know propensities in advance.
However, estimating them might be difficult in practice.
As demonstrated above, the use of inaccurate estimates can lead to results being far away from the optimal ones.

Therefore it would be useful if propensities could be estimated directly from a biased training set.
Unfortunately, this is an ill-defined problem because the absence of a label can be explained by
either a small conditional probability of the label or a low propensity or both.
The additional assumption needed for the propensity to be identifiable were studied before,
in the areas of learning from positive and unlabeled data~\citep{elkan_learning_2008},
and novelty detection~\citep{Blanchard_et_al_2010_SSND}.
The overview of the possible assumption is given by~\citet{Bekker_and_Davis_2020_PAU_Survey},
where the weakest of the assumptions requires
that the true distribution of negative samples for a given label
cannot contain the positive distribution~\citep{Blanchard_et_al_2010_SSND}.
In these areas and under compatible assumptions, many methods for estimating the error ratio or labels priors,
both directly related to propensity estimates, were proposed~\citep{Bekker_and_Davis_2020_PAU_Survey}.
Recently, \citep{Zhu_et_al_2020} and~\citep{Teisseyre_et_al_2020} have introduced methods for estimating the unbiased conditional label probabilities and propensities jointly on the biased training set.
We refer to such methods as Propensity Estimation via Joint Learning (PEJL).

Let us briefly describe the method of \citet{Teisseyre_et_al_2020}
(cf. Appendix for description of the method of \citet{Zhu_et_al_2020}).
It uses the fact
that minimization of logistic loss leads to estimation of the posterior probability.
Therefore, we can define the loss in the following way:
\begin{equation}
\label{eq:log-loss-ps-plug}
\ell(\tilde{y}_j, p_j, f_j(x)) \!=\!
    -\tilde{y}_j \log(p_j f_j(x)) \!-\! \left(1 \!-\!\tilde{y}_j\right) \log(1\!-\!p_j f_j(x))\,,
\end{equation}
where $p_j f_j(x)$ can be seen as an estimate of the actual,
ground-truth, conditional probability $\eta_j(x)$,
with $p_j$ being the propensity and $f_j(x)$ the estimate of the observed conditional probability,
analogously to~(\ref{eqn:prob_adjustment}).
This function can be optimized not only with respect to $f_j(x)$, but also to $p_j$. The outline of the alternative method of \citet{Zhu_et_al_2020} can be found in the appendix.

We evaluate this approach on Yahoo R3 dataset. The estimated values of $p_j$ are plotted on the \autoref{fig:yahoo-r3-ps} and the last row of \autoref{tab:results-yahoo-r3} presents the promising results of this approach. While the obtained estimates are overestimated, they capture the true trend. The predictive performance also looks promising, being only slightly worst than the best propensity model $\phi_{\text{R}}$. This is indeed encouraging as this method does not have access to the unbiased or bias-controlled data.
\autoref{fig:yahoo-r3-ps} also plots the obtained propensities for each label.

\subsection{Task losses for long-tails}
\label{sec:task-losses-for-long-tails}

It seems that \citet{Jain_et_al_2016} have introduced the propensity-scored losses
rather to "promote" long-tail labels than to deal with missing labels.
As such, the propensities can be seen as a kind of weighing approach that
gives higher importance to less popular labels.
Unfortunately, it is not clear why the weighing scheme used in~\citep{Jain_et_al_2016} should be preferred over
other ones.
Moreover, a weighing scheme does not have to be interpreted in terms of propensities.
Let us consider a weighted variant of $\precision[k]$:
\begin{equation}
\precision[k](\vec{y}, \hat{\vec{y}}) = k^{-1} \sum_{\mathclap{j \in \topk[k](\hat{\vec{y}})}} w_j \tilde{y}_j \,.
\end{equation}
This boils down to $\psprecision[k]$ when $w_j = \frac{1}{p_j}$ which also implies $w_j \ge 1$.
But one can use any weights that would represent the importance or gain of labels.
In such a case, the weighted $\precision[k]$ has a natural interpretation of being an unbiased estimate of the expected gain.
If tail labels are of our interest, then they should get higher weights,
but actual values are rather domain-specific
without a direct relation to propensities.

To finalize our discussion, let us mention several other task losses that can be used
as metrics that pay special attention to long-tail labels.
The macro $F_\beta$-measure defined as:
\begin{equation}
\fmacro[\beta] \left (\set{\vec{y}_i, \hat{\vec{y}}_i}_1^n \right ) =
\frac{1}{m} \sum_j \frac{(1+\beta^2) \sum_i y_{ij} \hat{y}_{ij}} { \beta^2 \sum_i y_{ij} + \sum_i \hat{y}_{ij}}\,,
\end{equation}
puts the same weight to each label and focuses on true positives.
Therefore, positive predictions on long-tail labels are important to obtain high values on this metric.
One can also consider an $@k$ version of this metric,
in which only top $k$ predictions are taken into account.

Another interesting metric, originally proposed for search engines~\citep{Radlinski_et_al_2008},
is abandonment$@k$ defined as:
\begin{equation}
\abandon[k](\vec{y}, \hat{\vec{y}}) = \indicator[\forall j \in \topk[k](\hat{\vec{y}}): \, y_j \ne 1 ]\,,
\end{equation}
which encounters no error if there is at least one correctly predicted label among the $k$ ones in the predicted set.
This untypical formulation enforces diversity in the predicted set.
Always predicting the two most popular but correlated,
labels might be less beneficial than predicting less popular but also non-overlapping labels.

Finally, we mention the coverage metric, which directly reflects the diversity of correctly predicted labels.
It is defined as a fraction of labels with at least one correct positive prediction:
\begin{equation}
\!\!\!\operatorname{cov}\!\left( \set{\vec{y}_i, \hat{\vec{y}}_i}_1^n \right) = {m}^{-1} \left | \{ j \in [m] \!:\! \exists i \in [n] \; \text{s.t.}\; y_{ij} = \hat{y}_{ij} = 1 \} \right | \,.
\end{equation}
This metric has already been suggested in the literature as an alternative~\citep{Babbar_Scholkopf_2019, Wei_and_Li_2020}.
It has also been used in the original paper of~\citet{Jain_et_al_2016},
but only to motivate the propensity-based metrics.

\section{Label frequency in XMLC datasets}

We show in \autoref{fig:labels-freqs} a log-log plot of the distribution of label frequencies
in popular benchmark datasets from the XMLC repository~\citep{Bhatia_et_al_2016}.
As also noted in other works~\citep{Bhatia_et_al_2015, Babbar_Scholkopf_2017},
the label frequencies are characterized by the long-tail.

\input{figures/lables-freqs}

\section{Statistics of missing-label datasets}

We present in~\autoref{tab:new-dataset-stats} the statistics of datasets created to experiment with propensities models and missing labels. The description of these datasets is given in Section~\ref{sec:recipes-to-follow}. Because the process of generating the biased training sets contains randomness, for the mean number of labels per example, we report the average value from all generated variants of the datasets.

\begin{table}[h!]
\caption{Characteristics of datasets used in the experiments.
We report the size of the \textit{biased} train set ($n^\text{tr.}$) and the size of the  test set ($n^\text{ts.}$), the total number of labels ($m$), and the mean number of labels per example in the \textit{biased} train set and the test set. Symbol $*$ denotes  the average value over all generated variants of the dataset, and $\dagger$ the value corrected by $p_j^c = r / m$, where $r$ is a number of labels sampled for labeling for each datapoint.}
\label{tab:new-dataset-stats}
\resizebox{\linewidth}{!}{
\pgfplotstabletypeset[
every head row/.style={
  before row={\toprule},
  after row={\midrule}},
  columns={dataset,n train,n test,m,avg l bs,avg l true},
every last row/.style={after row=\bottomrule},
columns/dataset/.style={string type, column type={l|}, column name={Dataset}},
columns/n train/.style={column name={$n^\text{tr.}$}},
columns/n test/.style={column name={$n^\text{ts.}$}},
columns/m/.style={column name={$m$}},
columns/avg l bs/.style={string type, column name={$\expectation[|\relevantlabels{x}|]^\text{tr.}$}},
columns/avg l true/.style={string type, column name={$\expectation[|\relevantlabels{x}|]^\text{ts.}$}}
]{data/datasets-new.txt}
}
\end{table}

\section{Estimation errors of propensity models}

\autoref{tab:est-error} presents the mean squared errors (MSE) of inverse propensity estimates ($\|\hat{p}_j^{-1} - \phi(\tilde \pi_j)^{-1}\|^2$) on the Yahoo R3 dataset
obtained by different propensity models described in Section~\ref{sec:recipes-to-follow}. Models marked as (fit.) have been fitted to the same data the error has been calculated on.
Since we repeated the experiment with the PEJL model several times, we report the average error.

\begin{table}[h!]
    \caption{MSE on inverse propensity estimates of different propensity models for the Yahoo-R3 dataset. Symbol $*$ denotes the average value over several runs.}
    \label{tab:est-error}
    \centering
    \small
    \begin{tabular}{l|c}
        \toprule
        Model & MSE \\
        \midrule
        $\phi_{\mathbf{1}}$ & 1663.94\\
        $\phi_{\xmlcpropensitymodel}$ & 1557.04 \\
        $\phi_{\xmlcpropensitymodel}$ (fit.) & 1259.78 \\
        $\phi_{\text{P}}$ (fit.) & 512.94 \\
        $\phi_{\text{R}}$ (fit.) & 516.85 \\
        \midrule
        $\phi_{\text{PEJL}}$ & $\approx1236^*$ \\
        \bottomrule
    \end{tabular}
\end{table}

\section{Details of model training}

We describe here the implementation and training procedures of classifiers used in the experiments in Section~\ref{sec:recipes-to-follow}.
For all the experiments, except the one with the PEJL model described in Section~\ref{sec:ps-joint-learning},
we train a linear model using the unbiased binary-cross entropy loss (\ref{eq:un-log-loss})
with propensities coming from different models $\phi$.
The weights of the linear models are initialized from the uniform distribution $\mathcal{U}(-\sqrt{k}, \sqrt{k})$,
where $k = 1/d$ with d being the number of features.

In the case of the PEJL approach, the same linear architecture is used but trained using~(\ref{eq:log-loss-ps-plug}).
To assure that estimated values of $p_j$ are in $[0, 1]$, they are modeled as a sigmoid transformation ($\sigma(\cdot)$) of $p'_j$ parameters (one per label). Parameters $p'_j$ are initialized from the uniform distribution $\mathcal{U}(-e, e)$.

For each experiment, 10\% of the biased training set serves as a validation set for the selection of hyperparameters and early stopping. All methods are implemented using PyTorch~\citepappendix{pytorch}.
Optimization is performed with the Adam optimizer~\citepappendix{Kingma_et_al_2015}.
Only two hyperparameters are tuned on the biased validation test, the learning rate (from set \{0.005, 0.01, 0.05, 0.1\}) and the weight decay (from set \{0, 1e-8, 1e-7, 1e-6\}). Experiments have been performed on a single machine with Intel Xeon Gold 5115 and NVIDIA V100 16GB.

\section{An alternative PEJL method}

Below we briefly describe the main idea behind the method of~\citet{Zhu_et_al_2020},
which can serve as an alternative to the approach of \citet{Teisseyre_et_al_2020}
describe in Section~\ref{sec:ps-joint-learning}.

Let $\vec{M} \in \{0, 1\}^m$ be a mask random variable
that expresses relation between $\vec{Y}$ and $\tilde{\vec{Y}}$,
i.e., $\tilde{\vec{Y}} = \vec{M} \odot \vec{Y}$.
We then have $\probability[\tilde{Y}_j = 1 | \vec{x}] = \probability[M_j = 1] \cdot \probability[Y = 1 | \vec{x}]$
and $\probability[M_j = 1] = p_j$ and $\probability[Y = 1 | \vec{x}] = \eta_j(\vec{x})$~\citep{Schultheis_Babbar_2021}.
Because $M_j$ is a Bernoulli variable as well as $Y_j$ and $\tilde{Y}_j$,
we end up with two models, the first one for $Y_j$ with known $p_j$, and the second for $M_j$ with known $\eta_j(\vec{x})$.
The first one can be obtained by minimizing~(\ref{eq:un-log-loss}).
For the second one, a parametric model $\phi(\theta_j)$ for $p_j$ can be learned by minimizing the following logistic loss:
\begin{equation}
\label{eq:ps-un-log-loss}
\!\ell(\tilde{y}_j, \eta_j(\vec{x}), \phi(\theta_j)) \!=\! -\frac{\tilde{y}_j}{\eta_j(\vec{x})} \log(\phi(\theta_j))
    - \left(1\!-\!\frac{\tilde{y}_j}{\eta_j(\vec{x})}\right) \log(1 - \phi(\theta_j))\,.
\end{equation}
We can learn $f_j(\vec{x})$ and $\phi(\theta_j)$ jointly by replacing $\eta_j(\vec{x})$ by $f_j(\vec{x})$ in (\ref{eq:ps-un-log-loss}),
and $p_j$ by $\phi(\theta_j)$ in (\ref{eq:un-log-loss}).
Since both $f_j(\vec{x})$ and $\phi(\theta_j)$ can be updated on a single example $\vec{x}$, to avoid estimation-training overlap problem,
the training data is split into two parts, and the training is performed with $\phi(\theta_j)$ fixed on one part and with $f_j(\vec{x})$ fixed on the second one.

\section{Additional assumptions for complex metrics}

Here we show an example which demonstrates that for a non-decomposable metric, additional assumptions on the
process of labels going missing are required.

Consider a setting with two class labels. Let the label-wise propensities for both labels be $p_1 = p_2 = 0.5$.
The desired unbiased loss function of $\ell$ can be parametrized for each prediction $\hat{\vec{y}}$ by four real numbers, $v_{\tilde{\vec{y}}} \coloneqq
\tilde{\ell}(\tilde{\vec{y}}, \hat{\vec{y}})$. We can consider two different scenarios. First, both labels always go missing at the
same time, second they go missing complementarily, corresponding to the probability distributions $\probability$ and
$\probability^\prime$. As the unbiasedness needs to hold for all potential distributions of true labels, it needs to hold in particular in the four cases in which the true label distribution is concentrated on a single point. By explicitly calculating expectations through reading off the probabilities from \autoref{tab:correlated-missing}, we can state the unbiasedness
requirement which is $\expectation[\tilde{\ell}(\tilde{\vec{Y}}, \hat{\vec{Y}})] = \expectation[\ell(\vec{Y}, \hat{\vec{Y}})] $:
\begin{equation}
\begin{aligned}
\ell((1,1), \hat{\vec{y}}) &= 0.5 (v_{11} + v_{00}) = 0.5(v_{10} + v_{01})\,,\\
  \ell((1,0), \hat{\vec{y}}) &= 0.5 v_{00} + 0.5 v_{10}\,,\\
  \ell((0,1), \hat{\vec{y}}) &= 0.5 v_{00} + 0.5 v_{01}\,,\\
  \ell((0,0), \hat{\vec{y}}) &= v_{00}\,.
\end{aligned}
\end{equation}
These are five linear equations with only four variables, so in general, there is no solution. If we additionally assume that
the labels go missing independently from each other, then the marginal propensities uniquely determine the full distribution.
In that case, unbiased estimates can be derived for general loss functions \citep{Schultheis_Babbar_2021}.

\begin{table}[h!]
  \caption{Different distributions with the same propensities.}
  \label{tab:correlated-missing}
  \centering
  \small
  \begin{tabular}{cc|cc|cc|c}
    \toprule
    $Y_1$ & $Y_2$ & $\tilde{Y}_1$ & $\tilde{Y}_2$ & $\probability[\tilde{Y}_1, \tilde{Y}_2]$ &
    $\probability^\prime[\tilde{Y}_1, \tilde{Y}_2]$ & $\tilde{\ell}(\tilde{\vec{y}}, \hat{\vec{y}})$ \\ \midrule
    1 & 1 & 1 & 1 & 0.5 & 0.0 & $v_{11}$ \\
    1 & 1 & 1 & 0 & 0.0 & 0.5 & $v_{10}$ \\
    1 & 1 & 0 & 1 & 0.0 & 0.5 & $v_{01}$ \\
    1 & 1 & 0 & 0 & 0.5 & 0.0 & $v_{00}$ \\ \midrule
    1 & 0 & 1 & 0 & 0.5 & 0.5 & $v_{10}$ \\
    1 & 0 & 0 & 0 & 0.5 & 0.5 & $v_{00}$ \\
    \midrule
    0 & 1 & 0 & 1 & 0.5 & 0.5 & $v_{01}$ \\
    0 & 1 & 0 & 0 & 0.5 & 0.5 & $v_{00}$ \\ \midrule
    0 & 0 & 0 & 0 & 1.0 & 1.0 & $v_{00}$ \\
    \bottomrule
  \end{tabular}
\end{table}

\section{Introduction}
\input{1-intro}

\section{Problem statement}
\label{sec:problem_statement}
We first define the problem of XMLC,
then the problem of missing labels,
and finally the problem of long-tail labels.

\subsection{Extreme multi-label classification}
\label{subsec:xmlc}
\input{2-xmlc}

\section{Critical view on the current approach to sparse labels in XMLC}
\label{sec:critical-view-on-propensities}

In this section we present an overview on the current state of addressing the
long-tail and missing-labels problems in XMLC. This is in large parts based on
the work of \citet{Jain_et_al_2016}, so we start by a recap of their findings.

\subsection{Current approach to missing labels and long tails}
\input{3-critical-view}

\section{Recipes to follow}
\label{sec:recipes-to-follow}

\input{4-recipes}

\section{Conclusions}

Despite our critical comments regarding~\citep{Jain_et_al_2016},
we still appreciate this contribution.
It was the first paper that brought direct attention to the problem of missing and long-tail labels in XMLC.
The original theoretical results concerning the propensity model have motivated a lot of research in this direction.
Nevertheless, we believe that missing labels and long-tail labels are rather orthogonal problems
that should be solved with different tools.
Obviously, labels gone missing may cause labels to be sparse,
but it does not mean that a blind propensity model may solve any of these problems.

\begin{acks}
Academy of Finland grant: Decision No. 347707. Computational experiments have been performed in Poznan Supercomputing and Networking Center.
\end{acks}

\bibliographystyle{ACM-Reference-Format}
\balance

\section{Reviewer \#1}
\begin{enumerate}

    \item Does the paper make novel contributions to data science research (methodology, process, evaluation, systems, and so on)?

        Disagree

    \item Is the paper properly contextualized, i.e., does the proposed work makes a nontrivial advance over relevant related work?

        Disagree

    \item Are the most important claims of the paper justified based on theoretical contribution(s) or experimental validation?

        Disagree

    \item Is the presentation of the paper (organization, writing style, formatting etc.) consistent with the high and well established SIGKDD standards?

        Neutral

    \item Please use this space to provide detailed feedback to the authors and justify your selection to the above questions. Please provide at least one paragraph of justification per question.

        1) The authors think the JPV propensity model does not fulfill natural conditions. However, the JPV propensity model is simple and efficient, which makes it easy to reproduce. The authors propose to consider the correlation between features and labels, as well as the dependence between one label and the others. This will add to the complexity of the XMLC model. TODO: check if the claim about correlation isn't too strong

        2) The authors suggest to use unbiased sets for validating. However, as the authors point out, it is nearly impossible to check the whole set of labels. In practice, the size of label space is up to million level, so designing a unbiased validation set is very expensive. TODO: we can add some kind of outlook at the end of the paper

        3) The authors demonstrate that the JPV model falls short on reliable and reproducible hyper-parameters, such as scaling behaviour and the estimation process. However, the proposed examples are not comprehensive. The authors should conduct more validation experiments to show the negative impact of unreliable hyper-parameters in XMLC models.
        Therefore, I think this paper has limited contributions, since the problem is not well-solved, and the proposed method is expensive to conduct. TODO: Maybe we should do some experiment with just JPV and different A, B parameters.

    \item Please make a recommendation regarding acceptance of the paper.

        Weak Reject

\end{enumerate}

\section{Reviewer \#4}
\begin{enumerate}

    \item Does the paper make novel contributions to data science research (methodology, process, evaluation, systems, and so on)?

        Strongly agree

    \item Is the paper properly contextualized, i.e., does the proposed work makes a nontrivial advance over relevant related work?

        Strongly Agree

    \item Are the most important claims of the paper justified based on theoretical contribution(s) or experimental validation?

        Agree

    \item Is the presentation of the paper (organization, writing style, formatting etc.) consistent with the high and well established SIGKDD standards?

        Strongly Agree

    \item Please use this space to provide detailed feedback to the authors and justify your selection to the above questions. Please provide at least one paragraph of justification per question.

        For extreme multi-label classification (XMLC), propensity is usually used as a performance metric, which targets two problems in XMLC: label missing and long-tail labels. This paper analyzes the propensity score and get the conclusion that the assumption on label missing is unrealistic, and the way to estimate parameters in the propensity score is problematic and cannot be generalized to other datasets. The resulted propensity score itself is problematic and may not be within a reasonable range. Finally, the paper proposes several recipes to follow in evaluate XMLC, including how to estimate the propensity score and other alternatives to the propensity score which could also weight the tail labels higher.

        Q1
        Propensity score is considered as one of the metrics to evaluate XMLC and is used widely as the original paper published in 2016 has been cited for more than 200 times. The paper challenges the reasonability of estimating and using propensity score and is potential to have large impact on ongoing works in this direction. Despite any controversy the paper may raise, the reasoning on why we should not use propensity score makes sense.

        Q2
        Although this is not a methodology paper, the paper has nontrivial advance over relevant works, as pointing out the possibility that relevant works are using an improper metric to evaluate its results.

        Q5
        The paper provides sufficient reasoning and examples to support its claim. In the "recipe" part, the paper even provides empirical justifications. Of course, the paper may not be comprehensive or perfect in the current phase, that is, it did not propose a metric to fully replace the propensity score. But this does not impact its contribution to the community.

        Q6
        The paper is well-written and easy to follow.

    \item Please make a recommendation regarding acceptance of the paper.

        Strong Accept

\end{enumerate}

\section{Reviewer \#5}
\begin{enumerate}

    \item Does the paper make novel contributions to data science research (methodology, process, evaluation, systems, and so on)?

        Agree

    \item Is the paper properly contextualized, i.e., does the proposed work makes a nontrivial advance over relevant related work?

        Agree

    \item Are the most important claims of the paper justified based on theoretical contribution(s) or experimental validation?

        Neutral

    \item Is the presentation of the paper (organization, writing style, formatting etc.) consistent with the high and well established SIGKDD standards?

        Agree

    \item Please use this space to provide detailed feedback to the authors and justify your selection to the above questions. Please provide at least one paragraph of justification per question.

        This work analyzes the issues of a model for the XMLC problem. The work strongly depends on the paper "Extreme Multi-Label Loss Functions for Recommendation" and may have novelty limitation to the certain paper. The paper can be greatly improved in the writing, because the paper is hard to follow.

        Below lists points related to the writing.
        1. What is "propensity", which is broadenly used in the paper and misses a basic explanation?

        2. The sentence "We require..." from line 197 to line 198 is hard to understand.

        3. Grammar mistake: "data are (is)" in line 221.

        4. The author should better give a global view of the JPV model at first before giving the details.

        5. Some definition is unclear. For example, the $y_j = 1[j \in L(x)]$ in line 136 needs guess to understand. Is the "1" a matrix or a function?

        6. Some sentences are too long to follow. For example, a sentence, "A first observation ..." from line 478 to line 481 is confusing.

    \item Please make a recommendation regarding acceptance of the paper.

        Neutral

\end{enumerate}

\section{Reviewer \#7}
\begin{enumerate}

    \item Does the paper make novel contributions to data science research (methodology, process, evaluation, systems, and so on)?

        Neutral

    \item Is the paper properly contextualized, i.e., does the proposed work makes a nontrivial advance over relevant related work?

        Neutral

    \item Are the most important claims of the paper justified based on theoretical contribution(s) or experimental validation?

        Neutral

    \item Is the presentation of the paper (organization, writing style, formatting etc.) consistent with the high and well established SIGKDD standards?

        Neutral

    \item Please use this space to provide detailed feedback to the authors and justify your selection to the above questions. Please provide at least one paragraph of justification per question.

        This paper reviews the propensity-scored loss function and evaluation metrics in extreme multi-label classification (XMLC). First, this paper discusses the flaws of the propensity-based approach. Then this paper presents some thoughts on how to improve those flaws.

        \#\# Strength

        1. This paper presents several disadvantages of the propensity model, which is beneficial to motivate the development of more reasonable alternatives;

        2. This paper constructs some new datasets for learning with missing labels in XMLC, and empirically shows the flaws of the propensity model on those datasets.

        \#\# Weakness

        1. Some flaws of the propensity model were already pointed out in previous literature. For instance, the issue of the estimation of the parameters $a$ and $b$ in Eq. (9) was mentioned in [1], and it also mentioned that converge may be a reasonable alternative evaluation metric. Additionally, regarding the use of propensity-scored precision (PSP@k) as evaluation metric, it may not completely unreasonable because propensities are calculated based on label frequencies, which can, to some extent, reflect the performance of models on tail labels. TODO: Add the cited paper, but I think doesn't require a lot of discussing.

        2. Regarding the recipes presented in this paper, it presents to use unbiased or bias-controlled test and validation set for model selection, however the unbiased validation set is still costly to obtain in XMLC regime; Moreover, the newly generated datasets and conducted experiments in this paper are not informative. This papers generates datasets with no more than 3000 labels and hence cannot simulate the task of XMLC very well. More large-scale datasets can be tested. Also, it is unclear how existing methods perform on newly generated datasets using suggested evaluation metrics, which is helpful for justifying that PSP@k is problematic.

        [1] Does Tail Label Help for Large-Scale Multi-Label Learning (TNNLS)

    \item Please make a recommendation regarding acceptance of the paper.

        Weak Reject

\end{enumerate}

\section{Meta-review}

\begin{enumerate}
    \item Please summarize the paper, the reviews, the ensuing reviewer discussions and your recommendation

    This paper was difficult to judge because it received such diverse reviews. On the surface, the "Strong Accept" can probably not outweigh the three other reviews, which are more on the borderline/reject side, but eventually I arrived at the conclusion that it actually should.

    Review \#5 was not particularly familiar with work in this area, and thus gave a neutral opinion. The criticism that the paper is not innovative does not hold (the cited paper is the paper that this paper criticisizes, so it is a very natural predecessor), and I also do not share the concerns about the readability (e.g., "data are" is certainly correct English). So I think this review should be disregarded (but I do appreciate the efforts of the reviewer, nevertheless).

    Reviewer \#7 has some concerns about the novelty, in particular that some of the criticism of propensity methods have been brought up before. In the discussion s/he stated that "Although the topic of this paper is interesting and meaningful, previous related works should be discussed." I strongly agree with this statement, but I believe that this can be easily fixed in a revision, and does not diminish the quality of this work, which, in my opinion, makes an excellent summary of the criticisms of propensity scores, and outlines some potential remedies.

    In that, I disagree with the reviewer's opinions that the contribution of the work reported in this paper is not sufficient to be published at a top-tier conference like KDD. I do believe that this paper substantially helps to understand our understanding of XMLC because it provides a sound analysis of the problem. This is the type of paper that I would like to give to my students (as opposed to papers that demonstrate yet another epsilon improvement in an experimental evaluation).

    Reviewer \#1 was not convinced by the criticism brought forward by this paper against the Jain et al. paper. In that, s/he seems to be an outsider as at least two of the other reviewers (\#4 and \#7) confirmed that they are valid (\#7 even stated that some of them are well-known). R\#1 also asks for more experiments (as \#7), but as explained above, I don't think that experiments are the main contribution of this paper. Also, this request is also not specific in terms of what hypothesis should be experimentally verified.

    So, overall, I side with the strongly positive reviewer. In particular, I perfectly agree with his/her summary stated in the discussion: "I agree with other reviewers that the paper may be weak in proposing other methods as the performance metric. But at least it contributes by saying a previously widely used one is unreasonable and such a contribution, if correct, should have a large impact on the community."

\item Please make a recommendation about the acceptance of the paper

    Accept
\end{enumerate}

\end{document}

redpajama_set_name: RedPajamaArXiv


There's something about walking in a leather goods store. That smell: rich, dark-stitched, with just a trace of border town recklessness thrown in. And just so we're clear, you won't find any Velcro or plastic snaps or plaid lining in these classics because we'd hate for you to get gut-shot in a border town. This is the wallet I carry on weekends when I'm going to be somewhere longer than overnight but less than five days. Just throw in what you need and go. It's a wallet for those between times, like when you're driving between Austin and Alpine.


text:
"Fact checkers" gang up to demand YouTube censors more "misinformation"
| Reclaim The Net
Third party "fact-checkers" appear determined not only to maintain their business – all too often consisting of what critics consider to be censorship delegated by Big Tech – but, like any rising industry, they would like to see that business grow.
It seems as if they see the opportunity for growth in focusing more on languages other than English and spreading their "fact-checking" operations to those markets as well.
That is one take on why as many as 80 such organizations have now come together to call on YouTube to step up moderation and censorship even further. They, however, frame their appeal as concern that the platform is "a major conduit" of harmful misinformation around the world and put an emphasis on non-English content.
The 80 organizations are based in more than 40 countries, and call themselves "independent, non-partisan." But most of them are obscure groups whose backing and funding is often nontransparent – making it difficult for anyone to "fact-check" their claim to independence and political neutrality.
Appearing to attempt to fan the flames of misinformation panic, the open letter addressed to CEO Susan Wojcicki accuses YouTube of being weaponized by "unscrupulous actors" who "manipulate and exploit others."
These "fact-checkers" believe that what YouTube is already doing is not enough and want Google's video giant to map out a more efficient plan for "policy and product interventions (…) and to do so with the world's independent, nonpartisan fact-checking organizations."
Once the culprits are identified in this way, "fact-checkers" want YouTube to more aggressively demonetize them, while giving these outfits more work to do. Other than stopping repeatedly flagged creators from monetizing through ads, the letter suggests preventing them from pointing viewers toward outside payment platforms.
In particular, they seem unhappy that some of the money that could go to them is spent by YouTube on developing censorship algorithms and automating the process.
Instead, the letter says that YouTube should collaborate more deeply with "fact-checkers" – and that means "systemtically investing in independent fact-checking efforts around the world."
Among the signatories are Africa Check, Rappler from the Philippines, France-based Science Feedback, India's Factly and Colombia's Colombiacheck.
The post "Fact checkers" gang up to demand YouTube censors more "misinformation" appeared first on Reclaim The Net.
Reclaim The Net is a free speech and online privacy organization that defends our individual liberty by pushing back against big tech and media gatekeepers. Much of their work focuses on exposing digital tyrants and promoting free speech and privacy-friendly alternative online services. Visit reclaimthenet.org
reclaimthenet.org/
This post has been republished with permission from a publicly-available RSS feed found on Reclaim The Net. The views expressed by the original author(s) do not necessarily reflect the opinions or views of The Libertarian Hub, its owners or administrators. Any images included in the original article belong to and are the sole responsibility of the original author/website. The Libertarian Hub makes no claims of ownership of any imported photos/images and shall not be held liable for any unintended copyright infringement. Submit a DCMA takedown request.
Categories Politics & Culture Tags coronavirus, Free Speech and Censorship, News Post navigation
Senators Introduce Bipartisan Legislation To End US Reliance On Chinese Minerals
GOP Senator Who Fauci Called a 'Moron' Is Set to Introduce the 'FAUCI Act'
Germany's Interior Ministry accuses Google and Apple of inciting "conspiracy theories" for allowing Telegram
Facebook apologizes after threatening UN Watch director with account deletion