text:
Greetings From Palau, The Micronesian Archipelago That Baseball Built
David Walter
Illustration: Angelica Alzona/GMG
KOROR, Palau—What would a country run by baseball players look like? Would it be a sabermetrics-driven technocracy? A clutch-obsessed theocracy? A cup-adjusting macho dystopia?
This isn't a thought experiment. It's happening right now in Palau, a tiny archipelago of some 20,000 souls located in the Western Pacific that is currently playing host to a radical experiment in letting the sluggers run the show.
Baseball has dominated the cultural and sporting life of Palau for almost 100 years, which is about four times longer than Palau's been an independent nation. Over the years, Palauans have shaped the game to fit their island lives. Kids learn to play with bamboo bats and coconut-leaf balls. Pitchers chew betel nut instead of dip. Monsoons rain out not just games and series, but entire seasons of league play. Local traditions of witchcraft have crossed over into the country's sporting life; even today, it's not uncommon for accusations of black magic to fly after particularly contentious games. (I've been reporting on and off from Palau for seven years, so I'm used to it. THIS pitcher's dad was a known wizard; THAT team's manager caught women from an opposing village burning leaves over home base.) Baseball as it's played in Palau is a decidedly Palauan thing.
But baseball has also shaped Palau. It's more than a national pastime here. It's an organizing principle—or, more accurately, a re-organizing principle. Before the 20th century, Palau was a matriarchy. Women controlled most aspects of society, and men were limited to fishing, fighting, and handling village-to-village diplomacy. Then colonial rule brought centralized government—and baseball—to the archipelago. Ever since, these two male-dominated worlds have fed on each other, with Palau's baseball leagues serving as a kind of farm system for government service. Scores of congressmen, senators, diplomats, and heads of state have passed through Palau's dugouts on their way to political power.
"Those men were nobodies before they stepped out to hit the ball. Baseball created a stage for [male] validation," a woman from a high-ranking clan told me when I asked for her thoughts on the game. "It's empowered men like nothing before."
It's hard to overstate baseball's effect on the male Palauan mind—and, by extension, on the body politic. Many countries still play the sports they first learned from their colonizers, of course, but Palau's remoteness has allowed baseball to flourish in a way that's remarkably independent of the global sports-industrial complex. American sluggers aren't revered by Palauans as legends in the way that NBA stars are in the Philippines. Nor have any washed-up MLB pros found second acts or adoring fans in the Palau Major Leagues in the way that, say, Bobby Valentine became a cuddly mascot in Japan. Instead, and despite a style of play that tends toward small-ball, all the game's glory has gone to male Palauans. Men who might rate elsewhere as ballboys have instead become authentic sultans of swing.
"I see Palauan men's passion for baseball as a way, temporarily, of gaining a respite from the dominating influence of women," wrote Donald R. Shuster, who produced the definitive history on Palauan baseball. "[For them,] baseball is liberating."
Look at it a certain way and the story of baseball in Palau is the heartwarming, underdog, Disney-ready tale of male liberation…that exactly no one is asking for in 2017. From a different angle, though, Palau's baseball history can be read a cautionary tale, and a parable of what happens when society's overlords let one gender suit up while the other rides the bench.
Hit that one to Tuvalu. (Photo by David Walter)
You might expect this story to begin with the Americans, who invented baseball in the 1800s, ruled Palau during the decades after World War II, and have their own history of resistance toward women in power. But the sport's roots in the Pacific actually begin with the Japanese: Baseball first came to Palau in the 1920s under the auspices of the Japanese Empire's colonial administration, which annexed Palau as part of a plan to fill the Pacific with Japanese settlers. The Pacific Islanders living on those islands and archipelagos were something of an afterthought in that colonial gambit; the Japanese saw the locals as easy targets for cultural assimilation. Under Japanese rule, Palauans were required to attend Japanese public school and pledge loyalty to the Emperor. Local religion was suppressed, and local men's clubs, called bai, were disbanded.
Baseball soon filled the cultural gap. In 1925, the Japanese taught the game to their Palauan servants in the colonial administration. The jobs held by these first Palauan players spoke to their low position in Japanese society: First baseman Lekeok worked a tea server; Bismark, the team's center fielder, was a "helper" at the Japanese communications office; Third baseman Ngirakesau served as a bicycle messenger for the Post Office.
On the field, however, the Palauans soon surpassed their colonial overlords. A 1928 series between Palau's all-star team and a Japanese squad resulted in a surprise win for the locals. The Japanese responded by swiftly banning all future interracial competition.
What the Japanese didn't do was stop promoting baseball altogether. Indeed, they continued to provide Palauans with free equipment and coaching. That's because baseball was more than just a pastime for the Palau's colonizers: "Yakyudo," or "the baseball way," held a spiritual pull for the Japanese, and they hoped to mold them Palauans into obedient colonial subjects by teaching them yakyudo. Baseball practice was a lesson in hierarchical discipline: "Anyone who was lazy got slapped," an early baseball player recalled in Shuster's history of the Palauan game. "In my time, we practiced like we were in military training. 50 balls would be hit to each infielder: to first, second, short, third."
Squad ca. 1926 (From Donald Shuster's Baseball In Palau: Passion For The Game, From 1925-2007)
World War II happened before Japan could complete its "civilizing mission." Yakyudo was blasted away in a hail of American bombs, along with most of Palau's schools, roads, houses and hospitals. The island's infrastructure would take decades to rebuild but baseball came back much sooner, and with the full support of the island's new U.S. Military overlords.
By 1948, the U.S. had organized village teams into an archipelago-wide pennant race. Baseball was a cornerstone of American efforts to remodel Palau's society—so much so that GIs used M-boats and army trucks to ferry village teams from island to island. These inter-village games were the start of what is now called the Palauan Major Leagues; at the time, though, they were simply a welcome diversion for shell-shocked Palauans—especially for Palauan men.
Even before the war, the old matriarchal system had left a lot of young men idle. Women controlled all property, money, and hereditary titles in a village. Men could traditionally make their names through fishing and military battles, but both activities were in decline as the U.S. pushed Palauans toward peaceful, more urbanized living.
Baseball soon became war by other means. "Young men were warriors! You had to be the macho guy," explained Johnson Toribiong, a legendary player-manager who in 2008 became Palau's third President since independence. "Instead of a spear, you could throw the ball. Or like a warrior, you can avoid that spear."
Baseball's militaristic turn also revived Palau's system of communal men's houses, or bai, which had suffered under Japanese rule. In the olden days, the bai housed a villages' warriors as they trained for battle. Under American rule, they were used to sequester players in the days before big baseball games. During these lockups the old bai rules for warding off dark magic were followed closely. "You don't sleep on your back, because the village god could step on your chest and you would have problems sleeping," said Toribiong. "You don't curse in the bai, you don't fart in the bai, you don't urinate near the bai, and so forth."
The dugout became a spiritual extension of the bai—a hallowed space for male camaraderie and valor. Women and children were banned from both. "That's to avoid changing your aggressive attitude to war by becoming loving to your child or becoming attracted to a woman," Toribiong told me. "You need people to come to the dugout and keep their focus on the fight."
Johnson Toribiong, Baseballer In Chief (Photo by David Walter)
Ironically, the colonial import of baseball also gave a boost to the Palauan independence movement. In the 1960s and '70s, America wanted to unite its Western Pacific territories into a single Micronesian federation. But they also organized pan-regional baseball tournaments which, counterproductively, made Palau hate its neighbors. The players from Guam, Palauans said, were rich and haughty. The squad from Chuuk played rough: during inter-island play in 1969, one Chuukese infielder embedded a shark's tooth his mitt and used it to slash at passing runners. The team from Pohnpei, meanwhile, had a reputation as supernatural cheats. Once, when they were ahead of Palau at the end of a game, Pohnpei tried to use a wizard to call down the clouds and cause an early rainout—and they might have succeeded, too, had Palau's own wizard not been on hand to counter the spell. (Maneuvers like these were considered a big enough problem that, when regional play was revived in the '90s, special rules were written to ban the use of black magic.)
As for Palauans? They were simply the best. "Palau, our home and people were a great motivating force for us," recalls Frank Kyota, a left fielder on the national team who is now a senator. "Our idea was, never lose! We're Palauan, we cannot lose!" Whole villages would empty to greet the national squad upon its return from overseas; on these occasions the women of the village prepared grand feasts of fish and taro while players regaled their neighbors with inning-by-inning highlights.
By 1978, Palau had rejected membership in the U.S.'s Micronesian federation and resolved to go it alone. The negotiating team leading this independence fight included several star baseball players and coaches—among them future President Toribiong, of the Airai Aces, and Joshua Koshiba, of the rival Aimeliik team, who would later become Palau's longest serving senator.
As independence day grew nearer, Palau's government bureaucracy continued to be filled with notable Palauan baseball players. At the vanguard were the player-managers—men like Koshiba, Toribiong, and Mlib Tmetuchl, Toribiong's successor on the Aces (and himself a future senator). In the years since, many more major-league ballplayers have became senators and representatives in the Olbiil Era Kelulau (House of Whispers), Palau's congress. Others filled administrative posts up and down the line, from Governorships to Ministerships to the head of Parks and Recreation department. Hersey Kyota, Palau's ambassador to the U.S. and the dean of the D.C. diplomatic corps, started out as a second baseman in the Palau major leagues. "It was just the natural transition of things. When we were growing up there was no other sport than baseball," said Frank Kyota, the senator, of the transition from ballplayer to politician. "It was sort of the only way you could break out in the community and be seen. That connects to everything."
Baseball lives here. (AP Photo/Itsuo Inouye, File
I've been reporting on Palauan politics since 2010, and in my trips to the country I've passed many a sweltering afternoon on the bleachers of the national stadium, cheeseburger in hand, cheering on high school and village teams. At night, over beers on the waterfront, I've heard proud retellings of the national squad's heroics and anti-heroics during tournaments overseas. The anti-heroics tend to stand out the most in my memory—a Palauan batter chasing a Guam pitcher into the stands, bat in hand, after a beaning; a Palauan baserunner kicking an infielder as he rounded first. Both of these men are now senators.
I've learned to keep my face noncommittal during allegations of black magic, which is the closest Palau's gotten to having a doping problem. I've seen boys playing pickup games on the grounds of empty bais with a green wall of jungle looming the background. I've loved it all, and so I arranged for a reporting trip in hopes of profiling the national team...only to be greeted with endless rain and rainouts.
As a result, I wound up focused on the off-field intrigue, cross-referencing playing rosters with government rolls and polling old-timers for their theories on how baseball and politics became so intertwined. It didn't take long to figure everything out. There's an undeniable symmetry at play in modern Palauan history. Before colonialism, exceptional men won glory by fighting in wars, then served their villages as "chiefs"—really, diplomatic messengers to other tribes—at the behest of the women who ran the show. When the colonizers came, Palau's systems of valor and village politics were dismantled, and replaced with baseball and modern government. Men shifted seamlessly from one realm to the other, not as the result of some misogynist conspiracy, but more as a result of the soft bigotry of like flocking to like. When colonial administrators wanted to play sports with the locals, they unthinkingly gravitated toward the men, not the women; likewise, when they set up a "modern" political bureaucracy, they just as unquestioningly filled open positions with the men with which they felt friendly.
On the surface, the new roles held by men in modern politics—governor, senator, president—may have seemed analogous to the diplomatic duties male chiefs had performed under the traditional system. In practice, however, modern politics has afforded men far more dealmaking power than they had ever enjoyed before. Women, for all their historical power in Palau, were left controlling an increasingly marginalized village system.
What's less clear to Palauans is what all this has done to the overall equality of the sexes. Viewed from the outside, Palau's clubhouse-to-statehouse pipeline seems to confirm the dangers of gender inequality in sports. Every year, it seems, brings a new study in the US affirming the link between participation in sports and leadership in society. That, we're told, is a big part of why Title IX matters: people who play sports are more likely to wield power later in life.
Many of the Palauan women I know, however, hesitate to draw similar conclusions about their own roles in sports and politics. If I've heard it once, I've heard it a thousand times: While Palauan women may look disempowered compared to the heyday of the matriarchy, they still feel like they pull the strings. Men may control government, they say, but women control the men.
"Baseball is when they first felt they were big men," said a female clan leader. But, she said, "We gave them that new stage. Just like we gave them the modern system, the government." Women, she said, had "never stepped out publicly," instead sending male chiefs as emissaries to to carry out women's decisions. "Women have always ruled from the back. We still feel in power."
Behind the scenes, I've seen women wear down their male family members until they've taken the woman's line on issues like land proposals or environmental laws. So maybe there's something to be said for the claim that Palauan women are leading from behind, and letting the men swing their bats around while the women quietly get on with the work at hand. At the same time, once men get into government—once they take those jobs as senators and presidents and governors, and spend their days in a statehouse located far away from their homesteads—these men have power that extends beyond the home or even the village.
The female leaders denying that men run the show are the same ones I've heard speaking out against the betrayals of male leaders. Those are the same men that, for instance, seem determined to build a giant casino in Palau despite popular opposition; they're the men who've taken lavish junkets abroad on the dime of land-hungry Chinese investors. They're the men accused one year of misusing political funds, then cleared the next, then accused again, then cleared again. It's men who get honored at overseas conferences for their activism on climate change, and men who are fêted at home with pennants and inaugural balls. It's the men who get the glory, men who hoist the trophies. So: what would a country run by baseball players look like? The same as any other, more or less.


text:
\section{Introduction}

Interpretable machine learning (ML) has attracted significant interest as %
ML models are used to support human decision making in societally critical domains such as justice systems and healthcare \cite{doshi2017towards,guidotti2019survey,lipton2016mythos}.
In these domains, full automation is often not desired and humans are the final decision makers
for legal and ethical reasons.
In fact, the Wisconsin Supreme Court ruled that
``a COMPAS risk assessment should
not be used to determine the severity of a sentence or whether
an offender is incarcerated'', but does not eliminate the use of ML models if ``judges be made aware of the limitations of risk assessment tools''~\citep{nytimes,wicourts}.
Therefore, it is crucial to {\em enhance} human performance with the assistance of machine learning models, e.g., by explaining the recommended decisions.

However, recent human-subject studies
tend to show limited effectiveness of explanations in improving human performance~\citep{bussone2015role,horne2019rating,lai+tan:19,weerts2019human}.
For instance,
Lai and Tan \citep{lai+tan:19}
show that explanations alone only slightly improve human performance in deceptive review detection;
Weerts et al. \citep{weerts2019human} similarly find that explanations do not improve human performance in predicting whether one's income exceeds 50,000 in the Adult dataset.
These studies
explain a machine prediction by revealing model internals, e.g., via
attributing importance weights to features and then visualizing feature importance.
We refer to such assistance as real-time assistance because they are provided as humans make individual decisions.

To understand such limited effectiveness, we argue that it is useful to distinguish
two {\em distinct} modes in which
ML models are being used: {\em emulating} and {\em discovering}.
In tasks such as object recognition \citep{deng2009imagenet,he2015delving}, %
datasets are crowdsourced because humans are considered the gold standard, and ML models are designed to emulate human intelligence.\footnote{As a corollary, it is usually considered overfitting the dataset when machine learning models outperform humans in these tasks.}
In contrast,
in the discovering mode, datasets are usually collected from observing social processes, e.g., whether a person commits crime on bail for bail decisions \citep{kleinberg2017human} and what the writer intention is for deceptive review detection~\citep{abouelenien2014deception,ott2011finding}.
ML models can thus often identify patterns that are unsalient, unknown, and even counterintuitive to humans,
and may even outperform humans in {\bf constrained} datasets \citep{kleinberg2017human,ott2011finding,tan2014effect}.
Notably, many critical policy decisions such as bail decisions resemble the discovering mode more than the emulating mode because policy decisions are usually challenging (to humans) in nature \citep{kleinberg2015prediction}.

Studies on how explanations affect human performance tend to employ these challenging tasks for humans (the {\em discovering} mode for ML models) because humans need {\em little} assistance to perform tasks in the emulating mode (except for scalability).
This observation highlights different roles of explanations in these two modes.
In the emulating mode, explanations can help debug and identify biases and robustness issues in the models for future {\em automation}.
In the discovering mode, if the patterns embedded in ML models can be elucidated for humans, they may enhance human knowledge and improve human decision making.\footnote{It is worth noting that these two modes represent two ends of a continuum, e.g., emulating experts lead to discoveries for novices.}
Moreover, it might help humans identify spurious patterns in ML models and account for potential mistakes to generalize beyond a {\bf constrained} dataset.

\begin{figure*}[t]
  \centering
  \includegraphics[width=0.98\textwidth]{tutorial.pdf}
  \vspace{-0.1in}
  \caption{Illustration of example-driven tutorials and guidelines shown to participants during the training phase: a) top 10 features of the
  %
  review text are highlighted in green and red ({\em signed highlights}), where green words are associated with genuine reviews
   and red words are associated with deceptive reviews;
   %
   %
b) participants are presented the actual label, the predicted label, and textual explanations for a review after choosing the label of the review in example-driven tutorials;
c) a list of guidelines for identifying deceptive reviews extracted from scientific papers. \label{fig:tutorial}}
\end{figure*}

To further illustrate the difficulty of interpreting explanations in the discovering mode,
\figref{fig:tutorial}(a) shows an example from a deceptive review detection task, where the goal is to distinguish deceptive reviews written by people who %
did not stay at
the hotel from genuine ones.
``Chicago'' is highly associated with deceptive reviews because people are
more likely to mention the city name instead of specific places when they imagine their experience.
Such a pattern can be hard to comprehend for humans, especially when
the highlights are shown as real-time assistance without any other information.

Instead of throwing people in at the deep end directly with real-time assistance,
we propose a novel training phase that can help humans understand the nature of a task and the patterns embedded in a model.
This training step is analogous to offline coaching and can be complementary to real-time assistance in explaining machine predictions.
We consider two types of model-driven tutorials:
1) guidelines extracted from scientific papers \citep{li2014towards,ott2013negative,ott2011finding} (\figref{fig:tutorial}(c)), which reflects the current practices of science communication;
2) example-driven tutorials where we
select examples from the training data and present them along with explanations in the form of highlights (\figref{fig:tutorial}(a)\&(b)).
We also develop a novel algorithm that incorporates spaced repetition to help humans understand the patterns in a
machine learning
model,
and conduct an
in-person user study to refine the design of our tutorials.

Our main contribution in this work is to design large-scale, randomized, pre-registered human-subject experiments to investigate whether tutorials provide useful training to humans, using the aforementioned deceptive review detection task as a testbed.
We choose this
task because 1) deceptive information including fake news is prevalent on the Internet \citep{allcott2017social,grinberg2019fake,lazer2018science,ott2012estimating} and mechanical turkers can provide a reasonable proxy for humans facing this challenge compared to other tasks such as bail decisions and medical diagnosis that require domain expertise;
2) while humans struggle with detecting deception~\citep{bond2006accuracy},
machine learning
models are able to learn useful patterns in constrained settings (in particular,
ML
models
achieve an accuracy of above 85\%
in our deceptive review detection task);
3) full automation might not be desired in this case because the government should not have the authority to automatically block information from individuals, and it is important to {\em enhance} human ability with a machine in the loop.
Specifically, we focus on the following three research questions:
\begin{itemize}[itemsep=-2pt,topsep=-4pt]
    \item {\bf RQ1:} Do model-driven tutorials improve human performance without any real-time assistance?
    \item {\bf RQ2:} How do varying levels of real-time assistance affect human performance {\em after} training?
    \item {\bf RQ3:} How do model complexity and
    %
    explanation methods affect human performance with/without training?
\end{itemize}
In all experiments, if training is provided, human subjects first go through a training phase with model-driven tutorials, and then enter the prediction phase to determine whether a review is deceptive or genuine.
The prediction phase allows us to evaluate human performance after training.

Our first experiment aims to compare the effectiveness of different model-driven tutorials.
Ideally, we would hope that these tutorials can help humans understand the patterns embedded in the
ML models well enough that they can perform decently in the prediction phase without any real-time assistance.
Our results
show that human performance after tutorials are always better than without training, and the differences are statistically significant for two types of tutorials.
However, the improvement is relatively limited: human performance reaches $\sim$60\%, while the ML models are above 85\%.
Meanwhile, there is no statistically significant difference between human performance after any type of tutorial, which suggests that all model-driven tutorials are similarly effective.

One possible reason for the limited improvement of human performance in Experiment 1 is that the patterns might be too complicated for humans to apply in the prediction phase without any real-time assistance.
Therefore, our second experiment is designed to understand the effect of tutorials with real-time assistance.
Inspired by Lai and Tan \cite{lai+tan:19}, we develop a spectrum
with varying levels of real-time assistance between full human agency and full automation (\figref{fig:spectrum}).
Our results demonstrate that real-time assistance can indeed significantly improve human performance to above 70\%.
However, compared to Lai and Tan \cite{lai+tan:19}, the best human performance is not significantly improved.\footnote{We only discuss qualitative differences from
\cite{lai+tan:19},
as these are
separate experiments subject to different randomization processes.}
It suggests that given real-time assistance, tutorials are mainly useful in %
that humans can perform similarly well in the prediction phase with only signed highlights,
thus retaining a
higher level of human agency.

Finally, in order to understand how our results generalize to different kinds of
models, we would like to examine the effect of model complexity and methods of deriving explanations.
Our first two experiments use a linear SVM classifier because linear models are typically deemed interpretable, but deep learning models are
increasingly prevalent because of their superior predictive power.
While it is well recognized that deep learning models are more complex,
it remains an open question how human performance changes with assistance from deep learning models (e.g., BERT) vs. simple models (e.g., linear SVM).
Our results show that tutorials and explanations of simple models lead to better human performance than deep learning models, which
highlights the tradeoff between {\em model complexity} and
{\em interpretability}.
We also show that for BERT, post-hoc signed explanations from LIME are more effective than built-in explanations derived from attention mechanisms.
Moreover, tutorials are effective in improving human performance for both kinds of models compared to without training.

Overall, our results show that model-driven tutorials can somewhat improve human performance with and without real-time assistance, and humans also find these tutorials useful.
However, the limited improvement also points to future directions of human-centered interpretable machine learning.
We highlight two implications here and present further discussions in the Discussion section\xspace.
First, it is important to explain beyond the surface patterns and facilitate humans in reasoning about why a feature is important.
A strategy is to develop interactive explanations that allow humans to explore the patterns
in both the training and the prediction phase.
Second, it is useful to bridge the gap between training and generalization in developing tutorials because the model behavior and performance in training data might differ from that on unseen data.
The ability to understand this difference is crucial for humans to calibrate trust and generalize beyond the constrained dataset.

\section{Related Work}

We
start by introducing recent methods for interpretable ML, and then discuss experimental studies on human interaction with explanations and predictions derived from ML models.
We end by summarizing related work on deception detection.

\subsection{Methods for interpretable machine learning}
A battery of studies propose various algorithms to explain a machine prediction by uncovering model internals (also known as local explanations) \citep{guidotti2019survey}.
Most relevant to our work is feature attribution that assigns an importance weight to each feature \citep{lei2016rationalizing,lundberg2017unified,ribeiro2016should,anchors:aaai18}.
For instance, Ribeiro et al. \citep{ribeiro2016should} propose LIME that fits a sparse linear model to approximate local machine predictions, and coefficients in this linear model are used as explanations.
Lai et al. \citep{lai+cai+tan:19} compare the built-in and post-hoc explanations methods in text classification and show that different methods lead to very different explanations, in particular, deep learning models lead to explanations with less consistency than simple models such as linear SVM.
Other popular approaches include 1) example-based \citep{kim2016examples,kim2014bayesian,mothilal2019explaining,russell2019efficient,wachter2017counterfactual}, e.g., counterfactual explanations find alternative examples that would have obtained a different prediction, and
2) rule-based \citep{andrews1995survey,guidotti2018lore} that summarizes local rules (e.g., via decision trees).
Notably,
SP-LIME is an algorithm that selects examples to provide a global understanding of the model \citep{ribeiro2016should}, which aligns with our goal of generating tutorials.
However,
to the best of our knowledge, there have not been any human-subject experiments with such example-driven tutorials.

\subsection{Human interaction with explanations and models}

The importance of human-subject experiments is increasingly recognized in understanding the effectiveness of explanations because they are ultimately used by humans.
In addition to
studies mentioned in the introduction, researchers have investigated other desiderata of explanations
\citep{binns2018s,cai2019human,green2019disparate,green2019principles,kunkel2019let,poursabzi2018manipulating,yin2019understanding}.
For instance, Binns et al. \citep{binns2018s} examine
perception of justice given multiple styles of explanations and conclude that there is no best approach to explaining algorithmic decisions.
Cai et al. \citep{cai2019human} show that a user-centered design improves human perception of an image-search tool's usefulness, but does not improve human performance.
Green and Chen \citep{green2019disparate} find that humans underperformed a risk assessment tool even when presented with its predictions, and exhibited behaviors that could exacerbate biases against minority groups.
Yin et al. \citep{yin2019understanding} examine the effect of stated accuracy and observed accuracy on humans' trust in models,
while Kunkel et al. \cite{kunkel2019let} study the effect of explanations on trust in recommender systems.
This line of work on trust also relates to the
literature on appropriate reliance
with general automation \citep{lee2004trust,lewandowsky2000dynamics}.
Retaining human agency is particularly important in societally critical domains where consequences can be dire.
Finally, Bansal et al. \cite{bansal2019beyond} provide feedback during decision making, which can be seen as a form of continuous learning.
Our focus
is to understand the effect of offline tutorials, which can be potentially combined with real-time assistance/feedback in practice.\footnote{Although feedback (e.g., true labels) on real decisions such as bail decisions can take a long time to observe.}

\subsection{Deception detection}
Deception is a ubiquitous phenomenon and has been studied in  many disciplines \citep{vrij2000detecting}.
In psychology, deception is defined as an act that is intended to foster in another person a belief or understanding which the deceiver considers false \citep{krauss1976modalities}.
Computer scientists have been developing machine learning models to identify deception in texts, images, and videos \citep{abouelenien2014deception,feng2012syntactic,feng2013detecting,jindal2008opinion,ott2011finding,perez2015verbal,wu2010distortion,yoo2009comparison}.
An important challenge in studying deception is to obtain groundtruth labels because it is well recognized that humans struggle at detecting deception \citep{bond2006accuracy}.
Ott et al. \citep{ott2011finding} created the first sizable dataset in deception detection by employing workers on Amazon Mechanical Turk to write imagined experiences in hotels.

As people increasingly rely on information on the Internet (e.g., online reviews for making purchase decisions \citep{chevalier2006effect,trusov2009effects,ye2011influence,zhang2010impact}),
deceptive information also becomes prevalent \citep{caspi2006online,ott2012estimating,shin2011prevalence}.
The issue of misinformation and fake news has also attracted significant attention from both the public and the research community \citep{farsetta2006fake,grinberg2019fake,lazer2018science,vosoughi2018spread,zhang2018structured}.
Our work employs the deceptive review detection task in Ott et al. \citep{ott2013negative,ott2011finding} to investigate the effectiveness of model-driven tutorials.
While this task is a constrained case of deception and may differ from intentionally malicious deception, it represents an important issue that people face on a daily basis and can potentially benefit from assistance from ML models.

\section{Methods}

In this section, we introduce the preliminaries for our prediction task, machine learning models, and explanation methods.
We then develop tutorials to help humans understand the embedded patterns in the
models in the training phase.
Finally, we present types of real-time assistance in the prediction phase.
A demo is available at \url{https://deception.machineintheloop.com}.

\subsection{Dataset, models, and explanations}

\para{Dataset and prediction task.}
We employ the deceptive review detection task developed by Ott et al. \citep{ott2013negative,ott2011finding}, consisting of
800 genuine and 800 deceptive hotel reviews for 20 hotels in Chicago.
The genuine reviews were extracted from TripAdvisor and the deceptive ones were written by turkers who were asked to imagine their experience.
We use 80\% of the reviews as the training set and the remaining 20\% as
the
test set.
We evaluate human performance based on their accuracy on sampled reviews from the test set.
The task
for both humans and ML models
is to determine whether a review is deceptive or genuine based on the text.

\para{Models.}
We consider
a linear SVM classifier with unigram bag-of-words as features, which represents a simple model, and BERT \citep{devlin2018bert}, which represents a deep learning model with state-of-the-art performance in many NLP tasks.
The hyperparameter for linear SVM was selected via 5-fold cross validation with the training set; BERT was fine-tuned on 70\% of the reviews
and
the other
10\% of the reviews
in the training set
were used as the development set for selecting hyperparameters.
Table~\ref{tb:accuracy} shows their accuracy
on the test set.

\begin{table}[h]
\vspace{-0.1in}
\small
\centering
\begin{tabular}{p{0.2\textwidth}r}
\toprule
Model &  Accuracy (\%) \\
\midrule
SVM & 86.3 \\
BERT & 90.9 \\
\bottomrule
\end{tabular}
\caption{Accuracy of machine learning models on the test set.}
\label{tb:accuracy}
\vspace{-0.1in}
\end{table}

\para{Methods of deriving explanations.}
We explain a machine prediction by highlighting the most important 10 words.
For linear SVM, we use the absolute value of coefficients to determine feature importance, and the highlights are signed because coefficients are either positive or negative.
For BERT, we consider two methods following Lai et al. \citep{lai+cai+tan:19}:
1) BERT attention based on the built-in mechanism of Transformer~\citep{vaswani2017attention} (specifically, feature importance is calculated using the average attention values of 12 heads used by the first token at the final layer; these highlights are unsigned because attention values
range between 0 and 1);
2) BERT LIME, where feature importance comes from LIME by fitting a sparse
linear model to approximate local model predictions (these highlights are signed as they come from coefficients in a linear model).

\subsection{Tutorial generation}

Our main innovation in this work is to introduce a training phase with {\em model-driven} tutorials before humans interact with ML models.
We consider the following two types of tutorials.

\noindent {\em Guidelines.}
We follow the current practice of science communication and summarize findings in scientific papers \cite{ott2013negative,ott2011finding,li2014towards} as a list of guidelines.
These guidelines are observations derived from the
ML model
(see ``\figref{fig:tutorial}(c)'') and paraphrased by us.
A ``Next'' button is enabled after a 30-second timer.
 %
  %
  %

\noindent {\em Example-driven tutorials.}
Inspired by Ribeiro et al. \citep{ribeiro2016should}, another way to give humans a global sense of a model is to present a sequence of examples along with predicted labels and explanations of predictions.
For each example in our tutorial, informed by our in-person user study, we
first ask participants to determine the label of the example, and then reveal the actual label and the predicted label along with explanations in the form of highlights.
The algorithm selects
10 examples that are representative of the patterns that the ML model identifies from the training set.\footnote{We chose 10 so that an experiment session finishes within a reasonable amount of time (30 minutes), and all examples happened to be classified correctly by the model (since machine performance is even better on the training set).}
There could be genuine insights as well as spurious patterns.
Ideally, these examples allow participants to understand the problem at hand and then apply the patterns, including correcting spurious ones, in the prediction phase.
\figref{fig:tutorial}(a)\&(b) presents an example review after
the label is chosen and the predicted label and its explanations are shown.
A ``Continue'' button is enabled after a 10-second timer.
See the supplementary material for screenshots.

\begin{figure*}[t]
  \centering
  \includegraphics[width=0.8\textwidth]{spectrum.pdf}
  \vspace{-0.1in}
  \caption{An adapted spectrum between full human agency and full automation from Lai and Tan [32].
  The order approximates our intuition, but the distance does not reflect linear changes in machine influence.
  In particular, guidelines do not necessarily increase the influence of predicted labels.
  %
  %
  }
  \label{fig:spectrum}
\end{figure*}

We consider the following algorithms for example selection:

\begin{itemize}[itemsep=-2pt,topsep=-2pt,leftmargin=*]
  \item {\bf Random.} 10 random examples are chosen.
  %
  \item {\bf SP-LIME.}
  Ribeiro et al. \citep{ribeiro2016should} propose SP-LIME to select examples with features that provide great coverage in the training set.
  To do that, the global importance of each feature is defined as $I_j = \sqrt{\sum_{i=1}^n W_{ij}}$, where $W_{ij}$ is the importance of feature $j$ in the $i$-th instance.
  Since we only highlight the top 10 features, $W_{ij}=0$ for any other features.
  %
  Then, 10 examples are selected to maximize the following objective function:
$\argmax_{S, |S| \leq B}  \sum_{j=1}^{d} \mathbb{1}(\exists i \in S: W_{ij} > 0) I_{j},$
where $B=10$ and $d$ represents the dimension of features.
This objective function presents a weighted coverage problem over all features, and is thus submodular.
A greedy algorithm provides a solution with a constant-factor approximation guarantee of $1 - 1/e$ to the optimum \citep{krause2014submodular}.
  %
  %
  %
  \item {\bf Spaced repetition (SR).}
  We propose this algorithm to leverage insights from the education literature regarding the effectiveness of spaced repetition (e.g., on long-term retention) \citep{kang2016spaced,tabibian2019enhancing}.
  Specifically,
  %
  %
  we develop the following novel objective function so that users can be exposed to important features repeatedly:
  $\argmax_{S, |S| \leq B} \sum_{j=1}^d U(\{W_{kj}\}_{1 \leq k \leq |S|})I_{j},$
  where $U(\{w_{kj}\}_{1 \leq k \leq |S|}) = \mathbb{1}(\max(\{k, W_{kj} > 0\}) - \min(\{k, W_{kj} > 0\}) \geq 3)$.
The key difference from SP-LIME is that the weight of a feature is included only if it is repeated in two examples with a gap of at least three.
   %
  %
  %
  %
  %
\end{itemize}

Finally, we consider the combination of guidelines and examples selected with spaced repetition by first showing the guidelines for 15 seconds, 10 examples selected with spaced repetition, and the guidelines again for 15 seconds.

\subsection{Real-time assistance}

\begin{figure}[t]
  \centering
  \includegraphics[width=0.45\textwidth]{unsigned_highlights.png}
  \vspace{-0.05in}
  \caption{Unsigned highlights for the example review in \figref{fig:tutorial}(a).}
  \vspace{-0.2in}
  \label{fig:unsigned-highlights}
\end{figure}

In addition to tutorials in the training phase, we introduce varying levels of real-time assistance in the prediction phase.
Inspired by Lai and Tan \citep{lai+tan:19}, we design six levels of real-time assistance, as illustrated in \figref{fig:spectrum}.

\begin{itemize}[itemsep=-2pt,topsep=-2pt]
  \item {\bf No machine assistance.} Participants are not exposed to any real-time machine assistance.
  \item {\bf Unsigned highlights.} Top 10 features are highlighted in shades of blue. The darker the color, the more important the feature.
  See \figref{fig:unsigned-highlights} for an example.
  %
  \item {\bf Signed highlights.} Top 10 features are highlighted in shades of green and red: green words are associated with genuine reviews, while red words are associated with deceptive reviews.
  %
  The darker the color, the more important the feature. See \figref{fig:tutorial}(a) for an example.\footnote{We use an attention check question to make sure that participants can distinguish red from green.}
  %
  \item {\bf Signed highlights + predicted label.}
  In addition to signed highlights, we display the predicted label.
  %
  \item {\bf Signed highlights + predicted label + guidelines.}
  %
  We additionally provide the option of revealing guidelines.
  %
  \item {\bf Signed highlights + predicted label + guidelines + accuracy statement}.
  %
  We further add an accuracy statement, ``It has an accuracy of approximately 86\%'', emphasizing the strong performance of the ML model.
  %
  %
\end{itemize}

These six levels gradually increase the amount of information and prime users towards machine predictions.
Ideally, we
hope to retain human agency
as much as possible
while achieving strong human performance.

\section{In-person User Study}

To obtain a qualitative understanding of human interaction with model-driven tutorials,
we conduct an in-person semi-structured user study.
This user study allows us to gather in-depth insights on how humans learn and apply our tutorials through interviews,
as well as feedback on the
interface before conducting large-scale, randomized experiments.

\subsection{Experimental design}
We employ a concurrent think-aloud process with participants~%
\citep{nielsen2002getting}.
Each participant went through a tutorial and determined the label of 20 reviews from the test set.
They were told to verbalize the reason before deciding on the label
both in the training and the prediction phase with the following syntax: I think the review is {\em predicted label} because {\em reason}.
After the prediction phase, we conducted an interview to gather general feedback on tutorials.
We manually transcribed the audio recordings after an initial pass with the Google Cloud API.

A total of 16 participants were recruited from mailing lists in our department:
3 were female and 13 were male, ranging between age 20 and 35.
All participants were engineering graduate students and most of them studied computer science.
Participants were invited to the lab where the study occurred.
Either a personal or a provided laptop was used.
Participants were compensated between \$15 and \$20 for \$10 every 30 minutes.
Four types of tutorials (guidelines, examples selected with SP-LIME\xspace, examples selected with SR, guidelines + examples selected with SR) were randomly assigned to participants and each tutorial type had a sample size of 4.
Thematic analysis was undertaken to identify common themes in participants' think-aloud processes.
Thematic codes were collectively coded by the first two authors.

\subsection{Results}

We summarize the key themes into the following three parts.

\para{Tutorial training and application.}
8 out of 8 participants with access to guidelines remembered a couple of ``rules'' and applied them in the prediction phase.
P13 said (the number is randomly assigned), ``I believe it is deceptive based on rule No. one and No. three, if I remembered them correctly, it just describes its experience, and does not have a lot of details''.

7 out of 12 participants exposed to selected examples
adopted pure memorization or pattern-matching during the prediction phase.
Participants remembered key deceptive words such as ``chicago''
to help them
decide the review label:
P2 said, ``My husband is deceptive, I is deceptive, Chicago is deceptive''.
Some participants were even able to generate similar theories to our guidelines without
exposure to it.
P14 commented, ``The review didn't have anything specific to offer'' before deciding that the respective review was deceptive.
However, reasoning about the patterns is generally challenging.
Quoting from P2, this is mainly because they ``can't seem to find a rhyme or reason for those words being genuine or deceptive''.

Participants
also created
theories such as length of review
when predicting.
P8 remarked, ``no one would take that much time to write a review so it won't cross more than 5 lines''.

\para{Improvements on tutorials.}
Participants
thought that the guidelines should be available during the prediction phase
to better assist them.
4 out of 4 participants felt that they were unable to remember
as
there were too many guidelines to be memorized.
P11 felt that ``the tutorial is helpful but it's just hard not being able to reference it''
and P9 said that he could ``keep checking if it is on the top right corner''.

12 out of 12 participants exposed to selected examples
expressed confusion about why the features were highlighted as deceptive or genuine but made up their own reasonings for ease of memory.
They felt that they would have learned better if some form of explanations were given to justify each feature's indication.
P16 remarked that ``it would be nice if it can let me know why exactly it thinks the word is deceptive''
and P10 commented that on top of the current explanations in selected examples, ``more detailed explanation would be helpful'' to help understand.

\para{Improvements on the interface.}
We found that
some participants thought that deceptive reviews are written by an AI without reading the instructions, which is false.
We thus introduced three additional questions for our large-scale experiments:
1) how are deceptive reviews defined in this study?; 2) identify the color that highlights a word; 3) reiterate the training process and ask user to answer true or false to ensure that the participants know which treatment they are exposed to.
We also changed the flow of showing explanations in the training phase: users need to first determine the label for a review before the explanations, the actual label, and the predicted label are shown for at least 10 seconds.
Refer to
the video
and detailed feedback
in the supplementary material.

\section{Experiment 1: Do Tutorials Improve Human Performance without any Real-time Assistance?}

As introduced in the Methods section\xspace, we hope to build tutorials that can help humans understand the embedded patterns in
ML models, which can sometimes be unsalient, unknown, or even counterintuitive to humans.
Ideally, humans reflect on these patterns from our tutorials and can apply them in
their decision making
without any further real-time assistance from
ML models.
Therefore, we start with
RQ1: do tutorials improve human performance without any real-time assistance?

\subsection{Experimental treatments \& hypotheses}

We consider the following treatments to examine the effectiveness of various tutorials proposed in the Methods section\xspace:
1) guidelines;
2) random examples;
3) examples selected with SP-LIME\xspace;
4) examples selected with SR;
5) guidelines + examples selected with SR.
All the tutorials and explanations in the tutorials are based on the linear SVM classifier in the Methods section\xspace.
After a training phase, participants will then decide whether a review is deceptive or genuine based on the text.
Note that ML models also rely exclusively on textual information.
In addition to these tutorials, we include a control setup where no training was provided to humans.

We hypothesize that 1) training is important for humans to understand this task, since it has been shown that humans struggle with deception detection \citep{bond2006accuracy};
2) it would be easier for participants to understand the patterns embedded in the ML model situated with examples;
3) carefully chosen examples provide more comprehensive coverage and can better familiarize participants with the patterns \citep{kang2016spaced,tabibian2019enhancing};
4) guidelines and examples have complementary effects in the training phase.
To summarize, our hypotheses in Experiment 1 are as follows:

\begin{itemize}[itemsep=-2pt,topsep=-2pt]
    \item ({\bf H1a}) Any tutorial treatment leads to better human performance than the control setup.
    \item ({\bf H1b}) Examples (including {\em random examples, examples selected with SP-LIME\xspace and SR}) lead to better human performance than {\em guidelines}.
    \item ({\bf H1c}) {\em Selected examples (with SP-LIME or SR)} lead to better human performance than {\em random examples}.
    \item ({\bf H1d}) {\em Examples selected with spaced repetition} lead to better human performance those selected with {\em SP-LIME\xspace}.
    \item ({\bf H1e}) {\em Guidelines + examples selected with SR} lead to the best performance.
\end{itemize}
These five hypotheses
were pre-registered on AsPredicted.\footnote{The anonymized pre-registration document is available at \url{https://aspredicted.org/blind.php?x=v8f7zh}.
A minor inconsistency
is that we did not experiment with ``guidelines + examples selected from SP-LIME'' as we hypothesized that SR is better.
}

\subsection{Experimental design}

To evaluate human performance under different experimental setups, participants were recruited via Amazon Mechanical Turk and filtered to include only individuals residing in the United States, with at least 50 Human Intelligence Tasks (HITs)  completed and 99\% of HITs approved.
Each participant is randomly assigned to one of the six conditions (five types of tutorials + control).
We did not allow any repeated participation.
We adopted this between-subject design
because exposure to any type of tutorial cannot be undone.

In our experiment, each participant finishes the following steps sequentially:
1) reading an explanation of the task and a consent form;
2) answering a few attention-check questions depending on the experimental condition assigned;
3) undergoing a set of tutorials if applicable (training phase);
4) predicting the labels of 20 randomly selected reviews in the test set (prediction phase);
5) completing an exit survey.
Participants who failed the attention-check questions are automatically disqualified from the study.
Based on feedback from our in-person user study, for each example in the tutorials, a participant
first chooses genuine or deceptive without any assistance, and then the answer is revealed and the predicted label and explanations are shown (\figref{fig:tutorial}(a)\&(b)).
In the exit survey, participants were asked to report basic demographic information, if the tutorial was helpful (yes or no), and feedback in free responses.\footnote{Feedback from Turkers generally confirmed findings in the in-person user study. See the supplementary material for an analysis.}

Each participant was compensated \$2.50 and an additional \$0.05 bonus for each correctly labeled test review.
80 subjects were recruited for each condition so that each review in the test set was labeled five times.
In total 480 subjects completed Experiment 1.
They were balanced on gender (224 females, 251 males, and 5 preferred not to answer).
Refer to the supplementary material for additional information about experiments (e.g., education background, time taken).

To quantify human performance, we measure it by the percentage of correctly labeled instances by humans.
In other words, the prediction phase provides an estimate of human accuracy through 20 samples.
In addition to this objective metric, we also report subject perception of tutorial usefulness reported in the exit surveys.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.35\textwidth]{exp1_acc.pdf}
  \caption{Human accuracy without any real-time assistance after different types of tutorials.
    Error bars represent standard errors.
  Human accuracy after tutorials is always better than that without any training.
  Differences are statistically significant between random and control, and guidelines and control based on post-hoc Tukey's HSD test.
  }
  \label{fig:exp1-acc}
\end{figure}

\begin{figure}[t]
  \centering
  \includegraphics[width=0.35\textwidth]{exp1_use.pdf}
  \caption{Subjective perception of tutorial usefulness.
    Error bars represent standard errors.
    Differences are statistically different in the following pairs based on post-hoc Tukey's HSD test: guidelines vs. random, random vs. SR+guidelines, and SR vs. SR+guidelines.}
  \label{fig:exp1-use}
  \vspace{-0.1in}
\end{figure}

\subsection{Results}
We first
present human accuracy in the prediction phase, an objective measurement of tutorial effectiveness.
Our results suggest that tutorials are useful to some extent:
all tutorials lead to better human performance ($\sim$60\%) than the control setup without any training (\figref{fig:exp1-acc}).
To formally compare the treatments, we conduct an one-way ANOVA
 and find
a statistically significant
effect
($\eta^2 = 0.033$; $p = \num{7.70e-3}$).
We further use post-hoc Tukey's HSD test to identify pairs of experimental conditions in which human performance exhibits significant differences.
The only statistically significant differences are {\em guidelines} vs. {\em control} ($p = \num{1.75e-2}$) and {\em random} vs. {\em control} ($p = \num{7.0e-3}$) (the difference between {\em guidelines+SR} and {\em control} is borderline significant with $p = 0.10$).

In other words, our experiment results provide partial support to {\bf H1a}, and reject all other hypotheses in Experiment 1.
These results suggest that although tutorials provide somewhat useful training, different tutorials are similarly effective.
The limited improvement in human performance across all tutorials indicates that the utility of tutorials is small.
We hypothesized that it is too challenging for humans to remember all the patterns after a short tutorial (supported by feedback from in-person user study), which motivated Experiment 2 to understand the effect of real-time assistance in conjunction with tutorials.
Another contributing factor certainly lies in the design of tutorials, which we will further discuss in the Discussion section\xspace.

As for subjective perception of tutorial usefulness,
we find that participants generally find our tutorials useful:
73.8\% of 400 participants reported that the tutorial was useful (excluding 80 participants in the control setup).
\figref{fig:exp1-use} shows the results by types of tutorials.
Among different treatments, participants in {\em guidelines} and {\em guidelines + examples selected with SR} find the tutorials most useful, as high as 90\% in {\em guidelines + examples selected with SR}.
Formally, post-hoc Tukey's HSD test shows that the differences between the following pairs are statistically different: {\em guidelines} vs. {\em random} ($p = 0.048$), {\em random} vs. {\em SR+guidelines} ($p < 0.001$), and {\em SR} vs. {\em SR+guidelines} ($p = 0.003$).
The difference between {\em SP-LIME} and {\em SR+guidelines} is borderline significant with $p = 0.078$.
These results suggest that tutorials provide strong positive effects in humans' subjective perception.

\section{Experiment 2: Human Performance with Varying Real-time Assistance after Tutorials}

Our second experiment is concerned with human performance with varying levels of real-time assistance after going through the training phase.
While Experiment 1 suggests that tutorials provide somewhat useful training, the improvement is limited without any real-time assistance.
We hypothesize that human performance could be further improved by introducing real-time assistance.
We adapt a spectrum with varying levels of real-time assistance from Lai and Tan \citep{lai+tan:19} (\figref{fig:spectrum}).
Moving along the spectrum, the influence of the machine generally becomes greater on the human
as more information from the model is presented.
For instance, a statement of strong machine performance is likely to bias humans towards machine predictions.
Lai and Tan \citep{lai+tan:19} find that there exists a tradeoff between human performance and human agency, i.e., as the real-time assistance gives stronger priming along the spectrum, human performance improves and human agency decreases.
Explanations such as highlighting important words can moderate this tradeoff {\em when predicated labels are given}.
It remains an open question how this tradeoff unfolds after training.

\subsection{Experimental treatments \& hypotheses}

All conditions in Experiment 2 used the {\em guidelines + selected examples with spaced repetition} tutorial in the training phase
because all tutorials are similarly effective and our participants find this one most useful in subjective perception.
To examine how humans perform under different levels of real-time assistance from machine learning models, we consider
the spectrum in \figref{fig:spectrum},
inspired by Lai and Tan \citep{lai+tan:19}.

We hypothesize that 1) real-time assistance results in
improved human performance, since it has been shown that highlights and predicted labels improve human performance \citep{lai+tan:19}; 2) signed highlights result in better human performance compared to unsigned highlights because signed highlights reveal information about directionality; 3) predicted labels result in better human performance compared to highlights alone; 4) guidelines and signed highlights might moderate the tradeoff between human performance and human agency while achieving the same effect as when
an accuracy statement is shown.
To summarize, our hypotheses
are as follows:
\begin{itemize}[itemsep=-2pt,topsep=-2pt]
    \item ({\bf H2a}) Real-time assistance leads to better human performance than no assistance.
    \item ({\bf H2b}) {\em Signed highlights} lead to better human performance than {\em unsigned highlights}.
    \item ({\bf H2c}) {\em Predicted label} leads to better human performance than highlights alone.
    \item ({\bf H2d}) {\em Signed highlights + predicted label + guidelines + accuracy statement} leads to the best performance.
    \item ({\bf H2e}) {\em Signed highlights + predicted label + guidelines}
    and {\em Signed highlights + predicted label} perform as well as {\em Signed highlights + predicted label + guidelines + accuracy statement}.
\end{itemize}
These five hypotheses were pre-registered on AsPredicted.\footnote{The anonymized pre-registration document is available at \url{http://aspredicted.org/blind.php?x=fi8kz8}.}

\subsection{Experimental design}
We adopted the same experimental design as stated in Experiment 1 except that real-assistance is provided in the prediction phase when applicable.
In total 480 subjects completed the experiment (80 participants in each type of real-time assistance).
They were balanced on gender (238 females, 237 males, and 5 preferred not to answer).
Refer to the supplementary material for additional information about experiments (e.g., education background, time taken).

Human performance is measured by the percentage of correctly predicted instances by humans, which provides an objective measure of human performance with real-time assistance.
We also consider the percentage of humans whose performance exceeds machine performance for the corresponding 20 reviews in the prediction phase.\footnote{We also pre-registered trust as a measure and present the results in the supplementary material for space reasons.}

\begin{figure}[t]
  \centering
  \includegraphics[width=0.4\textwidth]{exp2_acc.pdf}
  \caption{Human accuracy with varying levels of real-time assistance after training.
  Error bars represent standard errors.
  With the exception of {\em unsigned highlights}, human accuracy with real-time assistance is better than without real-time assistance.
  Differences between {\em no assistance} and any assistance with signed highlights are statistically significant based on post-hoc Tukey's HSD test.
  %
  }
  \label{fig:exp2-acc}
  \vspace{-0.1in}
\end{figure}

\subsection{Results}
We first present human accuracy in the prediction phase.
Our results suggest that real-time assistance is indeed effective: all the levels of real-time assistance except unsigned highlights lead to better human performance than the setup without machine assistance in \figref{fig:exp2-acc}.
To formally compare the treatments, we conduct an one-way ANOVA
and find a statistically significant
effect
($\eta^2 = 0.23$; $p = \num{5.15e-25}$).
We further use post-hoc Tukey's HSD test to identify pairs of experimental conditions in which human performance exhibits significant differences.
With the exception of {\em no assistance} vs. {\em unsigned highlights} ($p = 0.67$), differences in remaining setups compared to {\em no assistance} are all statistically significant ($p < 0.001$).
Moreover, the difference between {\em unsigned highlights} and {\em signed highlights} is significant ($p < 0.001$),
demonstrating the effectiveness of signed highlights.
Finally, the difference between {\em signed highlights} and any other real-time assistance with stronger priming ({\em signed highlights + predicted labels, signed highlights + predicted labels + guidelines, signed highlights + predicted labels + guidelines + accuracy statement}) is not significant.

In summary, our experimental results support {\bf H2a} with the exception of {\em unsigned highlights}, {\bf H2b}, {\bf H2e}, and reject {\bf H2c} and {\bf H2d} in Experiment 2 (note that {\em signed highlights + predicted label + guidelines + accuracy statement} indeed leads to the best performance but the difference with other methods is not always statistically significant).
These results suggest that {\em signed highlights}
provide sufficient information for improving human performance, and we do not gain much from presenting additional information with stronger priming.
While there is significant improvement in human performance with real-time assistance (from $\sim$60\% to $\sim$70\%),
the improvement is still limited compared to the machine performance, which is above 85\%.
This improvement is similar to results reported in Lai and Tan \citep{lai+tan:19}, which did not use any tutorials other than minimal examples to introduce the task.
These observations taken together suggest that the utility of our tutorials mainly lies in that humans can perform well with only signed highlights, a type of real-time assistance with relatively weak priming.

Another ambitious measurement
is
how frequent humans outperform the ML model.
It was rare in Experiment 1 (2 of 480, 0.4\%).
With effective real-time assistance (i.e., signed highlights included),
we find that 26 of 320 (8.1\%, 20 times the percentage in Experiment 1) of our participants are able to outperform the ML model.
The difference between 8.1\% and 0.4\% is statistically significant using chi-squared tests ($p < 0.001$).
This observation suggests that with the help of tutorial and real-time assistance, there exists hope for a synergy of {\em humans and AI} outperforming AI alone.
We hypothesize that facilitating hypothesis generation is important
and present detailed discussions in the Discussion section\xspace.

\section{Experiment 3: The Effect of Model Complexity and Methods of Deriving Explanations}

Our experiments so far are based on explanations (coefficients) from a linear SVM classifier.
Meanwhile, deep learning models are being widely adopted because of their superior predictive power.
However, it is also increasingly recognized that they might be more complex and harder to interpret for humans.
Our final experiment investigates how model complexity and methods of deriving explanations relate to human performance and effect of training.

\subsection{Experimental treatments \& hypotheses}
Participants are exposed to two different treatments: presence of training and methods of deriving highlights.
Where training is present,
we use the {\em selected examples with spaced repetition} tutorial in this experiment.
Note that example selection depends on the model and the explanation method (i.e., which features are considered important).
In comparison, guidelines are static and are extracted from papers based on linear SVM, so they are not appropriate here.
Based on results from Experiment 2, we adopted {\em signed highlights} as our real-time assistance in the prediction phase when applicable.\footnote{Since BERT performs better than linear SVM, only showing signed highlights also avoids the potential effect of predicted labels.}
To summarize, we consider the following setups to examine how humans perform when exposed to training and different methods of deriving explanations:
1) no training + SVM coefficients;
2) no training + BERT attention;
3) no training + BERT LIME;
4) training + SVM coefficients;
5) training + BERT attention;
6) training + BERT LIME.

Note that the deep learning model (BERT) leads to both different real-time assistance and examples selected for tutorials
because they consider different words important.
We can only use unsigned highlights for BERT attention because attention values range between 0 and 1.
Refer to the Methods section\xspace for details of BERT attention and BERT LIME.

We hypothesize that 1) SVM results in better performance compared to BERT, since it is a common assumption that linear models are more interpretable and it has been shown that SVM results in important features with lower entropy~\citep{lai+cai+tan:19}; 2) BERT LIME results in better performance compared to BERT attention because signed highlights can reveal more information about the underlying decision; 3) participants would perform better with training than without training.
To summarize, our hypotheses in Experiment 3 are as follows:
\begin{itemize}[itemsep=-4pt,topsep=-5pt]
    \item ({\bf H3a}) The simple model ({\em SVM}) leads to better human performance than the deep learning model ({\em BERT}).
    \item ({\bf H3b}) {\em BERT LIME} leads to better human performance than {\em BERT attention}.
    \item ({\bf H3c}) {\em Training} leads to better human performance than {\em without training}.
\end{itemize}
These three hypotheses were pre-registered on AsPredicted.\footnote{The anonymized pre-registration document is available at \url{http://aspredicted.org/blind.php?x=vy794a}.}

\subsection{Experimental design}
We adopted the same experimental design as
in Experiment 1.
In total 480 subjects completed the experiment (80 participants in each experimental setup).
They were balanced on gender (239 females, 240 males, and 1 preferred not to answer).
Refer to the supplementary material for additional information about experiments (e.g., education background, time taken).

To quantify human performance, we measure it by the percentage of correctly predicted instances by humans.
In addition to this objective metric, we also report subject perception of tutorial usefulness reported in the exit surveys (note that this is only applicable for the experimental setups with training).

\begin{figure}[t]
  \centering
  \includegraphics[width=0.4\textwidth]{exp3_acc.pdf}
  \caption{Human accuracy grouped by methods of deriving explanations.
  %
  Error bars represent standard errors.
  SVM explanations lead to
  %
  better human performance than explanations based on BERT.
  Training (second bar from the top in each method) also consistently improves human performance for all explanation methods.
  }
  \label{fig:exp3-acc}
\end{figure}

\begin{figure}[t]
  \centering
  \includegraphics[width=0.4\textwidth]{exp3_use.pdf}
  \caption{Human perception of tutorial usefulness.
    Error bars represent standard errors.
Participants are more likely to find SVM tutorials useful
 (differences between (SVM, BERT attention) and (SVM, BERT LIME) are statistically significant using post-hoc Tukey's HSD test).
}
  \label{fig:exp3-use}
  \vspace{-0.05in}
\end{figure}

\subsection{Results}
We first present human accuracy in the prediction phase.
Our results suggest that
methods of deriving explanations make a significant difference (\figref{fig:exp3-acc}): 1) human performance is consistently better when important words derived from the linear SVM
are highlighted as compared to deep models; 2)
BERT LIME leads to better human performance than BERT attention.
It also reinforces the point that training leads to better human performance as compared to no training: humans achieve better performance with training with any kind of explanation methods.
To formally compare the treatments, we conduct a two-way ANOVA
and find a statistically significant
effect of tutorials ($\eta^2 = 0.049$; $p = \num{1.50e-7}$) and methods of deriving explanations ($\eta^2 = 0.13$; $p = \num{4.66e-16}$).
Differences among all pairs of treatments are also statistically significant using post-hoc Tukey's HSD test ($p < 0.001$).\footnote{It is reduced to $t$-test for the training/no training treatment since the degree of freedom is 1.}

In other words, our experiment results provide support to all hypotheses in Experiment 3.
These results suggest that tutorials  are  indeed useful in improving human performance, albeit improvement is still limited in the sense that human performance is $\sim$70\% after training with real-time assistance, echoing results in Experiment 2.
It also suggests that simple models are preferred to deep learning models when serving as explanations to support human decision making.
Between explanations derived from post-hoc and built-in methods from BERT, attention
provides the least
value for humans,
again demonstrating the importance of signed highlights.

The effectiveness of training for simple models is further validated by subjective perception of tutorial usefulness.
\figref{fig:exp3-use} shows that participants are much more likely to find the tutorials derived from SVM explanations useful: 85\% of our participants find it useful.
The differences between the following pairs are statistically different using post-hoc Tukey's HSD test: {\em SVM} vs. {\em BERT attention} ($p < 0.001$) and {\em SVM} vs. {\em BERT LIME} ($p < 0.001$).
Interestingly, with real-time assistance, humans also find the tutorials more useful compared to the same tutorial in \figref{fig:exp1-use}.
These results underscore our findings in Experiment 3 that simple models provide more interpretable tutorials and explanations than deep models.

\section{Discussion}
In this paper,
we conduct the first large-scale, randomized, pre-registered human-subject experiments to investigate whether model-driven tutorials can help humans understand the patterns embedded in ML models and improve human performance.
We find that tutorials can indeed improve human performance to some extent,
with and without real-time assistance, and humans also find them useful.
Moreover, real-time assistance is crucial for further improving human performance in such challenging tasks.
Finally, we show that simple models like linear SVM generate more useful tutorials and explanations for humans than complex deep learning models.

\para{Towards human-centered tutorials.}
Both quantitative results from our randomized experiments and qualitative feedback from in-person user study demonstrate that humans can benefit from model-driven tutorials, which suggests that developing model-driven tutorials is a promising direction for future work in human-centered interpretable machine learning.

However, the improvement in human performance remains limited compared to machine performance in the deceptive review detection task.
In order to further advance the synergy between humans and AI,
we need to develop human-centered tutorials.
Many participants
commented that they could not understand why certain words were deceptive or genuine (an example reason could be that imaginative writing does not cover specific details).
These results highlight the importance of {\em facilitating hypothesis generation} in the tutorials.
It is insufficient to highlight important features via feature attribution methods,
and these tutorials need to also explain why some features are useful.
While it is challenging to develop automatic methods that can propose theories about particular features,
we might prompt humans to propose theories and evaluate them through the ML model.

Another reason that tutorials had limited improvement in human performance is that the tutorials failed to establish proper trust in machine predictions. %
It is important to highlight both strengths and caveats of ML models in the tutorials, echoing
recent work on
understanding trust
 \citep{kunkel2019let,yin2019understanding}.
A challenge lies in how to bridge the gap between training and generalization in tutorials, i.e., model behavior and performance in the tutorials might
differ from that in unseen data.

\para{Beyond static explanations.}
Another important direction is to design interactive explanations beyond static explanations such as simply highlighting important words.
Interactive explanations allow humans to experiment with their hypothesis about feature importance.
One strategy is to enable humans to inquire about the importance of any word in a review.
An alternative strategy is to assess model predictions of counterfactual examples.
For instance, humans can remove or add words/sentences in a review, which can help humans understand model behavior in new scenarios.

\para{Choice of tasks.}
We would like to highlight the importance of task choice in understanding human-AI interaction.
Deception detection might simply be too challenging a task for humans, and a short tutorial is insufficient to help humans understand the patterns embedded in ML models.
There may also exist significant variation between understanding text and interpreting images, because the former depends on culture
and life experience, while the latter relies on basic visual cognition.

We believe that it is important to study human-AI interaction in challenging tasks where human agency is important because the nature of explanations in decision making is distinct from that in debugging.
While machines excel at identifying patterns from existing datasets, humans might be able to complement ML models by deriving theories and appropriately correcting machine predictions in unseen data, e.g., spotting mistakes when machines apply patterns (``chicago'' becomes a specific comparison point for reviews about a hotel in New York City).
So there exists hope for further advancing human performance in these challenging tasks.

\para{Limitation of our samples.}
Our study is limited by our samples of human subjects.
The in-person user study was conducted with university students who tend to have a computer science education, and large-scale, randomized, pre-registered experiments were conducted with
Mechanical Turkers from the United States.
While our samples are likely to face the challenges of deception on the Internet and would benefit from enhancements in deception detection, they may not be representative of the general population.
The effectiveness of model-driven tutorials can also potentially depend on properties of the sample population.
In general, we did not find any consistent differences between demographic groups
based on age, gender, education background, and review experience
(see the supplementary material).
It is certainly possible that other demographic information could affect the effectiveness of tutorials.
We leave that
for future studies.

It is important to point out that our setup employs a random split to obtain training and testing data, which is a standard assumption in supervised machine learning.
While humans can ideally improve generalization in this case,
humans might be more likely to correct generalization errors in machine learning models when the testing distribution differs from training.
In that case, understanding the embedded patterns, especially spotting spurious ones, can help humans generalize these data-driven insights.

In summary, our work highlights the promise of (automatically) building model-driven tutorials to help humans understand the patterns embedded in ML models, especially in challenging tasks.
We hope to encourage future work on human-centered tutorials and explanations beyond static real-time assistance towards a synergy between humans and AI.

\para{Acknowledgments.}
We thank helpful comments from anonymous reviewers.
All experiments were approved by the University of Colorado IRB (18-0558).
This work was supported in part by NSF grants IIS-1837986, 1849931, and 1927322.

\balance{}

\bibliographystyle{SIGCHI-Reference-Format}

redpajama_set_name: RedPajamaArXiv


text:
home > shop > Dailuaine, Connoisseurs Choice
Dailuaine, Connoisseurs Choice
Rich with dried fruits and earthy aromas.
Alcohol %:
Gordon and MacPhail was founded in 1895, as a Family Grocers, Tea, Wine and Spirit Merchant. The founding fathers, James Gordon and John Alexander MacPhail who's approach was to personally select their stocks for their quality. The company started buying up casks of single malt whisky from local distilleries. They were one of the first companies to specialise in single malt whiskies long before individual distilleries started to do so. The used to bottle unde licence from famous distilleirs such as Macallan, Glenlivet, Glen Greant, Linkwood and Mortlach. Eventually they stated to fill their own casks, mostly ex sherry casks and age them for much longer than was the norm.
In the mid 1960's, they released what would become an icon in the world of single malts, namely their Connoisseurs Choice range. These were a range of single malts from different distilleries ladled under Gordon & MacPhail's own Connoisseurs Choice label. These were focused at the Italian, French, American and Dutch markets and ignited the interest in single malt whisky which is still present in these countries today in single malt whiskies.
Gordon & Macphail became a limited company in the late 1970's but is still family owned and managed to this day. www.gordonandmacphail.com
An aromatic whisky with dried fruit aromas - figs and dates. There are spices and earth aromas with a floral note developing. The palate is rich and coating with the dried fruits again evident, along with a slight peppery edge.

redpajama_set_name: RedPajamaCommonCrawl


text:
Q: cHow desactivate sound in my App with 1 button? I have some mp3 audios in some scenes, and I have 1 button in my main scene and I want when user press these button, my App had no mp3 sounds. Do I Have to duplicate my App without sounds? or how?
I have this but not works?
public class AudioApp : MonoBehaviour {

    public void SonidoApp() {
        PlayerPrefs.SetInt("MP3AudioState", 1);

    }

    public void SonidoAppmute() {
        PlayerPrefs.SetInt("MP3AudioState", 0);

    }

}

A: What you can do is when the user press the button you save a value in PlayerPrefs (0 for unmuted 1 for muted as an exemple):
PlayerPrefs.SetInt("MP3AudioState", 1);

And wherever you play the audio sound all you need to check is the value you saved:
if(PlayerPrefs.GetInt("MP3AudioState") == 0){ //Play the sound}

So if the value you saved was 0 at the time the sound wants to be played you let it pass if it's 1 you don't.

A: To disable all Audios in your scene, simply change the volume of the current AudioListener to zero.
The AudioListener is attached to your camera by default, so this could work for you:
Camera.main.GetComponent<AudioListener>.volume = 0;

redpajama_set_name: RedPajamaStackExchange


Browse profiles of Member singles here at Pagan Chat City that are tagged with Jewish. Meeting other singles that have like minded interests is a great way to come up with ideas to do on a first date. Signup for a Free Profile to Meet Someone Tonight!
If you wish to talk to any member on Pagan Chat City, you need to create a 100% Free Account to ensure you are who you say you are. Once you verified you are real, you can begin messaging to find out if there is any interest. It's very simple to meet just about anyone. Just click to send a flirt or message and wait for them to respond. After they respond and there is a common interest, then you can schedule to meet to find out if there is a genuine connection. Stop Waiting around! Open up your free profile below and get started today!


text:
Pseudocoremia berylia is a species of moth in the family Geometridae. It is endemic to New Zealand. This species was first described by George Howes in 1943.

References

Boarmiini
Moths of New Zealand
Endemic fauna of New Zealand
Moths described in 1953
Taxa named by George Howes (entomologist)
Endemic moths of New Zealand

redpajama_set_name: RedPajamaWikipedia


File Types: PSD,PDF,All Files; File Size: 4.02 MB; Layered: Yes; DPI: 300 DPI.
- Smart object & Editable text layers What's Included?
Keywords: business card, psd, corporate business card, creative, minimal, design, photoshop, minimal business card, editable, modern, simple, simplistic, business card template, business, corporate, template, logo, branding, customizable, landscape, fashion.


text:
Get the latest facts on abortion worldwide
February 2019 Policy Analysis
U.S. Health Care Reform Proposals: How They Address Sexual and Reproductive Health Care
Adam Sonfield,Guttmacher Institute
Leah H. Keller,Guttmacher Institute
First published online: February 28, 2019
Updated June 5, 2019.
As policymakers, advocates and health care analysts contemplate the future of the U.S. health care system, the Guttmacher Institute is evaluating how sexual and reproductive health and rights fit into future health care reform efforts.
Framework for Our Analysis
More to Be Done: Individuals' Needs for Sexual and Reproductive Health Coverage and Care
There are a wide array of sexual and reproductive health needs that the U.S. health care system should address: contraception, abortion, maternal and newborn care, infertility, reproductive cancers, sexual or intimate partner violence, HIV/AIDS and other STIs, and additional health needs that cut across categories.
Principles for Reform: New U.S. Health Care Proposals Cannot Overlook Sexual and Reproductive Health
Key principles for addressing sexual and reproductive health needs within the U.S. health care system include: comprehensive and universal coverage; unrestricted patient access to a broad network of qualified providers; and robust patient protections against barriers to care, violations of privacy and autonomy, and discrimination.
Analysis of Specific Proposals
In the resources below, we evaluate how well individual health care reform proposals address sexual and reproductive health and rights.
Medicare for America Act (H.R. 2452, 116th Congress)
Sponsored by Rep. Rosa DeLauro (D-CT); as of 5/1/2019
Creates a new national health insurance program available to U.S. residents to replace many existing public and private plans, but leaves employer-sponsored private coverage as an option. Read how this proposal addresses sexual and reproductive health and rights
Choose Medicare Act (S. 1261 and H.R. 2463, 116th Congress)
Sponsored by Sen. Jeff Merkley (D-OR) and Rep. Cedric L. Richmond (D-LA); as of 5/1/2019
Creates new public health plans (known as Medicare Part E plans) that would be available as options for people buying insurance through an employer or on their own through the Affordable Care Act's (ACA) health insurance marketplaces, with all of the ACA's subsidies to make coverage more affordable. Read how this proposal addresses sexual and reproductive health and rights
Medicare for All Act (S. 1129, 116th Congress)
Sponsored by Sen. Bernie Sanders (I-VT); as of 4/10/2019
Creates a federally run, nationwide health insurance program (often described as a "single-payer" program) to cover all U.S. residents, replacing existing private and public health insurance programs. Read how this proposal addresses sexual and reproductive health and rights
CHOICE Act (S. 1033 and H.R. 2085, 116th Congress)
Sponsored by Sen. Sheldon Whitehouse (D-RI) and Rep. Janice Schakowsky (D-IL); as of 4/4/2019
Creates a new public option for individual and small-group plans to be sold on the Affordable Care Act's (ACA) health insurance marketplaces, with all of the ACA's subsidies to make coverage more affordable. Read how this proposal addresses sexual and reproductive health and rights
Medicare-X Choice Act (S. 981, 116th Congress)
Sponsored by Sen. Michael F. Bennet (D-CO); as of 4/2/2019
Medicare for All Act (H.R. 1384, 116th Congress)
Sponsored by Rep. Pramila Jayapal (D-WA); as of 2/27/2019
State Public Option Act (S. 489 and H.R. 1277, 116th Congress)
Sponsored by Sen. Brian Schatz (D-HI) and Rep. Ben Ray Lujan (D-NM); as of 2/14/2019
Allows states to offer residents of all incomes the option of buying into Medicaid, as a public option that would compete with private plans in the Affordable Care Act's (ACA) health insurance marketplaces. Read how this proposal addresses sexual and reproductive health and rights
U.S. Policy Resources


We're looking for craft beer aficionados to join our team part time. If you're interested in applying, please send your resume to partna@comcast.net.
Many thanks to all of our customers who came out to our ribbon cutting ceremony yesterday afternoon with the Columbia County Chamber of Commerce!
With each growler or howler purchase, you will receive an extra loyalty club point. After you've collected ten points, you'll receive a 32 oz. refill for FREE! See our Loyalty Club page for details on how the program works.
AND we are giving away a free tasting glass with each growler purchase while supplies last.
Come in to the store by Sunday to take advantage of these specials and to help us celebrate our grand opening. You don't want to miss this!
When you order a growler of your favorite craft beer, there is the possibility that the keg will run out mid-pour.
At Tip Top Taps, if this happens to you, you will receive that partial growler for free with the purchase of another growler.
We call it "growler roulette"! It's like winning in Vegas!
Check out our beer menu to see what's currently on tap — and what's running out. You could be our next growler roulette winner!
Tip Top Taps was recently featured in The Buzz on Biz, an online publication featuring small businesses in the CSRA.
We are so thankful to the CSRA for welcoming us with open arms. It's been just over a week since we opened, and we're having a blast — and we hope our customers are, too. We're excited to be a part of the growing craft beer culture in the Augusta area.
Check our beer menu often to see what new beers are in stock and what's disappearing. You won't want to miss the chance to taste these delicious craft brews!
If you've visited the store, please consider leaving us a positive review on Yelp, Facebook, or your favorite review site, and contact us if your visit was less than stellar. We would love to hear from you.
It's tournament time! And to help you celebrate, Tip Top Taps will be open until 9 pm every night this week.
Pick up a few growlers for the ideal refreshment after a day at the course. They also make great gifts for your guests or your hosts during your stay in Augusta.
Check out our beer menu for current selections, and come in to the store to sample them and to fill up your growlers!
Tip Top Taps is Now Open!
We are excited to announce that Tip Top Taps is now open for business!
Come in to say hello, try a few beers, and buy some to take home with you!
Check out our beer selections to see what's currently on tap. You're welcome to try before you buy, and we're confident that you'll find something you love on our diverse menu.
We offer 32-ounce and 64-ounce growlers, filled with your choice of delicious craft beer from one of our 20 taps. Growlers are a convenient way to enjoy draught beer at home and without paying bar premiums.
If the draught options don't strike your fancy, we also stock bombers (large 22-ounce bottles) of a variety of other craft beers.
We are open from noon to 8 pm, 7 days a week. We are located at the corner of Washington Road and Belair Road in Evans, Georgia — just down the street from Evans Towne Center Park.
We look forward to seeing you at Tip Top Taps!
A new growler store in Evans, Georgia — opening soon!
We are working hard to get the new store ready for business. Transforming it from a convenience store to a craft beer mecca is no small task. We want Tip Top Taps to be a comfortable and inviting atmosphere for beer fans of all kinds.
To that end, a growler store is nothing without knowledgable beer lovers manning the taps! We are currently seeking employees (both part-time and full-time positions are available) to serve and evangelize a love of beer to our customers. If you're interested in applying for a position, please see the job post and send your resume to jobs@tiptoptaps.com.
We are also working on the list of beers we will have on tap when we open. If you have any special requests or suggestions, please send them to growlers@tiptoptaps.com or fill out our survey to give us an idea of what you like!
We will keep you posted on the status of Tip Top Taps in the coming weeks, and will announce our grand opening here and on our Facebook page. Stay tuned — and get ready for some awesome beer!


text:
You are here: Home » News and events » Job vacancies
Advertise your vacancy
Job vacancies in the migration sector
This page has details of job vacancies that we're aware of in the migration sector in Yorkshire and Humber. If you're interested in any of the vacancies, contact the organisations directly. Follow this link to find out how to advertise your vacancy on this website.
Migration Yorkshire [Leeds City Council]: Assistant finance support officer
Job title: Assistant finance support officer
Organisation: Migration Yorkshire [Leeds City Council]
Website: https://www.migrationyorkshire.org.uk
Salary: C1 £22,183.00 - £24,491.00
Additional information: 37 hours per week. Fixed term until December 2022
Closing date: 11.30 pm on 02 February 2021
Do you have excellent finance support skills? Are you looking for an interesting and varied role?
We're looking for an assistant finance support officer to work on the Refugee Integration Service [RIS], which is part of a large new project led by Migration Yorkshire, in which we'll provide wide-ranging support to improve the integration of refugees, with a strategic approach and front line services, underpinned by comprehensive evidence and refugee participation. The project is co-funded by the European Union Asylum, Migration and Integration Fund.
You'll be part of the Migration Yorkshire team and you'll provide support across RIS, including assisting in the monitoring of financial and performance management systems for the project and liaising with the UK Responsible Authority [UKRA] and partner organisations.
The key skills we are looking for include:
Experience of maintaining spreadsheets and databases for financial monitoring.
Experience of working with other finance officers in a range of public sector organisations at a similar level.
Experience of working in partnership with others to deliver work to set deadlines.
Ability to prepare clear and concise financial reports and summaries [including written reports].
Ability to assist in the monitoring, analysis and reporting on project performance.
This is a full time post with an option for some home working. The role requires travel across Yorkshire and Humber and occasionally beyond the region.
You'll be joining a friendly and growing team that is based in the centre of Leeds. Migration Yorkshire provides leadership and local support across the Yorkshire and Humber region, and is hosted by Leeds City Council.
For more information and to apply, go to the Leeds City Council jobs and careers website
Migration Yorkshire takes no responsibility for any errors contained in the jobs vacancy listings.
Back to top Print this page
Your contact for this page
Email the Web Manager
Download our guide to becoming a foster carer
Find out about fostering: Welcoming Young Refugees web pages
Do-it: national database of volunteering opportunities
TimeBank: local, national and international volunteering opportunities
Universal Jobmatch


I created this blog to share my experiences while I travel through the exciting path of pursuing an MBA. I have a feeling that this journey is going to be one hell of a ride and probably the most difficult of my journeys. I am not sure about where I end up but one thing I am pretty sure is that this journey is going to teach me a LOT.
OK, that's enough blah for one post. My plan is to write predominantly about the pieces in the MBA puzzle - GMAT preparation, school selection, and application process.


text:
May 20, 2019 California / Motion to Dismiss / Personal Jurisdiction / Pleadings Challenge
Specific Jurisdiction Established Under "Stream of Commerce Plus" Theory
CALIFORNIA – The plaintiff Thomas Toy alleged that his mesothelioma diagnosis was a result of asbestos exposure that incurred in multiple Navy shipyards to a variety of products throughout his machinist career. He claimed he was exposed to friction products in his role as a mechanic for the Army while stationed in Germany, Korea, and other U.S. locations and to construction products he used during home renovations. The defendant Viking Pump, Inc. moved to dismiss pursuant to Federal Rule 12(b)(6), arguing that the plaintiff failed to establish personal jurisdiction in his complaint, and submitted an affidavit alleging that Viking, a publicly traded company, was incorporated and established in Delaware and Iowa, respectively, and had manufacturing facilities in Canada, Ireland, and the United Kingdom.
The court first concluded that no general jurisdiction could be established given that Viking, a Delaware corporation, kept its principal place of business in Iowa. The court went on to conclude, however, that it had specific jurisdiction over Viking under the "stream of commerce plus" theory, despite having no manufacturing facilities in the places in which the plaintiff alleged exposure while in the military. The plaintiff could rely on the allegations in his complaint that:
Viking was a supplier of asbestos-containing pumps; and
Viking regularly conducted business in the California since Viking submitted no evidence that it supplied its products to the Navy "somewhere outside of California."
In sum, although the plaintiff's allegations were "meager," Viking's motion did not contradict the plaintiff's allegations. As a result, the Northern District of California denied Viking's motion to dismiss on personal jurisdiction grounds.
Read the case decision here.
Court Recommends Granting Five Defendants' Motions for Summary Judgment Based on Lack of Substantial Factor Causation
Insurer of Long Defunct Employer May Be Held Liable Under "Enhanced Benefits" in Worker's Compensation Statute


Apowersoft Streaming Audio Recorder is a powerful audio recording software with a wide range of editing and conversion capabilities. With it, you can record any audio playing on the computer (for example, streaming music from an internet website, online live shows or internet radio broadcast). You can also record sound from a microphone or any other audio devices that plug-into your computer. Apart from recording audio from computer, this Streaming Audio Reorder can also work as an ID3 tags identifier. It can automatically get the ID3 tags information like title, artist, album, genre and release year for your music file. Of course, if it failed to get the correct ID3 tags, you can add or edit ID3 tags manually with its inbuilt ID3 Tag Editor. And with its Schedule Tasks function, you are able to set any scheduled tasks at a time that is most convenient for you. Once preset the Start and End time, the application will run in the background of your computer automatically when the condition is triggered. Apowersoft Streaming Audio Recorder also provides the built-in audio converter. It supports converting audio files between any popular audio formats: MP3, WMA, WAV, AAC and OGG. Besides, it's also equipped with smart audio editor that helps you to edit any part of the audio file by cutting, pasting or deleting. Entertainment is another indispensable feature of this audio recorder. It supports the ability to listen to thousands of live radio stations directly on the application; you can enjoy the fun brought by listening to the latest music and the radio from various genres. Anyway, Streaming Audio Reorder can be seen as a good option for recording system sound or voice through microphone. As it's not only favored by the new learners for its simple and clear interface, but also welcomed by professionals for its high-quality and professional recordings.