text:
Filmmaker Javier Tellez On That Time He Shot a Human Cannonball Across the Border
By Iñaki Fernández de Retana
He makes movies with patients from psychiatric institutions around the world. He once fired a human cannonball over the border from Tijuana. He was expelled from the school of Fine Arts in his second year. Javier Téllez is an artist with an intense gaze and hurried speech. "I identify myself with a radical thought," he says.
Born in Valencia, Venezuela, in 1969, Téllez grew up among books, movies, and the mentally ill. His parents, both psychiatrists, had a library of 50,000 books in a house that also functioned as their office where patients would visit. His grandfather founded Capitol Cinema in Turmero in 1911, one of the first theaters in Venezuela. "I saw Jaws at least 20 times," he recalls.
He started drawing, then made films and art installations — mental illness would be the central theme of his work. After spending some years in Madrid and Berlin, Téllez currently lives in New York City.
The San Francisco Art Institute is currently showing "Forbidden Games are in the Labyrinth," an exhibit that consists of a 46-minute film, Dürer's Rhinoceros (2010), shot at the Hospital Miguel Bombarda in Lisbon, Portugal, and a giant installation, "Chess" (2014), that substitutes traditional chess pieces with psychiatric implements. The film revolves around the panopticon, an architectural space composed of cells arranged in a circle that was adopted in psychiatric institutions to allow observation of the internees.
We caught up with Téllez to talk about his new exhibit, working with mentally ill patients, and shooting a human cannonball across the U.S.-Mexico border.
You've worked in psychiatric institutions in Mexico City, Berlin, Tijuana, Aspen… What is the process like of working with patients?
In doing the first installations related to mental health in the early 90s, I realized that I was not authorized to discuss the topic, that I needed the patients' voice, the voice of those living with the mental illness in the psychiatric institutions. Otherwise it would be a projection of what I think about them. Instead of me representing them, I realized that it was better for them to participate.
It is a collaborative process. It is an open call to the patients of a mental institution to work on a film project. I come with an idea that changes completely when I get to know them and receive their feedback. In Aspen, Colorado, I was making a western version of the Ancient Greek tragedy "Oedipus Rex" by Sophocles. From that starting point, we constructed a screenplay and they worked as co-authors of the screenplay — we discussed the ideas, we picked the most interesting ones, we developed a screenplay and then they acted it out. They are the main actors of all my films. In some cases I make a rough cut of the film, I show it to them and they make comments, which I film and then add to the film in a subsequent cut.
Tell us about the time you fired a human cannonball over the border from Tijuana.
It was in '05. I called it "One Flew over the Void" ("Bala Perdida"), in reference to the film One Flew over the Cuckoo's Nest. I used the geopolitical border between USA and México as a metaphor for another boundary, the boundary between the normal and the pathological. I reached out to the CESAM, the Mental Health Center of Baja California, and worked in collaboration with the patients. The patients organized a demonstration in Playas de Tijuana. They also collaborated in a publicity campaign for TV and radio. And they symbolically accompanied the human cannonball in the flight. He fell on a net on the other side of the border, la migra was waiting.
In 2012, you shot a project about the conquest of Mexico. Can you describe the creative process for that film?
It was a film I made in the Hospital Fray Bernardino Alvarez in Mexico City with psychiatric patients. It consisted of staging the text about the conquest of Mexico that the French author Antonin Artaud wrote in the '30s. The film runs at three different times — Artaud's stay in Mexico, the conquest of Mexico itself, Artaud's piece, and the third is a fictionalization of the life of patients. They recreated life in a psychiatric ward. Like the piece that is being displayed here in San Francisco.
Do you consider your work to be political?
All work is political. Every art object has political content.
All work is political. Every art object has political content. I identify myself with a radical thought. I love what Swiss artist Paul Klee said: "Art does not reproduce the visible, it makes things visible." Making something visible is a political concept. Working with those who are excluded, and somehow, incorporating them into the public discourse.
What do you mean when you refer to "constructions of difference" in your work?
Rational thinking is built on the idea of the irrational. There is no rationality without irrationality. Without sense there is no nonsense. You cannot understand one without the other. Man builds these excluding taxonomies. To reinforce the idea of civilization, the idea of the barbaric was necessary. I am interested in breaking those boundaries that are ideological constructs.
You make a living off your art. Any advice for young artists?
It is difficult for a young artist. The most important thing is to continue working. Trying, somehow, to work outside the market system, which can be very destructive for someone young. One of the most important things for an artist is to feel that you are part of a tradition, that there is a dialogue, even with the dead. I can talk to Pasolini. It is important to know your history.
"Forbidden Games are in the Labyrinth" is being shown at the San Francisco Art Institute until December 13. For more info visit sfai.edu.
Forbidden Games are in the Labyrinth San Francisco Art Institute video art
INTERVIEW: Anitta Reflects on Past Year & Finding Joy
By Alex Ocho
Culture INTERVIEW: Jillian Mercado Discusses Humanizing the Disabled Community Through Technology 12.20.22
By Lyra Hale
Film INTERVIEW: Mariana Treviño on Working With Tom Hanks & the Collectiveness in 'A Man Called Otto' 01.12.23
By Kiko Martinez
Film INTERVIEW: Guillermo del Toro Talks 'Pinocchio', Sacrificing for Art & Inspiring Taylor Swift 12.12.22


text:
Fatal car accident claims young lives near Denver
On behalf of Patricia M. Jarzobski, P.C. posted in Wrongful Death on Wednesday, May 30, 2012.
Police are currently investigating a horrific car accident that took place a few hours outside of Denver, Colorado, last week. Purportedly, the wrecked vehicle was stolen from an owner in nearby Colorado Springs and was being driven by an 18-year-old male. The fatal car accident killed two teenagers and seriously injured three others.
Members of the Colorado state patrol were alerted to a vehicle traveling at a high rate of speed in the early hours of the morning after the driver passed state troopers on Highway 45. Eventually, the driver lost control the vehicle on making a left-hand turn. The vehicle rolled and three of the passengers were ejected.
The driver, along with a 16-year-old female passenger, was killed. Two 14-year-old girls and one 15-year-old female were seriously injured in the car accident. The injured passengers were taken to Parkview Medical Center for treatment. At this point, police are uncertain as to whether or not substance abuse was a factor in the incident.
In the wake of such a tragic incident, it's likely that the victimized families have a plethora of questions. Because the driver of the vehicle lost his life in the fatal car accident, the only hope these families may have for getting the answers they seek hinges on the outcome of the police investigation. Aside from that, it's possible those victimized will seek to pursue monetary damages for the pain and suffering they endured as a result of driver negligence. Accident victims in and near Denver, Colorado, may benefit from knowing that there are laws in place designed to protect families from financial downfall in the wake of tragedies like this one.
Source: Examiner, "Fatal crash in Pueblo County kills two teens - Denver Top News," Ken Green, May 28, 2012
Tags: Colorado, fatal car accident, stolen vehicle, wrongful death
Related Posts: Wrongful death claim possible in Colorado accident, Driver could face wrongful death lawsuit for trooper's death, Doctor may face wrongful death claim for mother and toddler, Wrongful death claim possible in DUI crash that killed 3
Car Accidents (231)
Truck Accidents (20)
Car accidents: Man injured in suspected DUI crash
Car accidents: 5 vehicle crash sends 4 to hospital
Car accidents: 2 dead, several injured after bus crash
Car accidents: Driver pleads guilty to murder
© 2019 by Patricia M. Jarzobski, P.C.. All rights reserved. Disclaimer

redpajama_set_name: RedPajamaCommonCrawl


UPM Pharmaceuticals is an independent CDMO specializing in formulation development, cGMP manufacturing, analytical methods development, stability testing and commercial manufacturing for oral solid dosage forms and semi-solid creams and ointments. The organization serves clients of all sizes, from virtual to big pharma.
UPM has grown significantly over the last five years, from 56 employees in 2013 to over 250 employees operating out of 500,000 square feet of space today. Leveraging the benefit of a long-term, multifaceted relationship, That's Nice has helped to support UPM across its marketing and business development efforts for more than seven years, collaborating with the organization's senior leadership team on a full range of research, strategy and creative efforts to continuously drive progressive growth for the family-owned CDMO.
UPM participates in several major industry events throughout the year, supported by Nice Exhibit event strategy and design. Beyond the brand's bold booth aesthetic, UPM and That's Nice work closely together to implement and maintain an ongoing and comprehensive approach that ensures the organization is optimizing opportunities and seeing a return on investment. This includes promotional pre- and post-show communications, PR and media opportunities, thought leadership platforms, lead generation and optimized awareness.
With decades of combined experience in pharmaceutical R&D and manufacturing, members of UPM's leadership team are highly regarded as thought leaders in the CDMO space. Pharma's Almanac, a research-driven content community powered by Nice Insight, the research division of That's Nice, provides pharmaceutical industry leaders with a platform to convey their subject matter expertise, while generating qualified leads based on who is viewing their published content. UPM has published on a range of topics, from quality compliance to formulation innovation and controlled substance manufacturing, garnering hundreds of views by pharmaceutical outsourcing professionals across major markets and a strong source of targeted lead generation.
UPM's published content has garnered hundreds of views by pharmaceutical outsourcing professionals across major markets.
A brand's website can be its biggest marketing asset, though companies often neglect to appreciate its full value. Using a well-established marketing automation system, UPM's website serves as a lead generation tool, providing insight into visitor behavior and a structure for lead conversion. With the support of That's Nice, UPM consistently develops, promotes and publishes new content through the website, with articles available through a gated access, which means visitors must provide some degree of trackable contact information in order to view and download content, transforming visits into leads.
With the support of Nice Insight's PTE/FTE market research and lead generation, UPM has identified a number of strategic growth opportunities around both new and existing business. Beginning with an inquiry, Nice Insight's market research team provides a wealth of information to support business development, from lead generation opportunities to content strategy.


text:
World news – June 23 2018
Aid agency SOS Children's Villages UK response to treatment of children at US-Mexico border
SOS Children's Villages UK CEO Alison Wallace said: "There is no place in any progressive society for the forcible separation of parent and child. The scenes we have witnessed over the past days on the US-Mexico border are unbearable snapshots of an inhumane and punishing immigration policy. There can be no justification, ever, for risking children's physical and mental wellbeing in exchange for political gain.
"SOS Children's Villages' work focusses on caring for children without parental support, and when we meet children who have been forcibly removed from parental care - whether through war, migration or persecution – they are often irreparably damaged by deep trauma.
"We welcome President Trump's reversal of his original devastating policy but it was cynically created to serve politics not people, and as such always ignored its inherent potential to harm the most vulnerable members of its target group.
"Those who spoke out in their thousands against the violation of children's rights - from across the political spectrum - have shown President Trump and the international community that we will not allow cruelty to children to be a by-product of political expediency."
For media enquiries please contact Lucy Prioli at Lucy.Prioli@sosuk.org or on 01223 222 974.
Sign up for our newsletter and receive the latest updates about our global projects, latest fundraising news and campaigns.
We will never trade or sell your information. To unsubscribe click the unsubscribe button at the bottom of your email or just let us know at hello@sosuk.org and we'll take you off the subscriber list. To find out how we use your data, read our full Privacy Policy.


Royal Enfield Classic Stealth Black on-road price in Gurgaon - ₹ 2,17,764 onwards. It is available in 2 versions and 1 colour. Classic Stealth Black is sold by 4 dealerships in Gurgaon. All the colour options and versions of Classic Stealth Black might not be available at all the dealerships in Gurgaon.
Get in touch with your nearest Royal Enfield service center in Gurgaon for service enquiries, service and repair costs and more!
You are currently viewing price of Royal Enfield Classic Stealth Black in Gurgaon. Not from Gurgaon? Select your city to see price in your city.


My name is Garet and I have had to overcome a lot of things in life.
For example, when I was 6 years old I underwent back surgery to help remedy my spina bifida.
But I have a strong faith, loving family, and compassionate friends. My own heart is as big as a house.
Ryan Collins has been like a Brother to me, and I also consider his parents to be a 2nd mom and dad.
OK that's enough about me!


Void Flyer (Su) A sarcesian can go 1 hour without breathing and can exist in a vacuum without suffering the associated environmental e ects. By spending 1 Resolve Point, a sarcesian can extend this duration to a number of hours equal to her CR, or she can double that by spending 2 Resolve Points. When in a vacuum, sarcesians automatically grow wings made from pure energy that grant them a supernatural fly speed of 120 feet (average maneuverability) but that work only in a vacuum.


Your main guidelines when you're working with different partners. The graphic charter provides all the necessary tools to ensure a perfect consistency between your business and your strategic positioning. It includes typography, color codes, usages and many other guides more than useful for any usages both internally and externally.


text:
How to profit from domain names
September 23, 2008 in Domains | Tags: Domains
MSM takes a look at the domain for sale .biz In an article titled How to profit from domain names the London Times looks at a couple examples of quick domain flips for profit.
"Two years ago Mark Richards noticed that no one had registered the .com and .co.uk addresses for "latelicence" and "latelicense". He bought the domains for £20 and sold all four as a package one month later for £6,000.
"And Peter Savage whose best return was http://www.fastpizzapizza.com. He registered the address for $20 after watching the success of Pizza.com, which sold in April for a record $2.6 million. A month later he sold his address on Sedo.co.uk, the website marketplace, for $5,600, earning him £3,122.
« Update on auctioning .info sunrise domain names
Moniker live domain auction sells ~$3,000,000 of domain names »


text:
Q: Player not moving I'm coding a game in pygame while learning OOP and the pygame.sprite.Sprite class, along with the attributes rect comes with, but for some reason, after coding this very simple code, the player doesn't move to the right, it only moves to the left, and without collision detection code, it stops moving once it's coordinates are equal to width.
Why is that?
import pygame
import random

width = 800
height = 600

black = (0, 0, 0)
green = (0, 255, 0)
blue = (0, 0, 255)
red = (255, 0, 0)
white = (255, 255, 255)

pygame.init()
pygame.mixer.init()
screen = pygame.display.set_mode((width, height))
pygame.display.set_caption("Shmup")
clock = pygame.time.Clock()

class Player(pygame.sprite.Sprite):
    def __init__(self):
        pygame.sprite.Sprite.__init__(self)
        self.image = pygame.Surface((50, 40))
        self.image.fill(green)
        self.rect = self.image.get_rect()
        self.rect.centerx = width/2
        self.rect.bottom = height - 10
        self.x_change = 0
        self.speedx = 0.2

    def change_left(self):
        self.x_change += -self.speedx

    def change_right(self):
        self.x_change += self.speedx

    def move_left(self):
        self.rect.x += self.x_change

    def move_right(self):
        self.rect.x += self.x_change

    def stop_move_left(self):
        self.x_change = 0

    def stop_move_right(self):
        self.x_change = 0

player = Player()
all_sprites = pygame.sprite.Group()
all_sprites.add(player)

exitGame = False
while not exitGame:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            exitGame = True
        if event.type == pygame.KEYDOWN:
            if event.key == pygame.K_a:
                player.change_left()
            if event.key == pygame.K_d:
                player.change_right()
        if event.type == pygame.KEYUP:
            if event.key == pygame.K_a:
                player.stop_move_left()
            if event.key == pygame.K_d:
                player.stop_move_right()

    player.move_left()
    player.move_right()
    screen.fill(white)
    all_sprites.draw(screen)
    pygame.display.update()

pygame.quit()
quit()

A: The problem with your code is that you are only incrementing or decrementing player.rect.x by 0.2, and pygame stores positions as integers. If the player.rect.x value is set to 280 and is decremented by 0.2, it should become 279.8, but it actually becomes 279, because of the conversion to an integer! The value does indeed change when moving left, but when moving right, the value is incremented by 0.2, but this just gets rounded back down to the original value.
Note: because of how the int() function works in Python, -0.2 gets rounded to 0, so that is why the block cannot move past the window boundary, even though there isn't any explicit "collision detection" code being executed.
The quick and easy solution to this problem is to change the self.speedx value to 1 or greater:
self.speedx = 1

I hope this answer helped you, and if you have any further questions relating to this answer, please feel free to leave a comment below!

A: If you actually want to use floating point numbers for the movement, you need to store the actual position as a separate attribute, change it first when the sprite moves and afterwards update the rect position as well (because the self.rect is used as the blit position).
I recommend storing the actual position and the velocity as pygame.math.Vector2 objects. Then you can just add the velocity to the position vector and set the rect.center (or one of the other rect attributes) to the new position.
import pygame
from pygame.math import Vector2

width = 800
height = 600

green = (0, 255, 0)
white = (255, 255, 255)

pygame.init()
screen = pygame.display.set_mode((width, height))
clock = pygame.time.Clock()

class Player(pygame.sprite.Sprite):
    def __init__(self):
        pygame.sprite.Sprite.__init__(self)
        self.image = pygame.Surface((50, 40))
        self.image.fill(green)
        self.rect = self.image.get_rect(midbottom=(width/2, height-10))
        self.pos = Vector2(self.rect.center)
        self.velocity = Vector2(0, 0)

    def update(self):
        self.pos += self.velocity
        self.rect.center = self.pos

player = Player()
all_sprites = pygame.sprite.Group(player)

exitGame = False
while not exitGame:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            exitGame = True
        elif event.type == pygame.KEYDOWN:
            if event.key == pygame.K_a:
                player.velocity.x = -0.8
            elif event.key == pygame.K_d:
                player.velocity.x = 0.8
        elif event.type == pygame.KEYUP:
            if event.key == pygame.K_a:
                player.velocity.x = 0
            elif event.key == pygame.K_d:
                player.velocity.x = 0

    all_sprites.update()  # Calls the `update` methods of the sprites.

    screen.fill(white)
    all_sprites.draw(screen)
    pygame.display.update()
    clock.tick(60)

pygame.quit()

redpajama_set_name: RedPajamaStackExchange


text:
Kaitlyn McInnis
Kaitlyn McInnis is a freelance travel and lifestyle writer with bylines in Condé Nast Traveler, Tatler Asia, CNN, The Points Guy, and more. Born and raised in Canada, she has been writing Canadian and international travel guides for Travel + Leisure for more than three years.
• Received a bachelor's degree in English literature and Irish studies
• Currently working toward a master's degree from the University of King's College in Nova Scotia, where she is working on a travel-focused nonfiction manuscript
About Travel + Leisure
Travel + Leisure, a Dotdash Meredith Brand, is a top travel media brand with a mission to inform and inspire passionate travelers. Our expert team includes a network of hundreds of writers and photographers across the globe, all providing a local eye on the best places to stay, eat, and explore. We reach an audience that takes 76 million round trips annually, offering valuable travel tips, ideas and inspiration, and products you need to get you to your destination — whether it's a small town or big city, beach or lake, national park or theme park, road trip, cruise, or long-haul flight, and everything in between. Learn more about us and our editorial process
Latest from Kaitlyn McInnis
The 10 Best Dress Shoes for Women of 2023
The 8 Best Lighted Makeup Mirrors for Travel of 2023
The Best Wool Clothing and Accessories of 2023
The 12 Best Women's Loafers of 2023
The 15 Best Travel Jewelry Cases of 2023
The 10 Best Portable Ice Makers of 2023
The 6 Best Rechargeable Hand Warmers of 2023, Tested and Reviewed
The 15 Best Travel Accessories for Women of 2023
The 11 Best Rain Boots for Women of 2023
The 12 Best Travel Safety Products of 2023
The 10 Best Compression Socks for Women of 2023
The 7 Best Travel Steamers of 2023
The 16 Most Comfortable Flip-flops for Men and Women of 2023
The 10 Best Portable Heated Blankets of 2023
The 13 Best Sunglasses for Travel of 2023
The 14 Best Luggage Tags of 2023
The 5 Best Kindle Devices for Reading on the Go in 2023
The Best White Sneakers for Travel
The 51 Best Travel-related Gifts Under $10 of 2023
The 13 Best Joggers to Add to Your Travel Wardrobe in 2023
The Best Moisturizing Skin Care Products for Travel
The 55 Best Gift Cards for Travel Lovers in 2023
The 8 Best Ice Scrapers of 2023
The 10 Best Travel Clothes and Accessories for Women of 2023


I believe that the purpose of it all is to find our way back to the essence of universal consciousness from different, unique subjective pathways. With a mind set of calmness, clarity (free of bias and limiting beliefs), gratitude and respect we open the channels and optimize awareness (consciousness) of self and of the greater environment. In so doing we re-connect with eternal source consciousness and thereby enhance it.


Geniusis an action-packed crime thriller from the makers of Gadar: Ek Prem Katha. The producer-director Anil Sharma is back with the story of another patriot after 11 years of making the thrilling love story Gadar: Ek Prem Katha. Also, this movie is a debut film of Utkarsh Sharma son of Anil Sharma.
Vasudev Shastri a.k.a. Genius (Utkarsh Sharma) is an engineering graduate looking for a job opportunity. A patriot by heart and engineer by the mind, he ends up joining Special Forces where he gets an opportunity to serve the nation. And he plays a crucial role in fighting with a dreaded villain Gangster Shah played by Nawazuddin Siddiqui.
Happy Phirr Bhag Jayegi is a sequel to 2016 movie Happy Bhag Jayegi in which Diana Penty played the lead role of Happy. The sequel starts with Harpreet Kaur a.k.a. Happy (Diana Penty) of the prequel. It is an extension of the Happy Bhag Jayegi movie.
Happy (Diana Penty) is on the run again but this time another Happy (Sonakshi Sinha) is caught due to the similarity in names. She's kidnapped and taken to China. During interrogation, she reveals her true identity but no-one is ready to believe on her story. The movie is like comedy-of-errors.
As a director, Mudassar Aziz feels it is more challenging to extend a story without changing the theme. He compares the story with other sequels where the characters remain same but the story is changed or characters changes but the story is kept same.
The Wife: How far could a wife go to help her husband?
A prolific writer is expecting Nobel Prize for his work and his dream comes true when he receives a phone call from the Nobel folks. Happy to receive the highest honor, he wants to share the news with everyone. But what he doesn't know that his wife has already got the news on the other line of the phone. It is the story of a wife who sacrificed her writing career to become caretaker of his husband.
Professor Joe Castleman is happy to get nominated for Nobel Prize for his novel. He received the news after spending a sleepless night. He was confident but a little bit worried about the Nobel community recognizing his work. The call regarding his selection for the coveted price comes as an early morning surprise and Joe becomes speechless after receiving the information.


Last Sunday, the reading from Mark's gospel brought us stories of Jesus healing one sick person after another, beginning at the local synagogue where he healed a possessed person, and continuing with Simon Peter's mother-in law who was in bed with a fever. Before long, if you remember, he had the whole town at his door, clamoring for his healing touch.
Today we have yet another healing, the healing of a leper, but this healing takes us into tricky territory because it involves more than healing: it also involves breaking the Law. The leper broke the Law and so did Jesus. Let me explain.
The Law, when it came to lepers, was clear. We heard it in today's reading from the Book of Leviticus. The Law said that lepers were to dwell apart, to remain as far as possible away from people and, if people should chance to come near, the leper was to make his whereabouts known by shouting "unclean, unclean!" There was good reason for this. Leprosy was thought to be highly contagious; people lived in dread of any sort of contact with lepers. So the leper in today's gospel story clearly broke the Law. By coming out of the shadows, walking right up to Jesus, kneeling down before him, and even daring to engage him in conversation, he broke the Law. And you can imagine how people must have reacted. Their reaction must have ranged from horror to indignation to fear for their lives - fear that they would now catch the dread disease.
But the leper was not the only one to break the Law in this story. Jesus - in allowing the leper to approach him, and then entering into a conversation with him, and doing the unthinkable by actually reaching out and touching him - Jesus also broke the law.
As so often in the gospels, Jesus, a devout Jew who revered the Law, simply refused to be bound or straight-jacketed by the Law. Everything he does in this encounter makes it clear that people are more important to him than laws, no matter how sacred. So, moved with pity, he allows the leper to come right up to him; he converses with him and goes even further by touching him. Jesus knew that this poor outcast of a man who had been living on the outer margins of society needed more than just physical healing - that he also needed a human encounter, human contact, human warmth. That's why he did the truly unthinkable - reaching out and touching him. In doing so, he not only gave him healing, he also gave him love and acceptance. He welcomed him back to the community.
I can't help but think of Pope Francis and his efforts to reach out to people in marriages not recognized by the Church for whatever reason. He knows that many of them feel like lepers and outcasts, and so he's intent on doing everything he can to reach out to them, to assure them that they are loved, part of the family, part of the community.
But none of this is about flaunting the Law. Jesus may bend the Law but he also honors it – just as Pope Francis honors Church teaching. Jesus honors the Law when he tells the man to go and show himself to the priest and to make the customary offering called for in the Law of Moses. Curiously, he also tells him to keep quiet about the healing. That's most likely because he doesn't want people to get a skewed, one-sided notion of what it means for him to be the Messiah.
But the healed leper does not keep quiet. Once he is healed, he can't contain himself. He immediately goes about spreading the story. And that's where this story gets a fascinating little twist to it. The more the news gets out that Jesus has healed a leper, the more he is swarmed wherever he goes. Everyone wants a piece of him, everyone wants his healing touch - so much so that, as Mark puts it, it becomes "impossible for Jesus to enter a town openly" and he has to remain outside the towns in deserted places. Do you see the twist here, the irony? Jesus is now the one who has to dwell apart from the community. In a sense, Jesus has become the leper!
As with the leper, Jesus puts us and our needs first, puts them before any laws or rules. He is here now to welcome us as he did the leper. His Word is full of power for our healing; his Eucharist is full of nourishment and strength for our journey. Like the leper, let us approach him with confidence and feel his healing touch!


text:
\section{Background and Motivation}

A frequently occurring classification problem consists in identifying items that pass a set of \textit{screening tests} (filters).
This is not only common in medical diagnosis but in many other fields as well, from database querying - where we filter tuples based on predicates \cite{optimalWidom2014}, to hotel search - where we filter places based on features of interest \cite{Lan_dynamicfilter_hcomp17}, to systematic literature reviews (SLR) - where we screen candidate papers based on a set of exclusion criteria to assess whether they are in scope for the review~\cite{Wallace2017crowdMLshort}.
The goal of this paper is to understand how, given a set of trained classifiers whose accuracy may or may not be known for the problem at hand (for a specific query predicate, hotel feature, or paper topic), we can combine machine learning (ML) and human (H) classifiers to create a hybrid classifier that screens items efficiently in terms of cost of querying the crowd, while ensuring an accuracy that is acceptable for the given problem.
We focus specifically on the common scenario of \textit{finite pool} problems, where the set of items to screen is limited and where therefore it may not be cost-effective to collect sufficient data to train accurate classifiers for each specific case.
To make the paper easier to read and the problem concrete, we will often take the example of SLRs mentioned above, which is rather challenging in that each SLR is different and each filtering predicate (called \textit{exclusion criterion} in that context) could be unique to each SLR (e.g., "exclude papers that do not study adults 85+ years old").

The area of crowd-only and of hybrid (ML+H) classification has received a lot of attention in the literature.
Research in crowdsourcing has identified how to best classify items given crowd votes~\cite{LiuWang_truelabel,Dong2013,whitehill2009whose}, including  work that focuses specifically on the \textit{screening} problem discussed here ~\cite{optimalWidom2014,Wallace2017crowdMLshort,Mortensen2016crowd,Krivosheev_hcomp17,Lan_dynamicfilter_hcomp17}.
Interesting ways to combine ML and H classifiers have also been proposed, for example by building ML classifiers that can also include crowd votes as features, whose purpose is not only that of providing better classification but also of weighing the value (vs the cost) of obtaining additional crowd votes~\cite{Kamar_2012_combining}.
Other hybrid approaches ask the crowd to extract interesting patterns to be then fed to algorithms for classification, as opposed to relying on ML to do this~\cite{flock_2015,Carlos_pattern}.

The approach we propose leverages the information provided by each kind of classifier (machine and human) to improve the effectiveness of the other kind so that they can be stronger together.
The algorithm is based on a probabilistic model that adapts to the specific characteristics of each item, screening filter,  workers' accuracy, and ML classifier available to identify \textit{what} to ask next for each item (which filter to test) and \textit{if} we should stop polling the crowd for a given item, either because we reached a decision or because we realize we cannot do so cheaply and confidently (that is, the problem of classifying that item is too hard for the crowd-machine ensemble).
The latter point is particularly important: there may be items, filters, or classification problems in general where it is questionable whether machines and/or the crowd can classify with acceptable accuracy.
If the algorithm can identify this subset of items and filters - or (item,filter) pairs - early and avoid spending money on them, then we know we can confidently try a crowd or hybrid approach as it will not waste money.

In essence, the approach - which we call \textit{hybrid shortest run} (HSR) -  works by considering the output of machine classifiers as a prior to the class probability for each item.
This information is combined with an adaptive crowdsourcing algorithm that refines the class probability by polling the crowd on the (item,filter) pair that makes it more likely to reach a decision cheaply.
In turn, classification information that is progressively obtained as we poll the crowd can be used to re-assess the performance of ML classifiers as well as create a model over the available classifiers for improved accuracy.

While many adaptive algorithms have been proposed (e.g., \cite{kamar_crowdonly_2013,POMDP_2013,Nushi2015CrowdAP}),  to the best of our knowledge this paper is the first to discuss hybrid classification in screening contexts, which has unique requirements and opportunities since, as we will see, the heart of the problem lies in identifying the filters (e.g., the exclusion criteria in SLR) that i) are most selective (screen out a large proportion of papers) and ii) that crowd or machines can classify accurately (can correctly determine if the exclusion criterion applies to a paper). If we can do so, we can focus on getting crowd votes for these filters first, leaving a smaller set of items for the filters the crowd finds hard to classify correctly or that do not have high selectivity.

We believe this paper is also the first to take a "black box" approach to  hybrid classification: since we tackle finite pool problems, we do not assume we can develop an accurate, dedicated classifiers for each classification problem, although we may have available (possibly weak) classifiers from similar problems - in our SLR example these may be neural nets developed for other SLRs in the same or different fields or for other exclusion criteria.
Rather, we identify how to leverage classifiers we are given and for which the accuracy for our task is unknown.
We do this by a combination of (minimal) testing to filter out very poor classifiers  and by combining the remaining ones to set a prior probability for whether a filter applies to an item. This helps us determine the most promising (item,filter) pairs for which asking a vote to the crowd has the highest probability of reaching a decision cheaply and confidently.

While it is not surprising that hybrid approaches have the potential to outperform human- or machine-only methods, the contribution we provide here lies in showing how we can cope with finite pools problems where potentially weak classifiers - and  even classifiers with unknown accuracy for the problem at hand - can be leveraged effectively, even in contexts characterized by very demanding requirements in terms of accuracy (as is the case for SLRs) and in the presence of difficult problems where also the crowd has low accuracy.
HSR always estimates the characteristics of the problem and the classifiers, and identifies which are the items and filters for which it can draw accurate conclusions cheaply, leaving the rest to expert classification.

We also study how the effect of incorporating classifiers (and ensembles of classifiers) into a crowdsourcing algorithm varies based on the different characteristics of the problem at hand (such as different selectivity of filters, accuracy of classifiers, correlation among base classifiers, amount of data available for testing classifiers and for training ensembles) and show why and under which conditions \textit{ensembles} of classifiers do or do not provide benefits over "simply" using the best available classifier at hand.

\input{sections/related}

\section{Model, Assumptions and Problem Statement}
We next define the problem we aim at solving and the model, which captures a few broad assumptions we make.

Specifically, we assume to have in input a set $(I,F,C,L)$ where $I$ is the set of items to classify (in our SLR example, these are papers to screen), $F$ denotes the filters (paper exclusion criteria), $C$ represents the set of ML or H classifiers, and $L=k*FE + FI$ is a loss function, modeled as a linear combination of false exclusions FE and false inclusions FI, which may carry a different relative weight $k$.
The reason for the weight $k$ is that in many contexts the two kinds of errors have very different implications.
This is for example the case when screening is performed to reduce the number of cases to be brought to human attention, such as potential credit card fraud, tweets/posts potentially linked to criminal activities, or SLRs: screening out an item is often more  ``costly" than erroneously bringing the item to human attention.

The (possibly empty) set of given ML classifiers is assumed to be trained to receive an (item, filter) pair and output whether the filter applies to the item (e.g., receive a paper and an exclusion criterion and assess if the criterion applies to the paper).
As mentioned, we do not discuss how to obtain classifiers (some examples are provided in the experiments section), as this is the subject of ample literature and it anyways depends on the problem domain.
We do  \textit{not} necessarily assume that the classifiers are trained or even tested in the specific problem at hand (that is, for the specific filters and items under consideration),  and therefore in general we do not even know their accuracy.

For human classifiers, we assume we have a set of generic workers of different and initially unknown accuracy and a set of high-accuracy but expensive experts.

Each classifier $c = \{ cost, a(f), \rho(f,C) \} \in C$ is associated with the cost of asking one vote on an (item, filter) pair, with a filter-specific estimated accuracy (a 2x2 confusion matrix capturing probability of correct decisions for positive and negative labels), and with its \textit{error correlation} $\rho$ with other classifiers.
The error correlation among two classifiers is the probability that both make an error given that one of them makes an error.
Because we have no knowledge on the classifiers' accuracy, we conservatively model it as a uniform $Beta(1,1)$ distribution for both positively and negatively labeled items (which means we do not even assume classifiers are better than random). If we do have additional information, this can be incorporated in the  $Beta$.
Consistently with crowdsourcing literature, we also assume that crowd workers opinions are independent (that is, they make independent errors, given an item, filter and true label).

Each filter $f$ is characterized by  \textit{difficulty} $d_f$ and \textit{power} $\theta_f$.
Difficulty reflects how easy it is for crowd workers to classify items correctly on that filter.
Following Whitehill \cite{whitehill2009whose}, we model difficulty as a  real number $d_f \in [0,+\infty)$ that, given an expected accuracy $\alpha_w$ of a worker $w$, skews the accuracy as follows:
\begin{equation}
\alpha_{f,w} = 0.5+ (\alpha_w-0.5) \cdot e^{-d_f}
\end{equation}
As the difficulty $d_f$ grows, $\alpha_{f,w}$ goes to 0.5, corresponding to random selection, which we consider to be the lowest accuracy level.
The power $\theta_f$is simply the proportion of items to which  filter $f$ applies. Notice that we do not assume in this paper that power affects difficulty and we also do not assume different accuracies based on the true label of the items.
Both difficulty and power are unknown and we assume no prior knowledge on them.

Given this model and assumptions, our goal is to identify an algorithm that can  efficiently (in terms of cost) query the classifiers available and aggregate results while achieving the quality goals, stated in terms of loss as well as of the classical precision and recall measures.

\section{Baseline Algorithms}
We next describe baseline crowd and machine classification algorithms on which we base the construction of the hybrid approach.
The crowd-based one is  \textit{shortest run} (SR)~\cite{Krivosheev_www18}, a recently developed algorithm that has been shown to perform well for multi-filter screening. We summarize SR here to make the paper self-contained. There are other reasons why SR is particularly suited for the hybrid approach we propose, and we get back to this in the following.

\subsection{Recap of Shortest Run}
The Short Adaptive  Multi-Run  algorithm (Shortest run, SR for short) identifies and continuously updates an individual strategy for each item to be screened by identifying the shortest path to decision.
In a nutshell, the idea borrows concepts from Partially Observable Markov Decision Processes (similarly to \cite{kamar_crowdonly_2013,POMDP_2013}), by assessing based on the current state (that is, the knowledge accumulated thus far at the \textit{global} level - power and difficulty  for each filter - and at the \textit{local} level - the votes on each specific item) which are the (item, filter) pairs to submit to the crowd for testing because they would be more likely to quickly (cheaply) lead to an accurate decision.  SR may also decide to quit trying the crowd classification approach on an item if it believes that it would be too expensive to reach an accurate decision.

SR proceeds by performing what authors call a \textit{baseline run} where a small set of items (usually 50) is screened with the standard approach of asking for labels on all filters (typically, 5 labels per item and filter) and classifying using a variant of Expectation Maximization.
The result of the run is information on the power $\theta_f$ and difficulty $d_f$ (and, correspondingly, workers' accuracy $\alpha_f$) for each filter.

This information is used to obtain a prior estimate on the probability that the \textit{next} vote for each (item i ,filter f) will be a vote to screen out the item (that is, a vote that the filter applies to the item). We get an out vote if the item is out (with probability $\theta_f$) and the worker answers correctly, or if the item is in (with probability $1-\theta_f$ and the worker answers incorrectly:

\begin{equation}
P(v_{i,f}=OUT) = \alpha_f \cdot \theta_f + (1- \alpha_f)(1-\theta_f)
\end{equation}

In addition, by applying Bayes, we know the probability of a filter $f$ screening (i.e., $P(i \in OUT_f)$) or  not screening ($P(i \in IN_f)$) an item $i$, once we obtain a label for the $(i,f)$ pair.

\begin{equation}\label{eq:bayespin}
\begin{split}
P(i \in IN_f | v_{i,f})= \frac{P(v_{i,f} | i \in IN_f ) * (1-\theta_f)} {P(v_{i,f})}
\end{split}
\end{equation}

Because an item is screened out if at least one filter applies, then:

\begin{equation}
	\begin{aligned}
		P(i \in OUT) = 1 - \prod_{f \in F} P(i \in IN_f)
	\end{aligned}
	\label{formula:p-out}
\end{equation}

An item is classified as out if $P(i \in OUT)$ is greater than a threshold $\overline{P_{out}}$.
These formulas allow SR to estimate both what the next vote can be for a pair $(i,f)$ and the impact  each vote has on  $P(i \in OUT)$.
These estimates are updated as votes come in, where in Formula \ref{eq:bayespin} the class probability, initially derived only from filter power and therefore equal for all unclassified items, is updated based on the votes obtained thus far for that item.

The above formulas can be easily extended to compute the probability that the next $n$ votes for an $(i,f)$ pair will be \textit{in} or  \textit{out} votes.
SR can in particular estimate the minimal number of votes $N^{min}_{i,f}$ in one direction (in or out) it needs to reach a decision, and the probability $P^{min}_{i,f}$ of getting such votes.
As $N^{min}_{i,f}$ grows and its probability shrinks, SR may decide that crowd classification cannot be done efficiently, and quits trying (the stopping criteria are based on threshold which can be set as discussed in \cite{Krivosheev_www18}).
Intuitively, items that are left unclassified are essentially not filtered out, so they contribute to a higher loss, which has to be weighted against the price one would incur by insisting with the crowd. This in general is part of the same trade-off of how much users are willing to spend for unit of loss.

\section{Hybrid SR algorithm}

\begin{figure}[htb]
  \centering
  \begin{minipage}{.7\linewidth}

\begin{algorithm}[H]
\nl \KwIn{Input: $I$, $C$, $F$, $T$, $\overline{P_{out}},\overline{P_{in}}$, $Budget$}
\nl \KwOut{$CI=\{CI_{in},CI_{out} \}$ \textit{\#classified items}}
\nl $CLF \leftarrow$ \textit{Select classifiers based on T tests}\\
\nl $C_{meta} \leftarrow$ \textit{Ensemble CLF}\\
\nl $\textbf{\textit{Prior}},  \hat{\theta^0} \leftarrow$ \textit{run ensemble on I}\\
\nl $CI \leftarrow \{\}$, $UI \leftarrow I$,$k \leftarrow 0$\\
\nl $Votes^0, \hat{\alpha^0} \leftarrow$ \textit{\textbf{Baseline run }}\\
\nl \textit{\#SRuns iterations}\\
\nl \While {$UI \not= \varnothing$ \textbf{and} $Budget \not= \varnothing$} {
\nl 	$k \leftarrow k + 1$\\
\nl     \For{$i \in UI$}{
\nl     	$f(i) \leftarrow $ \textit{assign filter on i | \textbf{Prior}, Votes}\\
\nl         \textit{do check stop condition on i | \textbf{Prior}, Votes}\\  }
\nl    $I^k \leftarrow$ \textit{items with highest prob of classification}\\
\nl    \For{$i \in I^k$} {
\nl    	$v_{i,f}^{k} \leftarrow$ \textit{collect a crowd vote on $f(i)$}\\
\nl		$Votes \leftarrow Votes \cup \{v_{i,f}^{k}\}$\\
\nl        \If{$P(i\in IN|Votes^{k}_{i}, \textbf{\textit{Prior}}) > \overline{P_{in}}$} {
\nl        	$CI_{in} \leftarrow CI_{in} \cup \{i \}$\\
\nl            $UI \leftarrow UI - \{i \}$\\
        }
\nl        \If{$P(i\in OUT|Votes^{k}_{i},\textbf{\textit{Prior}}) > \overline{P_{out}}$}{
\nl        	$CI_{out} \leftarrow CI_{out} \cup \{i \}$\\
\nl         $UI \leftarrow UI - \{i \}$\\
        }
    }
\nl    $\hat{\theta^k} \leftarrow$ update power\\
 }
\nl $CI^{difficult\_items} \leftarrow$ label $UI$ as \textit{"IN items"}\\
\nl \textbf{return} $CI, CI^{difficult\_items}$

\caption{\textbf{Hybrid SR Algorithm}}
	\hrulefill
	\label{alg:hsm_runs}

\end{algorithm}
  \end{minipage}
\end{figure}

While there is ample literature on how to generate a ``good" ensemble of classifiers \cite{Rokach2010Ensemble,Dietterich_ensemble}, the prior art on the general problem of combining an existing set of ML classifiers $C =\{c\}$ is relatively less abundant \cite{Dzeroski2004Stacking}.
We base the hybrid machine-crowd classification strategy on modifying SR.
In the following, we first motivate why we start from SR and then introduce hybrid shortest run (HSR) by presenting, at each step, first the intuition and then the formalization.

The reason for starting from SR is that i) it was designed for multi-filter screening and has shown to perform better than baseline algorithms for crowd multi-predicate classification, ii) it has a per paper and per item probabilistic model that can leverage prior knowledge on items and filters, and ML classifiers can provide such knowledge, and iii) the algorithm can adapt to work with different test items $T$ of different sizes.
This is important, as test items can help us filter out ML classifiers with an expected accuracy lower than a threshold $\overline{a}$ (to be tuned as discussed later), and the more extensive set of crowd-classified items from the baseline can be used to a) assess independence among ML classifiers (which is necessary if we want to pool votes from ML classifiers with simple algorithms such as majority voting or Naive Bayes), and b) build an ensemble model out of the ML classifiers where the output of each ML classifier is a feature.
This is similar to \textit{stacking}~\cite{Dzeroski2004Stacking} although we do not apply it over the training data used to build the individual classifiers (which we do not assume to have), but rather on the labels as estimated by the hybrid classification.

The Hybrid SR pseudo code is presented by Algorithm \ref{alg:hsm_runs}. The first step consists in filtering out classifiers that do not perform well for the case at hand. Specifically, we want to retain a classifier if we are confident it has an accuracy that is better than random.
In principle, every contribution that is better than random helps, especially if it is independent from the other classifiers and if we weigh the vote by the classifier accuracy.

As commonly done in crowdsourcing, we assume we have access to a high accuracy test dataset $T$ (obtained at a cost $C_T=ec*\left\vert{T}\right\vert$, that increases linearly by a factor $ec$ with the number of test examples, where $ec$ represents the cost for an ideal, expert screening).
We then assign a prior probability distribution to each ML classifier and for each filter (ML classifiers may perform differently for different filters), reflecting our belief of how well the classifier performs on the problem at hand.
As mentioned, in absence of additional information, we assume a
$Beta(1,1)$ uniform prior for a given filter.
Other choices of prior are possible, assuming more likely that classifiers will have an accuracy above 0.5 and that accuracies close to 0 or 1 are very unlikely (for example, a Beta (3,2) has such characteristics). If we have a non negligible number of test cases the impact of this choice on the performance of the algorithm is relatively small.
In the following for simplicity we assume a Beta (1,1) prior for all filters.

ML classifiers are tested with the gold dataset $T$ (line 3), resulting in posterior probability $Beta(1+correct\_answers, 1+failed\_answers)$ which has a known distribution.
We can, in particular, retain classifiers whose probability of being better than random is greater than a selection threshold $sc$, and this can be easily computed from the Beta distribution.
Then we build an ensemble of ML classifiers and run it on a whole unclassified pool of items $UI$ (lines 4,5).

Consistently with SR, we then perform a baseline run on $B$ items (on all filters), both to estimate crowd accuracy on each filters and to get data for the next step (line 7).

This information is used to inform a prior $p^{mc}_{i,f}$ for each item and filter in Equation~\ref{eq:bayespin}.
In other words, while SR takes as prior for each filter $f$ the proportion $1-\theta_f$ of items classified as ``in" for $f$ and computes an overall prior, here we use the baseline ensemble output for each prior $p^{mc}_{i,f}$, where the class probability is the confidence that the filter applies.
Equation~\ref{eq:bayespin} therefore becomes

\begin{equation}\label{eq:bayespin_ml}
\begin{split}
P(i \in IN_f | v_{i,f})= \frac{P(v_{i,f} | i \in IN_f ) * (1-p^{mc}_{i,f})} {P(v_{i,f})}
\end{split}
\end{equation}

Notice that the prior has three effects: 1) it concurs to determine the classification probability; 2) it also affects which (item, filter) pairs we pick next for obtaining the crowd vote, since Equation~\ref{eq:bayespin} also affects $N^{min}_{i,f}$; 3) it has an impact on the stop condition (line 13), i. e., whether to stop to iterate over an item due to its classification difficulty or to continue. Once the $(i,f)$ pairs to query next have determined, HSR proceeds to obtain the vote and iterates as per the SR algorithm (line 16).

We now present a series of experiments that, besides assessing the validity of HSR, help us understand its robustness as  parameters of the problem and of the algorithm change. Besides comparing it with crowd-only classification, we also compare with ML-only classification. For this baseline comparison we leverage a Naive Bayes classifier, where machines are considered to be independent and where their votes are weighted by the estimated classifiers accuracy $A_c = \{a_c\}$ (by means of tests $T$). The ensemble is therefore defined as follows:

\begin{equation}\label{eq:ensemble_naive_bayes}
\begin{split}
Class(l \in \{0,1\}) \propto \prod_{c \in C} a^{\delta(l, l_c)}_c \cdot (1-a_c)^{1-\delta(l, l_c)}
\end{split}
\end{equation}

where $	\delta(l,l_c) = \begin{cases} 1 & l = l_c\\0 & l \neq l_c\end{cases}$ is the Kronecker delta function\footnote{https://en.wikipedia.org/wiki/Kronecker\_delta}, and $l_c$ is a label from a classifier $c$.

\section{Analysis and Experiments}
The approach to analysis is based on both simulations and crowdsourcing experiments. The simulations are interesting because they allow us to understand the behavior of the approach under very different conditions, and assess the impact of the variation of each parameter, be it a parameter that describes the nature of the problem (such as the filters' power and difficulty) or a parameter of the algorithm, such as the classification threshold $\overline{P_{out}}$.

The crowdsourcing experiments allow us to get actual values of parameters for real scenarios and assess validity of results in that context - besides understanding a set of nuances important in the setup of crowdsourcing tasks.
We also leverage the data to build classifiers using commonly available techniques and use their accuracies to get a feel for the kind of accuracy we can achieve for the problem at hand.

\textit{Metrics.}
In the experiments there are three metrics we want to assess:
\begin{enumerate}
\item The \textit{loss}, computed as defined earlier, that quantifies our error weighted by how ``severe" false exclusions are considered (we can similarly use F$_\beta$ for this, though we choose loss in line with previous literature and also as it is more intuitive).
\item The \textit{recall}, which is as usual defined as \textit{true inclusions / (true inclusions + false exclusions)}
\item The relative cost of crowdsourcing, defined as the \textit{price ratio} of the cost of crowd classification versus expert classification.
\end{enumerate}

\begin{figure*}[hbt]
     \subfloat[\label{fig:loss-price} ]{%
		\includegraphics[width=.333\textwidth]{figs/base_lvp.png}
     }
     \subfloat[\label{fig:recall_vs_pr}]{%
       \includegraphics[width=.333\textwidth]{figs/base_rvp.png}
     }
     \subfloat[\label{fig:pr_vs_expertcost}]{%
       \includegraphics[width=.333\textwidth]{figs/price_expert_cost.png}
     }

     \subfloat[\label{fig:recall_vs_theta}]{%
       \includegraphics[width=.333\textwidth]{figs/recall_theta.png}
     }
     \subfloat[\label{fig:pr_vs_theta}]{%
       \includegraphics[width=.333\textwidth]{figs/price_theta.png}
     }
     \subfloat[\label{fig:vary_filter_params}]{%
       \includegraphics[width=.333\textwidth]{figs/vary_filter_params.png}
     }
    \label{fig:simulations}
\caption{Analysis of the hybrid algorithm as key parameters vary. Settings and description are provided in the text. Crowd-Ensemble refers to the SR algorithm, while Hybrid-ensemble is HSR.}
\end{figure*}

The rationale for using price ratio as a metric when comparing algorithms is as follows: we consider that the price of crowdsourcing is not just represented by the total number $CV$ of crowd votes asked:
In fact, for each false inclusion $FI$ (for each item the crowd fails to screen out) we need to incur in the expert screening cost, as we are leaving the item on the expert's desk.
If we fail to account for this cost we overestimate the performance of the algorithm (incidentally, this is true for SR as well, although this is not discussed in the original paper and price ratio is not used there as a metric).
We therefore compute the crowd cost as $CC = CV + FI *ec$\footnote{There are cases where the ``expert cost" cannot be easily computed as the cost of expert labor: for example, a quasi-exact medical screening test may present practical challenges and even risks. In these cases, either domain experts can map these considerations into an ``expert cost" metric, or the analysis needs to focus on precision, not price ratio.} and compare it with what the total expert cost $EC = \vert I \vert \cdot ec$ (recall that $I$ denotes the set of items to be classified) would be if experts did all the classification.  The price ratio is then $CC / EC$.
Notice that considering price ratio and loss also incorporates precision: lack of precision (false inclusions) means higher loss and higher classification cost (higher price ratio). For this reason we do not plot precision in the following.

To describe the results, we first describe the baseline parameter settings for the experiments, and then show how results changes as we vary each parameter.

\textit{Baseline experiment.}
The baseline experiment settings consists of a problem where we screen items via 4 filters, and where 30\% of the items survive the screening - if we classify them correctly.
We simulate 10 ML classifiers with accuracy randomly selected from a 0.5-0.95 range and on which the algorithm assumes no prior knowledge.
We screen them with T=50 tests, work the math with the Beta distribution as described earlier, and keep the classifiers with 0.95 probability of having an accuracy greater than 0.5. The classification threshold $\overline{P_{out}}$ is set at 0.99, the loss ratio $k$ in the loss function is set to 10, and the expert classification cost $ec$ is set to 20 times the crowd label cost. This latter value is estimated from~\cite{Mortensen2016crowd} for the case of SLRs.
Notice that expert screening cost is \textit{per item}, while the unit label cost is \textit{per label} on an (item,filter) pair.
We stress again that users only have to set parameters corresponding to their requirements in terms of loss function, and the algorithm works out the rest, including the expected price ratios and recalls corresponding to given thresholds $\overline{P_{out}}$.

In Figure~\ref{fig:loss-price} we compare the results of applying machine only (ensembled with Naive Bayes), crowd (with SR), and HSR to simulations of classifications for 1000 items.
The charts plot the mean loss by price ratio per item, obtained by averaging the results of 50 iterations.
Vertical bars denote standard deviation.
The different dots of the machines and hybrid curves denote different values for \textit{error} correlation among machines, simulated at 0, 0.2, 0.3, 0.5, 0.7, 0.9.
As we can see (Figure~\ref{fig:loss-price}), for a similar loss level, hybrid algorithms significantly outperform the crowd in terms of both price ratio and loss.
Not surprisingly, price ratio and loss worsen as error correlation increases, as ML classifiers tend to agree when they make mistakes.
However, even at high correlations we maintain performances that are superior to those of crowd-only.
As we do not consider crowd correlation, for the crowd we just have a point in the chart.
ML-only classification is competitive only with independent and accurate classifiers, and their performance quickly deteriorates even at small level of error correlation, as we do not assume we have strong classifiers.
In the case of SLRs, with expert accuracy in the 95-98\% range \citep{Krivosheev_www18,Mortensen2016crowd} and assuming that the items that survive the screening are in the 10-30\% range, the loss lies within 0.04 and 0.18 (with the settings of the experiment being closer to the 0.18 mark, slightly better than the result of HSR but a much higher cost).

Similarly, Figure \ref{fig:recall_vs_pr} shows the recall vs the price ratio. Recall is independent of the loss parameter $k$ and gives us a direct indication of our capacity to identify the items that pass the filters	 for a given monetary investment.
Notice that we disregard here the cost of obtaining the base classifiers, which, if they  are built from scratch for this specific SLR, needs to be factored in when estimating price and  assessing the best strategy. On the other hand, if built for the specific problem they are also likely to have higher accuracy. We do not discuss this further as again building the base classifier is orthogonal to our goal here.
We also observe that recall is high "by design" in that we fix the threshold $\overline{P_{out}}$ for considering an item as screened.

\textit{Effect of changes to the parameters.}
We now explore how results changes as we change either parameters that describe the nature of the problem (e.g., filter power, crowd accuracy, or loss and expert cost ratios) and the behavior of the algorithm (e.g., decisions on how many test data to obtain, or thresholds to set).
Here we focus on a few interesting variations, and refer the reader to our GitHub repository\footnote{https://github.com/Evgeneus/crowd-machine-collaboration-for-item-screening} for additional details, data and source code.
Notice that HSR is \textit{adaptive}, that is, it estimates the parameters of the problem and changes its behavior accordingly. For this reason, we expect the recall to remain high, while price ratio may change.

Fig \ref{fig:pr_vs_expertcost} plots how the price ratio improves with the expert cost $ec$. Recall (not plotted here) is essentially constant. For example, for a 0.5 correlation, it stays in the .96-.97 range for all $ec$.
The lines flatten for high $ec$ and asymptotically reach the minimal price ratio, which is the proportion of false inclusions.

Figures \ref{fig:recall_vs_theta} and \ref{fig:pr_vs_theta} show instead how recall and price ratio change with the proportion of items that should pass the screening (positive items). Specifically, in the chart we show the behavior when the power of one of the four filters grows.
The figures show that the performances worsen in high power situations. This is because high powers mean a prior $P (i \in OUT)$ that is very close to the $\overline{P_{out}}$ threshold, and as such it is more sensitive to  errors from the crowd in the first few votes. This is not a problem of HSR per se but is inherited from SR.
To correct it, it is sufficient to "tone down" the prior when the power estimate is too high.
Indeed, we observed experimentally that this can be achieved by structurally underestimating power by approximately 20\% of its value.

Finally, Figure~\ref{fig:vary_filter_params} shows how recall varies in the presence of one criteria that heavily differs from the other in terms of difficulty (accuracy) and power (selectivity of filters). As expected, recall remains constant because the algorithm adapts to the characteristics of the problem. Price ratio worsens as we go from an easy and powerful criteria to the most difficult case of low power, low accuracy filter where we have many incorrect votes, and even correct ones do not help us much because the screening power of this filter is low, which means we anyways need to query the other filters.
HSR results are robust to variations in the number of tests and in the confidence thresholds for keeping a ML classifier (not shown). Assuming a set of classifiers in the 0.3-0.7 range instead of 0.5-0.95 cuts approximately in half the gain with respect to SR.

\textbf{Experimental data and experiment design.}
We experimented both by using existing datasets and by running crowdsourcing experiments.
In all cases, data and code for the algorithms are available at the same url [omitted], along with other charts describing variations of behaviors with parameter values which we skip here as we find them  less informative.

\begin{figure}[hbt]
	\centering
    \begin{minipage}{.8\linewidth}		\includegraphics[width=0.7\textwidth]{figs/task3.png}

		\caption{Classification task for SLR (settings borrowed from \cite{Krivosheev_www18})}
\label{fig:task_design}
	\end{minipage}
\end{figure}

\begin{figure*}[hbt]
     \subfloat[\label{fig:recall-tests} ]{%
		\includegraphics[width=.383\textwidth]{figs/ensemble_algos_price_tests.png}
     }
     \subfloat[\label{fig:recall-price-acc}]{%
       \includegraphics[width=.383\textwidth]{figs/fig3b_ensemble_algo_acc05_095.png}
     }

     \subfloat[\label{fig:fig3c_ensemble_algo_acc05_075}]{%
       \includegraphics[width=.383\textwidth]{figs/fig3c_ensemble_algo_acc05_075.png}
     }
     \subfloat[\label{fig:semi-real-experiment}]{%
       \includegraphics[width=.358\textwidth]{figs/3dhcomp_old.png}
     }
    \label{fig:simulations}
\caption{Experimental results for basic and stacking models. (a): recall for SR (black), basic HSR (blue) and HSR where classifiers are combined with regression (yellow) as the size of the training set used for the regression model varies; (b) and (c): recall vs price ratio for different distribution of classifiers accuracies; (d) results based on parameters obtained from AMT experiments. The dots in figs b,c,d present the correlation level of machine classifiers. Corr is [0, 0.2, 0.3, 0.5, 0.7, 0.9] from left to right. }
\label{ff}
\end{figure*}

We first took an existing SLR dataset by Wallace and colleagues~\cite{Mortensen2016crowd}.
This includes over 20000 crowd votes over 4000 papers, with four filters.
We also run a set of experiments on AMT and CrowdFlower, with different designs, but specifically borrowing the design from \cite{Krivosheev_www18} for a fair comparison, and eventually built our own platform on top of AMT for increased flexibility and for being able to plug in the adaptive algorithms into the task assignment logic (\textit{url omitted}).
Specifically, the basic design included classifications of 100 papers for an SLR in the social informatics field, with three filters of different difficulty  (one of them very difficult, with a worker accuracy barely above 0.5). The filters were: "does the paper describe a study on 65+ older adults", "does the paper describe a study that uses technology", and "does the paper describe an \textit{intervention} type study" (the difficult one).
The task proceeded as depicted in Figure \ref{fig:task_design}. Workers with historical overall AMT accuracy over 70\% were invited to participate. A worker would see a description of one criterion, along with a positive and a negative example. They would then proceed to labeling papers based on that criterion.
The choice of a per-filter approach (asking for labels on many items for one filter) as opposed to per-item, is because it takes time to properly explain and understand a criterion (for example, explaining what an intervention is may be tricky and requires the worker to go through positive and negative examples carefully).
We screen workers with two test questions and consider the labels from the remaining workers. After initial experiments to estimate the time taken by workers, we tuned parameters to arrive at a pay rate of 10USD/hour. We obtained 10 labels for each item and filter for a total of 3000 crowd votes from 147 workers.

We then built classifiers for each filter using a variety of techniques (from KNN to random forest, variations of naive Bayes and logistic regression and others, with different kinds of document representations) and different sizes of training data to get realistic information on  classifier accuracies and correlations.
We obtained classifier accuracies in the 0.5-0.8 range, correlations in the 0.2-0.9 range, and crowd accuracy in the 0.55-0.8 range.
In this case, classifiers were obtained by training on the same SLR data using 20\% of the data, where the training set is obtained via crowdsourcing, so the savings for the experiment refers to the remaining portion of the items to be screened.

We then use these parameters obtained from experiments  to fuel various experiments and repetition.
We show the representative result in Figure ~\ref{ff}d where the blue line represent HSR. We ran experiments with different combinations of base classifiers, having different correlations among them as usual represented by dots in the line.
Notice that the standard deviation is very high in proportion to the scale of the chart, though we can see the impact of the hybrid approach. Higher correlations (moving from top to bottom) lead to slightly lower recall.
Incidentally, we also report that, consistently with expectations, errors made by HSR and by us (the experts, in this case) were about the same: half the time we disagreed with the crowd, either the crowd was right or the decision was questionable.

\textbf{Stacking classifiers.}
In the experiments above we naively included ML classifiers regardless of their error correlation. However, if two or more classifiers predict the same class (and especially if they make the same errors) then it does not help to pool their predictions - worse than that, when they make mistakes they make so by consensus thereby increasing our confidence in the incorrect prediction.
There are essentially two approaches to deal with this: the first is to filter out highly correlated classifiers.
If we denote with $c^j_{err}$ the event corresponding to classifier $c^j$ making an error, then we look to exclude classifiers where $P(c^j_{err} | c^k_{err})$ is high, and particularly above 0.5, otherwise we reinforce the error of $c^k$.
This can be done again in the test phase similarly to what we do for estimating accuracy, for example again by assigning a Beta prior to this conditional probability.
However, since errors might be rare, estimating error correlation requires more data points. If the initial (expert-provided) gold dataset is small, then we can proceed with HSR to get data points, initially with one or few ML classifiers, and then add more as we are confident of their low correlation.

However, we can also \textit{stack} the ML classifiers by learning a model, and let this model filter or downweight correlated classifiers.
Figure~\ref{ff}(a-c) show the performance of a logistic regression model built over the base ML classifiers and leveraged in HSR, compared with base HSR as from the previous section (blue) and with crowd-only ensemble (SR).
The plot assumes 5 ML classifiers with accuracies as before, and a varying size of training dataset.
In theory, the regression naturally copes with correlation and in the presence of a sufficiently large training set, it identifies how to effectively combine classifiers.
As above, the training set can be progressively obtained via SR, and we can use base classifiers to identify a dataset that has higher possibility to be balanced, to get a better training set.

The charts show that in practice stacking with regression offers no improvement with respect to the naive aggregation of classifiers, and improvements are also limited with respect to taking the single best classifier in the set as opposed to an ensemble (but notice that SR takes us already to 0.95 recall, so improving from that is challenging).

We found this result surprising, although consistent with the findings briefly mentioned by \cite{Dzeroski2004Stacking}, where however the problem is not discussed besides the mentioning of this observation.
To get clarity on this matter we investigated it further. Specifically, besides experimenting with classifiers of varying accuracies and correlation, we engaged to understand if the lack of effect is due to the specific way we use the output of classifiers - that is, to set prior probabilities (which in turn determine how to query the crowd for that item based on the HSR algorithm) as opposed to directly take a classification decision.

The results are shown in Figure \ref{fig:ensemble_simulations}. Figures a and b respectively show the performance of the ensembles when used as prior (Figure \ref{fig:ensemble_simulations}a) and when used directly to take final classification decisions, without resorting to the crowd (Figure \ref{fig:ensemble_simulations}b).
The different dots denote accuracy of the base classifiers, with recall and price ratio improving with accuracy as expected (with one interesting exception discussed later). The two figures are shown in the same scale to facilitate comparison. We adopt the price ratio  metric also for Figure \ref{fig:ensemble_simulations}b since classification errors mean that we incorrectly leave items for experts to classify, and this incurs in an unnecessary cost with respect to classifiers with perfect precision.
Figure \ref{fig:ensemble_simulations}b shows that when we use the ensemble to make predictions, in low correlation conditions both Naive Bayes (NB) and regression ensembles are superior to the single best classifier especially in conditions of higher accuracy of the base classifiers (the difference in recall is small and within a standard deviation, while the benefit in terms of price ratio is high).
With high correlation, regression ensemble and best classifiers offer similar performances, while NB drops significantly in recall, our main target metric. This is not surprising since in conditions of high correlation, ensembles with NB essentially reinforce the errors.
Indeed, as accuracy of base classifiers increases so does our confidence in their vote (and in the ensemble). Specifically, often this confidence will exceed our confidence threshold $\overline{P_{out}}$ for classifying items as out directly, which explains the drop in recall for the NB ensemble as accuracy increases.
In conditions of lower accuracy we take NB ensemble's predictions more cautiously, which explains the better recall (at the expense of price ratio).
The benefit of regression  however only manifests themselves once we have sufficient training data - in our experiments we started to observe a difference starting from training sets of 400 items.

When we leverage ensembles as prior (Figure \ref{fig:ensemble_simulations}a) as per HSR algorithm, these differences smooth out considerably and the points in the charts become closer. In HSR we never classify based on machines only, as we ask at least one vote from the crowd, which helps avoid such drops in recall.
Ensembles improve over crowd-only approaches as discussed, even with weak classifiers, but because taking classifiers' results as prior is conservative and because classification threshold are high, recall errors from classifiers have a chance to be corrected by the crowd.

\begin{figure*}[t]
     \subfloat[\label{fig:recall-tests} ]{%
		\includegraphics[width=.450\textwidth]{figs/fig4_hybrid_same_scale.png}
     }
     \subfloat[\label{fig:recall-price-acc}]{%
       \includegraphics[width=.450\textwidth]{figs/fig4_machines_same_scale.png}
     }
    \label{fig:ensemble_simulations_b}
\caption{Comparison of NB and regression ensembles when used as prior (left) and to classify items directly (right). The dots in charts represent different accuracy range of machine classifiers ([0.65, 0.75], [0.75, 0.85], [0.85, 0.95]), increasing from  right to left. Ensembles have been trained over a dataset of 500 items.}
\label{fig:ensemble_simulations}
\end{figure*}

\section{Conclusions}
The main result of this paper is a method for multi-filter classification that combines ML and human classifiers to achieve high level of classification accuracy for unit of cost.
We believe the main benefit lies in the ability of the algorithm to be robust to both the characteristics of the problem (such as  filters that are hard for the crowd to classify) and to weak ML classifiers.
In both cases, the algorithm works out to leverage the filter, items, and ML classifiers (over specific filters and items) and builds models of classifiers to make the most of what can be screened automatically or with the crowd, leaving the rest for other classification methods (such as expert classification) without compromising on quality.
Furthermore, the experimental results also helps us understand, in case we do have to build classifiers because we cannot transfer them from similar learning problems, what is the accuracy we should look for to get a benefit from hybrid classification.
While we focused on hybrid algorithms we also uncovered improvements to multi-filter crowd classification algorithms such as the opportunity to smooth the prior in high-power filter scenarios.

Although the evaluation has focused on SLR we see the approach as applicable to many multi-filter screening problems.
An extensive evaluation of the limits of the approach and the kinds of problems to which it applies is part of our current work.
In addition, we see active learning for ensembles in finite pool contexts such as the one described here as an interesting research problem, which is complicated by the likely presence of highly imbalanced datasets and by needs that are often conflicting, that of leveraging the crowd efficiently to screen items out but to also get training data classified as in. Another interesting thread of analysis is to see if crowd workers can, besides labeling (item,filter) pairs,  actually identify features that machines can then leverage for the problem at hand.

Finally, an interesting problem is the extent to which learning can be transfered across finite pool problems of the same kind (for example, different SLRs), especially in terms of identifying general approaches and methodologies that can be adopted in different domains.

\section{Related Work}

\subsection{Crowd Classification}

This paper researches a field that is abundant in prior art.
The problem of crowd classification has been studied for centuries, at least since the early work of the Marquis de Condorcet who, trying to make a case for democracy, proposed his \textit{Jury Theorem}\footnote{http://www.stat.berkeley.edu/~mossel/teach/ SocialChoiceNetworks10/ScribeAug31.pdf}, stating that if each juror in a jury (or each citizen, when taking a binary decision) has an error rate lower than 0.5 and if \textit{guilty} vs \textit{innocent} votes are independent, larger juries reach more accurate results, and approach perfection as the jury grows.

More recently, with the opportunity offered by crowdsourcing on the one hand and the need for large labeled dataset to support ML tasks on the other, many researchers from the AI, database, and human computation communities investigated the problem in detail, both to identify algorithms and to study theoretical bounds. Initial algorithms were based on variations of (weighted) majority voting, where votes are counted differently based on the estimated worker's accuracy.
Dawid and Skene \cite{DawidSkene_Confusion} in the late nineties started a thread of research based on modeling and iteratively estimating workers' accuracy and class labels, by adopting variants of Expectation Maximization \cite{EM_1977}.
From there, many important and useful extensions have been proposed by Whitehill \cite{whitehill2009whose}, Dong et al. \cite{Dong2013}, Li et al. \cite{Li_error_13},  Liu et al. \cite{LiuWang_truelabel,liu2013scoring} and many others.
Relatively recent approaches based on spectral methods \cite{karger2011iterative} and maximum entropy \cite{Zhou_spectral_13} have also been proposed, and belief propagation has  been  recently shown  \cite{Ok_belief_13} to be optimal under certain assumptions.

Prior art has also discussed how to continuously assess the estimated value of asking for more votes from the crowd versus taking a classification decision with the votes as available.
The approach is based on modeling the problem in terms of partially observable Markov Decision Processes ~\cite{kamar_crowdonly_2013,POMDP_2013}, where policies are progressively learned as votes become available.

\subsection{Crowd Classification in multi-predicate problems}
The set of classification problems we address here is that of \textit{finite pool} classification \cite{Nguyen2015}: we work on a finite set of items, and the classification criteria may be unique to the problem. SLRs are one case of such problems that has been thoroughly investigated by many researchers in recent years \cite{Mortensen2016crowd,Krivosheev_hcomp17,Nguyen-hcomp15,Sun16-hcomp}.

Mortensen and colleagues crowdsourced paper screening~\cite{Mortensen2016crowd} in four literature reviews, each with several criteria. They explore feasibility and costs of crowdsourcing, and they address the problem by measuring workers agreement.
Furthermore, they include a set of very interesting observations related to the importance of task design, as well as on the high degree of variability in workers's agreement from paper to paper and criteria to criteria (Fleiss' Kappa ranging from 0.5 to -0.03).

Krivosheev and colleagues~\cite{Krivosheev_hcomp17} also present a model and strategies for crowdsourcing SLR. An interesting aspect of the model and approach here is that the authors model cost and loss (error) resulting from crowdsourcing task, attempt to estimate them at the start, and provide authors with a price/error trade-off that can be used to decide how much to invest in the task.
They extend the work to the multi-filter screening case \cite{Krivosheev_www18}, also by introducing algorithms that efficiently query the crowd by focusing first on criteria that are more promising in terms of screening items correctly.

We borrow several concepts from this work and indeed we build over a slightly modified version of their algorithm.
However, crowdsourced classification algorithms for us  are to a large extent swappable components we leverage: we do not aim at improving crowd-only classification but rather to study how to efficiently combine ML and human classifiers, and specifically how to do so in a manner that is robust to weak classifiers to avoid compromising on accuracy and avoid spending money in classifications not likely to produce good results.

\medskip

Multi-filter classification has been also widely studied in search and information retrieval contexts. Indeed, an instance of such problem is that of selecting which predicates to apply first when filtering tuples in a query. The seminal work in this area is by Hellerstein and Stonebraker, discussing the problem of ranking predicates in query plans \cite{Hellerstein93predicates}.

More recently, Franklin et al. \cite{CrowdDB2011}, Park and Widom \cite{ParkWidom13} and Parameswaran et al \cite{crowdscreen12,optimalWidom2014} focused on a similar  problem but in the context of crowd-powered databases, while Anvur et al \cite{Eddies2000}, Babu et al \cite{Babu2004adaptive}, Lan et al \cite{Lan_dynamicfilter_hcomp17} presented \textit{adaptive} extensions that tune the algorithm  as the query progresses (see Deshpande et al \cite{DeshpandeAdaptive} for a review).
We differ in many ways with respect to the prior art in this area. First, again, we aim at combining ML classifiers with crowd.
Furthermore, we follow an  approach that makes initial estimates on the characteristics of the classification problem and then continuously revise the estimates, both in general and for each item to be classified, to identify an item and filter specific strategy to reduce errors and cost. Importantly, we identify the items on which we should give up trying, since it is more efficient to leave them for experts to screen.

Other prior work also addresses the issue of optimizations in terms of costs for obtaining labels and techniques to reduce cheating \cite{smyth1995inferring,karger2011budget,hirth2013analyzing,eickhoff2013increasing,hirth2011cost}. For example, Hirth and colleagues \cite{hirth2013analyzing} recommend specific cheating detection and task validation algorithms based on the cost structure of the task.
In this paper we do not optimize cheating detection or even task design to a large extent, in that those are orthogonal aspects with respect to the methods and optimizations discussed here.

\subsection{Hybrid Classification}
Just like crowd classifiers, ML classifiers are also ingredients of a solution we borrow from the state of the art. In the context of our problem the interesting issues are how to combine ML and humans, how to ensemble available classifiers, and how to balance the request cost of (crowd or expert) votes with the need of training and testing ML models.

Hybrid classification is  an increasingly active area of research (see \cite{Vaughan_crowd17} for a recent survey).
Many researchers study the problem in the context of clustering or entity resolution (\cite{crowdER_2012,VesdapuntER_vldb2014,Vinayak_crowdcluster_nips2016,Gomes_crowdclustering_nips2011}).
In general, many of these techniques operate by first classifying items automatically when ML classifiers are highly confident, and  by then leveraging ML to shape the kind of task proposed to the crowd (e.g., identifying potential clusters or matching items to propose to the crowd \cite{VesdapuntER_vldb2014,Vinayak_crowdcluster_nips2016}).
Another class of hybrid approaches, also extremely popular in industrial applications, are those where ML makes a proposal or a pre-filtering and humans confirm or refine. This happens in many fields, recently even in fashion \footnote{https://multithreaded.stitchfix.com/blog/2016/03/29/hcomp1/}.

Kamar and colleagues propose instead a promising approach where crowd features (votes, as well as potentially other aspects of the crowdsourcing process such as task completion times) and task features are combined into a broader set of features to be used to learn a model~\cite{Kamar_2012_combining}.
Researchers also explored using the crowd to extract features and patterns to then be leveraged by classifiers ~\cite{flock_2015,Carlos_pattern}.

While very interesting, these results are complementary to the work discussed here as we do not aim at developing a good base classifier but at leveraging effectively a set of arbitrarily weak classifiers we are given for the problem at hand.
Perhaps the most closely related prior art is the work by Nguyen et al. \cite{Nguyen2015}, who adopt a clever mix of crowd+expert+machine learning  with an active learning classifier, where papers to be labeled are iteratively chosen to minimize overall cost and loss, by comparing estimated loss of crowd classification versus expert classification. Our problem and approach differs, however, in that we focus on exploiting existing classifiers and in doing so for optimally selecting the (item,filter) pair to poll the crowd on.
It is indeed in the optimization of which question to ask the crowd that lies the essence of the approach we propose.

redpajama_set_name: RedPajamaArXiv


Hoyle Law, LLC provides comprehensive planning and litigation services to individuals and businesses in Ocean Township, Long Branch, Asbury Park, Oakhurst and surrounding communities of Monmouth County, Ocean County, Middlesex County and nearby counties Central of New Jersey.
I am estate attorney John Hoyle III, the founder of this law firm with more than 15 years of experience. I want you to know that our firm values the opportunity to serve your legal needs and hopes that this is the start of a long relationship. Contact us today and learn how our comprehensive solutions can apply to your situation.
Elder law - We provide single-source counsel for all legal issues pertaining to clients at retirement age and beyond, including estate planning, long-term care planning, and elder abuse.
Guardianship and conservatorship - We represent clients seeking to take control of an elderly parent's financial and medical affairs, or parties in guardianship disputes.
Estate planning — We go beyond the simple will to help clients protect assets, guide family members' decisions, and plan for ill health or incapacity. We can arrange a variety of trusts to accomplish specific goals.
Estate administration — We represent executors or appointed administrators in efficiently settling a loved one's estate.
Will contests and probate litigation — We represent named heirs, excluded heirs, creditors or executors in disputes over a will, including undue influence or breach of fiduciary duty.
Medicaid planning and nursing home finance — With advance planning, we can help clients plan for Medicaid eligibility without spending down all family assets. We also advise on alternatives for paying for long-term care.
Special needs trusts — Our understanding of Medicaid and Social Security rules enables clients to protect assets for an adult disabled child.
Contact our Monmouth County office online or call 732-988-9595 to arrange a consultation with an experienced lawyer at your convenience. As a small firm, we can respond quickly when urgent legal action is required.


text:
#include <Arduino.h>
#include <OneWire.h>
#include <DallasTemperature.h>
#include <ESP8266WiFi.h>

#include "config.h"

// amount of time in milliseconds to wait before closing connection with no data being receieved
#define IDLE_TIMEOUT_MS 3000

const int ONE_WIRE_PIN = D2;

// initialise OneWire library
OneWire oneWire(ONE_WIRE_PIN);

// initialise temperature library
DallasTemperature sensors(&oneWire);

const byte DEVICE_ADDRESS_LENGTH = 8;
byte deviceCount;
byte deviceIndex;

unsigned long interval; // polling interval in milliseconds
unsigned long lastRunTime;
unsigned long currentTime;

struct Reading {
  byte deviceAddress[DEVICE_ADDRESS_LENGTH];
  bool valid; // if a reading could be obtained from device
  float temperature; // measured in degrees Celsius
};

/**
 * Initialise all sensors on 1-Wire bus.
 */
void sensorInit() {
  Serial.print(F("Sensor bus on PIN "));
  Serial.println(ONE_WIRE_PIN, DEC);

  // start temperature library
  sensors.begin();

  deviceCount = sensors.getDeviceCount();
  Serial.print(F("Number of devices on bus: "));
  Serial.println(deviceCount, DEC);

  // report parasite power requirements
  Serial.print(F("Parasite power: "));
  if (sensors.isParasitePowerMode()) {
    Serial.println(F("on"));
  }
  else {
    Serial.println(F("off"));
  }

  Serial.println();
}

/**
 * Print the address of a device in hexadecimal format to the serial port.
 *
 * Example:
 *   { 0x28, 0xFF, 0x12, 0x34, 0x56, 0x78, 0x9A, 0xBC }
 */
void printAddress(byte address[DEVICE_ADDRESS_LENGTH]) {
  Serial.print(F("{ "));
  for (uint8_t i = 0; i < DEVICE_ADDRESS_LENGTH; i++) {
    Serial.print(F("0x"));
    // zero-pad the address if necessary
    if (address[i] < 16) {
      Serial.print(F("0"));
    }
    Serial.print(address[i], HEX);
    // seperate each octet with a comma
    if (i < 7) {
      Serial.print(F(", "));
    }
  }
  Serial.print(F(" }"));
}

/**
 * Return the address of a device as a hexadecimal string.
 *
 * Example:
 *   0x28ff123456789abc
 */
String formatAddress(byte address[8]) {
   String s = "0x";
   for (uint8_t i = 0; i < 8; i++) {
    // zero-pad the address if necessary
    if (address[i] < 16) {
      s += "0";
    }
    s += String(address[i], HEX);
   }
   return s;
}

void readSensors(Reading readings[]) {
  // call sensors.requestTemperatures() to issue a global temperature request to all devices in the bus
  Serial.println(F("Requesting temperatures..."));
  sensors.requestTemperatures();

  for (deviceIndex = 0; deviceIndex < deviceCount; deviceIndex++) {
    Reading* reading = &readings[deviceIndex];
    if (sensors.getAddress(reading->deviceAddress, deviceIndex)) {
      float temperature = sensors.getTempC(reading->deviceAddress);
      reading->valid = true;
      reading->temperature = temperature;
    } else {
      // unable to read from device; therefore clear values
      reading->valid = false;
      memset(reading->deviceAddress, 0, sizeof(reading->deviceAddress));
      reading->temperature = 0;
      Serial.print(F("Unable to read from device: "));
      Serial.println(deviceIndex);
    }
  }
}

void printReadings(Reading readings[]) {
  for (deviceIndex = 0; deviceIndex < deviceCount; deviceIndex++) {
      Reading reading = readings[deviceIndex];
      if (reading.valid) {
        Serial.print(F("Device "));
        Serial.print(deviceIndex);
        Serial.print(F("; address: "));
        printAddress(reading.deviceAddress);
        Serial.print(F("; temperature: "));
        Serial.println(reading.temperature);
      }
  }
}

void wifiConnect() {
  Serial.print(F("Connecting to Wi-Fi network: "));
  Serial.println(WLAN_SSID);

  WiFi.begin(WLAN_SSID, WLAN_PASS);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(F("."));
    delay(1000);
  }
  Serial.println(); // close off "connection dots" line

  Serial.print(F("Connected to Wi-Fi network with IP address: "));
  Serial.println(WiFi.localIP());
}

/**
 * Construct JSON message with sensor readings.
 *
 * Examples:
 *   { "readings": { "0x28ff123456789abc": 20.50 } }
 *   { "readings": { "0x28ff123456789abc": 18.06, "0x28ff123456789abd": 17.19 } }
 */
String constructMessage(Reading readings[]) {
  String message = "{ \"readings\": {";

  for (deviceIndex = 0; deviceIndex < deviceCount; deviceIndex++) {
      Reading reading = readings[deviceIndex];
      if (reading.valid) {
        String address = formatAddress(reading.deviceAddress);
        // readings are comma-separated
        message = message + " \"" + address + "\": " + String(reading.temperature) + ",";
      }
  }

  // remove trailing "," for last reading in comma-separated list
  if (message[message.length() - 1] == ',') {
    message.remove(message.length() - 1);
  }
  message += " } }";

  return message;
}

void sendReadings(Reading readings[]) {
  String message = constructMessage(readings);
  Serial.println(F("Sensor data message:"));
  Serial.println(message);

  WiFiClient client;

  Serial.print(F("Connecting to server "));
  Serial.print(API_HOST);
  Serial.print(F(" on port "));
  Serial.print(API_PORT);
  Serial.print(F(" with endpoint "));
  Serial.println(API_ENDPOINT);
  if (client.connect(API_HOST, API_PORT)) {
    // send request to server
    client.print(F("POST "));
    client.print(API_ENDPOINT);
    client.println(F(" HTTP/1.1"));
    client.print(F("Host: "));
    client.println(API_HOST);
    if (API_AUTH_VALUE != "") {
      Serial.println(F("Using HTTP Basic authentication"));
      client.print(F("Authorization: Basic "));
      client.println(API_AUTH_VALUE);
    } else {
      Serial.println(F("Not using authentication"));
    }
    client.println(F("Content-Type: application/json"));
    client.println(F("User-Agent: NodeMCU/1.0"));
    client.println(F("Connection: close"));
    client.print(F("Content-Length: "));
    client.println(message.length(), DEC);
    client.println();
    client.println(message);
    client.println();

    // read response data until either the connection is closed, or the idle timeout is reached
    Serial.println(F("-----------------------"));
    unsigned long lastRead = millis();
    while (client.connected() && (millis() - lastRead < IDLE_TIMEOUT_MS)) {
      while(client.available()) {
        char c = client.read();
        Serial.print(c);
        lastRead = millis();
      }
    }
    Serial.println(F("-----------------------"));
  } else {
    Serial.println(F("Unable to connect to server"));
  }

  // close connection to server
  client.stop();
}

/**
 * Read and process data from sensors (one cycle).
 * Will be performed in a loop.
 */
void run() {
  Reading readings[deviceCount];
  readSensors(readings);

  printReadings(readings);

  wifiConnect();
  sendReadings(readings);
  WiFi.disconnect();
}

void setup() {
  Serial.begin(115200);
  Serial.println(F("Starting..."));
  delay(100);

  sensorInit();

  // set WiFi to station mode and disconnect from an AP if it was previously connected
  WiFi.mode(WIFI_STA);
  if (WiFi.isConnected()) {
    WiFi.disconnect();
  }
  delay(100);

  // set polling interval
  interval = (unsigned long)POLLING_INTERVAL_S * 1000;
  Serial.print(F("Loop interval: "));
  Serial.print(POLLING_INTERVAL_S);
  Serial.print(F(" seconds => "));
  Serial.print(interval);
  Serial.println(F(" milliseconds"));

  // initialise to current time
  lastRunTime = millis();

  Serial.println(F("Setup done"));

  // start first run after setup
  run();
}

void loop() {
  // interval loop
  currentTime = millis();
  if (currentTime - lastRunTime > interval) {
    lastRunTime = currentTime;
    run();
  }
}

redpajama_set_name: RedPajamaGithub


Graduate research programs leading to the M.S. or Ph.D. degree are available for students choosing to study with CIIR faculty in conjunction with other Biomedical Sciences departments in the UVA School of Medicine. The graduate training program includes select coursework and laboratory research training experience in immunology, autoimmunity, inflammatory diseases or regenerative medicine.
Application to graduate programs in the UVA School of Medicine should be made through the Biomedical Sciences Graduate Consortium. CIIR faculty members include NIH funded academic and clinical investigators who are aligned with MII and/or MSTP programs in the Biomedical Sciences Graduate Consortium.