text:
Architecture is More Than Buildings
Martin A. Pedersen, one-time New Yorker and former editor of Metropolis and Graphis, now a New Orleans resident, just started a unique site whose formal title is the Common Edge Collaborative. It a nonprofit founded by his creative partner and friend, Steven Bingler, a New Orleans–based architect and planner; they'd met while covering the city's Katrina recovery for Metropolis. He'd been thinking for years about a project called the Common Edge that would cover his belief that contemporary architecture was almost terminally disconnected from the concerns of real people. And he thought there had to be a kind of Third Way: an architecture that fused the common truths, derived from 5,000 years of architectural thinking, with the cutting edge of innovation. Hence, the term Common Edge. It shouldn't be an either/or proposition. In December 2014 he and Bingler published a polemic in The New York Times that "stirred things up a bit and got us thinking about creating a platform for these ideas." I asked Pederson why architects, urbanists and designers in general should bookmark another site.
Common Edge is a nonprofit organization dedicated to reconnecting architecture and design with the public that it's meant to serve.
With Common Edge, what is the space that you are filling, and why does that space and audience need a voice?
We're deliberately defining our mission, public engagement, as broadly as possible, so that we ourselves can engage with a variety of issues: architecture, planning, cities, social justice, functionality. It's really trying to look at these issues from a user's, from a citizen's, viewpoint.
Architecture is as essential a design activity as we have. What do you think Common Edge is saying that both the architectural and lay communities need to know? It is absolutely one of the essential design activities, and yet you wouldn't know it. The dialogue around it, even though there are many, many terrific critics (some of them my friends), is a bit insular. It tends to exclude people who don't speak the code. One of the things I've been trying to do, in the early going, is recruit lay people to write about architecture. Most of them, tragically, say, "Oh, I couldn't possibly, what would I say? I'm not qualified to write about architecture." And my answer is, "Of course you're qualified! You're a user of the building!"
If the designers had been asked to create a setting that signals the chaos of this campaign, they could not have done it better.
That's valid but I think there may be too many amateur critics. You are a pro, but you say you're an amateur when it comes to the web. So, what are your primary concerns, and how are you getting them aired? At this point, we're so new and I'm such a digital infant that my primary concerns and my principal means of getting them aired is very much a work-in-progress. I spend far too much time at Facebook. I'm learning Twitter (overrated). I'm trying to write "stickier" internet-friendly headlines that don't embarrass me (this is definitely a work-in-progress). It's a slow process that requires patience, in a medium that doesn't reward patience, but we are getting some traction and appear to be gathering an audience.
You obviously focus a bit on our patron saint Jane Jacobs. How does she play a role in your editorial thinking? here's a lot on the site about Jane Jacobs for a rather fortuitous reason. May 4th was the 100th anniversary of her birth. She is, in my eyes, the most important American urbanist of the 20th century, and in many ways, the ultimate Common Edger. She was always about the wisdom of the crowd. The users of the city. I suspect, even with the passing of the centennial, her name will continue to percolate on the site, and for very [good] reason: Her ideas still matter.
With the insanity, buffoonery, cluelessness, bigotry and narcissism, Trump's candidacy is tapping into something else besides racism and xenophobia: economic displacement.
The wisdom of the crowd is an interesting concept but what about the wisdom of the editor—what are your favorite themes? I've always been partial to stories and essays about how cities form, how they develop, how they change, and what that means for the people who live in them.
What is on your coverage wishlist? Healthcare. Post-occupancy stories about celebrated buildings. The social justice issues surrounding the built environment. There's at least 20 more that I can't think of right now because I haven't drank enough coffee yet. It's a bottomless well. But for me, it's important that this is a group effort. The last thing I want this to be is me, pontificating on the state of design. I want to be part of a collection of diverse voices: designers and users. The public and the people who are supposed to be designing for them.
Get PRINT Delivered to Your Door Rather than focusing on the how-to of design, the experts who write for PRINT magazine cover the why–why the world of design looks the way it does, how it has evolved, and why the way it looks matters. Subscribe to PRINT today, and get in on the conversation of what the brightest minds in the field are talking about right now—essential insight that every designer should know to get ahead.
Treat yourself and your team to a year of PRINT for only $40—which includes the massive Regional Design Annual ($29.99 on newsstands).
#DailyHeller #MartinPedersen #StevenHeller #TheCommonEdge
RELATED POSTS My Favorite Dummy Brooklyn Street Art The Dean of Design The Bernini Of Cardboard Sculptures Narrative Of Things
About Steven Heller Steven Heller is the co-chair of the SVA MFA Designer /Designer as Author + Entrepreneur program, writes frequently for Wired and Design Observer. He is also the author of over 170 books on design and visual culture. He received the 1999 AIGA Medal and is the 2011 recipient of the Smithsonian National Design Award.View all posts by Steven Heller →


Project X Ltd. » Join us at MicroStrategy Symposium Toronto – Oct. 14th!
Join us at MicroStrategy Symposium Toronto – Oct. 14th!
We're excited to once again be sponsoring MicroStrategy Symposium Toronto coming up next week on Friday, October 14th. Taking place at the Sheraton Centre, MicroStrategy Symposium is your one-stop shop to network with Canadian BI professionals and learn about the latest developments in big data analytics, data visualization, dashboard design, mobile app development and more.
Businesses are racing to get value from their data, and they want it now. To that end, many organizations are looking to align their organizations to be more "agile" in their delivery models. In this informative panel, some of Canada's sharpest minds in the BI sector will discuss how savvy companies are moving away from the waterfall deliver methodology and moving towards a more rapid approach to delivering value.
Visit our booth and enter to win a DJI Phantom 3 Quadcopter!
This easy-to-fly drone takes stunning 2.7K HD videos and 12 Megapixel photos, and retails for over $600 – our most exciting giveaway yet! Just look for our Project X Ltd. drone taking flight – you can't miss us.


text:
$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)
require 'scalpel'
require 'fakefs/spec_helpers'

redpajama_set_name: RedPajamaGithub


text:
North Texas running back DeAndre Torrey celebrates after scoring during a game at Apogee Stadium.
DRC file photo
North Texas running back DeAndre Torrey (13) breaks a tackle as he heads for the end zone during the Mean Green's loss to Utah State in the New Mexico Bowl last season.
10 things North Texas fans should know about DeAndre Torrey
By Brett Vito Staff Writer bvito@dentonrc.com
Brett Vito
North Texas is expected to have one of the best teams in Conference USA this fall, and maybe one of its best in recent program history.
There are plenty of reasons that is the case from the return of head coach Seth Littrell to the talent of quarterback Mason Fine, the two-time C-USA offensive player of the year.
But there is more to UNT's team than just its coach and star player.
The Mean Green have several other talented veterans who will fill key roles this fall, including DeAndre Torrey.
Here are 10 things you might not know about the Mean Green's star running back:
Here's a sideline look at @TorreyBoy1's 11-yard TD run from earlier in the quarter #GMG #BeatRice pic.twitter.com/MMHDkVUksS
— North Texas Football (@MeanGreenFB) October 27, 2018
1. Torrey is coming off an impressive sophomore season
Torrey flew a little under the radar last season at UNT. That's understandable considering the Mean Green featured C-USA's best quarterback in Fine, one of the league's top NFL prospects in linebacker E.J. Ejiya and a couple of elite wide receivers in Rico Bussey Jr. and Jalen Guyton.
When one looks at what Torrey accomplished last season, it's apparent that his contributions were an overlooked aspect of UNT's success.
The Mean Green entered the season looking for a way to replace star running back Jeffery Wilson, who is now with the San Francisco 49ers.
Torrey emerged as the Mean Green's best option and rushed for 977 yards and scored a team-high 18 touchdowns.
2. Littrell made a late decision to recruit Torrey
All college coaching staffs lay out plans for recruiting classes in terms of how many players they intend to sign at each position.
UNT's changed those plans for its 2018 class once Littrell and his staff got a glimpse of Torrey. The 191-pound speedster was highly productive at Mississippi Gulf Coast Community College, where he rushed for 1,298 yards and 12 touchdowns in just nine games as a freshman in 2017.
Torrey was also offered a scholarship by Troy and South Alabama before signing with UNT.
3. The Georgia native has a unique background among UNT players
Torrey stands out among UNT players because of his background in addition to his talent.
The Mean Green focus their recruiting efforts in Texas. Torrey grew up in Gautier, Mississippi, before his short stint in junior college.
UNT has just 12 players on its roster who are not from either Texas or Oklahoma.
4. Torrey was a rare JC transfer with three years of eligibility
One of the reasons UNT was so intrigued by Torrey during his time in junior college is because he only needed to spend one season at that level before transferring to play at the Football Bowl Subdivision level.
Torrey was a full qualifier academically out of high school in Gautier. He went to junior college to improve his options in terms of where he could continue his college career.
5. Torrey scored more touchdowns than just about anyone in major college football last season
Torrey showed a nose for the end zone in his first season at UNT, when he piled up 18 total touchdowns.
He scored 15 rushing touchdowns, caught two touchdown passes and returned a kickoff for another score last season.
Only six players scored more touchdowns than Torrey last season at the Football Bowl Subdivision level, including Clemson running back Travis Etienne and Texas A&M running back Trayveon Williams.
FOOTBALL | Former @MGCCC_FOOTBALL All-American @TorreyBoy1 is getting it done at the #NextLevel: https://t.co/iswCbjCTww
— MGCCC Athletics (@MGCCCBulldogs) December 14, 2018
6. The departure of Tashard Choice leaves Torrey looking for a new mentor
Torrey credited his success last season in part to former UNT running backs coach Tashard Choice. The former Cowboys running back developed a rapport with Torrey and pushed him to capitalize on his potential.
Choice left UNT after last season to become the running backs coach at Georgia Tech, his alma mater.
Torrey will look to develop a similar relationship with Patrick Cobbs. The UNT Hall of Fame running back took over as the Mean Green's running backs coach in the offseason.
Deandre Torrey showed some serious burst last season by gaining 51.5% of his yards on 15+ yard runs. pic.twitter.com/9TC2tgx4nK
— PFF College (@PFF_College) July 2, 2019
7. Torrey was one of UNT's most explosive players last season
Torrey gained 51.5% of his yards last season on runs of at least 15 yards, the top total among Conference USA running backs, according to Pro Football Focus.
Torrey isn't the biggest running back out there but is quick and has breakaway speed.
It's #TopMomentTuesday! Today we take a deeper look at @TorreyBoy1's 92-yard TD run in the win over FAU last season. For a breakdown of the fourth-longest run in program history ⤵
📰:https://t.co/S2bt3uFGCr#GMG #UNiTeNorthTexas pic.twitter.com/CMXhKGUi36
— North Texas Football (@MeanGreenFB) June 18, 2019
8. Torrey broke off a TD run last fall that had a dramatic impact on two teams' seasons
Torrey displayed his breakaway speed on a play that helped UNT pull out a win that dramatically impacted two teams' seasons in 2018.
The Mean Green were locked in a tight game with Florida Atlantic in a rematch of the 2017 Conference USA title game on Nov. 15 at Apogee Stadium.
UNT led 34-31 with 4:21 remaining but was pinned at its 8-yard line. FAU brought a blitz. The Mean Green had the perfect play called when it handed the ball to Torrey, who broke free for a 92-yard touchdown.
UNT picked up its eighth win of the season, while FAU left Apogee stuck at five wins. The Mean Green went on to finish 9-4. FAU never recovered from its loss to UNT, dropped its season finale against Charlotte and missed the bowl season after finishing 5-7.
Torrey was named the Conference USA Offensive Player of the Week for his performance against the Owls.
9. The best games of Torrey's season all came late in the year
Torrey was having a solid debut season at UNT through the first five games of the year, when he rushed for 128 yards and four touchdowns.
It was the rest of the season that set Torrey apart. He rushed for at least 130 yards in each of UNT's last four regular season games.
Maybe you shouldn't kick it to Deandre Torrey 😳
A 96-yard kickoff return for @MeanGreenFB! pic.twitter.com/MYNHeOiKEg
— Conference USA (@ConferenceUSA) September 2, 2018
10. Torrey is also a great kick returner
While Torrey is best known for his exploits as a running back, that is not the only way in which he contributed to the Mean Green's success last season. He was also a standout kick returner.
Torrey returned a kickoff 96 yards for a touchdown in UNT's win over SMU and averaged 27.5 yards on 12 returns.
Torrey would have ranked second in C-USA in kick return average behind Kerrith Whyte of FAU if he had enough returns to qualify for the league leaders but had just 12 attempts on the year.
BRETT VITO can be reached at 940-566-6870 and via Twitter at @brettvito.
Deandre Torrey
Seth Littrell
Unt Hall Of Fame
Follow Brett Vito
From the AP

redpajama_set_name: RedPajamaCommonCrawl


As mentioned in our previous preview blog, we have been hard at work developing a new image compression tool for the latest WP-Optimize 2.3.0 (free + paid) release. Our image compression service is an easy to use and handy imaging tool that allows you to quickly and easily optimise, compress and resize images on your website.
As unnecessarily large website images can cause frustratingly slow website loading speeds, we identified the need for a new image compression tool within WP-Optimize. Large images can impact on a websites user experience, bounce rate and SEO performance, but with our new image compression options you can improve these factors without compromising quality.
By using the best-in-class Lossy and Lossless compression techniques; WP-Optimize can offer massive savings in image file size and saves the new compressed file in your image library.
If you have a lot of images on your website, you can also use our bulk editing option to compress as many images as you want at the same time – or even set-up 'Auto-Compress', which enables you to set WP-Optimize to automatically compress your images as and when they are uploaded.
In order to achieve big savings and increase speed, WP-Optimize gives you the option of choosing either Lossy or Lossless image compression. When using image compression methods, you may find that there may be a slight loss in image quality, but rest assured it will hardly even be noticeable. The Lossy compression method can achieve greater space savings when compared to Lossless, but Lossless compression allows you to keep file data and the original image quality.
For more information on the difference between Lossy and Lossless compression and what kind of results you can expect with WP-Optimize, be sure to check out our in-depth recent blog on this subject.
WP-Optimize doesn't limit you and will let you work with your favourite kind of images, allowing you to compress NG, JPG, GIF, BMP and TIF pictures up to 5mg in size. Our image optimisation service even allows you to backup your original files so you can quickly restore them in case you make a mistake or need the original full quality image.
You can also be assured that should you wish to remove the WP-Optimize plugin for whatever reason, your compressed images will remain unaffected, allowing you to keep all your new and original content and maintain full control over your website.
Be sure to let us know any feedback comments you may have in the comment section below.
The post WP-Optimize announce release of new leading image compression feature appeared first on UpdraftPlus. UpdraftPlus – Backup, restore and migration plugin for WordPress.
As part of the continual development of WP-Optimize, we are pleased to share with you a sneak peek of our new and soon to be available image optimization tool; Smush. Smush is a handy and simple to use tool that uses the latest and best-in-class lossy compression techniques to quickly and easily optimise, compress and resize any or all the images on your website.
With unnecessary large image files often slowing website loading speeds, we saw the need for an image compression option as we know how important loading speed is for a websites user experience and SEO ranking. With our leading image compression service we hope to improve the performance of your websites, but still give you strict quality control in choosing your preferred image compression quality setting.
The feature will be launched very soon, so be sure to keep an eye out for further updates and information about Smush.
The post WP-Optimize confirms it is developing a leading image compression feature appeared first on UpdraftPlus. UpdraftPlus – Backup, restore and migration plugin for WordPress.


How ghostly was it? At least enough for Google to initially fail to acknowledge its existence. Search engine wonks began sounding the alarms as early as May 5 when certain sites' rankings began to drop dramatically, but two weeks passed before Google officially confirmed the update to Search Engine Land.
But analysts are now pulling a Scooby Doo on the Phantom, unmasking the digital ghost to reveal an update with some pretty clear outcomes. Here's what content marketers need to know about the recently exposed Phantom Update.
So what is the update?
In other words, this update is all about rewarding sites that focused on improving the user experience and pushing quality content (and punishing those that have not).
That's good news for content marketers who have been following the wise advice of industry analysts and focused on producing quality content. Search Metrics reports that sites like Amazon.com, Genius.com, and Thesaurus.com were big winners, while Examiner.com, RottenTomatoes.com, Answers.com, and WikiHow.com all took significant hits.
Has my site been affected?
At this point, it's difficult to say. If your search ranking plummeted in May, well, there's a pretty good chance the Phantom Update took its toll. If it did, you're probably wondering: How is my content not quality? It's a legitimate question, and for some, what defines quality is up for debate.
Co-founder of HubPages Paul Edmondson is among those in disagreement with Google's assessment after the user-generated topical information site's traffic tumbled.
Ultimately, the low-quality culprits likely include redundant content, thin content, self-starting videos, banner ads, and 404 errors. If you said "Oops" more than once while reading that list, your site might be affected.
Filter user-generated content: Quora is proof that Phantom/Quality/Reverse Panda isn't about punishing so-called how to content. The Q&A site's ranking jumped with the recent update.
So if user-generated content is part of your strategy, improving your moderation strategy should be in your plans.
Get rid of annoying ad formats: When you've got awesome traffic and advertisers are clamoring to hand you cash, it's mighty tempting to shove in ads anywhere you can. But here's the thing: Not only do excessive and disruptive ads (pop-ups, "above-the-fold ads," and so on) annoy visitors, they apparently really annoy Google. And where's your traffic going to come without either of them?
And for the love of all that is holy, take down those auto-play videos. Like other disruptive ad formats, they're bad for the user experience, and they're bad for your Google rank.
Trash the thin content: According to Google, thin content can include automatically generated articles; doorway pages that exist only to connect visitors to a new page; thin affiliates (ads from which the host site makes money on sales, but for which the product adds no value to the site); or thin syndication, which includes content pulled in from article banks or RSS.
Clean up your page: A gratuitous amount of comments, lots of 404 errors, and other clutter all dampen the user experience. And guess what? Google doesn't like that either. Again, this is something that should be a priority anyway, whether Google punishes it or not.
Though the change had a large impact on many sites, the quality-first principles upon which the update was apparently based on have been around for a long time. That's why Search Engine Land's Barry Schwartz says there's not much more for sites to do than what they should have been doing in the first place. "Keep focusing on building out a better web site, aimed at your users and overall quality," he writes.
Still, it's a little unclear why Google felt the need to employ all the smoke and mirrors. Perhaps Google wanted to catch sites still landing top search rankings with poor quality content. Shady? Definitely. But Google has been prioritizing quality content since its 2011 Panda update. If you got caught by the Phantom Update, you likely had it coming.


text:
Nottingham Forest ace discusses injury troubles as he sends message to Reds fans
Midfielder Samba Sow has had spells on the sidelines with knee and hamstring problems during his time at the club
Sarah Clapson
Samba Sow of Nottingham Forest
Get the Reds transfer latest, team news, match updates and analysis delivered straight to your inbox
Nottingham Forest midfielder Samba Sow has spoken of his injury frustrations, after making his comeback from a hamstring problem at Reading.
The 31-year-old was dogged by knee troubles last season, and while he is confident those are behind him, he was frustrated at having to spend another spell on the sidelines.
Sow made his seventh appearance of the campaign against the Royals, coming off the bench as the Reds were beaten 2-0.
And he is determined to build up his fitness so he can play his part in helping Chris Hughton's side move away from the difficult position they find themselves in, at the bottom end of the Championship table.
"I'm feeling good. I'm just trying to get back to my best, to get fit and to help the team as well, because we need everybody now, in this situation," Sow told NottinghamshireLive.
"But I'm feeling good.
"I was so frustrated. I had my knee problem last year, then I came back from that and had my hamstring injury.
"But it is what it is. I have to accept that and just work harder to give everything I have on the pitch.
"The knee is okay now. I feel good about my knee now.
"I can keep training every time. It was difficult last year, but I can keep training now.
"I feel good, but, for sure, I need longer and more games to be fully fit."
Saturday's match was the first time Forest had played in front of supporters since March, due to the coronavirus pandemic.
Reading were allowed 2,000 fans in the Madejski Stadium, under tier two restrictions.
Nottingham's tier three status currently means games must continue to be played behind closed doors at the City Ground.
But Sow hopes that can change soon.
"It's been a long time without fans," he said. "Football without fans is nothing. It's not the football we love, and nobody wants that.
"Everybody was happy to have fans back, even if it was only 2,000 of them.
"When it came to the warm-up, you could tell it was different with the fans in.
Johnson, Grosicki, signings - Everything Nottingham Forest boss Chris Hughton has said on January transfers
Nottingham Forest owner Evangelos Marinakis speaks about coronavirus experience as he makes donation
"It will make a difference if we can get them back at the City Ground. We need them.
"We need everybody to help turn this situation around."
The Reds sit two points above the relegation zone, in 21st spot, ahead of Wednesday's trip to Norwich City.
They have gone five games without a win and without scoring a goal, in what is proving to be a tough season after coming so close to a top six finish last time around.
"That was last year. This is a new season. Football changes a lot, and we have to accept that," Sow said.
"The reality is we are in a bad position, and that is the only thing we need to think about."
City Ground
Samba Sow
Nottingham Forest FCNottingham Forest youngster completes permanent January transferThe Reds have announced young goalkeeper Jordan Wright has completed a January move
Nottingham Forest boss sends Djed Spence message amid Premier League transfer talk
Nottingham Forest FCOn-loan Middlesbrough defender has been linked with the likes of Arsenal, Tottenham Hotspur and Leeds United
Nottingham Forest sent clear Brennan Johnson and Joe Worrall January transfer message
Nottingham Forest FCNottingham Forest fans have had their say on how the club should approach the remainder of the January transfer window
Former Nottingham Forest defender headbutted London Uber driver in unprovoked drunken attack
Nottingham Forest FCEx-Red Gregor Robertson was so drunk he was 'struggling to walk'
Nottingham Forest's extra edge which can 'drain' Derby County in City Ground showdown
Nottingham Forest FCSteve Cooper's Reds host their bitter rivals in a much-anticipated lunchtime Championship clash on Saturday
'Merciless' killer who 'took an overdose' learns his fate


1 CHAR 2/134/42 28 July 1924 A letter from TH Deakin, mining engineer, to Winston Churchill.
2 CHAR 22/155/119 8 March 1927 A secret memorandum, entitled the Franchise Question, from Winston Churchill to the other members of the Cabinet.
3 CHAR 2/157/8C-D 23 January 1928 A letter from Frances Helen Pumfrey to the editor of the Daily Mail.
4 CHAR 8/619/19 27 March 1938 A newspaper article, in the Sunday Chronicle, entitled 'WOMEN can win WARS – without firing a shot' by Winston Churchill.
5 CHAR 23/14/33 7 July 1945 A secret memorandum, entitled Man-Power, from Winston Churchill to the other members of the Cabinet.


Smooth animations help make a UI feel great. Using Vue's transition system and re-usable functional components, you can easily control the motion of your application. Most components can have their transition altered through the transition prop.
Slide x transitions move along the horizontal axis.
Many of Vuetify's components contain a transition prop which allows you to specify your own.
Another example of the fade transition can be found on the Carousel component.
Programmatically control the transition origin with a simple prop.


While I never met Seymour, I have very much enjoyed getting to know so many of those he knew and worked with from his daughter Artemis to Brian Silverman, Gary Stager to Mitch Resnick (and no doubt many others that I know but didn't know he directly influenced too). They have been extremely supportive of CBM, our recent Wolfram Language and Mathematica, of helping us to take Seymour's and their thinking, successes, mistakes and experience to try to shift to a much better educational world.
That this post is brief is an indication that others can much better enunciate the broad array of Seymour's achievements than can I. Indeed I am already learning more as they do.
However about one aspect I am quite clear. Seymour's death does not end any aspect I've come across of his vision. Quite to the contrary, we are instead starting to see its manifestation more than ever: a mixed computer-human approach, recasting of subjects, increasing pressure for change in education (frustratingly slow though it can seem). I expect Seymour's fame outside education circles will posthumously grow.
We will work hard to carry key strands of his work forward.


text:
Hawkensian
Tag: frustrating
'Thirteen' will have you wanting to commit acts of extreme violence.
This film evokes a veritable plethora of negative emotion; I hated almost everything that occurred, but I didn't hate the film. This film explores the anguish caused by the immaturity of adolescence: the competition, the bullying, the need for conformity. Its characters are like those found in Larry Clark films, however 'Thirteen' has none of the sleazy, gratuitous perversion that Clark relishes. This film explores all the vices, it depicts the worst of impressionable adolescence, but the camera never lingers luridly.
The story is familiar, one that everyone can relate to. It's about Tracy (Evan Rachel Wood), a painfully impressionable teen who desperately wants to fit in with the 'cool' crowd, who ironically are anything but. The 'cool' kids are repugnantly crass – truly revolting people. Tracy unfortunately thinks otherwise, becoming best friends with the utterly amoral prize bitch Evie (Nikki Reed). Her influence upon Tracy is immediate, and their activities together soon escalate; Tracy quickly becomes a scheming, stupid and cruel too. Her descent is exasperating to watch, partly through how powerless Tracy's suffering mother (Holly Hunter) is.
The performances are remarkably natural, they really are. It will never age in this respect, peak realism is shown here. Hunter and Wood have scenes of intense emotion, the kind of emotion so desperately strong that it transcends everything. The excitable exchanges between Wood and Reed are also commendable for their credibility.
The film could be a eye-opener for teens that are experiencing similar situations. The film would assure them that Tracy isn't radical in any way, it would highlight that she's just in the midst of a predictable, cringe-worthy cycle that has been experienced by millions of all generations.
'Thirteen' will have you wanting to commit acts of extreme violence. If you can restrain your temperament throughout the many scenes of vast obnoxiousness, gross injustice and the scores of vulgar, idiotic characters – I salute you. All of this, however, is a testament to the power of the film. It's an unpleasant watch, but the provocation it causes is interesting, I've certainly written this review faster than usual.
Author hawkensianPosted on February 15, 2013 January 25, 2015 Categories 2000s, 70 - 79% Good, Film ReviewsTags 2003, adolescent, alcohol, anger, angst, divorce, drugs, film reviews, frustrating, high school, impressionable, infuriating, peer pressure, school, secondary school, sex, single parent, step father, stepdad, teenage girl, teenager, thirteenLeave a comment on Thirteen (2003)
The Loved Ones (2009)
'The Loved Ones' is ultimately an exercise in frustration and indignation.
'The Loved Ones' is a well-made film, but it's also hollow and nasty. It takes a developed, sympathetic character and subjects him to an array of torture and humiliation at the hands of Lola and 'Daddy', a vile father/daughter serial killing partnership.
The film drew significant parallels with 'The Texas Chainsaw Massacre', a film I like. The difference between the two is that 'The Loved Ones' is far crueler; the protracted scenes of humiliation and violence left me utterly indignant. 'Chainsaw Massacre' is also an iconic, genre defining piece of work; this film is merely one of its many imitators.
Tough films such as the fellow Australian horror 'Wolf Creek' serve as an endurance test; they're full of tension, suspense and eventually unflinching brutality. They're straightforward and not particularly good, but they do have a purpose, albeit a doubtful one. 'The Loved Ones' however is just overbearingly frustrating; how can anyone derive anything but negative emotion out of watching an innocent character being tied to a chair and tortured?
As I watched their exploits, I realised that Lola and 'Daddy' were such reprehensible characters that no come-uppance would be satisfying enough, my growing bloodlust would have only been satiated if I'd been able to jump into the frame and exact hyper-violent justice myself.
The effect the film had on me is clearly testament to the power of it. There's no doubt that it's taut, well made and well-acted, however 'The Loved Ones' is ultimately an exercise in frustration and indignation.
Author hawkensianPosted on February 12, 2013 June 28, 2014 Categories 2000s, 50 - 59% Pooey, Film ReviewsTags 2009, australia, australian, australian film, film reviews, frustrating, horror, horror film, indignant, john brumpton, robin mcleavy, sadistic, the loved ones, the texas chainsaw massacre, torture porn, xavier samuelLeave a comment on The Loved Ones (2009)
DMovies portfolio
Stalker (1979)
The 10 Most Underrated Movie Villains
Hacksaw Ridge (2016)
The Coming War on China (2016)
How the Father of Organised Crime Won His Freedom by Helping the US Government during WW2
The Alan Clarke Collection: Disruption
30 – 39% Just awful
40 – 49% Bad
50 – 59% Pooey
60 – 69% Okay
70 – 79% Good
80 – 89% Great
90 – 94% Excellent
95%+ Masterpiece
Blogs Quizzes and Miscellaneous
Forgotten and Unappreciated
Remembered and Over-appreciated
Zero Day (2003)
American Psycho (2000)
Sexy Beast (2000)
Monster (2003)
Bad Taste (1987)
ray winstone
Hawkensian Create a free website or blog at WordPress.com.


Course in Miracles promoter, Marianne Williamson, who is running in the 2020 presidential election, has big plans for the children of this country if she is elected. Among other things, part of those plans is to make sure children in American public schools are trained in mindfulness meditation. While tens of thousands of public schools in the U.S. are already teaching children mindfulness meditation, no doubt, with a New Age president, every school would be including it.
Williamson also promises to advocate for "universal pre-K." This would help make sure that children as young as 3 and 4 would be placed into the public school system. Advocates of universal pre-K believe they need more time with America's children and want to get their hands on them much earlier than 5 or 6 (Kindergarten).
Parents beware. If Marianne Williamson (or another candidate with her "values,") becomes president, this country's children will be at even greater risk than they are now. And you can be sure, she will not be advocating for homeschooling families.
She is a pseudo- Christian – and you are deceived.
Williamson is the average New Ager; a control freak, and delusional. Okay, so if she seized power via the power of the universe (which is probably the equivalent of a spent 9-Volt battery), she'd run things her way, or . . . what? What if she took over and people refuse to bow down to her New Age gods?
The objectives of and comparisons between New Agers and another manmade system are eerily similar. Once upon a time, there lived a vicious murderer, a spreader of blasphemous falsehoods. This control freak also had his crazy ideas and had tried to rule the world his way. But the devil's righthand man Calvin could not turn Geneva into the New Jerusalem (for thousands of obvious reasons). It was a "big fail," as the youngsters would say today.
There is one force at play; it has different faces.
Williamson poses as big a danger to society/the world as any false religion/cult. If Americans use their intelligence, this Williamson should get nowhere but Cuckooland where she can play with the other little control freaks.
A country should be lead by a proper leader; a strong president, one who has the needs of his people at the top of his agenda, and not one who has his/her own needs at the top of the agenda. Has someone said "Hitler"? "Calvin"? No difference.
in as much as ye have judged another ye have done it unto thee…Marianne has helped thousands of souls, and women heal from trauma, sexual abuse, addictions, and much more….all of you judgemental christians here, why dont u go out onto the streets and preach to hookers, drug addicts, lost souls and stop projecting your own beliefs, please! mindfulness meditation has helped hundreds of thousands of people worldwide, it speaks for itself! my god cant u do any better than this low vibratory one sentence judgements! how sad!
My grandchildren will not be apart of this insanity.You won't win!
She is not a New Ager; she was asked this question and she definitely said no. It's a great misunderstanding among many people. She cares about children's needs which have been greatly neglected in our society, unfortunately. FYI – mindfulness training is basically helping children to be in the present moment — not being attached to the past nor being stuck with the future (both are the reasons for fear and anxiety). Being in the present is where love is — love is not in fear and anxiety). They are encouraged to appreciate life now in the moment and pay attention to what is happening right now as often as possible. A Course in Miracles (ACIM) is all about removing blocks to awareness of love's presence and forgiving — the foundation of inner peace. This is the aim of ACIM. It's beautiful and greatly needed by all of us. Let people hear her views and decide for themselves. Peace.
Williamson and Oprah are New Age poster persons and have been for the longest time.
I think you better rethink your statement about Marianne not being a New Ager. The teachings of A Course in Miracles contradict Holy Scripture, Tradition, and Authentic Biblical Teaching; claiming that: Jesus was NOT God, the devil does not exist, and there's no such thing as evil, or sin."as the Course teaches that man's fall from Grace in the Garden of Eden never happened— and that Holy Scripture cannot be trusted, as Christ's apostles misinterpreted his message and taught in error.pgs. These false tenets alone clearly show that the channeled spirit who wrote them, while claiming to be Jesus, is in fact, a counterfeit Christ .
The teachings of A Course in Miracles have FAILED many, if not all, of the standard "tests of discernment" the Church has used for centuries, to distinguish a true and Godly spirit from a false one.
She is a scary person! We must protect our babies and children from this woman.


text:
\section{Introduction}

In this paper, we investigate the \emph{misinformation filter bubble} creation and bursting on YouTube. In our \emph{auditing study} we simulate user behavior on the YouTube platform, record platform responses (e.g., search results, recommendations) and manually annotate them for the presence of misinformative content. Then, we quantify the dynamics of misinformation filter bubble creation and also dynamics of bubble bursting, which is the novel aspect of the study. With this paper, we publish the implementation of the experimental infrastructure and also the data we collected\footnote{Available at \url{https://github.com/kinit-sk/yaudit-recsys-2021}}.

Our study adds to the previous works~\cite{Hussein2020, Papadamou2020, Spinelli2020, AbulFottouh2020, Ribeiro2020} that used \emph{audits} to quantify the portion of misinformative content being recommended on social media platforms. We directly build on works~\cite{Hussein2020, Papadamou2020, Spinelli2020} that observed and quantified the creation of misinformative filter bubbles on YouTube.

The general motivation of our work is to emphasize the \emph{need for independent oversight of personalization behavior of large platforms}. In the past, platforms have been accused of being contributors to the misinformation spreading due to their personalization routines. Simultaneously, they have been reluctant to revise these routines~\cite{zuboff2019age, vaidhyanathan2018antisocial}. And when they promise some changes, there is a lack of effective public oversight that could quantitatively evaluate their fulfillment. Auditing studies are tools that may improve such oversight.

While previous works investigated how a user can enter a filter bubble, no audits have covered \emph{if}, \emph{how} or with what \emph{``effort''} can the user ``burst'' (exit or lessen) the bubble.
Multiple studies demonstrated that watching a series of misinformative videos would strengthen the further presence of such content in recommendations~\cite{Hussein2020, Papadamou2020, AbulFottouh2020}, or that following a path of the ``up next'' videos can bring the user to a very dubious content~\cite{Spinelli2020}. However, no studies investigated what type of user's watching behavior (e.g., switching to credible news videos or conspiracy debunking videos) would be needed to lessen the amount of misinformative content recommended to the user.
Such knowledge would indeed be valuable. Not just for the sake of knowledge about the inner workings of YouTube's personalization, but also to improve the social, educational, or psychological strategies for building up resilience against misinformation.

\begin{quote}
    \it
    \textbf{As the first contribution}, this paper reports on the behavior of YouTube's personalization in a situation when a user with misinformation promoting watch history (i.e., with a developed misinformation filter bubble) starts to watch content debunking the misinformation (in an attempt to burst that misinformation filter bubble). The key finding is that watching misinformation debunking videos (e.g., credible news, scientific content) generally improves the situation (in terms of recommended items or search result personalization), albeit with varying effects and forms, mainly depending on particular misinformation topic.
\end{quote}

We aligned our methodology with previous works, most notably with the work of Hussein et al.~\cite{Hussein2020} who also investigated the creation of misinformation filter bubbles using user simulation. \emph{As part of our study, we replicated parts of Hussein's study}. We have done this for the sake of replication and to bootstrap bots with history of watching misinformation promoting videos. We re-used maximum of Hussein's seed data (topics, queries, videos), used similar scenarios and the same data annotation scheme. Therefore, we were able to directly compare the outcomes of both studies (e.g., on the number of observed misinformative videos present in recommendations or search results). Due to recent changes in YouTube policies~\cite{YouTube2020policies}, we expected to see less filter bubble creation behavior than Hussein et al. However, this was generally not the case.

\begin{quote}
    \it
    \textbf{As the second contribution}, we report changes in misinformation video occurrences on YouTube, which took place since the study of Hussein et al.~\cite{Hussein2020} (mid 2019). We observe worse situation regarding the topics of vaccination and (partially) 9/11 conspiracies and some improvements (less misinformation) for moon landing or chemtrails conspiracies.
\end{quote}

\section{Background: Filter Bubbles and Misinformation}

To some extent, \emph{intellectual isolation} is a natural human defense against information overload~\cite{mutz2011communication} and provides us with stronger inner confidence~\cite{festinger1957theory}. However, it also comprises negative effects such as selective exposure (focusing on information that is in accordance with one's worldview) or confirmation bias~\cite{del2016spreading, RePEc:now:jlqjps:100.00016037}. In social media, intellectual isolation contributes to the creation of \emph{echo chambers}~\cite{bessi2016personality}: the same ideas are repeated, mutually confirmed and amplified in relatively closed homogeneous groups. Polarization and fragmentation of the society increases~\cite{journals/corr/ZolloNVBMSCQ15,sunstein1999law}.

The negative effects of echo chambers can be amplified by \emph{filter bubbles}. Filter bubbles (as states of intellectual isolation) were firstly recognized by Pariser~\cite{pariser2011filter} as a negative consequence of personalization in social media and search engines. Researchers~\cite{pariser2011filter, sunstein1999law} agree that algorithms of such platforms support cognitive bias, as users are presented with the content that complies with their hitherto attitudes. Besides that, this effect also has ethical implications. Users are often unaware of the existence of filter bubbles, as well as of the information that was filtered out. Moreover, personalization and recommendation tailored to the users' interests can escalate the problems with misinformation~\cite{Spinelli2020}.

\emph{Misinformation} is a false or inaccurate information that is spread regardless of an intention to deceive. Due to significant negative consequences of misinformation on our society (especially during the ongoing COVID-19 pandemic), tackling misinformation attracted a plethora of research efforts (see~\cite{Zannettou2019,Zhou2020} for recent surveys). While the majority of such research focuses on various characterization studies~\cite{simko2021study} or detection methods~\cite{pecher2021fireant,srba2019monant}, the studies investigating the relation between misinformation and adaptive systems are still relatively rare (e.g.,~\cite{Hussein2020,Papadamou2020}).

We denote filter bubbles that are characterized by the presence of misinformative content as \emph{misinformation filter bubbles}. They are states of intellectual isolation in false beliefs or a manipulated perceptions of reality. Analogically to \emph{topical} filter bubbles, misinformation filter bubbles can be characterized by a high homogeneity of recommendations/search results that share the same positive stance towards misinformation. In other words, the content adaptively presented to a user in a misinformation filter bubble supports one or several false claims/narratives. The proportion of such content represents how deep inside the bubble the user is.

To prevent misinformation and misinformation filter bubbles, social media conduct various countermeasures. These are usually reactions to public outcry or are required by legislation, e.g., EU's Code of practice on disinformation\footnote{\url{https://digital-strategy.ec.europa.eu/en/policies/code-practice-disinformation}}. Currently, the effectiveness of such countermeasures is evaluated mainly by self-evaluated reports. However, such reports are difficult to verify since social media are reluctant to provide access to their data for independent research.

The verification of countermeasures is further complicated by interference of psychological factors. For example, some researchers argue that cognitive bias is more influential than algorithms when it comes to intellectual isolation~\cite{del2016spreading, bakshy2015exposure}. To separate these influences, researchers employ platform \emph{audits}, such as the one in this paper.

\section{Related work: Audits of Adaptive Systems}

In this context, an audit is a systematic statistical probing of an online platform, used to uncover socially problematic behavior underlying its algorithms~\cite{Sandvig2014Audits,Hussein2020}. Audits come in multiple forms~\cite{Sandvig2014Audits} and two of them are also suitable to investigate the effect of (misinformation) filter bubbles: \emph{crowdsourcing audits} and \emph{sockpuppeting audits}.

Crowdsourcing audit studies are conducted using real user data. Silva et al.~\cite{Silva2020} developed a browser extension to collect personalized ads with real users on Facebook. Hannak et al.~\cite{Hannak2013} recruited Mechanical Turk users to run search queries and collected their personalized results. However, such auditing methodology suffers from a lack of isolation (users may be influenced by additional factors, e.g. confirmation bias). Moreover, uncontrolled environment makes comparisons difficult or unfeasible; it is difficult to keep users active; audits also raise several privacy issues.

Sockpuppeting audits solve these problems by employing non-human bots that impersonate the behavior of users in a predefined controlled way~\cite{Sandvig2014Audits}. To achieve representative and meaningful results in sockpuppeting audits, researchers need to tackle several methodological challenges~\cite{Hussein2020}. First is the selection of appropriate seed data (e.g., the initial activity of bots, search queries). Second, the experimental setup must measure the real influence of the investigated phenomena. At the same time, it must minimize confounding factors and noise (e.g., of name, gender or geolocation~\cite{Hannak2013}). Another challenge is how to appropriately label the presence of the audited phenomena (expert-based/crowdsourced~\cite{Hussein2020,Silva2020} or automatic labeling~\cite{Papadamou2020} can be employed).

Audits can be further distinguished by the social media they are applied on (e.g., social networking sites~\cite{Silva2020,Papadamou2020,Hussein2020}, search engines~\cite{Metaxa2019,Le2019,Robertson2018}, e-commerce sites~\cite{Juneja2021}), by adaptive systems being investigated (e.g., recommendations~\cite{Hussein2020,Spinelli2020,Papadamou2020}, up-next recommendation~\cite{Hussein2020}, search results~\cite{Papadamou2020,Hussein2020,Le2019,Metaxa2019,Robertson2018}, autocomplete~\cite{Robertson2018}) and by phenomena being studied (e.g., misinformation~\cite{Hussein2020,Papadamou2020}, political bias~\cite{Le2019,Metaxa2019}, political ads~\cite{Silva2020}). In our study, we focus specifically on misinformation filter bubbles in the context of the online video platform YouTube and its recommender and search system. As argued by Spinelli et al.~\cite{Spinelli2020}, YouTube is an important case to study as a significant source of socially-generated content and because of its opaque recommendation policies.
Some information about the inner workings of YouTube adaptive systems are provided by research papers published at RecSys conference~\cite{Covington2016,Zhao2019} or blogs~\cite{YouTube2020policies} published directly by the platform, nevertheless, a detailed information is unknown. Therefore, we feel a need to conduct independent auditing studies on undesired phenomena like unintended creation of misinformation filter bubbles.

The existing studies confirmed the effects of filter bubbles in YouTube recommendations and search results. Spinelli et al.~\cite{Spinelli2020} found that chains of recommendations lead away from reliable sources and toward extreme and unscientific viewpoints. Similarly, Ribeiro et al.~\cite{Ribeiro2020} concluded that YouTube's recommendation contributes to further radicalization of users and found paths from large media channels to extreme content through recommendation. Abul-Fottouh et al.~\cite{AbulFottouh2020} confirmed a homophily effect in which anti-vaccine videos were more likely to recommend other anti-vaccine videos than pro-vaccine ones and vice versa.

Recently, we can observe first audits focused specifically on misinformation filter bubbles. Hussein et al.~\cite{Hussein2020} and Papadomou et al.~\cite{Papadamou2020} found that YouTube mitigates pseudoscientific content in some handpicked topics such as COVID-19. Hussein et al.~\cite{Hussein2020} found that demographics and geolocation (within the US) affect personalization only after having acquired some watch history. These studies provide evidence of the existence and properties of misinformation filter bubbles on YouTube. From the properties that remain uninvestigated, we specifically address two. Firstly, the adaptive systems used by YouTube are in continuous development and improvement. Information on how YouTube proceeds in countering misinformation is needed. Secondly, while the existing studies focused on misinformation filter bubble creation, we do not have the same perspective on the inverse process -- filter bubble bursting.
\section{Study design and methodology}

To investigate the dynamics of bursting out of a misinformation filter bubble, we conducted an agent-based  sockpuppeting audit study. The study took place on YouTube, but its methodology and implementation can be generalized to any adaptive service, where recommendations can be user-observed.

In the study, we let a series of agents (bots) pose as YouTube users. The agents performed pre-defined sequences of video watches and query searches. They also recorded items they saw: recommended videos and search results. The pre-defined actions were designed to first \emph{invoke the misinformation filter bubble effect} by purposefully watching videos with (or leaning towards) misinformative content. Then, agents tried to \emph{mitigate the bubble effect} by watching videos with trustworthy (misinformation debunking) content. Between their actions, the agents were idle for some time to prevent possible carry-over effects. The degree of how deep inside a bubble the agent is was observed through the number and rank of misinformative videos offered to them.

The secondary outcome is the partial replication of a previous study done by Hussein et al.~\cite{Hussein2020} (denoted onwards as the \emph{reference study}). This replication allowed us to draw direct comparisons between quantities of misinformative content that agents encountered now (March 2021) and during the reference study done in mid 2019.

\subsection{Research Questions, Hypotheses and Metrics}
\label{ssec:research_questions}

\textbf{RQ1 (comparison to the reference study):} \emph{Has YouTube's personalization behavior changed with regards to misinformative videos since the reference study?} In particular, we seek to validate the following hypothesis:
\begin{itemize}
    \item \textbf{H1.1:} Compared on \emph{SERP-MS} and \emph{normalized score} metrics (see below), we would see better scores (after constructing a promoting watch history) than in the reference study in both search and recommendations (given YouTube's pledges~\cite{YouTube2020policies}).
\end{itemize}

\textbf{RQ2 (bubble bursting dynamics):} \emph{How does the effect of misinformation filter bubbles change, when debunking videos are watched?} The ``means of bubble bursting'' would be implicit user feedback -- watching misinformation debunking videos. In particular, we seek to validate the following hypotheses:
\begin{itemize}
    \item \textbf{H2.0:} Watching videos belonging to promoting misinformation stance leads to their increased presence in both search results and recommendations (worse SERP-MS and normalized score metrics).
    \item \textbf{H2.1:} Watching the sequence of misinformation debunking videos after the sequence of misinformation promoting videos will improve the metrics \emph{in comparison to the end of the promoting sequence}.
    \item \textbf{H2.2:} Watching the sequence of misinformation debunking videos after the sequence of misinformation promoting videos will improve the metrics \emph{in comparison to the start of the experiment}.
\end{itemize}

The metrics we use -- \emph{SERP-MS} and \emph{normalized score} -- are drawn directly from the reference study. Both metrics quantify misinformation prevalence in a given list of items (videos), which are annotated as either \emph{promoting} (value 1), \emph{debunking} (value -1) or \emph{neutral} (value 0). The output of both metrics is, similarly, from the $\left \langle -1, 1  \right \rangle$ interval. Lists populated mostly with debunking content would receive values close to -1, with promoting close to 1 and with balanced or mostly neutral, close to 0. In other words, a score closer to -1 means better score.
\begin{description}
    \item [Normalized score.] A metric computed as average of individual annotations of items present in the list. It is suited for unordered, shorter lists (in our case, recommendations).
    \item [SERP-MS (Search result page misinformation score).] A metric capturing amount of misinformation and its rank. It is suited for longer, ordered lists (in our case, search results). It is computed as $SERP\mbox{-}MS = \frac{\sum_{r=1}^{n}(x_i*(n-r+1))}{\frac{n*(n+1)}{2}}$, where $x_i$ is annotation value, $r$ search result rank and $n$ number of search results in the list~\cite{Hussein2020}.
\end{description}

\subsection{Experiments scenarios}

We let agents interact with YouTube following a \emph{scenario} composed of four phases, as depicted in Figure \ref{fig:agent-scenario}.

\emph{Phase 0: Agent initialization.} At the start of a run, the agent fetches its desired configuration, including the YouTube user account and various controlled variables (the variable values are explained further below). Also, the agent fetches $\tau \in T$, a topic with which it will work (e.g., ``9/11''). The agent fetches $V_{prom}$ and $V_{deb}$, which are lists of $n_{prom}=40$ and $n_{deb}=40$ most popular videos promoting, respectively debunking, misinformation within topic $\tau$. Afterward, it fetches $Q$, a set of $n_{q}=5$ search queries related to the particular $\tau$ (e.g., ``9/11 conspiracy``). The agent configures and opens a browser in incognito mode, visits YouTube, logs in using the given user account, and accepts cookies. Finally, the agent creates a neutral baseline by visiting the homepage and saving videos, and performing a search phase. In the \emph{search phase}, the agent randomly iterates through search queries in $Q$, executes each query on YouTube, and saves the search results. To prevent any carry-over effect between search queries, the agent waits for $t_{wait}=20$ minutes after each query.

\emph{Phase 1 (promoting): Create the filter bubble.} For creating a filter bubble effect, the agent randomly iterates through $V_{prom}$ and ``watches'' each video for $t_{watch}=30$ minutes (or less, if the video is shorter). Immediately after watching a video, the agent saves video recommendations on that video's page and visits the YouTube homepage, saving video recommendations listed there as well. After every $f_{q}=2$ videos, the agent performs another search phase.

\emph{Phase 2 (debunking): Burst the filter bubble.} The agent follows the same steps as in phase 2. The only difference is the use of $V_{deb}$ instead of $V_{prom}$.

\emph{Phase 3: Tear-down.} In this phase, the agent clears YouTube history (using Google's ``my activity`` section), making the used user account ready for the next run.

For each selected topic, we run the scenario 10 times (in parallel). This way, we were able to deal with recommendation noise present at the platform. In order to run our experiments multiple times, we used the \emph{reset} (delete all history) button provided by Google instead of creating a new user profile for each run.
Before deciding to use the \emph{reset} button in our study, we first performed a short verification study to see whether using this button really deletes the whole history and resets the personalization on YouTube. We randomly selected few topics, from which we manually watched few videos (5 for each). Then, we used the reset button and evaluated the difference between videos appearing on the YouTube homepage, recommendations, and search. We found no carry-over effects.

\begin{figure*}
\centering
\includegraphics[width=\textwidth]{images/agent-scenario-small.pdf}
\caption{Agent scenario for creating and bursting misinformation filter bubbles}
\label{fig:agent-scenario}
\Description[The figure illustrates the bot scenario described in text]{The figure illustrates the four phases of the bot scenario, as is described in the text, in a form of diagram.}
\end{figure*}

We needed to set up several attributes of agents (e.g., YouTube user profiles). For \emph{geolocation}, we use N.~Virginia to allow for better comparison with the reference study. The date of birth for all accounts was arbitrarily set to 6.6.1990 to represent a person roughly 30 years old. The gender was set as ``rather not say'' to prevent any personalization based on gender. The names chosen for the accounts were composed randomly of the most common surnames and unisex given names used in the US.

There were also \emph{process parameters} that we needed to keep constant. These include 1) $n_{prom}=40$ and $n_{deb}=40$ representing the number of seed videos used in promoting and debunking phases; 2) $t_{watch}=30$ representing the maximum watching time in minutes for every video; 3) $n_{q}=5$ representing the number of queries used; 4) $t_{wait}=20$ representing the wait time in minutes between query yields and 5) $f_{q}=2$ representing the number of videos to watch between search phases.

Values of the \emph{process parameters} greatly influence the total running time and results of the experiment. Yet, determining them was not straightforward given many unknown properties of the environment (first and foremost YouTube's algorithms). For example, prior to the experiment, it was unclear how often we need to probe for changes in recommendations and search result personalizations to answer our research questions.

Therefore, \emph{we run a pre-study in which we determined the best parameter setup}. Measuring the Levenshtein distance between ordered results and overlap of lists of recommended videos we determined to run 10 individual agents for each topic, as we observed instability between repeated runs (e.g., the same configuration yielded \(\sim70\%\) of the same recommended videos). For the $n_{prom}$ and $n_{deb}$ parameters, we observed that in some cases, a filter bubble could be detected after 20 watched videos. Yet in others, it was 30 or more. Due to this inconsistency, we opted to watch 40 videos for a phase. To determine the optimal value of $t_{watch}$, we first calculated the average running time of our seed videos. Most of the videos (\(\sim85\%\)) had a running time of about 30 minutes or shorter, so 30 minutes became the baseline value. In addition, we compared the results obtained by watching only 30 minutes with results from watching the whole video regardless of its length, but found no apparent differences.

To determine the number of queries $n_{q}$ and periodicity of searches $f_{q}$, we ran the scenario with all seed queries introduced by the reference study and used them after every seed video. We observed that the difference in search results between successive seed videos was not significant. As the choice of search queries and the frequency of their use greatly prolonged the overall running time of the agents, we opted to run the search phase after every second video. In addition, we opted to use only 5 queries per topic.

The only parameter not set by a pre-study is $t_{wait}$, which we set to 20 minutes based on previous studies. These found that the carry-over effect (which we wanted to avoid) is visible for 11 minutes after the search~\cite{Hannak2013, Hussein2020}.

\subsection{Seed Data}

We used 5 topics in our study (same as the reference study): 1) \emph{9/11 conspiracies} claiming that authorities either knew about (or orchestrated) the attack, or that the fall of the twin towers was a result of a controlled demolition, 2)\emph{moon landing conspiracies} claiming the landing was staged by NASA and in reality did not happen, 3) \emph{chemtrails conspiracy} claiming that the trails behind aircraft are purposefully composed of dangerous chemicals, 4) \emph{flat earth conspiracy} claiming that we are being lied to about the spherical nature of Earth and 5) \emph{vaccines conspiracy} claiming that vaccines are harmful, causing various range of diseases, such as autism. The narratives associated with the topics are \emph{popular} (persistently discussed), while at the same time, \emph{demonstrably false}, as determined by the reference study~\cite{Hussein2020}.

For each topic, the experiment required two sets of seed videos. The \emph{promoting} set, used to construct a misinformation filter bubble (its videos have a promoting stance towards the conspiratorial narrative or present misinformation). And the \emph{debunking} set, aimed to burst the bubble (and contains videos disproving the conspiratorial narratives).

As a basis for our seed data sets we used data already published in the reference study, which the authors either used as seed data, or collected and annotated. To make sure we use adequate seed data, we re-annotated all of them.

The number of seed videos collected this way was insufficient for some topics (we required twice as many seed videos as the reference study). To collect more, we used an extended version of the seed video identification methodology of the reference study. Following is the list of approaches we used (in a descending order of priority): YouTube search, other search engines (Google search, Bing video search, Yahoo video search), YouTube channel references, recommendations, YouTube homepage, and known misinformation websites. To minimize any biases, we used a maximum of 3 videos from the same channel.

As for search queries, we required fewer of them than the reference study. We selected a subset based on their popularity on YouTube. Some examples of the used queries are: "\emph{9/11 conspiracy}", "\emph{Chemtrails}", "\emph{flat earth proof}", "\emph{anti vaccination}", "\emph{moon landing fake}".

\subsection{Data collection and annotation}
\label{sec:annotation-methodology}

Agents collect videos from three main components on YouTube: 1) \emph{recommendations} appearing next to videos presently watched, 2) \emph{home page} videos and 3) \emph{search results}. In case of recommendations, we collect 20 videos that YouTube normally displays next to a currently watched video (in rare cases, less than 20 videos are recommended). For home page videos and search results, we collect all videos appearing with the given resolution, but no less than 20. In case when less than 20 videos appear, the agent scrolled further down on the page to load more videos.

For each video encountered, the agent collects metadata: 1) \emph{YouTube video ID}, 2) \emph{position} of the video in the list, and 3) \emph{presence of a warning/clarification message} that appears with problematic topics such as COVID-19. Other metadata, such as video \emph{title}, \emph{channel} or \emph{description}, are collected using the YouTube API.

To annotate the collected videos for the presence of misinformation, we used an extended version of the methodology proposed in the reference study. Each video was viewed and annotated by the authors of this study using a code ranging from -1 to 10. The videos are annotated as \emph{debunking} (code -1), when their narrative provides arguments against the misinformation related to the particular topic (such as \emph{"The Side Effects of Vaccines - How High is the Risk?"}), \emph{neutral} (code 0) when the narrative discusses the related misinformation but does not present a stance towards it (such as \emph{"Flat Earthers vs Scientists: Can We Trust Science? | Middle Ground"}), and \emph{promoting} (code 1), when the narrative promotes the related misinformation (such as \emph{"MIND BLOWING CONSPIRACY THEORIES"}). The codes 2, 3, and 4 have the same meaning as codes -1, 0, and 1, but are used in cases when they discuss misinformation not related to the topic of the run (e.g., video dealing with climate crisis misinformation encountered during a flat earth audit). The code 5 is applied to videos that do not contain any misinformation views (such as \emph{"Gordon's Guide To Bacon"}). This includes completely unrelated videos (e.g., music or reality show videos), but also videos that are related to the general audit topic, but not misinformation (e.g., original news coverage of 9/11 events). In rare cases of videos that are not in English and do not provide English subtitles, code 6 is assigned. Also rare are the cases when the narrative of the video cannot be determined with enough confidence (code 7). Videos removed from YouTube (before they are annotated) are coded as 8. Finally, as an extension of the approach used in the reference study, we use codes 9 and 10 to denote videos that specifically mention misinformation but rather than debunk them, they mock them (9 for related misinformation, 10 for unrelated misinformation, for example \emph{"The Most Deluded Flat Earther in Existence!"}). Mocking videos are a distinct (and often popular) category, which we wanted to investigate separately (however, for the purposes of analysis, they are treated as debunking videos).

To determine how many annotators are needed per video, we first re-annotated the seed videos released by the reference study. Each was annotated by at least two authors, and the annotations were compared between each other and with annotations from the reference study. We achieved Cohen's kappa value of 0.815 between us and 0.688 with the reference study. We identified characteristics of edge cases. Following the re-annotation and the findings from it, when annotating our collected videos, we assign only one annotator per collected video with instructions to indicate and comment if an edge case video is encountered. These were then reviewed by another annotator.

For the purpose of this study and to evaluate our hypotheses, we annotated the following subset of collected videos:
\begin{itemize}
    \item All recorded \emph{search results}.
    \item Videos recommended for first 2 seed videos at the start of the run and last 2 seed videos of both phases (resulting in 6 sets of annotated videos per topic). This selection was a compromise between representativeness, correspondence to the reference study, and our capacities.
    \item We have \emph{not} annotated the \emph{home page videos} for the purpose of this study. These videos were the most numerous, the most heterogeneous, and with little overlap across bots and seed videos.
\end{itemize}

\subsection{Data ethics assessment}

To consider various ethical issues regarding the research of misinformative content, we carried out a series of data ethics workshops. We explored questions related to data ethics issues~\cite{Tranberg2020Dataethics} within our audit and its impact on stakeholders. Based on the topics that emerged during the data ethics workshops, we identified different stakeholder groups. The most affected ones were platform users, annotators, content creators, and other researchers. For every stakeholder group, we devised different engagement strategies and specific action steps. Our main task was to devise countermeasures to the most prominent risks that could emerge for these stakeholder groups.

First, we were concerned about the risk of unjustified flagging of the content as misinformation and their creators as conspirators. To minimize this risk, we decided to report hesitations in the annotation process. These hesitations were consequently back-checked by other annotators and independently validated until the consensus was reached. One of our main concerns was also not to harm or delude other users of the platform. To avoid disproportional boost of the misinformation content by our activity, we select the videos with at least 1000 views and warn annotators not to watch videos online more than one time, or in case of back-checks, two times. After each round, we reset user account and delete the watch history.

Other concerns were connected to the deterioration of well-being of human annotators. Specifically, that their decision-making abilities would be negatively affected after a long annotation process. We proposed the daily routines for annotation, including the breaks during the process and advised to monitor any changes in annotators beliefs. Our annotators also underwent the survey on their tendency to believe in conspiracy theories\footnote{\url{https://openpsychometrics.org/tests/GCBS/}} and none of them showed such tendency at the end of the study.

\subsection{A note on comparability with the reference study by Hussein et al.}
\label{comparison-explained}

In order to be able to draw comparisons, we kept the methodology of our study as compatible as possible with the previous study by Hussein et al.~\cite{Hussein2020}. We shared the general approach of prompting YouTube with implicit feedback: both studies used similar scenarios of watching a series of misinformation promoting videos and recording search results and recommended videos. We re-used the topics, a subset (for scaling reasons) of search queries, and all available seed videos (complementing the rest by using a similar approach as the reference study). Moreover, both studies used the same coding scheme, metrics, sleep times, and annotated a similar number of videos.

We should also note differences between the studies, which mainly source from different original motivations for our study. For instance, no significant effects of demographics and geolocation of the agents were found in the reference study, so we only controlled these. In Hussein's experiments, all videos were first ``watched'' and only then all search queries were fired. In our study, we fired all queries after watching \emph{every 2nd} video (with the motivation to get data from the entire run, not just the start and end moment). The reference study created genuine 150 accounts on YouTube, while we used fewer accounts and took advantage of the browsing history reset option. In some aspects, our study had a larger scale: we executed 10 runs for each topic instead of one (to reduce possible noise) and used twice as many seed videos (to make sure that filter bubbles develop). There were also technical differences between the setups, as we used our own implementation of agents (e.g., different browser, ad-blocking software).

Given the methodological alignment (and despite the differences), we are confident to directly compare some of the outcomes of both studies, namely quantity of misinformative content appearing at the end of the promoting phases.
\section{Results and findings}

Following the study design, we executed the study between March 2nd and March 31st, 2021. Together, we executed 50 bot runs (10 for each topic). On average, runs for a single topic took 5 days (bots for a topic ran in parallel). The bots watched 3951 videos (collected 78763 recommendations associated with them, 8526 of them unique), executed 10075 queries (collected 201404 search results, 942 of them unique), and visited homepage 3990 times (collected 116479 videos there, 9977 of them unique). Overall, we recorded 17405 unique videos originating from 6342 channels.

Using the selection strategy and annotation scheme described in Section~\ref{sec:annotation-methodology}, 5 annotators annotated unique 2914 videos (covering 255844 appearances). In total, 244 videos were identified as promoting misinformation (related or unrelated to respective topics), 628 as debunking (including mocking videos), 184 as neutral, 1829 as not about misinformation. Other videos (unknown, non-English, or removed) numbered 29.

We report the results according to research questions and hypotheses defined in Section~\ref{ssec:research_questions}.
SERP-MS score metrics are reported for search results and mean normalized scores for recommendations.
Since the metrics are not normally distributed with some samples of unequal sizes, we make use of non-parametric statistical tests.
Pairwise tests are performed using two-sided Mann-Whitney U test.
In cases where multiple comparisons by topics are performed, Bonferroni correction is applied on the significance level (in that case \(\alpha=0.05\) is divided by number of topics \(n_T=5\), resulting in \(\alpha=0.01\)).

\subsection{RQ1: Has YouTube's personalization behavior changed since the reference study?}

Overall, we see a small change in the mean SERP-MS score across the same search queries in our and reference data: mean SERP-MS worsened from -0.46 (std 0.42) in reference data to -0.42 mean (std 0.3) in our data.
However, the distributions are not statistically significantly different (n.s.d.).
There is a similar small change towards the promoting spectrum in up-next (first result in recommendation list) and top-5 recommendations (following 5 recommendations).
We compared the up-next and top-5 recommendations together (as top-6 recommendations) using last 10 watched promoting videos in reference watch experiments and last two watched videos in our promoting phase.
We see mean normalized score worsened from -0.07 (std 0.27) in reference data to -0.04 (std 0.31) in our data.
These distributions are also not significantly different (U=45781.5, n.s.d.)

More considerable shifts in the data can be observed when looking at individual topics.
Table~\ref{tab:hussein-comparison-search} shows a comparison of SERP-MS scores for top-10 search results between our and reference data.
Improvement can be seen within certain queries for the chemtrails conspiracy that show a large decrease in the number of promoting videos.
The reference study reported that this topic receives significantly more misinformative search results compared to all other topics.
In our experiments, their proportion was lower than in the 9/11 conspiracy.
On the other hand, search results for flat earth conspiracy worsened.
Queries such as ``flat earth british'' resulted in more promoting videos, likely due to new content on channels with similar names.
Within the anti-vaccination topic, there is an increase in neutral videos (from 12\% to 35\%) and thus a drop in debunking videos (from 85\% to 61\%).
This may relate to new content regarding COVID-19.

Table~\ref{tab:hussein-comparison-recommendations} shows a comparison of normalized scores for up-next and top-5 recommendations.
Only the moon landing and anti-vaccination topics come from statistically significantly different distributions.
Similar to search results, recommendations for the 9/11 and anti-vaccination conspiracy topics worsened.
There were more promoting videos on the 9/11 topic (27\% instead of 18\%).
In the anti-vaccination topic, we observed a drop in debunking videos (from 29\% to 9\%) and a subsequent increase in neutral (from 70\% to 78\%) and promoting videos (from 1\% to 8\%).
The change within the anti-vaccination controversy is even more pronounced when looking at up-next recommendations separately.
Within up-next, the proportion of debunking videos drops from 77\% to 19\%, neutral videos increase from 22\% to 70\%, and promoting increase from 1 to 11\%.
On the other hand, in the moon landing topic, we see much more debunking video recommendations---40\% instead of 23\% in reference data.

These results bring up a need to distinguish between \emph{endogenous} (changes in algorithms, policy decisions made by platforms to hide certain content) and \emph{exogenous} factors (changes in content, external events, behavior of content creators) as discussed by Metaxa et al.~\cite{Metaxa2019}.
Our observations show that search results and recommendations were in part influenced by exogenous changes in content on YouTube.
Within the chemtrails conspiracy, we observed results related to a new song by Lana del Rey that mentions ``Chemtrails'' in its name.
Search results and recommendations in the anti-vaccination topic seem to be influenced by COVID-19.
Flat earth conspiracy videos were influenced by an increased amount of activity within a single conspiratorial channel.

\begin{table*}[]
\caption{
    Comparison of SERP-MS scores for top-10 search results with data from the reference study.
    The scores range from $\left \langle -1, 1 \right \rangle$, where -1 denotes a debunking and 1 a promoting stance towards the conspiracy.
    Only search results from queries that were executed both by the reference study and us are considered.
}
\small
\label{tab:hussein-comparison-search}
\begin{tabularx}{\textwidth}{lllp{16mm}X}
\toprule
Topic            & Hussein              & Ours   & Change & Inspection \\ \midrule

9/11             & -0.16                & -0.06 & No (n.s.d.)
& Smaller changes that depend on search query.  \\

Chemtrails       & -0.2                 & -0.47 & No (n.s.d.)
& Drop in promoting videos (from 45\% to 12\%) in 2 queries. \\

Flat earth       & -0.58                & -0.41 & No (n.s.d.)
& 2 queries worsen a lot due to new content. Other queries improve. \\

Moon landing     & -0.6                 & -0.59 & No (n.s.d.)
& Smaller decrease in number of neutral and increase of debunking videos. \\

Anti-vaccination & -0.8                 & -0.63 & \textcolor{red}{Worse} (U=324,p=$1.3\mathrm{e}{-9}$)
& Drop in number of debunking and increase in number of neutral videos.  \\ \bottomrule
\end{tabularx}
\end{table*}

\begin{table*}[]
\caption{
    Comparison of normalized scores for up-next and top-5 recommendations with data from the reference study.
    Normalized scores range from $\left \langle -1, 1 \right \rangle$, where -1 denotes a debunking and 1 a promoting stance towards the conspiracy.
    Last 10 out of 20 watched videos in reference data are considered.
    Last 2 out of 40 watched videos in our data are considered.
}
\small
\label{tab:hussein-comparison-recommendations}
\begin{tabularx}{\textwidth}{llllX}
\toprule
Topic            & Hussein & Ours   & Change  & Inspection \\ \midrule
9/11             & 0.14    & 0.26  & No (n.s.d.)                   & Similar distribution, more promoting videos. \\
Chemtrails       & 0.05    & 0.03  & No (n.s.d.)                   & More neutral results. \\
Flat earth       & -0.16   & -0.15 & No (n.s.d.)                   & Similar distribution. \\
Moon landing     & -0.08   & -0.32 & \textcolor{DarkGreen}{Better} (U=2954.5,p=$8\mathrm{e}{-6}$)   & More debunking videos. \\
Anti-vaccination & -0.28   & -0    & \textcolor{red}{Worse} (U=664,p=$1.6\mathrm{e}{-9}$)       & Less debunking videos, more neutral and promoting. \\ \bottomrule
\end{tabularx}
\end{table*}

\subsection{RQ2: What is the effect of watching debunking videos after the promoting phase?}

\begin{table*}[]
\caption{
Comparison of SERP-MS scores for top-10 search results in promoting and debunking phase of our experiment.
Three points are compared: start of promoting phase (S1), end of promoting phase (E1), end of debunking phase (E2).
}
\small
\label{tab:comparison-search}
\begin{tabularx}{\textwidth}{p{10mm}p{11mm}p{37mm}X}
\toprule
Topic            & SERP-MS & Change & Inspection \\ \midrule
9/11             &
S1: -0.07 \newline E1: -0.06 \newline E2: -0.11
&
S1--E1: n.s.d. \newline
E1--E2: n.s.d. \newline
S1--E2: n.s.d.
&
E2: More debunking videos in one query (30\% instead of 12\% at S1 and 11\% at E1 in query ``9/11'').
\\ \hline
Chemtrails       &
S1: -0.45 \newline E1: -0.47 \newline E2: -0.49
&
S1--E1: n.s.d. \newline
E1--E2: n.s.d. \newline
S1--E2: \textcolor{DarkGreen}{better} (U=915,p=0.0097)
&
E2: The ``Chemtrail'' search query showed an increase in number of debunking videos (from 66\% at S1 and 69\% at E1 to 80\%) and a decrease in promoting (from 10\% to 0\%).
\\ \hline
Flat earth       &
S1: -0.27 \newline E1: -0.41 \newline E2: -0.45
&
S1--E1: \textcolor{DarkGreen}{better} (U=762.5,p=0.0004) \newline
E1--E2: n.s.d. \newline
S1--E2: \textcolor{DarkGreen}{better} (U=704.5,p=0.0001)
&
E1: Change goes against expectations.
Promoting videos disappear in 3 search queries and decrease in another one (from 36\% to 30\%).
\newline
E2: Similar change as in E1 with a further decrease in promoting videos in one query (from 30\% to 22\%) and reordered videos in another.
\\ \hline
Moon landing     &
S1: -0.57 \newline E1: -0.57 \newline E2: -0.59
&
S1--E1: n.s.d. \newline
E1--E2: n.s.d. \newline
S1--E2: \textcolor{DarkGreen}{better} (U=900,p=0.0068)
&
E2: Reordered search results in ``moan hoax'' query---debunking videos moved higher.
\\ \hline
Anti-vacc. &
S1: -0.6 \newline E1: -0.63 \newline E2: -0.68
&
S1--E1: n.s.d. \newline
E1--E2: \textcolor{DarkGreen}{better} (U=699.5,p=0.0054) \newline
S1--E2: \textcolor{DarkGreen}{better} (U=641.5,p=0.0001)
&
E2: Increase in debunking videos across multiple queries (from 60\% at S1 and 61\% at E1 to 67\%).
\\ \bottomrule
\end{tabularx}
\end{table*}

\begin{table*}[]
\caption{
Comparison of changes in average normalized scores for top-10 recommendations in promoting and debunking phase of our experiment.
Three points are compared: start of promoting phase (S1), end of promoting phase (E1), end of debunking phase (E2).
}
\small
\label{tab:comparison-recommendations}
\begin{tabularx}{\textwidth}{p{10mm}p{11mm}p{37mm}X}
\toprule
Topic            & Score & Change & Inspection \\ \midrule
9/11             &
S1: 0.1 \newline E1: 0.42 \newline E2: 0.07
&
S1--E1: \textcolor{red}{worse} (U=45.5,p=$2.6\mathrm{e}{-5}$) \newline
E1--E2: \textcolor{DarkGreen}{better} (U=28,p=$2.9\mathrm{e}{-6}$) \newline
S1--E2: n.s.d.
&
E1: Number of promoting videos increased (from 14\% to 43\%) and neutral videos decreased (from 83\% to 56\%).
\newline
E2: The numbers of promoting and neutral videos returned to levels comparable to start (13\% and 82\%).
\\ \hline
Chemtrails       &
S1: 0 \newline E1: 0.05 \newline E2: -0.15
&
S1--E1: n.s.d. \newline
E1--E2: \textcolor{DarkGreen}{better} (U=323, p=0.0006) \newline
S1--E2: \textcolor{DarkGreen}{better} (U=330, p=0.0002)
&
E2: There is an increase in a number of debunking videos (from 0\% at S1 and 3\% at E1 to 19\%).
In return, we end up in a state that is better than at the start.
\\ \hline
Flat earth       &
S1: -0.17 \newline E1: -0.06 \newline E2: -0.47
&
S1--E1: n.s.d. \newline
E1--E2: \textcolor{DarkGreen}{better} (U=375, p=$1.8\mathrm{e}{-6}$) \newline
S1--E2: \textcolor{DarkGreen}{better} (U=347, p=0.0001)
&
E2: Similar to the Chemtrails conspiracy, there is an increase in number of debunking videos (from 19\% at S1 and 16\% at E1 to 48\%).
\\ \hline
Moon landing     &
S1: -0.2 \newline E1: -0.4 \newline E2: -0.42
&
S1--E1: n.s.d. \newline
E1--E2: n.s.d. \newline
S1--E2: n.s.d.
&
E1: Mean normalized scores changes against expectation and improves (but not significantly).
\\ \hline
Anti-vacc. &
S1: -0.1 \newline E1: 0.04 \newline E2: -0.37
&
S1--E1: \textcolor{red}{worse} (U=74.5,p=0.0008) \newline
E1--E2: \textcolor{DarkGreen}{better} (U=310,p=$2.5\mathrm{e}{-6}$) \newline
S1--E2: \textcolor{DarkGreen}{better} (U=307.5,p=0.0002)
&
E1: Increase in number of promoting videos (from 2\% to 13\%).
\newline
E2: Increase of debunking videos (from 12\% at S1 and 9\% at E1 to 37\%) and disappearance of promoting (from 2\% at S1 and 13\% at E1 to 0\%).
\\ \bottomrule
\end{tabularx}
\end{table*}

Answering this question requires three comparisons:
\begin{enumerate}
    \item comparison of metrics between start of promoting phase (S1) and end of promoting phase (E1),
    \item comparison of metrics between end of promoting phase (E1) and end of debunking phase (E2),
    \item comparison of metrics between start of promoting phase (S1) and end of debunking phase (E2).
\end{enumerate}

\emph{Comparison (1)} shows changes in search results and recommendations after watching promoting videos (E1) compared to the start of the experiment (S1).
If there was a misinformation bubble created, we would expect the metrics to worsen due to watching promoting videos.
Regarding search results, the distribution of SERP-MS scores between S1 and E1 is indeed significantly different (MW U=34118.5, p-value=0.028).
However, the score actually improves---mean SERP-MS score changed from -0.39 (std 0.28) to -0.42 (std 0.3).
Table~\ref{tab:comparison-search} shows the change for individual topics.
Only the flat earth conspiracy shows significant differences and improved the SERP-MS score due to a decrease in promoting and an increase of debunking videos.
Top-10 recommendations also change their distribution of normalized scores significantly at E1 compared to S1 (MW U=4085, p-value=0.0397).
We observe that the mean normalized score worsens from -0.07 (std 0.24) to 0.01 (std 0.31).
Looking at individual topics in Table~\ref{tab:comparison-recommendations}, we can see that the change is significant in topics 9/11 and anti-vaccination that gain more promoting videos.

\emph{Comparison (2)} relates the change in search results and recommendations between the end of promoting phase (E1) and the end of debunking phase (E2).
We expect the metrics would improve due to watching debunking videos, i.e., that we would observe misinformation bubble bursting.
However, SERP-MS scores in search results between E1 and E2 are not from statistically significantly different distributions, which is consistent with the fact that we did not observe misinformation bubble creation in search results in the first place.
Table~\ref{tab:comparison-search} shows that only a single topic---anti-vaccination---significantly changed its distribution and improved its mean score.
Nevertheless, we see minor improvements in SERP-MS scores also in other topics.
Top-10 recommendations show more considerable differences and their overall distribution is significantly different comparing E1 and E2 (MW U=7179.5, p-value=$1.8\mathrm{e}{-9}$).
Mean normalized score improves from 0.01 (std 0.31) to -0.27 (std 0.27).
Table~\ref{tab:comparison-recommendations} shows significantly different distributions for all topics except for moon landing conspiracy.
All topics show an improvement in normalized scores.
The 9/11 topic shows a decrease in promoting videos, while other topics show an increase in the number of debunking videos.

\emph{Comparison (3)} shows differences between the start (S1) and end of the experiment (E2).
We expect the metrics would improve due to watching debunking videos despite watching promoting videos before that.
The distribution of SERP-MS scores in search results is statistically significantly different when comparing S1 and E2 (MW U=36515, p-value=0.0002).
Overall, we see an improvement in mean SERP-MS score from -0.39 (std 0.28) to -0.46 (std 0.29).
In contrast with comparison (2), Table~\ref{tab:comparison-search} shows that all topics except 9/11 significantly changed their distributions.
All topics show an improvement according to our expectations.
The improvement is due to increases in debunking videos, decreases in promoting videos, or reordered search results in some search queries.
Similarly, top-10 recommendations at E2 come from a significantly different distribution than at S1 (MW U=6940.5, p-value=$2.9\mathrm{e}{-7}$).
Mean normalized score improves from -0.07 (std 0.24) to -0.27 (std 0.27).
Table~\ref{tab:comparison-recommendations} shows a significant difference in distributions for all topics except for 9/11 and moon landing conspiracies.
Mean normalized scores improve compared to S1 in all topics except for 9/11.
Nevertheless, the numbers of promoting and neutral videos in 9/11 topic at E2 are comparable to S1.
Other topics show increases in the numbers of debunking videos.

\section{Discussion and Conclusions}

In the paper, we presented an audit of misinformation present in search results and recommendations on the video-sharing platform YouTube. To support reproducibility, we publish the collected data and source codes for the experiment.

We aimed at verifying a hypothesis that there is less misinformation present in both search results and recommendations after recent changes in YouTube policies~\cite{YouTube2020policies} (H1.1). The comparison was done against a study done in mid 2019 by Hussein et al.~\cite{Hussein2020}. We were interested, whether we could still observe the formation of misinformation bubbles after watching videos promoting conspiracy theories (H2.0). In contrast to the previous studies, we also examined bubble bursting behavior. Namely, we aimed to verify whether misinformation bubbles could be burst if we watched videos debunking conspiracy theories (H2.1). We also hypothesized that watching debunking videos (even after a previous sequence of promoting videos) would still decrease the amount of misinformation compared to the initial state with no watch history at the start of the study (H2.2).

Regarding hypothesis H1.1, we did not find a significantly different amount of misinformation in search results in comparison to the reference study. A single topic (anti-vaccination) showed a statistically significant difference. However, it did not agree with the hypothesis as the metric \emph{worsened} due to more neutral and less debunking videos. Recommendations showed significant differences across multiple topics but were not significantly different overall. A single topic (moon landing) improved normalized scores of recommendation in agreement with the hypothesis. Yet, the anti-vaccination topic worsened its scores.
We suspect the changes in search results and recommendations were influenced mostly by changes in content. Overall, our results did not show a significant improvement in the fight against misinformation on the platform, as stated in the hypothesis.

We did not observe the creation of misinformation filter bubbles in search results (H2.0) despite watching promoting videos. On the other hand, recommendations behaved according to our hypothesis, and their overall normalized scores worsened. Since there was no filter bubble creation effect in search results, we did not observe any bubble bursting effect there. Results did not show a statistically significant difference between the end of promoting phase and the end of the debunking phase. Only a single topic (anti-vaccination) showed a statistically significant difference and an improvement following the hypothesis H2.1. Recommendations showed more considerable differences that were statistically significant and confirmed the hypothesis. Lastly, we showed that watching debunking videos decreases the number of misinformation videos both in search results and recommendations, which confirms our hypothesis H2.2. We observed an improvement of SERP-MS scores in all topics except for one and an improvement of normalized scores for recommendations in most topics.

Based on our results, we can conclude that users, even with a watch history of promoting conspiracy theories, do not get enclosed in a misinformation filter bubble \emph{when they search} on YouTube. However, we do observe this effect in video recommendations with varying degrees depending on the topic. However, \emph{watching debunking videos helps in practically all cases} to decrease the amount of misinformation that the users see. Additionally, although we expected to see less misinformation than the previous studies reported, this was in general not the case. Worsening in the anti-vaccination topic was partially expected due to the COVID-19 pandemic. However, it is interesting that we also observed a worse situation with the 9/11 topic. In fact, this topic served as a sort of a gateway to misinformation videos on other topics.

A limitation of our results lies with the limited amount of topics that we investigated -- these did not include, for example, recent QAnon conspiracy and COVID-19 related conspiracies were present only through anti-vaccination narratives. However, our topics were explicitly selected to allow comparison with the reference study. Next, we included only a limited set of agent interactions with the platform (search and video watching). Real users also like or dislike videos, subscribe to channels, leave comments or click on the search results or recommendations. A more human-like bot simulation, with these interactions and possible inclusion of human biases bursting remains our future work.

Nevertheless, our audit showed that YouTube (similar to other platforms), despite their best efforts so far, can still promote misinformation seeking behavior to some extent. The results also motivate the need for independent continuous and automatic audits of YouTube and other social media platforms~\cite{Simko2021}, since we observed that the amount of misinformation in a topic could change over time due to endogenous as well as exogenous factors.

redpajama_set_name: RedPajamaArXiv


text:
New York governor requests to exclude state from offshore drilling program
By Reuters Staff
(Reuters) - New York Governor Andrew Cuomo on Friday said he had formally asked for the state to be excluded from a federal offshore drilling program that he said would threaten its ocean resources and endanger efforts toward a cleaner energy economy.
FILE PHOTO - New York Governor Andrew Cuomo speaks during a dedication ceremony for the new Governor Mario M. Cuomo Bridge that is to replace the current Tappan Zee Bridge over the Hudson River in Tarrytown, New York, U.S., August 24, 2017. REUTERS/Mike Segar
"New York State strongly opposes the Department of the Interior's National Outer Continental Shelf Oil and Gas Leasing Program as it poses an unacceptable threat to New York's ocean resources, to our economy and to the future of our children," Cuomo said in announcing the exclusion request.
The five-year program, launched by the federal government in early January, proposes to make over 90 percent of the total U.S. offshore acreage available to oil and gas drilling. The plan would open two areas of the North Atlantic coast adjacent to New York State for fossil fuel exploration, according to a statement from Cuomo's office.
"As the number three ocean economy in the nation, New York stands to lose nearly 320,000 jobs and billions of dollars generated through tourism and fishing industries should the exclusion not be granted," Cuomo said.
"Instead of protecting our waters from another oil spill, like the one that devastated the Gulf, this new federal plan only increases the chances of another disaster taking place," he said.
Florida was granted exemption from the program on the grounds that the state relies heavily on tourism.
Interior Department spokeswoman Heather Swift did not immediately respond to a request for comment. New York is one of several states that has asked to be exempted from the drilling plan, and Interior has said it is considering the requests and holding discussions with states as it finalizes the proposal over the coming months.
Officials with at least two states, California and Washington, promised to take additional steps toward thwarting the plan. In letters to the Interior Department last month, they said they would block permits for transporting oil from new offshore rigs through their states.
In an effort to promote renewable energy in the New York, Cuomo also announced $1.4 billion in awards for 26 large-scale renewable energy projects across the state.
The awards, driven by the Clean Energy Standard mandate to obtain 50 percent of New York's electricity from renewable energy sources by 2030, are expected to generate enough clean, renewable energy to power more than 430,000 homes and create over 3,000 short- and long-term well-paying jobs, the statement said.
Reporting by Apeksha Nair in Bengaluru and Devika Krishna Kumar and Laila Kearney in New York; Editing by Dan Grebler