text:
The ACC's 2011 Top 10 Games to Watch
The start of the 2011 college football season is less than 20 days away. With kickoff right around the corner, Athlon takes a look at the top 10 must-see games in the ACC for 2011.
1. Oklahoma vs. Florida State (Week 3)
Sept. 17, All-Time Series: Oklahoma leads 5-1
There's a lot at stake in this Week 3 matchup. Oklahoma and Florida State are national title contenders, and a marquee non-conference win would go a long way towards an undefeated season. The Sooners easily handled the Seminoles last year, winning 47-17 in Norman. Florida State will be hungry to avenge last season's embarrassment and a victory over Oklahoma would be a statement win in Jimbo Fisher's second season in Tallahassee. The Sooners suffered a blow when linebacker Travis Lewis was lost for at least the first month of the season due to a foot injury suffered in fall camp. Although Lewis is out, Oklahoma's linebacking corps remains one of the best in the Big 12, especially with the rapid development of sophomores Tom Wort and Corey Nelson. Much of Florida State's national title hopes will hinge on quarterback EJ Manuel, who enters his first year as the starter. The junior has six starts over the last two years and there's no question he has the talent, but has yet to prove himself over a full season.
Athlon's Very Early Prediction:Seminoles by 3
2. Miami vs. Virginia Tech (Week 6)
Oct. 8, All-Time Series: Miami leads 17-11
Is Miami back? The Hurricanes have a tricky slate in the first half of the season, starting with the opener against Maryland, a non-conference matchup with Ohio State and this road date at Virginia Tech. The Hokies have won four out of the last five in this series, including the last two in Blacksburg. Will Miami's quarterback situation be solved by this game? If the Hurricanes are still trying to decide between Jacory Harris and Stephen Morris, the Hokies should have a huge advantage. Virginia Tech also has quarterback question marks, with Logan Thomas stepping in for Tyrod Taylor. Thomas has the physical tools to succeed, but needs more experience. By Week 6, the ACC should have a pretty good read on Thomas.
Athlon's Very Early Prediction: Hokies by 7
3. Florida State vs. Florida (Week 13)
Nov. 26, All-Time Series: Florida leads 33-20-2
Could this game be Florida State's final stumbling block to a berth in the national title? If the Seminoles knock off Oklahoma in mid-September, don't be surprised if they are undefeated and ranked among the top two teams in the nation. This will also be the first matchup as head coaches in this rivalry between Florida's Will Muschamp and Florida State's Jimbo Fisher. The coaches became close friends during their time as assistants under Nick Saban. The Gators were embarrassed in Tallahassee last year and will be looking to return the favor, while possibly ruining Florida State's season in the annual battle for bragging rights in the Sunshine State.
Athlon's Very Early Prediction: Seminoles by 6
4. Miami vs. Florida State (Week 11)
Nov. 12, All-Time Series: Miami leads 31-24
This series was dominated by Miami in the early 2000s, but Florida State has won four of the last six games. The Seminoles have lost the last two games in Tallahassee, including a 38-34 shootout in 2009. Florida State thoroughly handled the Hurricanes last year, winning 45-17, easily the most lopsided final in this series since a 47-0 win by the Seminoles in 1997. The Hurricanes should improve under Al Golden as the year progresses, but in Tallahassee, the Seminoles will likely make it two in a row against Miami.
Athlon's Very Early Prediction:Seminoles by 10
5. Ohio State vs. Miami (Week 3)
Sept. 17, All-Time Series: Ohio State leads 3-1
When Miami fans think of Ohio State, flashbacks of the 2003 Fiesta Bowl immediately come to mind. The Hurricanes lost a 31-24 battle for the national title that year, which included a controversial pass interference penalty in the first overtime to keep Ohio State's championship hopes alive. These two teams met last year, with Ohio State winning 36-24. This matchup in Week 3 will be a test for new Ohio State coach Luke Fickell, especially with an inexperienced quarterback under center. Even with the losses of coach Jim Tressel and quarterback Terrelle Pryor in the offseason, the Buckeyes still have enough talent to win a Big Ten title.
Athlon's Very Early Prediction: Miami by 5
6. Florida State vs. Clemson (Week 4)
Sept. 24, All-Time Series: Florida State leads 17-7
This matchup has trap game written all over. If Florida State manages to beat Oklahoma one week earlier, the Tigers should be able to pounce on a marquee victory. Clemson has won the last four matchups between these two teams in Death Valley in this series, but the Seminoles have won two out of the last three overall. Considering Clemson's success at home and Florida State's game against Oklahoma a week prior, don't be surprised if this is one of the upsets for Week 4.
Athlon's Very Early Prediction: Clemson by 3
7. Clemson vs. South Carolina (Week 13)
Nov. 26, All-Time Series: Clemson leads 65-39-4
The battle for bragging rights in the Palmetto State should be an attractive matchup this year. The Gamecocks are expected to contend for the SEC East title, while Clemson has the talent to surprise in the ACC. The Tigers are breaking in a new offensive scheme, and the progress of quarterback Tajh Boyd will be critical to contending for the conference title. Clemson has lost the last two in this series, including a 29-7 defeat last year. However, the Tigers have won six of the last overall nine matchups.
Athlon's Very Early Prediction: South Carolina by 7
8. Maryland vs. Miami (Week 1)
Sept. 5, All-Time Series: Tied 8-8
Miami is back on Labor Day for another ACC opener, this time against Maryland - not Florida State. Although this one might not have the national appeal of Florida State-Miami, there's plenty of intrigue for this matchup. This game will feature two new coaches – Maryland's Randy Edsall and Miami's Al Golden – and both teams will be looking to get off to a fast start in conference play. Keep a close watch on the Hurricanes' quarterback play, which could be the deciding factor in this ACC opener.
9. Maryland vs. West Virginia (Week 3)
Sept. 17, All-Time Series: West Virginia leads 24-21-2
Even though these two teams are separated by just over 200 miles, they have only met 47 times. The Mountaineers crushed the Terrapins last year, winning 31-17 in Morgantown. West Virginia has won the last five games this series, with the last Maryland victory coming in 2004 in the Gator Bowl. Since the Terrapins play in the Labor Day opener against Miami, they have a bye in Week 2, which will allow two weeks to prepare for the Mountaineers. Maryland could catch West Virginia looking ahead, as the Mountaineers host LSU in Week 4.
Athlon's Very Early Prediction: Mountaineers by 6
10. Maryland vs. Notre Dame (Week 11)
Nov. 12, All-Time Series: Notre Dame leads 1-0
The only matchup between these two teams occurred in 2002, with Notre Dame scoring an easy 22-0 win over Maryland in the Kickoff Classic. This season's matchup will take place in at FedEx Field Landover, Md. Although the Terrapins will have a short drive to this game, expect to see a pro-Notre Dame crowd. The Irish have BCS bowl hopes, but a loss to Maryland late in the year could be damaging.
Athlon's Very Early Prediction: Notre Dame by 8
Other Top Games to Watch
Northwestern at Boston College (Sept. 3)
Auburn at Clemson (Sept. 17)
NC State at Cincinnati (Sept. 22)
Clemson at Virginia Tech (Oct. 1)
Boston College at Clemson (Oct. 8)
Miami at North Carolina (Oct. 15)
NC State at Florida State (Oct. 29)
Florida State at Boston College (Nov. 3)
North Carolina at NC State (Nov. 5)
Virginia Tech at Georgia Tech (Nov. 10)
NC State at Boston College (Nov. 12)
Clemson at NC State (Nov. 19)
Boston College at Notre Dame (Nov. 19)
Virginia Tech vs. Virginia (Nov. 26)
Maryland at NC State (Nov. 26)
Georgia at Georgia Tech (Nov. 26)
Other ACC Content
Breaking Down Miami's QB Battle
Ranking the Best Units in the ACC
Athlon's 2011 ACC Predictions
Athlon's 2011 All-ACC Team
Ranking the ACC's Top 25 Players for 2011
Predicting the ACC's Breakout Players for 2011


Okay, I know this reply sounds odd, given that the caller probably has an aggressive dog she wants help with and knows my specialty is animal behavior, but that's what I wanted to say. For nearly a century, the term in psychology used for changing the behavior of animals and people using the scientific principles that guide learning and behavior has been called "behavior modification." This term describes a range of modifications, from the modification of simple behaviors, such as getting children in a classroom to be respectful and quiet, to handling anger management, tantrums, and depression.
In the dog obedience world, one or two celebrity dog trainers or animal handlers (who have probably never taken a psychology course in their lives or at least never paid attention in class) come along and decide they will elevate the status of what they do by calling their practices canine rehabilitation. Their goal was to distinguish training of simple behaviors (such as sit and lie down or training dogs to greet nicely instead of jumping) from fixing more complex behaviors such as attacking people and other dogs.
If you're going to make up some terminology, make sure the term you decide on is not already in use in the field for something else! The word rehabilitation already has an official definition and veterinarians and human physical therapists can get certification in canine or animal rehabilitation through at least one veterinary university, the University of Tennessee as well as through another well-established veterinary organization, the Canine Rehabilitation Institute.
What's more, animal rehabilitation is not related to animal behavior. Rather, it's actually physical therapy. That is, animal rehabilitation uses exercises, ultrasound, TENS and other non-invasive technologies to help animals recover from physical injuries or prevent or slow physical processes down. These exercises include, but are not limited to, balance exercises, passive range of motion exercises, swimming, and water walking. You know, the types of things you do when you rupture your cruciate or are trying to deal with sciatica or shoulder pain. So, in a nutshell, the person or people who popularized the term rehabilitation, meaning fixing of behavior problems in dogs, put little thought into their choice of the term and had a narrow view of the field of dog handling, care and training. All they needed to do to realize the they had chosen the wrong term was a quick Google search.
Are Dogs with Behavioral Problems Morally Bad or Physically Addicted?
The second problem with the term rehabilitation for problem dogs is that it implies the dog is a criminal or an addict. As a reminder, with human behavioral problems and disorders, the word behavior modification is used to describe the scientific techniques used to modify behavior. These are techniques that take advantage of what we know about animal and human learning and cognition. The word rehabilitation is most commonly used for those undergoing treatment for alcohol or other drug addictions but it's also used for treatment of criminals. Clearly, dogs with aggression issues or other extreme "misbehavior" issues are not alcoholics or drug addicts. And while they may be breaking the house rules or other human rules, they are not doing so out of a moral choice or even a real understanding of human societal laws. Their behavior reflects the way a dog naturally would behave in the "wild" when it is afraid of something, or when guarding its possessions. Once again, various behavior modification techniques provide the tools to fix these behaviors.
How Do I Train My Dog to Stop Chasing My Chickens, Cat, Rat…And Our Other Pets?
Of course, if people don't know the principles that guide learning and behavior in dogs and other animals, they might be inclined to make up a word as if they had invented an entire field of training. However, doing so would be a red flag to those in the know and would indicate that these people don't know much outside what they do themselves.
If rehabilitation is not a good word to use, how about using the term dog psychology? Well, to me this term is hokey because it conjures images of a dog reclining on a couch telling stories of his childhood. I have no problem with someone using this if they have actually taken psychology courses and understand what current psychology is. For instance, use of Freudian psychology, which was the popular psychology before psychology actually earned its place as a true science, would not represent what's current in psychology today.
For now, use whatever terminology you like. I can't make everyone use what's appropriate. However, be aware if you're a trainer or animal care professional and use the term rehabilitation instead of behavior modification when talking to me, I will assume you gained your knowledge about behavior from what you learned on T.V. or your knowledge has remained static over the years.
I'd love to be able to call myself a "Dog Psychologist", but alas somebody else has popularised the phrase and given it the effective definition of "somebody who makes it up as they go along", and I want to distance myself as much as possible from that methodology!
The number of times I have explained the benefits of reward versus dominance-based methods, only to have the dog owner tell me that's just the difference between "dog training" and "dog psychology". Maybe it would be easier if I got "I actually have a psychology degree" tattooed on my forehead!


text:
Topics Overview Practice Lawyers Careers Contact
Inquiry by Telephone
Takakuni Ujike
Bar Admission:
2009 (Daiichi Tokyo Bar Association)
2020 Munich Intellectual Property Law Center (MIPLC) at Max Planck Institute for Innovation and Competition (Munich, Germany; LL.M. in Intellectual Property and Competition Law)
2009 Legal Training and Research Institute of the Supreme Court of Japan
2008 Keio University, School of Law (J.D.)
2005 Osaka University, Faculty of Medicine, Division of Health Sciences, Course
of Radiological Technology (BHSc)
2020- Mr. Ujike currently works at a law firm in Munich, Germany
2013- Keiwa Sogo Law Offices
2010-2013 Hashidate Law Office; during the period Mr. Ujike also served as
an (part-time) in-house legal counsel at one of the largest asset management
companies in Japan
Main Areas of Practice:
Intellectual Property, IT, Finance, M&A, Medical Field/Healthcare, General
Corporate, International Transactions, Litigation/ADR
Back to Lawyers List
Please click below for any inquiries.
Copyright 2010~2019 Keiwa Sogo Law Offices. All rights reserved.
Copyright Keiwa Sogo Law Offices.

redpajama_set_name: RedPajamaCommonCrawl


text:
RETURN TO MASSIF CENTRAL
May 19, 2014 by Massif Central | TJP
Going West – Tour of California Round-up
"Go West, young man, go West and grow up with the country" – Horace Greeley
Read The Jersey Pocket Tour of California preview here.
On the surface of things there is a lot that it could be easy to be disparaging about in the Tour of California. The irony-laden fact that the race is sponsored by the makers of EPO has been covered many times before, for example. There is also the abundance of long straight highways used, the awful stop-motion quality of the on-board camera feed and the somewhat presumptuous assertion that the eight day 2.HC category race is the 'fourth Grand Tour'. But on the other hand, you have one of the most professionally run races in the calendar, the sublimeness of Pacific Highway 1, the very best online coverage I have ever seen and a strong rider line-up that a number of 'bigger' races would love to able to attract. Add in the fact that the favourable time difference means that the majority of live racing is on in the evening for British audiences and you have all the ingredients for an accessible, engaging and enthralling race.
Those of us who chose to follow the sun-baked Californian race more closely than the damp Giro stages last week were amply rewarded. We had British winners on 3 stages; we had young riders showing their elders that the next generation is coming through strong and sure; we had old hands still chancing their arms and we had star turns taking deserved bows as they crossed the line after epic efforts.
The first thing we had to contend with was how to watch the race. I've been tweeting the praises of the Amgen Tour Tracker (web and app) most of this week for it's excellent information and relative lack of adverts, but as so often in life you had to take the rough with the smooth. In this case that meant opting for the Phil & Paul (or more accurately in this case Paul & Phil) commentary over the much more engaging and informative pairing of Matt Stephens and Brian Smith over on Eurosport. Over the week Stephens and Smith re-animated the often staid arena of cycling commentary offering the kind of relevant and up-to-date insight that is so often missing these days. I often ended up watching the Tour Tracker video for the sidebar info that came with it, whilst playing the Eurosport audio through another device.
Stage 1 kicked things off nicely: A Mark Cavendish stage win and young British prospect Tao Geoghagen-Hart's top four on GC – thanks to bonifications earned throughout a long breakaway – made things look quite rosy in the bright Californian sun. Tao also secured the best Young Rider jersey for his efforts and made a significant mark in his first big race for his new Bissell Development team. Later in the week we saw videos of Tao talking about the new wireless (hell, yeah!) shifting system which they are testing for SRAM. This lad has a big future in cycling, writing and/or presenting. Take note. Cav's win over John Degenkolb was very close and neither knew who had taken it. Phones were produced, Tour Trackers were checked and a beautifully clear finish line photo was seen by the racers in around 20 seconds. We did get a little tired of still images as the week progressed with the live pictures often freezing due to transmission problems but that one was a peach.
Another young name that came to our attention this week was Lawson Craddock. Riding on Degenkolb's Giant Shimano team, the 22 year old was hugely consistent throughout the week and rode impressively in the Stage 2 Time Trial to take the Young Rider jersey from Tao with a 13th place overall finish, beating more notable names such as Sagan, Ten Dam, Van Avermaet and Terpstra. It was enough for the youngster to hold onto that jersey and strong finishes later in the week would elevate him to 3rd place on GC by the end of the race. The real star of the Time Trial though was Wiggins who went around the Folsom Circuit at a blistering pace to beat closest finisher New Zealander Rohan Dennis by 44seconds. Taylor Phinney was expected to do better than 3rd at 52seconds back and, though Wiggins took the Yellow Jersey from Cavendish, the young American stole the social media limelight by turning to give a single red rose to podium girl Allison Steinkamp as he departed the podium.
Stage 3 brought the first mountain test and Garmin-Sharp's Dennis looked to threaten Wiggins on the final climb up Mount Diablo. In a very un Sky-like fashion, it was Sir Brad himself who led the reducing peloton up the slope from the bottom. Despite having no teammates to count on, he looked the personification of assuredness and calm as he smoothly pedalled at high cadence with the air of someone leading out a Sunday club run. Dennis jumped away in the final kilometre to claw back 20 of the seconds he had lost to Wiggins in the Time Trial and, in doing so, made the next few days of racing far more interesting.
The run down Pacific Highway 1 from Monterey to Cambria on Stage 4 was a real treat for the eyes with the rugged coastline providing the most awe-inspiring backdrop. It certainly inspired the five man breakaway who held off the charging peloton just long enough to contest the stage win themselves. Comprised entirely of second tier teams and with a couple of skinny young lads often on the front, it reminded me of last year's Specialized ad where the young boy is being pursued by Tom Boonen. Tornado Tom was indeed at the head of the chase but even his thunderous thighs couldn't catch the plucky escapees. Will Routley of Optum took the win.
We were treated to an even greater escape on Stage 5 when Taylor Phinney – perhaps anxious to get back onto the podium to meet Miss Steinkamp again – drove off the front after the last climb of the day and plunged down and along the last 12km to Santa Barbara alone. He TT'd out a gap of 35 seconds by the finish and had time to literally take a bow as he crossed the line. It was a mighty effort and an impressive way for the US to record their only win of the week.
Stage 6 was the second (and last) big mountain finish. The aptly named hors category Mountain High was the destination and Garmin and others were looking to end Sir Brad's time in yellow. Dennis couldn't make an attack though and it was left to Tom Danielson to bridge up to the breakaway after earlier attacks from Ben King and George Bennett had shaken things up a bit. This time though Wiggins had a teammate with him and Joe Dombrowski put in a mammoth shift bringing his leader up the final ramps. The stage was won by OGE's Columbian climber Chaves but Wiggins managed to come home in 5th alongside fellow Brit Adam Yates, who had ignited a splintering of the chasers in the final yards. The small split gained the race leader a couple more precious seconds and the race was nearly in the bag.
Peter Sagan won the sprint after a lumpy Stage 7 into Pasadena had put Cavendish 6'21" behind by the end. Thor Hushovd looked to have been on for the win but Sagan popped out from a hidden viewpoint and shot past the Norwegian champion to rescue an otherwise mediocre week. Danny Van Poppel of Trek Factory racing continued his good week with a 3rd place beating Degenkolb, who had made it to the sprint intact. All that remained then was for a repeat of the Stage 1 showdown on the final stage with Cav and Degenkolb again going head to head. There were worries about Cavendish who had lost his lead-out man Renshaw in the lead-up and who later said he wasn't feeling too good at the stage start but he produced a great final burst to win by a clearer margin than before. No phones were needed this time. There was some disappointment on the last stage with Tao Geoghagen-Hart crashing hard and coming in 14 minutes down but greater concern was for Belkin's Moreno Hofland who also crashed breaking a vertebra and ribs. He will be in hospital for a week before he can fly home.
Brad's podium smile said everything that was needed about winning a race that was high on his and his team's agenda. He says that his next race will now be the Tour de France and that he is fully committed to supporting Chris Froome. Indeed, his leading of the peloton up Mount Diablo earlier in the week almost looked to be specific training for leading Froome up some of the Alpine climbs in France in July. The Tour of California came of age this year with it's status much elevated by this edition. It also won great respect for being a race where youngsters are being given a chance and are really grasping it. It was hugely refreshing to see. We tip our ten gallon cycling caps to y'all and, like many top riders, I suspect we will be back for more next year.
AmgenTourofCalifornia.com
This entry was posted in Pro cycling and tagged adam yates, california, cavendish, OPQS, sagan, tao, team sky, tour, wiggins. Bookmark the permalink.
Pantani:The Accidental Death of a Cyclist – Film Review
Giro d'Italia – Rest Day Roundup #2
One thought on "Going West – Tour of California Round-up"
Pingback: Tour of California - Page 4 - London Fixed-gear and Single-speed
View themassifcentral's profile on Facebook
View massif_central's profile on Twitter
View the_massif_central's profile on Instagram
The Massif Central Blog


text:
\section{Introduction}
\label{sec:introduction}}

\IEEEPARstart{F}{ollowing} Kodak's invention of the first megapixel sensor in 1986, digital photography has slowly grown to substitute its analog predecessor, playing a key role in the early 21st century digital revolution and social transformation \cite{peres2014concise, serafinelli2018digital}. As a relevant example, photography has modified the way mobile phones are used, as their integration of digital cameras has at once fostered an exponential growth of the photos that are shot and uploaded to the Internet every year, as well as a paradigm shift in mobile communications, which today rely on high quality multimedia \cite{yin2013socialized,Big_data_infrastructure_I,borcoci2010novel,rainer2016statistically}. These phenomena have proven to be game-changers for both how people communicate and the bloom of new fields of research, as both academia and industry have put to good use such plethora of visual data to train, develop and apply Artificial Intelligence (AI) models to a variety of different problems (e.g., face recognition, autonomous driving) \cite{lemley2017deep,li2019deep, chen2020learning,vaccaro2020image, 2020potential}.

Now, while a wealth of research is being devoted to the processing and analysis of digital images, much has to be done regarding analog ones, mainly because printed images representing a place (or an environment) at a given time may be: (i) scattered in numerous public and private collections, (ii) of variable quality, and, (iii) damaged due to hard or continued use or exposure. In addition, any analysis by means of image processing and computer vision algorithms requires the cumbersome and potentially quality degrading initial digitization step.

However, despite the complications and challenges brought on by analog photographs, they nevertheless represent an unparalleled source of information regarding the recent past: in fact, no other visual media has been used as pervasively to capture the world throughout the 20th century, as the availability of consumer grade photo cameras supported the spread and popularity of vernacular photography practices (e.g., travel photos, family snapshots, photos of friends and classes) \cite{mitman2016documenting,vernacular_moma}.

Family photo albums represent an example of vernacular photography that has drawn the attention of researchers and public institutions. A recent work defines family photo albums \textit{a globally circulating form that not only takes locally specific forms but also ``produces localities'' that creates and negotiate individual stories} \cite{sandbye2014looking}. Along the same lines, in another relevant contribution, family albums \textit{represent a reference point for the conservation, transmission and development of a community Social Heritage} \cite{calanca2011italians}. In essence, scholars from different fields agree in identifying such type of photography collections as capable of capturing salient features regarding the evolution of local communities in space and time.

However, a large-scale analysis of such collections of photos is often impeded by their numbers, as it would be exceedingly burdensome for historians and anthropologists to manually verify the characteristics of more than a few hundreds, considering also that in many cases no associated descriptions are available. This is why contributions in this field, normally, base their findings on the study of small corpora of photos~\cite{sandbye2014looking, calanca2011italians}.

This work addresses such problem, as it focuses on learning how to recognize a set of salient features of socio-historical interest exploiting the IMAGO collection of family photos started in year 2004 at the Rimini Campus of the University of Bologna~\cite{calanca2011italians}.
In addition, this work represents a first contribution for the following:

\begin{itemize}
    \item The introduction of a ``Family Album'' collection comprising over 80,000 photos taken between 1845 and 2009, belonging to ca 1,500 families, primarily from the Emilia-Romagna and immediately neighboring regions in Italy. 16,642 of such images have been labelled;
    \item The investigation of a new classification problem, to the best of our knowledge, regarding the identification of the socio-historical context of an image (according to classes individuated in socio-historical literature). In this case a model, given a photo, is requested to provide the socio-historical context of its subject. This amounts to quite a complicated task, naturally prone to ambiguities, as it can be easily seen from Fig.~\ref{fig_sim_semantic_fight} (Free-Time and Fashion, for example, share some common tracts, making it difficult to correctly discriminate between them);
    \item A thorough analysis using different deep learning architectures for both the image dating and the socio-historical context classification task. For photography and socio-historical scholars it is important to underline that the results of such analysis have been obtained with the sole use of the images at hand, without having the opportunity of resorting to external and additional sources of information.
\end{itemize}

\begin{figure}[t]
\captionsetup[subfigure]{labelformat=empty}
   \centering
   \begin{subfigure}{0.48\linewidth}
     \centering
     \includegraphics[width=\linewidth]{images/dataset_samples/freetime.png}
     \caption{Free-time}
   \end{subfigure}
   \hfill
   \begin{subfigure}{0.48\linewidth}
     \centering
     \includegraphics[width=\linewidth]{images/dataset_samples/fashion.png}
     \caption{Fashion}
   \end{subfigure}
\caption{Similarity between pictures labelled as belonging to different socio-historical classes.}
\label{fig_sim_semantic_fight}
\end{figure}

The rest of the paper is organized as follows: in Section~\ref{sec:related-work} we review the state of the art that falls closest to this contribution. Section~\ref{sec:imago} provides a description and the main characteristics of the IMAGO dataset. Section~\ref{sec:model} and Section~\ref{sec:results} present and validate several deep neural networks models applied to the proposed dataset, in order to highlight the difficulties and define an evaluation baseline. Later, in Section~\ref{sec:discussion} a review of the patterns learned by the deep learning models is provided. Finally, in Section~\ref{sec:conclusions} an overall discussion is carried out, along with possible directions of future works.

\section{Related Work}
\label{sec:related-work}

Only a few works have proposed so far the analysis of collections of vernacular photographs, also taking into account analog ones~\cite{fernando2014color,ginosar2015century,salem2016analyzing,muller2017picture}. The attention of such contributions has focused mostly on the dating of the images inside the collections. In the following, a discussion is offered regarding how such task was implemented.

In~\cite{fernando2014color}, the authors addressed the problem of dating color photos taken between 1930 and 1970, belonging to a combination of vernacular and landscape photography, considering a dataset of 1,325 photos. To this aim, they verified whether specific characteristics of the device used to acquire the photographs, as the distribution of derivatives and the angles drawn by three consecutive pixels in the RGB space, could be put to good use to date a photograph.
An accuracy of 85.5\% was achieved for the decades considered in the classification task. The proposed solution, though, is device dependent and may return inaccurate results when pictures are shot with particularly old cameras.

Another work employed a deep learning approach to analyze and date 37,921 historical frontal-facing American high school yearbook photos taken from 1928 to 2010~\cite{ginosar2015century}. Here, a Convolutional Neural Network (CNN) architecture was built to analyze people's faces and predict the year in which a photo was taken. The author observed the performance of the dating task to be gender-dependent: (i) considering the ability to individuate the exact year, a 10.9\% accuracy was obtained with women and 5.5\% with men, while (ii) 65.3\% and 46.4\% values, for women and men, respectively, were obtained accepting a $\pm$ 5 years tolerance.

Along the same lines, also the authors of~\cite{salem2016analyzing} presented a dataset containing images of students taken from high school yearbooks, covering in this case the 1950 to 2014 time span (considering 1,400 photos per year). Their results confirm that human appearance is strongly related to time. Moreover, they also resorted to CNNs to estimate when an image was captured, in order to evaluate the quality of color vs grayscale images containing the following features: (i) faces, (ii) torsos (i.e., upper bodies including people's faces), and, (iii) random regions of images. The best performance was obtained considering color images portraying the torso of people. Thanks to the size and balance in time of the dataset and to the fact that it represents the torso of people in homogeneous environments, the authors were able to obtain an accuracy of 41.6\% for the exact year estimation problem and a 75.8\% for the $\pm$ 5 years one.

In~\cite{muller2017picture}, instead, the dating task was implemented through the analysis of images belonging to years 1930 through 1999. As for~\cite{fernando2014color}, vernacular and landscape photos are considered, amounting to at most 25,000 pictures per year. The authors proposed different baselines relying on deep CNNs, considering the dating as both a regression and a classification task. With this work the authors were able to reach fair results, precisely 51.8\% on the decades dating task, with heterogeneous types of photos.

The first four rows of Table~\ref{table:dataset_information_comparison} report the main characteristics (image content, number of images and covered time span) of the original photo archives employed in this Section (in most cases only specific selections of such archives have been analyzed by means of image processing techniques). To provide an easy to see comparison, the same information, regarding the IMAGO collection (i.e., the collection originating the dataset analyzed in this work), is provided in the last row of the same Table.

\begin{table}[h!]\large
\renewcommand{\arraystretch}{1.6}
\center
\begin{adjustbox}{width=\linewidth}
\begin{tabular}{|l|l|l|l|l|}
\hline
\textbf{Original dataset} & \textbf{Characteristics} & \textbf{Cardinality} & \textbf{Period} \\
\hline
\cite[B. Fernando et al.]{fernando2014color} &  \vtop{\hbox{\strut vernacular and}\hbox{\strut landscape photos}} & 1,325 & 1930 - 1979 \\
\hline
\cite[S. Ginosar et al.]{ginosar2015century} & \vtop{\hbox{\strut high school}\hbox{\strut yearbooks portraits}} & 168,055
& 1905 - 2013
\\
\hline
\cite[T. Salem et al.]{salem2016analyzing} & \vtop{\hbox{\strut high school}\hbox{\strut yearbooks portraits}} & ca 600,000
& 1912 - 2014
\\
\hline
\cite[E. M\"{u}ller et al.]{muller2017picture} & \vtop{\hbox{\strut vernacular and}\hbox{\strut landscape photos}} & 1,029,710 & 1930 - 1999 \\
\hline
 \textbf{IMAGO collection} & \vtop{\hbox{\strut \textbf{family album}}\hbox{\strut \textbf{analog photos}}} & \textbf{ca 80,000} & \textbf{1845 - 2009} \\
\hline
\end{tabular}
\end{adjustbox}
\caption{Original dataset comparison.}
\label{table:dataset_information_comparison}
\end{table}

It is now important to highlight that all of the contributions discussed in this Section focused on the dating of vernacular photographs shot in heterogeneous settings. None, however, to the best of our knowledge, has considered implementing the classification of such kind of photos according to their context. In this respect, the contribution presented in this paper not only performs the dating of single images, but also analyzes the socio-historical context of the photos belonging to family albums. Nevertheless, for the dating task, we also combined three different features, extracted from the same photo (the whole image along with the faces and the people's full figures it contains) and observed that the accuracy of the proposed models increases as more people are represented in a photo. Finally, the dataset analyzed in this work is the only one solely composed by analog photos.

\begin{figure*}[t]
   \centering
   \begin{subfigure}{0.45\linewidth}
     \centering
     \includegraphics[width=\linewidth]{images/distributions/distribution_total_images_semantic_years.png}
     \caption{Classes distribution}
     \label{fig:distr_imago_normal}
   \end{subfigure}
   \hfill
   \begin{subfigure}{0.5\linewidth}
     \centering
     \includegraphics[width=\linewidth]{images/dataset_samples/IMAGO_SAMPLE.jpg}
     \caption{Sample images}
     \label{fig:samples_imago}
   \end{subfigure}
   \caption{IMAGO characteristics.}
\end{figure*}

\section{IMAGO dataset}
\label{sec:imago}

The IMAGO project is a digital collection of analog family photos year by year gathered and conserved by the Department for Life Quality Studies of the University of Bologna (a general overview may be found at
\href{http://imago.unibo.it}{\texttt{imago.unibo.it}}). On the whole, it comprises ca 80,000 photos, taken between 1845 and 2009, belonging to ca 1,500 family albums. 16,642 images have been labeled by the bachelor students in ``Fashion Cultures and Practices'' course, under the supervision of the socio-historical faculty. The metadata associated to each image includes its shooting year, a short textual description, the location (city and nation) where it was taken and a unique socio-historical label describing the category a scene belongs to~\cite{calanca2011italians}. The aforementioned subset of the IMAGO collection composed by 16,642 labelled images was considered as our dataset, namely the IMAGO dataset\footnote{The IMAGO resources, comprehending the dataset and the models, are available upon request.}.
From now on, we will consider the IMAGO dataset focusing on the individuation of the year and the socio-historical category (as described by the associated label). To this aim, the fourteen socio-historical categories, which have been individuated in~\cite{imago_historical}, have been grouped according to their visual similarities:

\begin{enumerate}[label=(\alph*)]
    \item \textit{Work}: photos belonging to this class are mostly characterized by people sitting and/or standing in workplaces and wearing work clothes;
    \item \textit{Free-time}, which includes the \textit{Free-time \& Holidays} categories: such category investigates the forms and ways of experiencing leisure time, reconstructing, wherever possible, generational and gender differences. It also includes images representing people which make new experiences, visiting far off landmarks, expanding social relationships and interacting with nature;
    \item \textit{Motorization}: although often closely related to the \textit{Free-time} category, this class has been distinguished as it includes symbolic objects such as cars and motorcycles, which represent a social and historical landmark;
    \item \textit{Music}: as for the \textit{Motorization} one, this class may also include scenes from leisure time, characterized in this case by the appearance of musical instruments or events;
    \item \textit{Fashion}, which includes the \textit{Fashion \& Traditions} categories: in this, clothing represents a mirror of the articulated intertwining of socio-economic, political and cultural phenomena. This class is characterized by the presence of symbolic objects and clothes, such as suits, trousers, skirts and coats;
    \item \textit{Affectivity}, which includes the \textit{Affectivity \& Friendship \& Family} categories: such photographs are characterized by the presence of people (e.g., couples, friends, families or colleagues) bound by inter-personal relationships;
    \item \textit{Rites}, which includes the \textit{Rites \& Marriages} categories: in the pictures comprised in this category, it is possible to find portraits pertaining the sacred and/or celebratory events which characterize the life of a family;
    \item \textit{School}: this class includes all the photos which represent schools, often characterized by symbolic objects (e.g., desk, blackboard) or group of students;
    \item \textit{Politics}: this class contains photos related to political gatherings, demonstrations and events.
\end{enumerate}

Fig.~\ref{fig:distr_imago_normal} reports the number of labelled images available per year in the 1930 to 1999 time frame (out of such time interval, the number of available images is too little to be represented visually). Such Figure also exhibits the unbalance observed both in terms of number of photos per year (the greatest share is concentrated between 1950 and 1980) and in terms of their socio-historical classification, as the dominant classes result to be the \textit{Affectivity}, \textit{Fashion} and \textit{Free-time} ones. Fig.~\ref{fig:samples_imago} shows four exemplar images from the IMAGO dataset which belong to different decades and represent different socio-historical contexts.
From this Figure, it is possible to appreciate the different characteristics that can be found in each photo (e.g., number of people, clothing, colors and location).

The following Section reports upon the deep leaning models exploited in our analysis.

\section{Deep Learning models}
\label{sec:model}
In this Section we firstly describe the image pre-processing actions carried out on the IMAGO dataset. Secondly, we detail how the training, validation and test sets have been composed to perform a fair evaluation of the models. Finally, we describe the considered deep neural network models along with their architecture, the training settings and the evaluation protocol.

\subsection{Pre-processing}
\label{subsec:pre_processing}
The pre-processing phase aimed at: (i) isolating the regions of interest which could enhance the performance of the selected deep learning models, (ii) improving the quality of the images composing the dataset, resorting to different computer vision algorithms.

\begin{figure*}[t]
   \begin{subfigure}{0.45\linewidth}
     \includegraphics[width=\linewidth]{images/dataset_samples/sample_faces.png}
   \caption{IMAGO-FACES}
     \label{fig:sample_imago_face}
   \end{subfigure}
   \hfill
   \begin{subfigure}{0.5\linewidth}
     \includegraphics[width=\linewidth]{images/dataset_samples/sample_people.png}
   \caption{IMAGO-PEOPLE}
     \label{fig:sample_imago_people}
   \end{subfigure}
   \caption{IMAGO-FACES and IMAGO-PEOPLE samples.}
\end{figure*}

As reported in Section~\ref{sec:related-work}, both faces and people represent regions of interest to be exploited for the dating analysis~\cite{salem2016analyzing,ginosar2015century}.
Following such insight, we created the IMAGO-FACES and the IMAGO-PEOPLE datasets, comprising each over 60,000 samples: the first composed of individual faces, the second of single person's full figure images. In particular, we processed each image of the IMAGO dataset using an open-source implementation of YOLO-FACE~\cite{yolo_face} and YOLO~\cite{yolo}, respectively.

The IMAGO-FACES dataset has been constructed accounting for the number of people portrayed in a photo. In fact, adopting a fixed size bounding box it may be possible to loose relevant details (e.g., hairstyle) or to include pixels related to the faces of other people. To avoid such problem, an adaptive strategy has been adopted: the size of the bounding box used to crop a face depends on the number of people portrayed in a photo, the greater the number of people, the smaller the bounding box. In this way, it was possible to extract the shoulders and the full head of a single person even when a picture portrayed tens of people. Fig.~\ref{fig:sample_imago_face} shows some sample images taken from the IMAGO-FACES dataset considering different decades and different socio-historical contexts. The construction of the IMAGO-PEOPLE dataset follows the same criteria employed for IMAGO-FACES, though, images can present different aspect ratios (i.e., people may be standing or sitting in photos). Fig.~\ref{fig:sample_imago_people} shows exemplar images from IMAGO-PEOPLE. It is possible to appreciate that IMAGO-PEOPLE includes details that are not present in IMAGO-FACES (e.g., the clothing of a person).

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{images/models/image_interchange_classifier.png}
    \caption{Model architectures for single-input classifiers.}
    \label{fig:model_single_image}
\end{figure}

Finally, we verified the utility of performing denoising and super resolution operations, as all the images considered in this work derive from scans of analog prints. For denoising we tested the neural network model from~\cite{zhang2018ffdnet} and the Bilateral Filter~\cite{bilateral_filtering}. For super resolution, we used an open-source implementation of the ESRGAN model~\cite{wang2018esrgan} within the Image Restoration Toolbox~\cite{kair_toolbox}. The overall improvement obtained adopting such strategies revealed to be negligible, we hence opted for an analysis based on the original scans of analog photos.

\subsection{Generation of train, validation and test subsets}
\label{subsec:train_validation_test}

All relevant datasets (IMAGO, IMAGO-FACES and IMAGO-PEOPLE) have been partitioned in three subsets or groups of pictures: 80\% for training and 20\% for testing. In addition, 10\% of the training images is parted to be used as the validation subset for the tuning of model hyperparameters. In particular, for each image in the train set of IMAGO, the faces and the people there portrayed are extracted and added to the corresponding IMAGO-FACES and IMAGO-PEOPLE subsets, respectively. This process is repeated also for the validation and test sets, as it guarantees that none of the training samples may end in the validation and test sets.

\begin{figure*}
    \centering
     \includegraphics[width=0.86\linewidth]{images/models/ensemble_classifier.png}
     \caption{Dating task: Ensemble classifier model architecture.}
    \label{fig:combined_model_ensemble}
\end{figure*}

\begin{figure*}
    \centering
     \includegraphics[width=0.86\linewidth]{images/models/real_merge_classifier.png}
     \caption{Dating task: Merged classifier model architecture.}
    \label{fig:combined_model_merged}
\end{figure*}

\subsection{Model architectures}
\label{subsec:model_architectures}

In this work we considered single and multi-input deep learning architectures for both the dating and socio-historical context classification tasks. The former type of models analyzes the images pertaining one of the datasets of interest in isolation, the latter, instead, operates on their combination (i.e., images in IMAGO, IMAGO-FACES and IMAGO-PEOPLE).

The deep neural network models exploited in our experiments are three well-known CNN architectures pre-trained on ImageNet~\cite{imagenet}: (i) Resnet50~\cite{resnet}, (ii) InceptionV3~\cite{inceptionv3} and (iii) DenseNet121~\cite{densenet121}. Each of such models was modified replacing the top-level classifier with a new classification layer, whose structure depends on the task (i.e., the number of output classes) and whose weights have been randomly initialized.
The pre-trained convolutional layers have been specifically fine-tuned for the given input data and task (dating or socio-historical context classification).

For what regards the single-input classifiers, one has been trained per each dataset. In particular, Fig.~\ref{fig:model_single_image} exhibits the architecture laid for IMAGO, IMAGO-FACES and IMAGO-PEOPLE.

The multi-input deep learning architectures result from the combination of the single-input models: the aim is to verify whether the classification performance improves with the combination of their knowledge.
We resorted at first to an Ensemble model which puts together the single-input classifiers introduced before, without performing any additional training. Adopting such architecture, shown in Fig.~\ref{fig:combined_model_ensemble}, the score assigned to each image belonging to IMAGO, and to its corresponding faces (in IMAGO-FACES) and full figures (in IMAGO-PEOPLE), is stored and subsequently combined to improve the class prediction. In fact, as a picture could contain more than one person, multiple IMAGO-FACES and IMAGO-PEOPLE images could stem from a single IMAGO sample.
Thus, each image in IMAGE-FACES and IMAGO-PEOPLE, produces a classification score vector by means of their respective single-input classifiers (one per each branch). The score vectors obtained are averaged out and normalized. The three vectors, resulting from the three models, are finally averaged out providing the output from which the most probable class is inferred.

We then defined the multi-input Merged model shown in Fig.~\ref{fig:combined_model_merged}, which with respect to the Ensemble model aims at not only putting together different sources of information, but also at learning how. Hence, a new training session is carried out as the newly introduced network is asked to learn how to perform such combination.
As before, the already trained single-input classifiers are employed, one along each branch (IMAGO, IMAGE-FACES, IMAGO-PEOPLE), but the classification layer is removed, preserving the CNN backbone.
The cardinality of the different features depends on the number of faces/people (in IMAGE-FACES, IMAGO-PEOPLE) portrayed in an image. For this reason the average of such feature vectors is computed, to be able to combine them with the vector obtained from the original image (in IMAGO). The three resulting feature vectors, one for the original image, one for faces and one for people, are then linearly combined by means of a weighted sum, whose weights are the real scalars $\alpha$, $\beta$ and $\gamma$, whose values are learned during the training phase.
The feature vector resulting from the linear combination is then fed to a fully connected layer with a softmax activation yielding the final probability vector and, consequently, the most probable class. Summing up, this model differs from the Ensemble classifier as it extracts the feature vectors instead of the scores from the pre-trained single-input classifiers and uses trainable weights to balance their importance. Moreover, it contains an additional dense classification layer which is trained from scratch observing the whole training set.

\subsection{Training settings}
During training we applied random cropping and horizontal flipping in order to make the model less prone to overfitting. Each model has been fine-tuned using a weighted cross entropy loss and an Adam optimizer with a learning rate of 1e-4 and a weight decay of 5e-4.
We set the batch size to 32 for the training of the IMAGO classifier and to 64 for the training of the IMAGO-FACES and IMAGO-PEOPLE models.

\subsection{Model comparison}
To compare the performance of single and multi-input classifiers, we need to establish an evaluation metric for those photos that contains more than one person.

Simply, for IMAGO-FACES and IMAGO-PEOPLE classifiers, we evaluate the accuracy not for a single face or person, but agglomerating votes for each of these in one picture. This means that, if a picture contains $n$ persons, the  IMAGO-FACES (and IMAGO-PEOPLE) classifier prediction would be considered correct if the unique vote that derive from the combination of the $n$ extracted is correct. Practically, we average the probability vectors returned by the single-input classifier for each image and then we compute the most probable class.

\section{Experimental Results}
\label{sec:results}
This Section reports on the results obtained for both the dating and the socio-historical context classification tasks described so far.
For what concerns image dating, 15,673 pictures, covering the 1930 to 1999 temporal interval, have been employed to reduce the imbalance problem of the IMAGO dataset that is even more challenging than the one exhibited in Fig. \ref{fig:distr_imago_normal}. The entire dataset (16,642 photos spanning over the 1845-2009 time period) is instead used during the analysis of the socio-historical context task.

\subsection{Dating task results}
\label{sec:results_dating}
The evaluation of the performance of the dating model is computed exploiting time distances, as also reported in~\cite{salem2016analyzing,ginosar2015century}. The time distance defines the tolerance accepted in predictions with respect to the actual year. For example, if a photo was labeled with year 1932 and the model returned 1927 (or even 1937), this would be considered correct for those cases where the time distance was set to values equal or greater than 5 and wrong otherwise.
In this work, model accuracies were computed considering temporal distances of 0, 5 and 10 years.

The results are reported in Table~\ref{tab:results_classifiers_dating}.
It is possible to appreciate that different baseline models return similar accuracy values. In addition, the same baselines returned comparable results when exploiting different datasets (i.e., IMAGO, IMAGO-FACES and IMAGO-PEOPLE). This may depend on many factors, including the people's appearance (e.g., dresses, hairstyle, earrings, trousers), as discussed in Section \ref{sec:discussion}. When comparing the results of the different approaches, multi-input improves with respect to single-input and the Merged one emerges above all. Evidently, the complementary knowledge exploited with multi-input models contributes to such results. This is found observing both the negative trend of the mean errors and the positive one of accuracies.

\begin{table*}[h] \large
\centering
\renewcommand{\arraystretch}{2}
\begin{adjustbox}{width=\linewidth}
     \begin{tabular}{cc|ccc|ccc|ccc|ccc|ccc|}
        \multicolumn{2}{c}{} & \multicolumn{15}{c}{\textbf{Classifier}} \\
        \Cline{2pt}{3-17}
        \multicolumn{2}{c}{} & \multicolumn{3}{|c}{\textbf{Image}} & \multicolumn{3}{|c}{\textbf{Face}} & \multicolumn{3}{|c}{\textbf{People}} & \multicolumn{3}{|c}{\textbf{Ensemble}} & \multicolumn{3}{|c|}{\textbf{Merged}} \\
        \Cline{2pt}{3-17}
        \multicolumn{2}{c|}{} & \textbf{ResNet50} & \textbf{InceptionV3} & \textbf{DenseNet121} & \textbf{ResNet50} & \textbf{InceptionV3} & \textbf{DenseNet121} & \textbf{ResNet50} & \textbf{InceptionV3} & \textbf{DenseNet121} & \textbf{ResNet50} & \textbf{InceptionV3} & \textbf{DenseNet121} & \textbf{ResNet50} & \textbf{InceptionV3} & \textbf{DenseNet121} \\
        \specialrule{2pt}{1pt}{1pt}
        \multirow[c]{3}{*}{\textbf{Accuracy}} &
        \textbf{d = 0} &
        \textbf{11.31} & 10.45 & 10.68 &
        \textbf{15.01} & 14.60 & 12.91 &
        \textbf{15.77} & 12.56 & 13.99 &
        \textbf{17.78} & 13.58 & 15.68 &
        \textbf{18.71} & 17.14 & 16.22 \\
        \cline{2-17}
        & \textbf{d = 5} &
        \textbf{62.56} & 61.38 & 60.77 &
        \textbf{58.09 }& 56.95 & 57.81 &
        \textbf{62.40} & 60.04 & 59.69 &
        \textbf{63.99} & 59.94 & 63.00 &
        \textbf{67.59} & 67.56 & 66.67 \\
        \cline{2-17}
        & \textbf{d = 10} &
        82.54 & \textbf{82.82} & 82.47 &
        78.39 & 78.46 & \textbf{79.70} &
       \textbf{ 82.47} & 81.39 & 81.42 &
        \textbf{83.33}& 80.69 & 82.98 &
        86.17 & \textbf{86.30} & 86.07 \\
        \specialrule{2pt}{1pt}{1pt}
        \multirow[c]{1}{*}{\vtop{\hbox{\strut \textbf{Error}}\hbox{\strut \textbf{(d = 0)}}}} &
        \textbf{Mean} &
        \textbf{6.01} $\pm$ \textbf{6.76}& 6.20 $\pm$ 6.89 & 6.13 $\pm$ 6.74 &
        6.56 $\pm$ 7.13 & 7.08 $\pm$ 7.20 & \textbf{6.54 $\pm$ 6.92} &
        6.82 $\pm$ 5.98 & \textbf{6.20 $\pm$ 6.75} & 6.23 $\pm$ 6.82 &
        \textbf{5.63 $\pm$ 6.51} & 6.34 $\pm$ 7.08 & 5.86 $\pm$ 6.68 &
        \textbf{5.06 $\pm$ 6.01} & 5.14 $\pm$ 5.92 & 5.22 $\pm$ 5.95 \\
        \specialrule{2pt}{1pt}{1pt}
    \end{tabular}
\end{adjustbox}
\caption{Accuracy: dating models accuracies for different time distances (d = 0, d = 5, d = 10). Error (d = 0): Mean errors computed for a time distance equal to 0.}
\label{tab:results_classifiers_dating}
\end{table*}

In the analyses that follow, the Merged model was selected utilizing as backbone reference the ResNet50. Such choices were made because the Merged model provided the best results and the ResNet50 the best trade-off between accuracy and model-dimensionality ~\cite{coleman2019analysis}.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.85\linewidth]{images/results/dating/conf_matrix_results_merged_without_values0cm.png}
    \caption{Confusion matrix for the dating task obtained using the Merged classifier considering a time distance equal to 0.}
    \label{fig:merged_classifier_dating}
\end{figure}

Fig.~\ref{fig:merged_classifier_dating} shows the confusion matrix considering a time distance equal to 0.
The diagonal structure demonstrates that the confusion mostly occurs between neighboring years, except for the initial and the final decades, as also highlighted in other works~\cite{ginosar2015century}.
The confusion created within the first 20 years could be caused by the low quality of the images and the limited number of samples representing those years.
This phenomenon could also be explained considering two additional facts: (i) during those decades photos mainly fall in two classes (i.e., \textit{Fashion} and \textit{Affectivity}, as clear from Figure \ref{fig:distr_imago_normal}), and (ii) the percentage of photos shot in a professional setting was much higher than in the decades that followed. The confusion created within the last 20 years, instead, could be related to the fact that the number of images for these years is very limited (as shown in Fig.~\ref{fig:distr_imago_normal}).

Nevertheless, it is interesting to observe the information provided in Fig.~\ref{fig:merged_classifier_dating_decades}, where the model accuracy and the number of samples per decade are reported. This Figure confirms the finding exhibited by the confusion matrix, the model accuracy improves after the 50's.
Fig.~\ref{fig:merged_classifier_dating_decades} also shows that, despite a reduction in terms of available samples per decade after the 80's, the performance of the model has a slight decrease.
In fact, it is evident that accuracy generally improves after the 50's (also when the number of samples drops), again, this could be related to the fact that the images are of better quality with respect to the previous decades.

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{images/results/dating/result_merged0_decades.png}
    \caption{Merged model accuracy (red line) and number of samples (blue line) by decade for a time distance equal to 0.}
    \label{fig:merged_classifier_dating_decades}
\end{figure}

Another behavior that was observed is that the accuracy of the model grows as the number of faces and/or people increases within a photo. To further verify such phenomenon, we decided to use the Ensemble classifier (described in Fig.~\ref{fig:combined_model_ensemble}), as this model let us isolate the contribution of each single input branch (IMAGO, IMAGO-FACES, IMAGO-PEOPLE). To this aim, an experiment was conducted considering a test subset characterized by a fixed and constant number $n$ of faces/people across all images. To weigh the role of the number of faces/people, fixing $n = 8$, the accuracy values were computed considering $k$ faces/people, with $k$ growing from $1$ to $n$. In essence, for all those images which comprised 8 faces/people, we computed the dating accuracy that could be obtained as a function of the number of faces/people considered for its evaluation.
To ensure the completeness and fairness of this experiment, all possible combinations of the $k$ faces/people were considered. Accuracy results have been grouped by $k$ and are reported in Table~\ref{tab:increasing_face_people_accuracies_dating_classifier}. In order to verify that the increasing dating accuracy was solely due to an increase of the number of considered faces/people, we proceeded to isolate the influence of the original image (IMAGO dataset).
To this aim, the date of an image was evaluated, including and excluding the contribution of the IMAGO classifier in Fig.~ \ref{fig:combined_model_ensemble}: (i) verifying the contributions of the IMAGO-FACES and IMAGO-PEOPLE classifiers in isolation and (ii) verifying the contributions of the IMAGO-FACES and IMAGO-PEOPLE classifiers while both active.
While confirming that a higher number of faces or people can improve the overall accuracy, it is also possible to see that the contribution of the original image decreases with a higher number of faces and people. It is worth highlighting that this experiment has been conducted on a subset of the test set.

\begin{table*}[h] \large
\centering
\renewcommand{\arraystretch}{2}
\begin{adjustbox}{width=0.8\linewidth}
     \begin{tabular}{cc|c|c|c|c|c|c|}
        \multicolumn{2}{c}{} & \multicolumn{6}{c}{\textbf{Ensemble classifier with ResNet50 backbone}} \\
        \Cline{2pt}{3-8}
        \multicolumn{2}{c|}{} &  \multicolumn{2}{c|}{\textbf{IMAGO-FACE}} &  \multicolumn{2}{c|}{\textbf{IMAGO-PEOPLE}} &  \multicolumn{2}{c|}{\textbf{IMAGO-FACE w/ IMAGO-PEOPLE}} \\
        \Cline{2pt}{3-8}
        \multicolumn{1}{c}{} & \textbf{Number of faces/people} & \textbf{w/o IMAGO} & \textbf{w/ IMAGO} & \textbf{w/o IMAGO} & \textbf{w/ IMAGO} & \textbf{w/o IMAGO} & \textbf{w/ IMAGO} \\
        \specialrule{2pt}{1pt}{1pt}
        \multirow[c]{8}{*}{\textbf{Accuracy}} & 1 &  15.25 & 16.75 & 13.99 & 15.03 & 24.55 & 24.78 \\
        \cline{2-8}
         & 2 & 15.57 & 17.21 & 18.03 & 19.05 & 26.62 & 27.50 \\
        \cline{2-8}
         & 3 & 17.04 & 18.50 & 18.77 & 19.49 & 28.06 & 28.79 \\
        \cline{2-8}
         & 4 & 17.54 & 19.31 & 19.49 & 19.54 & 29.60 & 30.12 \\
        \cline{2-8}
         & 5 & 17.61 & 19.57 & 19.96 & 19.45 & 30.68 & 31.02 \\
        \cline{2-8}
         & 6 & 17.93 & 20.00 & 20.24 & 19.43 & 31.39 & 31.60 \\
         \cline{2-8}
         & 7 & 17.75 & 20.25 & 20.39 & 20.09 & 31.92 & 31.99 \\
         \cline{2-8}
         & 8 & 18.00 & 20.00 & 20.24 & 21.43 & 33.33 & 33.33 \\
        \specialrule{2pt}{1pt}{1pt}
    \end{tabular}
\end{adjustbox}
\caption{Accuracy results for increasing number of faces/people exploiting the Ensemble classifier with ResNet50 backbone for a time distance equal to 0.}
\label{tab:increasing_face_people_accuracies_dating_classifier}
\end{table*}

\subsection{Socio-historical context classification task results}
\label{subsec:result_socio_historical}
In this Section we analyze the results that were obtained for the socio-historical context classification task using state of the art CNN architectures.
The results are reported in Table~\ref{tab:results_classifiers_socio_cultural}, in terms of top-$k$ accuracy: if the correct class is not the one with the highest predicted probability, but falls among the $k$ with the highest predicted probabilities, it will be counted as correct. It is interesting to compare the performance of the single-input classifiers in such terms: the IMAGO classifier exhibits a top-3 accuracy of 92.85\% with a ResNet50 backbone, while, in the same setting, the other single-input classifiers obtain a lower performance.
The implications of such result on the socio-historical analysis is discussed in Section~\ref{subsec:socio_gradcam}.

\begin{table*}[h] \large
\centering
\renewcommand{\arraystretch}{1.8}
\begin{adjustbox}{width=0.8\linewidth}
     \begin{tabular}{c|ccc|ccc|ccc|}
        \multicolumn{1}{c}{} & \multicolumn{9}{c}{\textbf{Classifier}} \\
        \Cline{2pt}{2-10}
        \multicolumn{1}{c}{} & \multicolumn{3}{|c}{\textbf{IMAGO}} & \multicolumn{3}{|c}{\textbf{IMAGO-FACE}} & \multicolumn{3}{|c|}{\textbf{IMAGO-PEOPLE}} \\
        \Cline{2pt}{2-10}
        \multicolumn{1}{c|}{} & \textbf{ResNet50} & \textbf{InceptionV3} & \textbf{DenseNet121} & \textbf{ResNet50} & \textbf{InceptionV3} & \textbf{DenseNet121} & \textbf{ResNet50} & \textbf{InceptionV3} & \textbf{DenseNet121} \\
        \specialrule{2pt}{1pt}{1pt}
        \textbf{Top-1} &
        \textbf{64.35} & 64.08 & 63.72 &
        41.30 & \textbf{43.37} & 41.39 &
        56.54 & \textbf{57.20} & 55.64 \\
        \hline
        \textbf{Top-2} &
        \textbf{85.00} & 83.83  & 83.38 &
        65.55 & \textbf{65.92} & 65.28 &
        78.48 & 77.34 & \textbf{78.90} \\
        \hline
        \textbf{Top-3} &
        \textbf{92.85} & 92.28 & 92.37 &
        \textbf{82.75 }& 81.18 & 82.66 &
        \textbf{89.90} & 89.36 & 89.87 \\
        \hline
        \textbf{Top-4} &
        96.66 & \textbf{96.75} & 96.54 &
        90.86 & 90.20 & \textbf{91.28} &
        \textbf{94.74} & 94.47 & 94.53 \\
        \hline
        \textbf{Top-5} &
        98.35 & \textbf{98.53} & 98.47 &
        94.98 & 95.01 & \textbf{95.31} &
        \textbf{97.42} & 97.14 & 97.26 \\
        \specialrule{2pt}{1pt}{1pt}
    \end{tabular}
\end{adjustbox}
\caption{Socio-historical models accuracies for an increasing Top-$k$ classification ($k$ ranging from 1 to 5).}
\label{tab:results_classifiers_socio_cultural}
\end{table*}

Please note that we here concentrated on the use of single-input architectures analyzing the IMAGO dataset, as multi-input ones did not provide any significant contribution to the fulfillment of the task at hand. Why this occurs may be explained as follows: the socio-historical context represented by an image may be inferred with a greater precision when it is analyzed as a whole, including its spatial relationships, objects and locations, in addition to how people look. In other words, the performance of the IMAGO classifier naturally emerges with respect to the results obtained utilizing the other single-input ones (IMAGO-FACES, IMAGO-PEOPLE), as shown in Table~\ref{tab:results_classifiers_socio_cultural}. In essence, the spatial relationships that are broken when constituting the IMAGO-FACES and IMAGO-PEOPLE datasets are key for this task. This is also why multi-input classifiers may not perform as well as the IMAGO classifier.
Reminding that the final output of multi-input classifiers is a combination of the knowledge acquired along the three branches, we can observe that the information acquired with the IMAGO classifier may be corrupted by the information extracted from the other single-input classifiers.

\begin{figure}[h]
    \centering
    \includegraphics[width=\linewidth]{images/results/socio_historical_context/conf_matrix_results_total_decades.png}
    \caption{Confusion matrix for total image classifier.}
    \label{fig:conf_matrix_semantic}
\end{figure}

Fig.~\ref{fig:conf_matrix_semantic} shows the confusion matrix obtained with the IMAGO classifier. It is possible to observe that three classes are the responsible for the largest share of misclassifications, in fact, the model is often misled to choose one among \textit{Fashion}, \textit{Affectivity} and \textit{Free-time}, instead of the correct class.
This can be justified considering that all the other classes contain images which are not easily separable from the three aforementioned ones. Indeed, pictures belonging to other classes can be visually similar (e.g., a model at work and/or people embracing during a concert, could be classified as \textit{Fashion} and \textit{Affectivity} while belonging to the \textit{Work} and \textit{Music} classes, respectively) and hence difficult to discriminate.
This is also true for the \textit{Free-time} class, which can be considered the most general one because photos that fall within this category are characterized by people that are involved in different kind of activities in heterogeneous environments.
Another interesting fact is that the \textit{Work} class is mostly confused with the \textit{Fashion} and \textit{Affectivity} ones. This could be explained since the images that fall within the \textit{Work} class represent groups of people wearing work-suits and/or being in working environments (e.g., people in a cafeteria, market or factory).
A similar phenomenon takes place for the \textit{Music} class. In particular, the performance of the model related to this class is slightly better than the one exhibited for the \textit{Work} one. This may be traced back to the confusion existing among the \textit{Free-Time}, \textit{Work}, \textit{Fashion} and \textit{Affectivity} classes. This confusion was generated by the fact that many photos belonging to the \textit{Music} class included different aspects of daily life: (i) a band playing at a concert which may be classified as \textit{Music} or \textit{Work}, or, (ii) a group of friends at a concert which could belong to \textit{Music} or \textit{Free-Time} depending on the perspective. This will be further analyzed in Section \ref{subsec:socio_gradcam}.

\section{What do the models learn?}
\label{sec:discussion}

The aim now is to understand which cues led the trained models to determine the date of a picture or to individuate its socio-historical context. To do so, in the following the Grad-Cam algorithm~\cite{grad_cam} is applied: in brief, this algorithm delimits the areas of an image exploited by deep learning models for the classification.

\subsection{Dating task}

\begin{figure*}[h]
\centering
\begin{adjustbox}{width=0.85\linewidth}
\begin{tabular}{m{12mm}!{\vrule width 2pt}m{24mm}!{\vrule width 2pt}m{24mm}m{24mm}!{\vrule width 2pt}m{24mm}m{24mm}!{\vrule width 2pt}}
    \multicolumn{1}{c!{\vrule width 2pt}}{} & \multicolumn{1}{c!{\vrule width 2pt}}{\textbf{IMAGO}} & \multicolumn{2}{c!{\vrule width 2pt}}{\textbf{IMAGO-FACES}} & \multicolumn{2}{c!{\vrule width 2pt}}{\textbf{IMAGO-PEOPLE}} \\
    \textbf{1930} &
    \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1930/1_1931_1930_35873_pnp.jpg}
    & \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1930/10_1931_1941_35873_pnp_face_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1930/10_1931_1941_35873_pnp_face_1.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1930/1_1931_1930_35873_pnp_person_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1930/13_1931_1944_35873_pnp_person_1.jpg} \\
    \textbf{1940} &
    \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1940/0_1941_1941_271_mq.jpg}
    & \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1940/0_1941_1941_271_mq_face_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1940/0_1941_1941_271_mq_face_1.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1940/0_1941_1941_271_mq_person_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1940/0_1941_1941_271_mq_person_1.jpg} \\
    \textbf{1950} & \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1950/2_1954_1956_16985_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1950/5_1954_1949_16985_pnp_face_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1950/1_1954_1955_16985_pnp_face_1.jpg}&   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1950/7_1954_1961_16985_pnp_person_0.jpg}&   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1950/3_1954_1951_16985_pnp_person_1.jpg} \\
    \textbf{1960} &  \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1960/0_1967_1967_32222_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1960/3_1967_1964_32222_pnp_face_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1960/4_1967_1963_32222_pnp_face_1.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1960/1_1967_1966_32222_pnp_person_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1960/1_1967_1966_32222_pnp_person_1.jpg} \\
    \textbf{1970} & \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1970/0_1973_1973_4731_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1970/1_1973_1974_4731_pnp_face_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1970/0_1973_1973_4731_pnp_face_1.jpg} &
    \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1970/0_1973_1973_4731_pnp_person_0.jpg} &
    \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1970/0_1973_1973_4731_pnp_person_1.jpg} \\
    \textbf{1980} & \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1980/0_1980_1980_15907_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1980/1_1980_1979_15907_pnp_face_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1980/0_1980_1980_15907_pnp_face_1.jpg} &
    \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1980/1_1980_1979_15907_pnp_face_2.jpg} &  \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1980/0_1980_1980_15907_pnp_person_1.jpg} \\
    \textbf{1990} & \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1990/0_1998_1998_33393_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1990/0_1998_1998_33393_pnp_face_0.jpg} &   \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1990/0_1998_1998_33393_pnp_face_1.jpg} &
    \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1990/0_1998_1998_33393_pnp_person_0.jpg} &  \includegraphics[width=24mm,height=24mm]{images/gradcam_dating/1990/0_1998_1998_33393_pnp_person_1.jpg} \\
\end{tabular}
\end{adjustbox}
\caption{Grad-Cam analysis of different images within IMAGO, and their respective pictures in IMAGO-FACES and IMAGO-PEOPLE, spreaded over the twentieth century.}
\label{fig:gradcam_dating}
\end{figure*}

Resorting to the Grad-Cam algorithm, it is possible to individuate the cues that led the deep learning models employed in this work to determine the specific year of a photo. From a socio-historical perspective, this analysis may help verifying whether such cues correspond to visual factors which are recognized as representative of a specific year or period.

To this aim, we proceeded analyzing the grad-cam versions of the correctly classified photos, reported in Fig.~\ref{fig:gradcam_dating}, which, for the sake of correctness and simplicity, have all been produced starting from IMAGO dataset images representing two people. In particular, each row corresponds to a specific decade and includes the grad-cam of an IMAGO image in the first column, the grad-cams of the two corresponding IMAGO-FACES images in the second and third columns and, finally, the grad-cams of the two derived IMAGO-PEOPLE images in the fourth and fifth columns, respectively.

From this Figure, it is possible to see that the single-input classifiers (IMAGO, IMAGO-FACES and IMAGO-PEOPLE, respectively), which processed the whole image, its faces and its people's full figures, focused on different regions (red areas in Fig.~\ref{fig:gradcam_dating}). This corroborates the finding that images contain complementary knowledge, located in different segments, that can be exploited to improve the performance of the dating model, as discussed in Section \ref{sec:results_dating}. In other words, not only the faces of the people represented in a given image, but also their full figures and the whole photo can support a more accurate dating.
This argues for the fact, which is perfectly comprehensible in socio-historical terms, that also clothing and furniture components, as well as their style, play an important role in the temporal classification of a picture.

\begin{figure*}[h]
\centering
\begin{adjustbox}{width=0.85\linewidth}
\begin{tabular}{m{22mm}!{\vrule width 2pt}m{24mm}m{24mm}m{24mm}m{24mm}m{24mm}!{\vrule width 2pt}}
    \multicolumn{1}{c!{\vrule width 2pt}}{} & \multicolumn{5}{c!{\vrule width 2pt}}{\textbf{IMAGO}} \\
    \textbf{Affectivity} &
    \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/AFFETTIVITA_AFFETTIVITA_13122_pnp.jpg} &
    \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/AFFETTIVITA_AFFETTIVITA_10263_pnp.jpg} &
    \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/AFFETTIVITA_AFFETTIVITA_18696_pnp.jpg} &
    \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/AFFETTIVITA_AFFETTIVITA_3798_pnp.jpg} &
    \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/AFFETTIVITA_AFFETTIVITA_9778_pnp.jpg} \\
    \textbf{Fashion} &
    \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MODE_MODE_6193_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MODE_MODE_6414_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MODE_MODE_666_mq.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MODE_MODE_726_mq.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MODE_MODE_7450_pnp.jpg} \\
    \textbf{Motorization} & \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MOTORIZZAZIONE_MOTORIZZAZIONE_10698_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MOTORIZZAZIONE_MOTORIZZAZIONE_3625_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MOTORIZZAZIONE_MOTORIZZAZIONE_3729_pnp.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MOTORIZZAZIONE_MOTORIZZAZIONE_375_mq.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MOTORIZZAZIONE_MOTORIZZAZIONE_4516_pnp.jpg} \\
    \textbf{Music} &  \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MUSICA_MUSICA_11318_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MUSICA_MUSICA_13125_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MUSICA_MUSICA_33477_pnp.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MUSICA_MUSICA_493_mq.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/MUSICA_MUSICA_ma025_g.jpg} \\
    \textbf{Politics} & \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/POLITICA_POLITICA_11060_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/POLITICA_POLITICA_11073_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/POLITICA_POLITICA_15023_pnp.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/POLITICA_POLITICA_16724_pnp.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/POLITICA_POLITICA_16735_pnp.jpg} \\
    \textbf{Rites} & \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/RITI_RITI_11050_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/RITI_RITI_4250_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/RITI_RITI_4548_pnp.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/RITI_RITI_642_mq.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/RITI_RITI_613_mq.jpg}  \\
    \textbf{School} & \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/SCUOLA_SCUOLA_11602_pnp.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/SCUOLA_SCUOLA_22278.jpg} &   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/SCUOLA_SCUOLA_32179_pnp.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/SCUOLA_SCUOLA_33118_pnp.jpg}&   \includegraphics[width=24mm,height=24mm]{images/grad_cam_semantic/SCUOLA_SCUOLA_6751_pnp.jpg} \\
\end{tabular}
\end{adjustbox}
\caption{Grad-Cam analysis of socio-historical context of pictures within IMAGO.}
\label{fig:gradcam_semantic}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=.49\linewidth]{images/grad_cam_semantic_wrong/AFFETTIVITA_MOTORIZZAZIONE_7599_pnp_probability_classes.png}
\includegraphics[width=.49\linewidth]{images/grad_cam_semantic_wrong/LAVORO_SCUOLA_cs006_g_probability_classes.png}
\caption{Grad-Cam analysis related to wrong classifications: these images exhibit the semantic overlapping of different socio-historical classes (\textit{Affectivity} recognized as \textit{Motorization} and \textit{Work} recognized as \textit{School}).}
\label{fig:wrong_gradcam_classification_1}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[width=.49\linewidth]{images/grad_cam_semantic_wrong/MOTORIZZAZIONE_MODE_6789_pnp_probability_classes.png}
\includegraphics[width=.49\linewidth]{images/grad_cam_semantic_wrong/AFFETTIVITA_MODE_12181_pnp_probability_classes.png}
\caption{Grad-Cam analysis related to wrong classifications: these images exhibit the bias of the neural network for different socio-historical classes (\textit{Motorization} and \textit{Free-time} recognized as \textit{Fashion}).}
\label{fig:wrong_gradcam_classification_2}
\end{figure*}

\subsection{Socio-historical context classification task}
\label{subsec:socio_gradcam}

Considering now the socio-historical context classification task, we show a sample of correctly classified IMAGO images processed by the Grad-Cam algorithm in Fig.~\ref{fig:gradcam_semantic}. Per each row, starting from the top, five exemplary images belonging to the \textit{Affectivity}, \textit{Fashion}, \textit{Motorization}, \textit{Music}, \textit{Politics}, \textit{Rites} and \textit{School} classes are produced. Such images are representative of the concrete and abstract concepts which the IMAGO classifier learned for each class.

More in detail, specific gestures which characterize the affection between people (e.g., hugs, kisses, hold a baby, handshakes) as shown in the first row of Fig.~\ref{fig:gradcam_semantic} are characteristic of the \textit{Affectivity} class.
Specific objects like earrings, necklaces and lapels, but also particular hairstyles, are used to classify a picture as belonging to the \textit{Fashion} class (second row of the Figure).
All kinds of vehicles, as well as musical instruments, are put to good use to recognize a given picture as member of the \textit{Motorization} or the \textit{Music} classes, shown in the third and fourth rows, respectively.
The presence of a political banner is typical of pictures in the \textit{Politics} class (fifth row).
The model distinguishes the objects and gestures (e.g., white dress, flowers, pour a drink, cheers) that characterize the \textit{Rites} class (sixth row). Finally, children wearing school uniforms, as well as school gear (e.g., books, pens, desks) are used to recognize pictures in the \textit{School} class (last row). Please note that the same type of analysis was not possible for the \textit{Free-time} and \textit{Work} classes. In particular, for the former it was not possible to individuate distinguishing cues which clearly helped identifying the images there contained (confirming the quantitative results reported in Section \ref{subsec:result_socio_historical}). The latter class, instead, was difficult to correctly recognize as its cues overlapped with the \textit{Affectivity}, \textit{Fashion} and \textit{Free-time} classes, as exhibited in Fig. \ref{fig:conf_matrix_semantic}.

It is not surprising that the model was able to correctly classify pictures belonging to the \textit{Motorization} and \textit{Music} classes, as these are clearly characterized by specific objects, but, more importantly, already part of the model pre-trained on ImageNet. However, also for the majority of the other classes (not studied so far in literature, to the best of our knowledge), the model is able to isolate and focus on the details which distinguish them.

However, all that is gold does not glitter. Fig.~\ref{fig:wrong_gradcam_classification_1} reports a few misclassified examples. From the leftmost picture and its probability histogram, it is possible to see that a photo containing a car was classified as belonging to the \textit{Motorization} class, but the ground truth label assigned to the picture was \textit{Affectivity} (two people are hugging each other in front of the car).
Instead, the rightmost picture and its corresponding probability histogram show that a picture depicting a school class was classified as belonging to  \textit{School}, while the actual one was \textit{Work}  (a teacher is standing in the rightmost part of the picture).
Such ambiguities may be traced back to the fact that the IMAGO dataset has been labeled by human classifiers which could adopt different points of view. In fact, the leftmost picture presented in Fig.~\ref{fig:wrong_gradcam_classification_1}, may be classified as belonging to the \textit{Motorization} class or to the \textit{Affectivity} one, without be mistaken in any of the two cases. Likewise, the rightmost picture could be classified as belonging to the \textit{Work} class, when adopting a teacher's point of view, but also to the \textit{School} one, when instead adopting a students' perspective. We are evidently experiencing a well known phenomenon in cataloguing studies, known as the observer's bias \cite{observer_bias}. In such setting, also the deep learning model becomes an observer and as such provides classifications which reflect the knowledge it has acquired. The existence of the observer's bias emphasizes the importance of the Top-$k$ classification (described in Section \ref{subsec:result_socio_historical}) and may open to the possibility of investigating the use of  multiple socio-historical labels for an image.

Not related to the observer's bias, it turns out that there have also been misclassifications due to neural network biases. Two representative examples are reported in Fig.~\ref{fig:wrong_gradcam_classification_2}: both of them were classified belonging to the \textit{Fashion} class but the leftmost picture should belong to \textit{Motorization} while the rightmost to \textit{Affectivity}.
The classes provided by the model in this case are quite off target, since the leftmost picture depicts a man standing by a plane, while the rightmost contains a couple of people hugging each other. These errors were driven by the fact that the model here focused on secondary details of the pictures (i.e., fashion aspects), rather than on the main ones (i.e., the structure of the plane and the hugging gesture).

\section{Conclusions and future works}
\label{sec:conclusions}
In this paper we introduced the IMAGO dataset, which is composed by pictures belonging to family albums. We trained and tested single and multi-input deep learning models which exploited different regions of a given photo to identify the year in which it was taken. We showed how a higher number of faces and people positively affected the final accuracy of the dating model. Furthermore, we exhibited the existence of complementary information that could be exploited by the model to improve the accuracy of the dating task.

In addition, to the best of our knowledge, this is the first work to introduce a socio-historical context classification task of family album images. This task consists in identifying the sociological and historical context of a picture, according to the definitions provided by socio-historical scholars~\cite{calanca2011italians}. We observed that the complementary knowledge learned in advance from people's faces and full figures did not help improving the classification accuracy, as relevant regions and their spatial relationships may be located throughout a photo. We also observed that the deep learning models, trained for the socio-historical context classification task, were able to correctly classify abstract concepts (i.e., affectivity, politics, rites) by focusing on relevant and actual cues (e.g, kisses and hugs for the affection, banners for the politics class, marriage cakes and toasts for the rites class).

For the possible developments of this work, we resort to what Antonio Gramsci said: ``The starting-point of critical elaboration is the consciousness of what one really is, and is 'knowing thyself' as a product of the historical processes to date, which has deposited in you an infinity of traces, without leaving an inventory.'' Indeed, when focusing on the image dating or socio-historical classification task, scholars perform analyses which resort at once to different sources of information (e.g., journals, magazines, archival documents), as well as to the traces mentioned by Antonio Gramsci. The same objectives, at the state of the art, are pursued in a completely different way when resorting to computer vision techniques, which are nowadays limited to what can be directly learnt from an image. In essence, historical information cannot be surely drawn when solely analyzing one of the traces that have been left on the ground, but should also consider the influence of space, time and of the human observer. Interestingly, in this work we were able to highlight how such traces also permeated the socio-historical classification.
Clearly, this is only one step in the direction of creating holistic models which may take into account all the processes involved in the complex socio-historical domain.

Finally, this line of work may benefit from further investigations which included: (i) a larger and more balanced amount of data, (ii) an additional exploitation of the complementarity between the different image regions and (iii) a more thorough analysis of the concept of deep neural networks viewed as observers within the socio-historical context classification task.

\ifCLASSOPTIONcompsoc
  \section*{Acknowledgments}
\else
  \section*{Acknowledgment}
\fi

This work was supported by the University of Bologna with the Alma Attrezzature 2017 grant and by AEFFE S.p.a. and the Golinelli Foundation with the funding of two Ph.D. scholarships.

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\bibliographystyle{IEEEtran}

redpajama_set_name: RedPajamaArXiv


Szabolcs Brickner – Alferink Artists Management B.V.
Belgium's Queen Elisabeth Competition, one of the world's most prestigious classical music competitions for young performers. Szabolcs Brickner sings roles such as Tamino (Mozart – Die Zauberflöte), Oronte (Händel – Alcina), Temistocle title role (J.C. Bach), Belmonte (Mozart – Die Entführung aus dem Serail), Don Ottavio (Mozart – Don Giovanni), Orfeo (Haydn – Orfeo ed Eurydice, L'anima del filosofo), Jaquino (Beethoven – Fidelio), Nemorino (Donizetti – L'elisir d'amore), Roméo (Gounod – Roméo et Juliette), Lensky (Tchaikovsky – Eugen Onegin), Alfredo (Verdi – La Traviata), Werther title role (Massenet), and he is a regular guest at the Bayerische Staatsoper, Frankfurt Oper, Nationaltheater Weimar, Nationaltheater Mannheim, Hungarian State Opera House, Opéra du Rhin in Strasbourg, Gergiev Festival Rotterdam, a.o. The tenor's concert repertoire includes Handel's Saul and Messiah, Magnificat by Carl Philipp Emanuel Bach, Mozart's Krönungsmesse and Requiem, Die Schöpfung and Die Jahreszeiten by Joseph Haydn, Beethoven's 9th Symphony, Mendelssohn's Elias, Dichterliebe by Schumann, Verdi's Requiem, and Mahler and Richard Strauss Lieder with orchestra. Szabolcs Brickner works with orchestras such as Orchestre de Paris, La Monnaie Symphony Orchestra, La Petite Bande, the Orchestra of the Opéra Royal de Wallonie, Danish Radio Sinfonietta, the Orchestre Philharmonique du Luxembourg, the National Orchestra of Belgium, the Orquestra Ciudad de Granada, Rotterdam Philharmonic Orchestra, the National Philharmonic Orchestra of Hungary, the Budapest Philharmonic Orchestra, the Israel Philharmonic Orchestra, and with conductors such as Giuliano Carella, Nicolas Chalvin, Adam Fischer, Valery Gergiev, René Jacobs, Nicola Luisotti, Kurt Masur, Günter Neuhold, Paolo Olmi, Kazushi Ono, Helmuth Rilling, Michel Tabachnik, and Christopher Warren-Green. Recent and future productions include a.o. Die Zauberflöte, La Traviata, Der Prolog and Quint (Britten – The Turn of the Screw) and Der Prinz (Prokofiev – L'amour des trois oranges) in Mannheim, Die Entführung aus dem Serail, L'elisir d'amore and Alfred (J. Strauss – Die Fledermaus) in Budapest, and Don Giovanni in Toulon. Since season 15/16 Szabolcs Brickner is part of the ensemble of Volksoper Wien, where he sings a.o. the role of Alfredo and Tamino.


Our company is a manufacturer of women clothing since the last 30 years. We have focused on making sure that her company provides women with dazzling affordable luxury apparel. Our company has become leading brand that is sold at the finest specialty and department stores around the world. We offer products have distinctive style, excellent quality and unequaled craftsmanship all combined with genuine value.


Get Ready for these 2019 Bathroom Trends!
Are you ready to ring in the new year with a new bathroom? New trends are popping up just in time for a complete bathroom remodel or even just new coat of paint. Check out some of these new bathroom trends to see if there are any that inspire you for 2019!
Traditional bathrooms are a thing of the past. Instead, spa styles with clean and minimal appearances are more popular. Designers are starting to favor transitional bathrooms with classic and clean lines, wood cabinets, and painted walls. If you prefer a cool-temperature color palette and brushed nickel fixtures, the transitional trend might be for you.
If you prefer a more contemporary style, you might want to consider installing a soaking tub for a sculptural effect. Contemporary bathrooms also have porcelain tile covering larger surface areas for easy cleaning and frameless cabinetry.
Freestanding vanities, linen storage cabinets, and open shelving are in while medicine cabinets and over-the-toilet storage are out. Undermount sinks are also popular. The overall design for bathrooms is focused on sleek, clean lines that eliminate the appearance of clutter and gives the space a feeling of zen relaxation.
There are many trends that let homeowners explore different options for sprucing up their existing space. For example, converting all light bulbs to LEDs and installing energy-efficient fans are a big step in sustainability. Many homeowners are installing water-saving shower fittings in guest bathrooms to be more eco-friendly.
The wet room, instead of a closed-in shower, is becoming more popular because it follows the spa trend. Rather than having just a shower or a bathtub, the entire bathroom becomes the shower and a linear drain helps control water. Combined with radiant floor heating and steam showers, you could take your bathroom from a simple lavatory to an oasis of comfort.


opportunity to provide its workforce with required skill sets and knowledge to enable them to contribute substantially to the economic growth of India.
period of 2010-2014 reports that over 109.73 million additional skilled manpower will be required by 2022 across different sectors.
In the last two years, the government has taken a host of initiatives to channelize the efforts and provide impetus to the Skill Development ecosystem.
To steer and coordinate the current skilling initiatives with quality deliverables, the Ministry of Skill Development and Entrepreneurship (MSDE) was created in November 2014 to drive the 'Skill India' agenda in mission mode.
Central Government Programs and NSDC associated training partners in the private sector.
MSDE has made great strides in a relatively short span of time and the key policy initiatives have been launched to support the skilling mandate.
Launched in July 2015 with the mission statement "Kaushal Bharat Kushal Bharat', PMKVY is the flagship scheme of MSDE aimed to benefit 1 crore youth. PMKVY is being implemented through National Skill Development Corporation (NSDC).
A new version of PMKVY was launched on July 13, 2016 with modification and continuation of PMKVY as 'Skill Development Component' of umbrella PMKVY with a target to skill 10 million people over four years (2016-2020) at an outlay of INR 12,000 crore.
• A total of 19.8 lakh candidates (17.8 lakhs in fresh training and 1.8 lakhs under Recognition of Prior Learning (RPL)) have been enrolled, out of which 17.93 lakh have been trained and 11.9 lakh have been certified. The details of sector- wise breakup is mentioned in Annexure-I.
• The number of ITIs in the country has increased from 10,750 in May 2014 to over 13,105 in May 2016. More than 1141 new ITIs have been added and 1.73 lakh seats have been increased in the last one year.
• 15,000 instructors have been trained by Central Institutes of Directorate General of Training (DGT).
• Distance learning infrastructure has been created and over 18000 trainers have been trained in last two years.
• MSDE has also enabled opening up of 8 new Regional Vocational Training Institutes (RVTIs) for women in skill development. Five of which were opened during FY 2015- 16, 2 of which have been operational since August 2015, 3 are to be opened in FY 2016- 17.
• 2,33,000 youth were awarded certificates from ITIs, and 18,000 plus graduating students received job offer letters on the occasion of World Youth Skills Day.
Read full article on Achievement of Skill Development Sector.


text:
Emeritus the Venerable Mina Smallman
Huntingdon Road
CB2 0DF
Mina Smallman is a retired Anglican priest and former High School teacher. She studied at the Central School of Speech and Drama. Before being ordained at St Paul's Cathedral, she was a former Assistant Principal of John Kelly Girls' Technology College. She served as the Archdeacon of Southend in the Diocese of Chelmsford from 2013-2016. In recognition of her work, she was awarded the title of Emeritus the Venerable.
In 2020, two of Ms Smallman's daughters, Nicole Smallman and Bibaa Henry, were found murdered in a park in north London. In July 2020, Danyal Hussein was convicted of their murders and sentenced to life imprisonment. Ms Smallman and her husband Chris have criticised the Metropolitan Police Service for being initially dismissive and mishandling the missing persons investigation, questioning whether the murder of women of colour is treated differently by police and society at large. In December 2021, two Metropolitan Police Officers who took photographs of the murdered sisters and shared the images to WhatsApp – conduct described by Ms Smallman as "a betrayal of catastrophic proportions" – were convicted and jailed for 33 months.
Mina Smallman has partnered with Stacey Dooley to tell her story in the documentary, Two Daughters for BBC Two and iPlayer. She was included in the BBC's 100 Women List for 2021, for her trailblazing work as the first female Church of England archdeacon from a black or ethnic minority background, and for her campaigning to make UK streets safer and to reform the police.