RICHARD FOURIE has a number of good rides at Kenilworth today.
Richard Fourie has been riding exceptionally well and has notched up an impressive record in his last 30 rides with seven winners and 13 places.
And he looks the force to be followed at Kenilworth on Wednesday when he teams up once more with trainer Glen Kotzen.
The first two races are filled to the rafters with first timers so could prove very tricky – follow the money. Fourie makes for great Eachway bets in Race 3 aboard Lindleys Lane and in Race 5 on Shizam.
Up next he rides Elusive Singer in Race 6, a MR 72 Handicap over 1000m. This Kotzen-trained gelding wasn't disgraced last time out when only a 1,10-length second to Straat-Kind. That run suggests the drop in trip should prove healthy for him here. He shoulders a modest 56.5kg and will jump from a not so bad No 7 draw. His biggest danger could come by way of Prince Alfred who left quite an impression in his last start when skipping 2.25 lengths clear of the field. This Adam Marcus-trained three-year-old will however shoulder 60kg – 3.5kg more than Elusive Singer which could be a massive disadvantage.
Silvan Star has been extremely unlucky over her last two starts, she could be returning to winning ways in Race 7, a FM 89 Handicap over 1600m. She meets stronger now but could have enough to edge out this field. She's well drawn in stall No 5 and will carry 59.5kg. The return to her favoured trip of 1600m will make her very hard to beat. Should she fluff her lines the newly formed alliance of Candice Bass-Robinson and Aldo Domeyer could pick up the pieces with Miss Marker who has run against stronger and held her own.
To close off the meeting Fourie will steer home Quickfire who improved on her debut run last time out when winning comfortably by 1.25 lengths. This three-year-old son of Querari is in pursuit of a quick double of wins and could prove the right horse in this competitive 12-horse race. She's well drawn in gate No six and will have 58.5kg on her back.


May 9th, 2000 / 5:55 p.m.
Mr. Speaker, I want to say a few words in support of the hon. member's motion. Two of my colleagues from New Brunswick have already spoken to this motion, and I want to say a few words on behalf of seasonal workers in the province of Newfoundland and Labrador.
First, let me make one thing absolutely crystal clear. The Liberal government's changes to the unemployment insurance system, the so-called employment insurance system, were and still are a direct attack on seasonal workers in Atlantic Canada in particular and in rural Canada in general.
The insurance system for the unemployed has been gutted. We are all very much aware of that. The changes to the UI system, the so-called reforms, introduced by the Liberals make it harder to qualify for benefits. When one does qualify, it is for fewer benefits and for a shorter period of time.
I can see why they dropped the word unemployment from the title of the new program. In 1989, 87% of Canada's unemployed qualified for benefits. Today only 36% of Canada's unemployed qualify for benefits in a system that has a multibillion dollar surplus.
It is no longer workers' insurance against becoming unemployed. It has become a system which has forced 50,000 Newfoundlanders to pull up stakes and move to other provinces in central and western Canada.
If people cannot or will not move from their home communities in rural Newfoundland, there is only one option. If work is not available, then they have to go on social assistance. For those who cannot move away it has become welfare insurance, not employment insurance.
In rural Newfoundland and many other places in Atlantic and rural Canada there are not many job options available. Most of the work which is available is seasonal work. At one time people could work seasonally, draw unemployment insurance and head out to look for jobs elsewhere. If they could not find jobs they would have the assurance of returning to their seasonal work when it became available.
Let me remind hon. members that it is not a crime to be a fisherman. It is not a crime to be a logger. It is not a crime to be a construction worker. These are trades that make valuable contributions to the people of Newfoundland and valuable contributions to the people of Canada. They should be recognized that way, but it has all changed.
If people are lucky enough to find seasonal work and they do qualify for EI benefits, then in all likelihood the benefits will not carry them to the point of returning to the seasonal work. People will live part of the year with no income at all. At that point they must deplete their savings. They probably have to turn to extended family for support and help, or they have to move away to find work. If that option is not available, they have to go on social assistance.
It would not sound so bad if the EI system were broke, depleted of benefits by the unemployed, but the system is not broke. It is awash with cash. The system is not broke, it is broken. It was deliberately broken by the governing Liberals. That was a piece of cold, calculated, social engineering on the part of the Liberals.
Their plan proved to be quite effective. Their strategy was to tighten up the EI system, making it impossible for someone to earn a livelihood at seasonal work, which would force the migration of Atlantic Canadians to areas of higher employment, and let welfare take care of the rest of the people who could not move or who would not move.
That piece of social engineering has cost my province about 10,000 people a year over the past few years. It has cost the Liberal Party all of its seats in Atlantic Canada. It will continue to cost the Liberal Party all of its seats in Atlantic Canada unless it does something about the EI system.
This piece of social engineering has cost the province of Newfoundland and Labrador $1 billion in lost EI revenues since the Liberals came to power in 1993. The city of St. John's, just to give a small indication of how it is spread out, is losing $78 million annually. My riding of St. John's East, which is comprised of the eastern part of the city of St. John's and the rural area along Conception Bay, is losing $50.2 million annually. St. John's West, where the Liberals hope to win the byelection on Monday, is losing $56.3 million annually.
Let us look at the riding of Burin—St. George's. Joining the Liberal side was supposed to be good for that riding. How much is it costing the people of Burin—St. George's in lost EI revenue? It is costing them $81.7 million a year.
These are not numbers to which only accountants can relate.
The people kicked out of the EI system have either moved away or they have had to go on social assistance. For the most part there was no new work in the area.
EI cheques, believe it or not, were not replaced by cheques from shiny, new, year-round factories. This money has simply gone from the local scene. It is sitting in the pot right here in Ottawa to help the finance minister balance his budget. This is money no longer being spent in restaurants, gas stations and grocery stores in Atlantic Canada and rural Canada. As I said, in Newfoundland we have lost $1 billion in EI revenue. In an economy the size of ours that has to hurt, and it has hurt many families and many communities.
Much has been said about the Liberal Party wanting to regain seats in Atlantic Canada in the next election. The Liberals have been promising to restore some of the benefits which they took away under the old UI system, but so far all they have come up with are a lot of empty words, a lot of empty rhetoric. I have a feeling that is the way it will remain.
In the last budget the government made a lot of extending EI sponsored maternity leave from six months to a full year. What the government did not say in the budget was that with its new hours based EI system only 31% of unemployed Canadian women actually qualify for these benefits. Given the fact that it is harder to qualify for maternity benefits than regular benefits, that is very cold comfort indeed.
The government's so-called reforms to the UI system and its creation of the new EI system was a deliberate and well planned attack on rural and Atlantic Canada, and unfortunately Canadian women as well. These changes might have gone down well in the more prosperous parts of the country, but they have been devastating to Atlantic Canada in general and to Newfoundland in particular.
Seasonal work is not a crime. As I said a moment ago, it is not a crime to be a logger, it is not a crime to be a fisherman and it is not a crime to be a construction worker. It certainly is not a valid excuse to force the migration of rural Canadians. However, this is what the government has done, and quite deliberately.
At the last election the Liberals paid a very stiff price for what they did to Atlantic Canadians. If they want votes in Atlantic Canada and in the province of Newfoundland next time they had better change their tune. The fishermen, the loggers and the construction workers of Atlantic Canada will not stand for what the government has done to them over the last three, four or five years.
I support the motion and I commend the member for having brought it to the floor of the House.


Apple today seeded the first beta of an upcoming iOS 10 update to public beta testers for testing purposes, just under two weeks after releasing iOS 10 to the public and one day after providing the iOS 10.1 beta to developers.
Beta testers who have signed up for Apple's beta testing program will receive the iOS 10 beta update over-the-air after installing the proper certificate on their iOS device.
iOS 10.1 introduces a "Portrait" mode for the iPhone 7 Plus, which was first shown off when the iPhone 7 and iPhone 7 Plus debuted on September 7. Portrait mode is designed to mimic the kind of shallow depth of field images that can be taken with a high-end DSLR, with a front subject that stands out over a blurred background.
The two cameras in the iPhone 7 Plus capture images, which are scanned by the built-in image signal processor. Machine learning techniques are use to recognize people and/or foreground images, keeping people and main objects in while applying an artful blue or "bokeh" to the background.
Portrait mode is an iPhone 7 Plus-only feature because it requires two images to create a depth map. iOS 10.1 also likely includes bug fixes and behind-the-scenes updates to address issues that have popped up since release.
iOS 10 offers features like a redesigned Lock screen, a Siri SDK to allow third-party apps to integrate with Siri, and a completely overhauled Messages app with stickers, a full App Store, Digital Touch, Bubble Effects, and more. Other new additions include a dedicated "Home" app for HomeKit users, new facial and object recognition capabilities in Photos, and redesigned Maps and Apple Music apps.
Ah, how could I forget!! Does anyone want to buy my 7+ 128GB? I need to buy an S7 now.
Let's see how funny you are when Galaxy S8 comes out this fall?
People were also saying "wait until the Note 7 is released. It's gonna destroy the competition."
As far as I recall, it only destroyed hotel rooms.
If you think that ANY smartphone has come close to ANY half decent DSLR, then you know nothing about photography or physics.
As for 'surpassed' sure. And let me guess, it also transforms into a space ship and can fly you to the moon? I'll join you there for cocktails this evening.
I have to say that I'm massively impressed with Apple's recent commitment to software QA and stability. I don't remember so many betas and GMs being thrown about before release.
Plus the more devices out there, the more bugs people will encounter. Apple's growth over the last 5 years has been exponential.
They've learned a lot from buggy disasters on launch date. iOS 7 and OS X Lion immediately spring to mind.
Is there anything else new in 10.1? Sounds like its only a useful update for those of us that have 7 Pluses.
Did you see this TechCrunch article ('https://techcrunch.com/2016/09/21/hands-on-with-the-iphone-7-plus-crazy-new-portrait-mode/') on the iPhone's Portrait mode? Some of their shots are damn good for a smartphone, especially the dogs and the mom holding a baby. The latter shows off how the depth mapping applies bokeh in proportion to the distance from the object in focus. In other scenes the effect fails but remember that this is Apple's first implementation which is still in beta. With hundreds of thousands field testing the effect it should improve dramatically for the iPhone 8.
Huawei also uses the same technique on their dual lens cameras so there will be competition to push Apple as well. I've not seen many shots by the Huawei but it also looks quite good and stores the depth maps so that one can manipulate the depth of field in post. Too bad Apple's so concentrated on dumbing down their features because that would be awesome. Maybe a 3rd party app will offer that feature.
And I'm sad, that iPhone are not able to do this.
It's not perfect, but I'm sure it will improve over time. As it is, since you get both the original and depth mapped image you can open them as layers in Photoshop and clean them up using a layer mask. Here is an example of the original off the camera and a 30 second 'fix' in Photoshop. This was just a test snapshot.
Yes no problem, just go to SamsungRumors and ask there.
I'm sure a lot of Samsung fans are on the market searching for iPhones to buy after their expolsive Samsung device tried to kill them.
Any Developer here can tell us if there are any significant bug fixes or performance improvements in this version?
I have an iPhone 6S so I don't care about portrait photos.
Is this combining images from the two lenses to build the portrait with the extra bokeh or is this just an algorithm to apply Gaussian blur or something like that to the background layer and then merges the images?
Read the Tech Crunch article. It's way more complex and way better than either of those.
Wrong in every way except the fast autofocus (which this also has).


PRO – Professional Referral Organization is the newest industry specific, referral sharing organization in the Skokie Chamber of Commerce & Industry.
During each meeting, presentations are given by two members to spotlight their businesses. The goal is for the members to teach the others how to be their sales force.
PRO meets every Wednesday morning from 7:00 a.m. to 8:30 a.m. for networking, breakfast and passing qualified leads among the organization's members. Currently, PRO meeting are held at Maggiano's Little Italy, 4999 Old Orchard Center, Skokie.
For further information about PRO or PRO Membership, contact the Skokie Chamber of Commerce & Industry, (847) 673-0240.


Ofgem has revealed 11 projects that are eligible to compete for a share of the annual £64m of funding from the £500m Low Carbon Networks Fund.
The Low Carbon Networks Fund was set up by Ofgem as part of the 2010-2015 electricity distribution price control. It will encourage Britain's electricity distribution network owners (DNOs) to develop innovative projects that will help energy networks and consumers make the transition to a low carbon economy.
Projects could see DNOs partnered with energy suppliers, technology providers or others to explore how electricity networks can help facilitate and enable low carbon and energy saving initiatives such as electric vehicles, or investigate the potential of the planned roll-out of smart meters.
The 11 projects are from Western Power Distribution, Southern Electric Power Distribution, Scottish Hydro Electric Power, SP Manweb, SP Distribution, two from Electricity North West, EDF Energy, two from Central Networks, Northern Electric and Yorkshire Electricity.
Each of Britain's seven electricity distribution companies (DNOs) submitted at least one project in the first stage of the annual competition in June. The proposals were screened by Ofgem to ensure they were eligible to be considered for funding.
The DNOs can now develop these proposals for entry into the final evaluation stage of the competition next month.
The submitted proposals will be scrutinised by an expert panel advising the Gas and Electricity Markets Authority, Ofgem's governing body, which will make the final decision on the projects that should be selected for funding. The winning projects will be announced in December.


Beginning producers are welcome! We combine theory with practice and offer practical knowledge about the preparation of a professional film development process.
Show us your point of view! DOK PRO Programme is the best way to make a professional documentary film.
SCRIPT – simple way to your screenplay! SCRIPT Programme helps participants develop their ideas into the first or second draft of the script.
DEVELOPMENT LAB Programme is an intensive course for film directors at a masterclass level.
Special award for experimental films, founded by Museum of Modern Art in Warsaw, Polish Film Institute & Wajda School.
Are you interested in documentary filmmaking? Film Kindergarten is a programme dedicated for high school students.
Discover the visual style for your film with EKRAN+ European Training Programme. Success at international film festivals starts in Wajda Studio!
Wajda School's online program is the first step on beginner's way to making movies.
Joint project of CANAL+ and WAJDA SCHOOL designed to seek & create premium tv series for Polish division of CANAL+.
Renowned Polish contest for screenwriters at the beginning of their professional career.


Even as we arrive at an actual mathematical formula for lasting love, we remain tragicomically unskilled at anticipating — to say nothing of domesticating — the unpredictable, nonlinear dynamics of the human heart.
That's what novelist and Believer magazine founding editor Heidi Julavits, who joins the ranks of history's notable diarists, touches on with equal parts gentleness and precision in a couple of related meditations from the kaleidoscopically illuminating The Folded Clock: A Diary (public library).
The marriages that last are the ones in which the two members regularly develop (but do not act upon) extramarital infatuations.
This, of course, makes sense — we know that love is a mode of "interbeing" and a "dynamic interaction" in which the opportunity to choose each other over and over is what sustains the longevity of a couple's bond.
Crushes thrive in small spaces. Humans must be programmed to respond positively when faced with a small sampling of other humans in, say, caves.
This has happened to me many times. It happened to me on a canoe trip; the minute we returned to civilization, I recanted my crush on the guy I'd angled to sit next to at the nightly campfires. I have been so cognizant of this phenomenon, and its inevitability, that I got nervous in college while waiting to hear where in France I was to spend my semester abroad, because I knew that a guy my friend was dating, someone I'd always found abstractly cute, was also going to France. Fortunately we were sent to different cities. Had we been in the same city, I am certain we would have fallen in love, or the sort of love that occurs in those situations, call it what you will, probably a mistake. This is also why I get nervous about going to art colonies, especially now that I am happily married to a man I met at an art colony. I don't want to fall for anyone else — I am pointedly not looking to fall for anyone — but these situations conspire against our best intentions. Art colonies, often located in remote woods or on beautiful estates, are communities in which all the residents sever ties to the real world within hours of arrival; they are like singles mixers for the married or otherwise spoken for. (I was married when I met my now-husband, who was otherwise spoken for.) When I arrive at a colony these days, I take a measure of the room, I identify the potential problems, I reinforce my weak spots, and then I relax.
Illustration from 'The Missing Piece Meets the Big O,' Shel Silverstein's minimalist allegory of true love. Click image for more.
Unlike Thoreau, I have no fondness for sparse living. I do not covet hardship. I liked the idea of Walden, however, because it was written in a cabin in the woods. It's a sort-of nature book that took place (at least the writing did) inside. Interiors are where I do my exploring. Interiors are my nature. I am an outdoorsman of the indoors… When I am there I am happiest. In my outbuilding I am sucking out optimum marrow.
Couple with some actual Thoreau, then fortify with the great Zen teacher Thich Nhat Hanh on how to love, Dan Savage on the unsettling secret of lasting love, Wendell Berry on freedom and marriage, and Kafka's beautiful and heartbreaking love letters.


iTunes is the world's easiest way to organize and add to your digital media collection. We are unable to find iTunes on your computer. To download and subscribe to The Documentary Podcast by BBC World Service, get iTunes now.... Not by trying to shoot from a mile away and get dodged, but by having already prepared a landmine to jump close to the pharah, try to shoot her, and use your other CD'ed land mine to take out half her HP after your attempt to hit her with your gun.
Fictional character biography Early life. Thea Queen was born on January 21, 1995 in Starling City to Robert (Jamey Sheridan) and Moira Queen (Susanna Thompson) (although it is later revealed that Thea is Malcolm Merlyn's (John Barrowman) biological daughter) and the younger half-sister of Oliver Queen (Stephen Amell). how to make fbi allow unsigned cia Fictional character biography Early life. Thea Queen was born on January 21, 1995 in Starling City to Robert (Jamey Sheridan) and Moira Queen (Susanna Thompson) (although it is later revealed that Thea is Malcolm Merlyn's (John Barrowman) biological daughter) and the younger half-sister of Oliver Queen (Stephen Amell).
Landmines are one of the few dangers out there that you can profit from. Disarm one before it blows, and you can sell it for plenty of caps. Disarm one before it blows, and you can sell it for plenty of caps.
Images. Samantha Clayton (died May 17, 2017) was an ex-lover of Oliver Queen and the mother of their son, William Clayton. Samantha was paid $1 million by Oliver's mother, Moira Queen, to pretend she had had a miscarriage and move back to Central City (though she never cashed the check).


text:
\section{Introduction}
Sign language is one of the most commonly-used communication tools for the deaf community in their daily life. It mainly conveys information by both manual components (hand/arm gestures), and non-manual components (facial expressions, head movements, and body postures)~\cite{dreuw2007speech,ong2005automatic}. However, mastering this language is rather difficult and time-consuming for the hearing people, thus hindering direct communications between two groups. To relieve this problem, isolated sign language recognition tries to classify a video segment into an independent gloss\footnote{Gloss is the atomic lexical unit to annotate sign languages.}. Continuous sign language recognition (CSLR) progresses by sequentially translating image streams into a series of glosses to express a complete sentence, more prospective towards bridging the communication gap.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/figure1.pdf}
    \caption{Visualization of class activation maps with Grad-CAM~\cite{selvaraju2017grad} for VAC~\cite{Min_2021_ICCV} (baseline). Top: Original frames. Bottom: activation maps. It's observed that without extra supervision, it fails to locate discriminative face and hand regions precisely. }
    \label{fig1}
  \end{figure}

In sign language, the left hand, right hand, and face play the most important role in expressing glosses. Mostly, they convey the information through horizontal/vertical hand movements, finger activities, and static gestures, assisted with facial expressions and mouth shapes to holistically deliver messages~\cite{dreuw2007speech,ong2005automatic}. As a result, hand and face, are always especially leveraged and incorporated in sign language systems. In isolated sign language recognition, early methods~\cite{freeman1995orientation,sun2013discriminative} leveraged hand-crafted features to describe the gestures and motion of both hands. Recent methods either choose to build a pure pose-based system~\cite{tunga2021pose,hu2021signbert} based on detected keypoints for both hands and face, or construct appearance-based systems~\cite{hu2021hand,boukhayma20193d} with cropped patches for hands and face as collaborative inputs. In CSLR, CNN-LSTM-HMM~\cite{koller2019weakly} builds a multi-stream (hands and face) Hidden-Markov-Model (HMM) to integrate multiple visual inputs to boost recognition accuracy. STMC~\cite{zhou2020spatial} explicitly inserts a pose-estimation network and uses the detected regions (hand and face) as multiple cues to perform recognition. More recently, C$^2$SLR~\cite{zuo2022c2slr} leverages the pre-extracted pose keypoints heatmaps as additional supervision to guide models to focus on hand and face areas.

Although it has been proven effective to incorporate hand and face features to improve recognition performance for sign language systems, previous methods usually come at huge computations with increased training complexity, and rely on additional pose estimation networks or extra expensive supervision (e.g., heatmaps). However, without these supervision signals, we find current methods~\cite{Min_2021_ICCV,hao2021self,cheng2020fully} in CSLR fail to precisely locate the hand and face regions (Fig.~\ref{fig1}), thus unable to effectively leverage these features. To more effectively excavate these key cues but avoid introducing huge computations or relying on expensive supervision, we propose a self-emphasizing network (SEN) to explicitly emphasize informative spatial regions in a self-motivated way. Specifically, SEN first employs a lightweight subnetwork to incorporate local spatial-temporal features to identify informative regions, and then dynamically emphasizes or suppresses input features via attention maps.

It's also observed that not all frames contribute equally to recognition. For example, frames with hand/arm movements of the signer are usually more important than those transitional frames. We present a temporal self-emphasizing module to emphasize those discriminative frames and suppress redundant ones dynamically. Remarkably, SEN yields new state-of-the-art accuracy upon four large-scale CSLR datasets, especially outperforming previous methods equipped with hand and face features, even though they always come at huge computations and rely on expensive supervision. Visualizations verify the effects of SEN in emphasizing spatial and temporal features.

\section{Related Work}
\subsection{Continuous Sign Language Recognition}
Sign language recognition methods can be roughly categorized into isolated sign language recognition~\cite{tunga2021pose,hu2021signbert,hu2021hand} and continuous sign language recognition~\cite{pu2019iterative,cheng2020fully,cui2019deep,niu2020stochastic,Min_2021_ICCV} (CSLR), and we focus on the latter in this paper. CSLR tries to translate image frames into corresponding glosses in a weakly-supervised way: only sentence-level label is provided. Early methods in CSLR usually depend on hand-crafted features~\cite{gao2004chinese,freeman1995orientation} to provide visual information, especially body gestures, hands, and face, or rely on HMM-based systems~\cite{koller2016deepsign,han2009modelling,koller2017re,koller2015continuous} to perform temporal modeling and then translate sentences step by step. The HMM-based methods typically first employ a feature extractor to capture visual representations and then adopt an HMM to perform long-term temporal modeling. The recent success of convolutional neural networks (CNNs) and recurrent neural networks brings huge progress for CSLR. The widely-used CTC loss~\cite{graves2006connectionist} enables end-to-end training for recent methods by aligning target glosses with inputs.

Especially, hands and face are paid close attention to by recent methods. For example, CNN-LSTM-HMM~\cite{koller2019weakly} employs a multi-stream HMM (including hands and face) to integrate multiple visual inputs to improve recognition accuracy. STMC~\cite{zhou2020spatial} utilizes a pose-estimation network to estimate human body keypoints and then sends cropped patches (including hands and face) for integration. More recently, C$^2$SLR~\cite{zuo2022c2slr} leverages the pre-extracted pose keypoints as supervision to guide the model. Despite high accuracy, they consume huge additional computations and training complexity.

Practically, recent methods~\cite{pu2019iterative,pu2020boosting,cheng2020fully,cui2019deep,niu2020stochastic,Min_2021_ICCV} usually first employ a feature extractor to capture frame-wise visual representations for each frame, and then adopt 1D CNN and BiLSTM to perform short-term and long-term temporal modeling, respectively. However, several methods~\cite{pu2019iterative,cui2019deep} found in such conditions the feature extractor is not well trained and propose the iterative training strategy to refine the feature extractor, but consume much more computations. More recent methods try to directly enhance the feature extractor by adding visual alignment losses~\cite{Min_2021_ICCV} or adopt pseudo label~\cite{cheng2020fully,hao2021self} for supervision. We propose the self-emphasizing network to emphasize informative spatial features, which can be viewed to enhance the feature extractor in a self-motivated way.

\subsection{Spatial Attention }
Spatial attention has been proven to be effective in many fields including image classification~\cite{cao2019gcnet,hu2018gather,woo2018cbam,hu2018squeeze}, scene segmentation~\cite{fu2019dual} and video classification~\cite{wang2018non}. SENet~\cite{hu2018squeeze}, CBAM~\cite{woo2018cbam}, SKNet~\cite{li2019selective} and ECA-Net~\cite{wang2020eca} devise lightweight channel attention modules for image classification. The widely used self-attention operator~\cite{wang2018non} employs dot-product feature similarities to build attention maps and aggregate long-term dependencies. However, the calculation complexity of the self-attention operator is quadratic to the incorporated pixels, incurring a heavy burden for video-based tasks~\cite{wang2018non}. Instead of feature similarities, our SEN employs a learnable subnetwork to aggregate local spatial-temporal representations and generates spatial attention maps for each frame, much more lightweight than self-attention operators. Some works also propose to leverage external supervision to guide the spatial attention module. For example, GALA~\cite{linsley2018learning} collects click maps from games to supervise the spatial attention for image classification. A relation-guided spatial attention module~\cite{li2020relation} is designed to explore the discriminative regions globally for Video-Based Person Re-Identification. MGAN~\cite{pang2019mask} introduces an attention network to emphasize visible pedestrian regions by modulating full body features. In contrast to external supervision, our self-emphasizing network strengthens informative spatial regions in a self-motivated way, thus greatly lowering required computations and training complexity.

\section{Method}
\subsection{Framework Overview}
As shown in fig.~\ref{fig2}, the backbone of CSLR models is consisted of a feature extractor (2D CNN\footnote{Here we only consider the feature extractor based on 2D CNN, because recent findings~\cite{adaloglou2021comprehensive,zuo2022c2slr} show 3D CNN can not provide as precise gloss boundaries as 2D CNN, and lead to lower accuracy. }), a 1D CNN, a BiLSTM, and a classifier (a fully connected layer) to perform prediction. Given a sign language video with $T$ input frames $x = \{x_{t}\}_{t=1}^T \in \mathcal{R}^{T \times 3\times H_0 \times W_0} $, a CSLR model aims to translate the input video into a series of glosses $y=\{ y_i\}_{i=1}^{N}$ to express a sentence, with $N$ denoting the length of the label sequence. Specifically, the feature extractor first processes input frames into frame-wise features $v = \{v_t\}_{t=1}^{T} \in \mathcal{R}^{T\times d}$. Then the 1D CNN and BiLSTM perform short-term and long-term temporal modeling based on these extracted visual representations, respectively. Finally, the classifier employs widely-used CTC loss to predict the probability of target gloss sequence $p(y|x)$.

To emphasize the informative spatial and temporal features for CSLR models, we present a spatial self-emphasizing module (SSEM) and a temporal self-emphasizing module (TSEM). Specifically, we incorporate them into the feature extractor to operate on each frame. Fig.~\ref{fig2} shows an example of a common feature extractor consisting of multiple stages with several blocks in each. We sequentially place the SSEM and TSEM before the $3\times 3$ spatial convolution in each block to emphasize informative spatial and temporal features, respectively. When designing the architecture, efficiency is our core consideration, to avoid heavy computational burdens like previous methods~\cite{zhou2020spatial,zuo2022c2slr} based on heavy pose-estimation networks or expensive heatmaps. We next introduce our SSEM and TSEM, respectively.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figures/figure2.pdf}
\caption{A overview for our SEN. It first employs a feature extractor (2D CNN) to capture frame-wise features, and then adopts a 1D CNN and a BiLSTM to perform short-term and long-term temporal modeling, respectively, followed by a classifier to predict sentences. We place our proposed spatial self-emphasizing module (SSTM) and temporal self-emphasizing module (TSEM) into each block of the feature extractor to emphasize the spatial and temporal features, respectively.}
\label{fig2}
\end{figure}
\subsection{Spatial Self-Emphasizing Module (SSEM)}
From fig.~\ref{fig1}, we argue current CSLR models fail to effectively leverage the informative spatial features, e.g., hands and face. We try to enhance the capacity of the feature extractor of CSLR models to incorporate such discriminative features without affecting its original spatial modeling ability. Practically, our SSEM is designed to first leverage the closely correlated local spatial-temporal features to identify the informative regions for each frame, and then augment original representations in the form of attention maps.

As shown in fig.~\ref{fig3}, SSEM first projects the input features $s = \{s_t\}_{t=1}^T \in \mathcal{R}^{T \times C\times H \times W}$ into $s_r\in \mathcal{R}^{T \times C/r\times H \times W}$ to decrease the computational costs brought by SSEM, with $r$ the reduction factor as 16 by default.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.8\linewidth]{figures/figure3.pdf}
  \caption{Illustration for our spatial self-emphasizing module (SSEM).}
  \label{fig3}
  \end{figure}

The frame-wise features $s$ in the feature extractor are independently extracted for each frame by 2D convolutions, failing to incorporate local spatial-temporal features to distinguish the informative spatial regions. Besides, as the signer has to throw his/her arms and hands to express glosses, the informative regions in adjacent frames are always misaligned. Thus, we devise a multi-scale architecture to perceive spatial-temporal features in a large neighborhood to help identify informative regions.

Instead of a large spatial-temporal convolution kernel, we employ $N$ parallel factorized branches with group-wise convolutions of progressive dilation rates to lower computations and increase the model capacity. As shown in fig.~\ref{fig3}, these $N$ branches own the same spatial-temporal kernel size $K_t\times K_s\times K_s$, with different spatial dilation rates $[1\cdots N]$. Features from different branches are multiplied with learnable factors $\{\sigma_{1}\dots \sigma_k\}$ to control the importance of different branches via gradient-based backward propagation, and are then added to mix information from different receptive fields. This multi-scale architecture is expressed as:

\begin{equation}
\label{e1}
s_m = \sum_{i=1}^{N}{\sigma_i \times {\rm Conv}_i(s_r)}
\end{equation}
where the group-wise convolution ${\rm Conv}_i$ at different levels captures spatial-temporal features from different receptive fields, with dilation rate $(1,i,i)$.

Especially, as the channels are downsized by $r$ times in SSEM and we employ group-wise convolutions with small spatial-temporal kernels to capture multi-scale features, the overall architecture is rather lightweight with few (\textless 0.1\%) extra computations compared to the original model, as demonstrated in our ablative experiments.

Next, $s_m$ is sent into a $1\times 1 \times 1$ convolution to project channels back into $C$, and then passed through a sigmoid activation function to generate attention maps $M_s\in \mathcal{R}^{T \times C\times H \times W}$ with values ranging between $[0, 1 ]$ as:
\begin{equation}
\label{e2}
M_s = {\rm Sigmoid}({\rm Conv}_{1\times 1\times 1}(s_m))
\end{equation}

Finally, the attention maps $M_s$ are used to emphasize informative spatial regions for input features. To avoid hurting original representations and degrading accuracy, we propose to emphasize input features via a residual way as:
\begin{equation}
\label{e3}
u = (M_s-0.5\times \mathds{1})\odot s+s
\end{equation}
where $\odot$ denotes element-wise multiplication and $u$ is the output.

In specific, we first subtract $0.5\times \mathds{1}$ from the attention maps $M_s$, with $\mathds{1}\in \mathcal{R}^{T \times C\times H \times W}$ denoting an all-one matrix, to change the range of values in $M_s$ into $[-0.5, 0.5]$. Then we element-wisely multiply the resulting attention maps with input features $s$ to dynamically emphasize the informative regions and suppress unnecessary areas. Here, the values in $M_s$ larger than 0 would strengthen the corresponding inputs features, otherwise they would weaken the input features. Finally, we add the modulated features with input features $s$ to emphasize or suppress certain spatial features, but avoid hurting original representations.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.75\linewidth]{figures/figure4.pdf}
  \caption{Illustration for our temporal self-emphasizing module (TSEM).}
  \label{fig4}
  \end{figure}

\subsection{Temporal Self-Emphasizing Module}
We argue that not all frames in a video contribute equally to recognition, where some frames are more discriminative than others. For example, frames in which the signer moves his/her arms to express a sign are usually more important than those transitional frames or idle frames with meaningless contents. However, the feature extractor only employs 2D spatial convolutions to capture spatial features for each frame, equally treating frames without considering their temporal correlations. We propose a temporal self-emphasizing module (TSEM) to adaptively emphasize discriminative frames and suppress redundant ones.

As shown in fig.~\ref{fig4}, input features $u \in \mathcal{R}^{T \times C\times H \times W}$ first undergo a global average pooling layer to eliminate the spatial dimension, i.e., $H$ and $W$. Then these features pass through a convolution with kernel size of 1 to reduce channels by $r$ times into $u_r \in \mathcal{R}^{T \times C/r}$ as:
\begin{equation}
  \label{e4}
  u_r = {\rm Conv}_{K=1}({\rm AvgPool}(u))
  \end{equation}
where $K$ denotes the kernel size. To better exploit local temporal movements to identify the discriminative frames, we leverage the temporal difference operator to incorporate motion information between adjacent frames to make decisions better. Specially, we calculate the difference between two adjacent frames for $u_r$ as approximate motion information, and then concatenate it with appearance features $u_r$ as :
\begin{equation}
\label{e5}
u_m = {\rm Concat}([u_r, u_r(t+1)-u_r])
\end{equation}

Next, we send $u_m$ into a 1D temporal convolution with kernel size of $P_t$ to capture the short-term temporal information. As the size of $u_m$ is rather small, we here employ a normal temporal convolution instead of a multi-scale architecture. The features then undergo a convolution with kernel size of 1 to project channels back into $C$, and pass through a sigmoid activation function to generate attention maps $M_t \in \mathcal{R}^{T \times C}$ as:
\begin{equation}
  \label{e5}
  M_t = {\rm Sigmoid}({\rm Conv}_{K=1}(u_m))
  \end{equation}

Finally, we employ $M_t $ to emphasize the discriminative features for input $u$ in a residual way as :
\begin{equation}
  \label{e6}
  o = (M_t-0.5\times \mathds{1})\odot u+u
  \end{equation}
where $\odot$ denotes element-wise multiplication and $o$ is the output.

\section{Experiments}
\subsection{Experimental Setup}
\subsubsection{Datasets.} \textbf{PHOENIX14}~\cite{koller2015continuous} and \textbf{PHOENIX14-T}~\cite{camgoz2018neural} are both recorded from a German weather forecast broadcast before a clean background with a resolution of 210 $\times$ 260. They contain 6841/8247 sentences with a vocabulary of 1295/1085 signs, divided into 5672/7096 training samples, 540/519 development (Dev) samples and 629/642 testing (Test) samples.

\textbf{CSL-Daily}~\cite{zhou2021improving} is recorded indoor with 20654 sentences, divided into 18401 training samples, 1077 development (Dev) samples and 1176 testing (Test) samples.

\textbf{CSL}~\cite{huang2018video} is collected in the laboratory environment by fifty signers with a vocabulary size of 178 with 100 sentences. It contains 25000 videos, divided into training and testing sets by a ratio of 8:2.

\subsubsection{Training details.} We adopt ResNet18~\cite{he2016deep} as the 2D CNN with ImageNet~\cite{deng2009imagenet} pretrained weights. We place SSEM and TSEM before the second convolution in each block. The 1D CNN consists of a sequence of \{K5, P2, K5, P2\} layers where $K$ and $P$ denotes a 1D convolutional layer and a pooling layer with kernel size of 5 and 2, respectively. We then adopt a two-layer BiLSTM with 1024 hidden states and a fully connected layer for prediction. We train our model for 80 epochs with initial learning rate 0.0001 decayed by 5 after 40 and 60 epochs. Adam optimizer is adopted with weight decay 0.001 and batch size 2. All frames are first resized to 256$\times$256 and then randomly cropped to 224$\times$224, with 50\% horizontal flip and $\pm$20\% random temporal scaling during training. During inference, a central 224$\times$224 crop is simply selected. We use VE and VA losses from VAC~\cite{Min_2021_ICCV} for extra supervision.

\subsubsection{Evaluation Metric.} We use Word Error Rate (WER) as the evaluation metric, which is defined as the minimal summation of the \textbf{sub}stitution, \textbf{ins}ertion, and \textbf{del}etion operations to convert the predicted sentence to the reference sentence, as:
\begin{equation}
\label{e11}
\rm WER = \frac{ \#sub+\#ins+\#del}{\#reference}.
\end{equation}
Note that the \textbf{lower} WER, the \textbf{better} accuracy.

\begin{table}[t]
  \centering
  \begin{tabular}{cccc}
  \hline
  Configurations & FLOPs & Dev(\%) & Test(\%)\\
  \hline
  - & 3.64G& 21.2 & 22.3\\
  $K_t$=9, $K_s$=3, $N$=\textbf{1} & +0.4M  & 20.5  & 22.0 \\
  $K_t$=9, $K_s$=3, $N$=\textbf{2} & +0.6M& 20.2 & 21.8 \\
  $K_t$=9, $K_s$=3, $N$=\textbf{3} & +0.8M & \textbf{19.9}  & \textbf{21.4} \\
  $K_t$=9, $K_s$=3, $N$=\textbf{4} & +1.0M & 20.2  & 21.7 \\
  $K_t$=\textbf{7}, $K_s$=3, $N$=3 & +0.7M & 20.2  & 21.6 \\
  $K_t$=\textbf{11}, $K_s$=3, $N$=3 & +1.0M & 20.3  & 21.8 \\
  $K_t$=9, $K_s$=\textbf{7}, $N$=1 & +2.9M & 20.5 & 22.0 \\
  \hline
  \end{tabular}
  \caption{Ablations for the multi-scale architecture of SSEM on the PHOENIX14 dataset.}
  \label{tab1}
  \end{table}

\iffalse
\begin{table}[t]
    \centering
    \begin{tabular}{ccc}
    \hline
    Configurations & Dev(\%) & Test(\%)\\
    \hline
    - & 21.2 & 22.3\\
    Stage 1  & 20.6  & 21.9 \\
    Stage 1-2 & 20.4 & 21.7 \\
    Stage 1-3 & 20.1  & 21.6 \\
    Stage 1-4 & \textbf{19.9}  & \textbf{21.4} \\
    \hline
    \end{tabular}
    \caption{Ablations for the numbers of SSEMs on the PHOENIX14 dataset.}
    \label{tab2}
  \end{table}
\fi

  \begin{table}[t]
    \centering
    \begin{tabular}{ccc}
    \hline
    Configurations & Dev(\%) & Test(\%)\\
    \hline
    - & 21.2 & 22.3\\
    $M_s \odot s$  & 22.3  & 23.4 \\
    $M_s \odot s + s$ & 20.6 & 21.7 \\
    $(M_s -0.5\times \mathds{1}) \odot s$ & 20.2  & 21.5 \\
    $(M_s -0.5\times \mathds{1}) \odot s +s $ & \textbf{19.9}  & \textbf{21.4}\\
    \hline
    \end{tabular}
    \caption{Ablations for the implementations of SSEM to augment input features on the PHOENIX14 dataset.}
    \label{tab3}
    \end{table}

\begin{table}[t]
  \centering
  \begin{tabular}{ccc}
  \hline
  Configurations & Dev(\%) & Test(\%)\\
  \hline
  - & 19.9 & 21.4 \\
  \hline
  $u_r$ & 19.8  & 21.2 \\
  ${\rm Concat}([u_r, u_r(t+1)-u_r])$ & \textbf{19.5}  & \textbf{21.0} \\
  \hline
  $P_t$ = 7 & 19.6  & 21.2 \\
  $P_t$ = 9 & \textbf{19.5} & \textbf{21.0} \\
  $P_t$ = 11 & 19.7  & 21.3 \\
  \hline
  \end{tabular}
  \caption{Ablations for TSEM on the PHOENIX14 dataset.}
  \label{tab4}
  \end{table}

\begin{table}[t]
  \centering
  \begin{tabular}{ccc}
  \hline
  Configurations & Dev(\%) & Test(\%)\\
  \hline
  - & 21.2 & 22.3\\
  SSEM  & 19.9  & 21.4\\
  TSEM & 20.5 & 21.7 \\
  SSEM + TSEM & 19.8  & 21.4 \\
  TSEM + SSEM & 19.6  & 21.2 \\
  Parallelled & \textbf{19.5}  & \textbf{21.0} \\
  \hline
  \end{tabular}
  \caption{Ablations for the effectiveness of SSEM and TSEM on the PHOENIX14 dataset.}
  \label{tab5}
  \end{table}

\begin{table}[t]
  \centering
  \setlength\tabcolsep{3pt}
  \begin{tabular}{lcc}
  \hline
  Methods & Dev(\%) & Test(\%)\\
  \hline
  - & 21.2 & 22.3\\
  w/ SENet~\cite{hu2018squeeze}  & 20.7  & 21.6 \\
  w/ CBAM~\cite{woo2018cbam} & 20.5 & 21.3 \\
  \hline
  CNN+HMM+LSTM~\cite{koller2019weakly} & 26.0  & 26.0 \\
  STMC~\cite{zhou2020spatial} & 21.1  & 20.7 \\
  C$^2$SLR~\cite{zuo2022c2slr} & 20.5  & 20.4 \\
  \hline
  SEN & \textbf{19.5} & \textbf{21.0} \\
  \hline
  \end{tabular}
  \caption{Comparison with other methods of channel attention or hand and face features on the PHOENIX14 dataset.}
  \label{tab6}
  \end{table}

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{figures/spatial_figure.pdf}
  \caption{Visualizations of class activation maps by Grad-CAM~\cite{selvaraju2017grad}. Top: raw frames; Middle: class activation maps of our baseline; Bottom: class activation maps of our SEN. Our baseline usually focuses on nowhere or only attends to a single hand or face. Our SEN could generally focus on the human body (light yellow areas) and pays special attention to informative regions like hands and face (dark red areas).}
  \label{fig5}
  \end{figure}

\subsection{Ablation Study}
We perform ablation studies on the PHOENIX14 dataset and report on both development (Dev) and testing (Test) sets.

\subsubsection{Effects of the multi-scale architecture of SSEM.}
Tab.~\ref{tab1} ablates the implementations for the multi-scale architecture of SSEM. Our baseline achieves 21.2\% and 22.3\% WER on the Dev and Test Set. When fixing $K_t$=9, $K_s$=3 and varying the number of branches to expand spatial receptive fields, it's observed larger $N$ consistently brings better performance. When $N$ reaches 3, it brings no more performance gain. We set $N$ as 3 by default and test the effects of $K_t$. One can see that either increasing $K_t$ to 11 or decreasing $K_t$ to 7 achieves worse performance. We thus adopt $K_t$ as 9 by default. Notably, one can find SSEM brings few extra computations compared to our baseline. For example, the best-performing SSEM with $K_t$=9, $K_s$=3 and $N$=3 only owns 0.8M (\textless 0.1\%) extra FLOPs, which can be neglected compared to 364G FLOPs of our baseline model. Finally, we compare our proposed multi-scale architecture with a normal implementation of more computations. The receptive field of SSEM with $K_t$=9, $K_s$=3 and $N$=3 is identical to a normal convolution with $K_t$=9 and $K_s$=7. As shown in the bottom of tab.~\ref{tab1}, a normal convolution not only brings more computations than SSEM, but also performs worse, verifying the effectiveness of our architecture.

\subsubsection{Implementations of SSEM to augment inputs features.} Tab.~\ref{tab3} ablates the implementations of SSEM to augment original features. It's first observed directly multiplying the attention maps $M_s$ with input features $s$ severely degrades performance, attributed to destroying input features distributions. Implemented in a residual way by adding $s$, $M_s \odot s + s$ could notably relieve such phenomenon and achieves +0.6\% \& +0.6\% on the Dev and Test Sets. Further, we first subtract $0.5\times \mathds{1}$ from the attention maps $M_s$ to emphasize or suppress certain positions, and then element-wisely multiply it with $s$. This implementation bring +1.0\% \& +0.8\% performance boost. Finally, we update this implementation in a residual way by adding input features $s$ as $(M_s -0.5\times \mathds{1}) \odot s +s $, achieving notable performance boost by +1.3\% \& +0.9\%.

\subsubsection{Study on TSEM.} Tab.~\ref{tab4} ablates the configurations for TSEM. We here adopt SSEM as our baseline and ablate the configurations for TSEM. It's first noticed that combining motion information by concatenating $u_r(t+1)-u_r$ with $u_r$ slightly outperforms only using $u_r$ to capture short-term temporal dependencies, verifying the effectiveness of local motion information. Next, when varying $P_t$, it's observed $P_t$=9 achieves the best performance among $P_t$=[7,9,11], which is adopted by default in the following.

\subsubsection{Study on the effectiveness of SSEM and TSEM.} Tab.~\ref{tab5} studies how to combine SSEM with TSEM. We first notice that only using SSEM or TSEM could already bring a notable performance boost, by +1.3\& +0.9\% and +0.7 \& +0.6\% on the Dev and Test Sets, respectively. When further combining SSEM with TSEM by sequentially placing SSEM before TSEM (SSEM+TSEM), placing TSEM before SSEM (TSEM+SSEM) or paralleling TSEM and TSEM, it's observed SSEM+TSEM performs best with +1.7\% \& +1.3\% performance boost on the Dev and Test Sets, respectively, adopted as the default setting.

\subsubsection{Comparison with other methods.} We compare our SEN with related well-known channel attention methods like SENet~\cite{hu2018squeeze} and CBAM~\cite{woo2018cbam}, and previous CSLR methods equipped with hand and face features by extra pose-estimation networks or pre-extracted heatmaps. In the upper part of tab.~\ref{tab6}, one can see SEN largely outperforms these channel attention methods, for its superior ability to emphasize informative hand and face features. In the bottom part of tab.~\ref{tab6}, it's observed SEN greatly surpasses previous CSLR methods equipped with hand and face features, even though they employ extra heavy networks or expensive supervision. These results verify the effectiveness of our SEN in leveraging hand and face features.

\subsection{Visualizations}
\subsubsection{Visualization for SSEM.}
We sample a few frames for expressing a gloss and plot the class activation maps for our baseline and SEN with Grad-CAM~\cite{selvaraju2017grad} in fig.~\ref{fig5}. The activation maps generated by our baseline usually focus on nowhere or only attend to a single hand or face, failing to fully focus on the informative regions (e.g., hands and face). Instead, our SEN could generally focus on the human body (light yellow areas), and pays special attention to those discriminative regions like hands and face (dark red areas). These visualizations show that without additional expensive supervision, our SEN could still effectively leverage the informative spatial features in a self-supervised way.

\begin{table*}[t]
  \centering
  \setlength\tabcolsep{3pt}
  \begin{tabular}{ccccccccc}
  \hline
  \multirow{3}{*}{Methods} &\multirow{3}{*}{Backbone} & \multicolumn{4}{c}{PHOENIX14} & \multicolumn{2}{c}{PHOENIX14-T} \\
  & &\multicolumn{2}{c}{Dev(\%)} & \multicolumn{2}{c}{Test(\%)} &  \multirow{2}{*}{Dev(\%)} & \multirow{2}{*}{Test(\%)}\\
  & &del/ins & WER & del/ins& WER & & \\
  \hline

  Align-iOpt~\cite{pu2019iterative}& 3D-ResNet &12.6/2 & 37.1& 13.0/2.5 & 36.7 & -&-\\
  Re-Sign~\cite{koller2017re}& GoogLeNet&- & 27.1 &- &26.8 &- &-\\
  SFL~\cite{niu2020stochastic}& ResNet18 & 7.9/6.5 & 26.2 & 7.5/6.3& 26.8 & 25.1&26.1\\

  FCN~\cite{cheng2020fully}& Custom & - & 23.7 & -& 23.9 & 23.3& 25.1\\
  CMA~\cite{pu2020boosting} & GoogLeNet & 7.3/2.7 & 21.3 & 7.3/2.4 & 21.9  & -&-\\
  VAC~\cite{Min_2021_ICCV}& ResNet18 & 7.9/2.5 & 21.2 &8.4/2.6 & 22.3 &- &-\\
  SMKD~\cite{hao2021self}& ResNet18 &6.8/2.5 &20.8 &6.3/2.3 & 21.0 & 20.8 & 22.4\\
  \hline
  SLT$^*$~\cite{camgoz2018neural}& GoogLeNet  & - & - & - & - & 24.5 & 24.6\\
  CNN+LSTM+HMM$^*$~\cite{koller2019weakly}& GoogLeNet  & - &26.0 & - & 26.0 & 22.1 & 24.1 \\
  DNF$^*$~\cite{cui2019deep}& GoogLeNet  & 7.3/3.3 &23.1& 6.7/3.3 & 22.9 & - & -\\
  STMC$^*$~\cite{zhou2020spatial}& VGG11 & 7.7/3.4 &21.1 & 7.4/2.6 & 20.7 & 19.6 & 21.0\\
  C$^2$SLR$^*$~\cite{zuo2022c2slr} & ResNet18 & - & 20.5 &- & 20.4 & 20.2 & 20.4  \\
  \hline
  Baseline & ResNet18 & 7.9/2.5 & 21.2 &8.4/2.6 & 22.3 & 21.1 & 22.8\\
  \textbf{SEN (Ours)} & ResNet18 & 5.8/2.6  &\textbf{19.5} &   7.3/4.0 & \textbf{21.0}  & \textbf{19.3} & \textbf{20.7} \\
  \hline
  \end{tabular}
  \caption{Comparison with state-of-the-art methods on the PHOENIX14 and PHOENIX14-T datasets. $*$ indicates extra clues such as face or hand features are included by additional networks or pre-extracted heatmaps.}
  \label{tab7}
\end{table*}

\subsubsection{Visualization for TSEM.} We visualize the temporal attention maps of TSEM in fig~\ref{fig6}. We sample several frames corresponding to an output gloss 'nord' as an example. The darker color, the higher weight. One can find that TSEM tends to allocate higher weights for frames with rapid movements (the latter two frames in the first line; the middle three frames in the second line). TSEM assigns lower weights for static frames with few body movements. Such observation is consistent with our habits, as humans always pay more attention to those moving objects in the visual field to capture key movements. Those frames can also be considered conveying more important pattern for expressing a sign.

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{figures/temporal_figure.pdf}
  \caption{Visualizations of temporal attention maps for TSEM. One can find that TSEM highlight frames with rapid movements and suppress those static frames.}
  \label{fig6}
  \end{figure}

\subsection{Comparison with State-of-the-Art Methods}
\textbf{PHOENIX14} and \textbf{PHOENIX14-T}. Tab.~\ref{tab7} shows a comprehensive comparison between our SEN and other state-of-the-art methods. We notice that with few extra computations, SEN could outperform other state-of-the-art methods upon both datasets. Especially, SEN outperforms previous CSLR methods equipped with hand and faces acquired by heavy pose-estimation networks or pre-extracted heatmaps (notated with *), without additional expensive supervision.

\textbf{CSL-Daily}. CSL-Daily is a recently released large-scale dataset with the largest vocabulary size (2k) among commonly-used CSLR datasets, covering daily contents. Tab.~\ref{tab8} shows that our SEN achieves new state-of-the-art accuracy upon this challenging dataset with large progresses, which generalizes well upon real-world scenarios.

\textbf{CSL}. As shown in tab.~\ref{tab9}, our SEN could achieve extreme superior accuracy (0.8\% WER) upon this well-examined dataset, outperforming existing CSLR methods.

\begin{table}[t]
  \centering
  \setlength\tabcolsep{2pt}
  \begin{tabular}{cccc}
  \hline
  Methods&  Dev(\%) & Test(\%)\\
  \hline

  LS-HAN~\cite{huang2018video}  & 39.0  & 39.4\\
  TIN-Iterative~\cite{cui2019deep}  & 32.8  & 32.4\\
  Joint-SLRT~\cite{camgoz2020sign}  & 33.1  & 32.0 \\
  FCN~\cite{cheng2020fully} & 33.2  & 32.5 \\
  BN-TIN~\cite{zhou2021improving} & 33.6  & 33.1 \\
  \hline
  Baseline & 32.8 & 32.3\\
  \textbf{SEN(Ours)} & \textbf{31.1} & \textbf{30.7} \\
  \hline
  \end{tabular}
  \caption{Comparison with state-of-the-art methods on the CSL-Daily dataset~\cite{zhou2021improving}.}
  \label{tab8}
  \end{table}

\begin{table}[t]
  \centering
  \setlength\tabcolsep{2pt}
  \begin{tabular}{cc}
    \hline
    Methods&  WER(\%)\\
    \hline

    SubUNet~\cite{cihan2017subunets}   & 11.0\\
    SF-Net~\cite{yang2019sf} & 3.8 \\
    FCN~\cite{cheng2020fully}   & 3.0 \\
    STMC~\cite{zhou2020spatial}  & 2.1 \\
    VAC~\cite{Min_2021_ICCV} & 1.6 \\
    C$^2$SLR~\cite{zuo2022c2slr} & 0.9 \\
    \hline
    Baseline  & 3.5\\
    \textbf{SEN(Ours)} & \textbf{0.8} \\
    \hline
    \end{tabular}
    \caption{Comparison with state-of-the-art methods on the CSL dataset~\cite{huang2018video}.}
    \label{tab9}

  \end{table}

\section{Conclusion}
This paper proposes a self-motivated architecture, coined as SEN, to adaptively emphasize informative spatial and temporal features. Without extra expensive supervision, SEN outperforms existing CSLR methods upon four CSLR datasets. Visualizations confirm the effectiveness of SEN in leveraging discriminative hand and face features.

\section{Introduction}
Sign language is one of the most commonly-used communication tools for the deaf community in their daily life. It mainly conveys information by both manual components (hand/arm gestures), and non-manual components (facial expressions, head movements, and body postures)~\cite{dreuw2007speech,ong2005automatic}. However, mastering this language is rather difficult and time-consuming for the hearing people, thus hindering direct communications between two groups. To relieve this problem, isolated sign language recognition tries to classify a video segment into an independent gloss\footnote{Gloss is the atomic lexical unit to annotate sign languages.}. Continuous sign language recognition (CSLR) progresses by sequentially translating image streams into a series of glosses to express a complete sentence, more prospective towards bridging the communication gap.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/figure1.pdf}
    \caption{Visualization of class activation maps with Grad-CAM~\cite{selvaraju2017grad} for VAC~\cite{Min_2021_ICCV} (baseline). Top: Original frames. Bottom: activation maps. It's observed that without extra supervision, it fails to locate discriminative face and hand regions precisely. }
    \label{fig1}
  \end{figure}

In sign language, the left hand, right hand, and face play the most important role in expressing glosses. Mostly, they convey the information through horizontal/vertical hand movements, finger activities, and static gestures, assisted with facial expressions and mouth shapes to holistically deliver messages~\cite{dreuw2007speech,ong2005automatic}. As a result, hand and face, are always especially leveraged and incorporated in sign language systems. In isolated sign language recognition, early methods~\cite{freeman1995orientation,sun2013discriminative} leveraged hand-crafted features to describe the gestures and motion of both hands. Recent methods either choose to build a pure pose-based system~\cite{tunga2021pose,hu2021signbert} based on detected keypoints for both hands and face, or construct appearance-based systems~\cite{hu2021hand,boukhayma20193d} with cropped patches for hands and face as collaborative inputs. In CSLR, CNN-LSTM-HMM~\cite{koller2019weakly} builds a multi-stream (hands and face) Hidden-Markov-Model (HMM) to integrate multiple visual inputs to boost recognition accuracy. STMC~\cite{zhou2020spatial} explicitly inserts a pose-estimation network and uses the detected regions (hand and face) as multiple cues to perform recognition. More recently, C$^2$SLR~\cite{zuo2022c2slr} leverages the pre-extracted pose keypoints heatmaps as additional supervision to guide models to focus on hand and face areas.

Although it has been proven effective to incorporate hand and face features to improve recognition performance for sign language systems, previous methods usually come at huge computations with increased training complexity, and rely on additional pose estimation networks or extra expensive supervision (e.g., heatmaps). However, without these supervision signals, we find current methods~\cite{Min_2021_ICCV,hao2021self,cheng2020fully} in CSLR fail to precisely locate the hand and face regions (Fig.~\ref{fig1}), thus unable to effectively leverage these features. To more effectively excavate these key cues but avoid introducing huge computations or relying on expensive supervision, we propose a self-emphasizing network (SEN) to explicitly emphasize informative spatial regions in a self-motivated way. Specifically, SEN first employs a lightweight subnetwork to incorporate local spatial-temporal features to identify informative regions, and then dynamically emphasizes or suppresses input features via attention maps.

It's also observed that not all frames contribute equally to recognition. For example, frames with hand/arm movements of the signer are usually more important than those transitional frames. We present a temporal self-emphasizing module to emphasize those discriminative frames and suppress redundant ones dynamically. Remarkably, SEN yields new state-of-the-art accuracy upon four large-scale CSLR datasets, especially outperforming previous methods equipped with hand and face features, even though they always come at huge computations and rely on expensive supervision. Visualizations verify the effects of SEN in emphasizing spatial and temporal features.

\section{Related Work}
\subsection{Continuous Sign Language Recognition}
Sign language recognition methods can be roughly categorized into isolated sign language recognition~\cite{tunga2021pose,hu2021signbert,hu2021hand} and continuous sign language recognition~\cite{pu2019iterative,cheng2020fully,cui2019deep,niu2020stochastic,Min_2021_ICCV} (CSLR), and we focus on the latter in this paper. CSLR tries to translate image frames into corresponding glosses in a weakly-supervised way: only sentence-level label is provided. Early methods in CSLR usually depend on hand-crafted features~\cite{gao2004chinese,freeman1995orientation} to provide visual information, especially body gestures, hands, and face, or rely on HMM-based systems~\cite{koller2016deepsign,han2009modelling,koller2017re,koller2015continuous} to perform temporal modeling and then translate sentences step by step. The HMM-based methods typically first employ a feature extractor to capture visual representations and then adopt an HMM to perform long-term temporal modeling. The recent success of convolutional neural networks (CNNs) and recurrent neural networks brings huge progress for CSLR. The widely-used CTC loss~\cite{graves2006connectionist} enables end-to-end training for recent methods by aligning target glosses with inputs.

Especially, hands and face are paid close attention to by recent methods. For example, CNN-LSTM-HMM~\cite{koller2019weakly} employs a multi-stream HMM (including hands and face) to integrate multiple visual inputs to improve recognition accuracy. STMC~\cite{zhou2020spatial} utilizes a pose-estimation network to estimate human body keypoints and then sends cropped patches (including hands and face) for integration. More recently, C$^2$SLR~\cite{zuo2022c2slr} leverages the pre-extracted pose keypoints as supervision to guide the model. Despite high accuracy, they consume huge additional computations and training complexity.

Practically, recent methods~\cite{pu2019iterative,pu2020boosting,cheng2020fully,cui2019deep,niu2020stochastic,Min_2021_ICCV} usually first employ a feature extractor to capture frame-wise visual representations for each frame, and then adopt 1D CNN and BiLSTM to perform short-term and long-term temporal modeling, respectively. However, several methods~\cite{pu2019iterative,cui2019deep} found in such conditions the feature extractor is not well trained and propose the iterative training strategy to refine the feature extractor, but consume much more computations. More recent methods try to directly enhance the feature extractor by adding visual alignment losses~\cite{Min_2021_ICCV} or adopt pseudo label~\cite{cheng2020fully,hao2021self} for supervision. We propose the self-emphasizing network to emphasize informative spatial features, which can be viewed to enhance the feature extractor in a self-motivated way.

\subsection{Spatial Attention }
Spatial attention has been proven to be effective in many fields including image classification~\cite{cao2019gcnet,hu2018gather,woo2018cbam,hu2018squeeze}, scene segmentation~\cite{fu2019dual} and video classification~\cite{wang2018non}. SENet~\cite{hu2018squeeze}, CBAM~\cite{woo2018cbam}, SKNet~\cite{li2019selective} and ECA-Net~\cite{wang2020eca} devise lightweight channel attention modules for image classification. The widely used self-attention operator~\cite{wang2018non} employs dot-product feature similarities to build attention maps and aggregate long-term dependencies. However, the calculation complexity of the self-attention operator is quadratic to the incorporated pixels, incurring a heavy burden for video-based tasks~\cite{wang2018non}. Instead of feature similarities, our SEN employs a learnable subnetwork to aggregate local spatial-temporal representations and generates spatial attention maps for each frame, much more lightweight than self-attention operators. Some works also propose to leverage external supervision to guide the spatial attention module. For example, GALA~\cite{linsley2018learning} collects click maps from games to supervise the spatial attention for image classification. A relation-guided spatial attention module~\cite{li2020relation} is designed to explore the discriminative regions globally for Video-Based Person Re-Identification. MGAN~\cite{pang2019mask} introduces an attention network to emphasize visible pedestrian regions by modulating full body features. In contrast to external supervision, our self-emphasizing network strengthens informative spatial regions in a self-motivated way, thus greatly lowering required computations and training complexity.

\section{Method}
\subsection{Framework Overview}
As shown in fig.~\ref{fig2}, the backbone of CSLR models is consisted of a feature extractor (2D CNN\footnote{Here we only consider the feature extractor based on 2D CNN, because recent findings~\cite{adaloglou2021comprehensive,zuo2022c2slr} show 3D CNN can not provide as precise gloss boundaries as 2D CNN, and lead to lower accuracy. }), a 1D CNN, a BiLSTM, and a classifier (a fully connected layer) to perform prediction. Given a sign language video with $T$ input frames $x = \{x_{t}\}_{t=1}^T \in \mathcal{R}^{T \times 3\times H_0 \times W_0} $, a CSLR model aims to translate the input video into a series of glosses $y=\{ y_i\}_{i=1}^{N}$ to express a sentence, with $N$ denoting the length of the label sequence. Specifically, the feature extractor first processes input frames into frame-wise features $v = \{v_t\}_{t=1}^{T} \in \mathcal{R}^{T\times d}$. Then the 1D CNN and BiLSTM perform short-term and long-term temporal modeling based on these extracted visual representations, respectively. Finally, the classifier employs widely-used CTC loss to predict the probability of target gloss sequence $p(y|x)$.

To emphasize the informative spatial and temporal features for CSLR models, we present a spatial self-emphasizing module (SSEM) and a temporal self-emphasizing module (TSEM). Specifically, we incorporate them into the feature extractor to operate on each frame. Fig.~\ref{fig2} shows an example of a common feature extractor consisting of multiple stages with several blocks in each. We place the SSEM and TSEM in parallel before the $3\times 3$ spatial convolution in each block to emphasize informative spatial and temporal features, respectively. When designing the architecture, efficiency is our core consideration, to avoid heavy computational burdens like previous methods~\cite{zhou2020spatial,zuo2022c2slr} based on heavy pose-estimation networks or expensive heatmaps. We next introduce our SSEM and TSEM, respectively.

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figures/figure2.pdf}
\caption{A overview for our SEN. It first employs a feature extractor (2D CNN) to capture frame-wise features, and then adopts a 1D CNN and a BiLSTM to perform short-term and long-term temporal modeling, respectively, followed by a classifier to predict sentences. We place our proposed spatial self-emphasizing module (SSTM) and temporal self-emphasizing module (TSEM) into each block of the feature extractor to emphasize the spatial and temporal features, respectively.}
\label{fig2}
\end{figure}
\subsection{Spatial Self-Emphasizing Module (SSEM)}
From fig.~\ref{fig1}, we argue current CSLR models fail to effectively leverage the informative spatial features, e.g., hands and face. We try to enhance the capacity of the feature extractor of CSLR models to incorporate such discriminative features without affecting its original spatial modeling ability. Practically, our SSEM is designed to first leverage the closely correlated local spatial-temporal features to identify the informative regions for each frame, and then augment original representations in the form of attention maps.

As shown in fig.~\ref{fig3}, SSEM first projects the input features $s = \{s_t\}_{t=1}^T \in \mathcal{R}^{T \times C\times H \times W}$ into $s_r\in \mathcal{R}^{T \times C/r\times H \times W}$ to decrease the computational costs brought by SSEM, with $r$ the reduction factor as 16 by default.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.8\linewidth]{figures/figure3.pdf}
  \caption{Illustration for our spatial self-emphasizing module (SSEM).}
  \label{fig3}
  \end{figure}

The frame-wise features $s$ in the feature extractor are independently extracted for each frame by 2D convolutions, failing to incorporate local spatial-temporal features to distinguish the informative spatial regions. Besides, as the signer has to throw his/her arms and hands to express glosses, the informative regions in adjacent frames are always misaligned. Thus, we devise a multi-scale architecture to perceive spatial-temporal features in a large neighborhood to help identify informative regions.

Instead of a large spatial-temporal convolution kernel, we employ $N$ parallel factorized branches with group-wise convolutions of progressive dilation rates to lower computations and increase the model capacity. As shown in fig.~\ref{fig3}, these $N$ branches own the same spatial-temporal kernel size $K_t\times K_s\times K_s$, with different spatial dilation rates $[1\cdots N]$. Features from different branches are multiplied with learnable factors $\{\sigma_{1}\dots \sigma_k\}$ to control the importance of different branches via gradient-based backward propagation, and are then added to mix information from different receptive fields. This multi-scale architecture is expressed as:

\begin{equation}
\label{e1}
s_m = \sum_{i=1}^{N}{\sigma_i \times {\rm Conv}_i(s_r)}
\end{equation}
where the group-wise convolution ${\rm Conv}_i$ at different levels captures spatial-temporal features from different receptive fields, with dilation rate $(1,i,i)$.

Especially, as the channels are downsized by $r$ times in SSEM and we employ group-wise convolutions with small spatial-temporal kernels to capture multi-scale features, the overall architecture is rather lightweight with few (\textless 0.1\%) extra computations compared to the original model, as demonstrated in our ablative experiments.

Next, $s_m$ is sent into a $1\times 1 \times 1$ convolution to project channels back into $C$, and then passed through a sigmoid activation function to generate attention maps $M_s\in \mathcal{R}^{T \times C\times H \times W}$ with values ranging between $[0, 1 ]$ as:
\begin{equation}
\label{e2}
M_s = {\rm Sigmoid}({\rm Conv}_{1\times 1\times 1}(s_m))
\end{equation}

Finally, the attention maps $M_s$ are used to emphasize informative spatial regions for input features. To avoid hurting original representations and degrading accuracy, we propose to emphasize input features via a residual way as:
\begin{equation}
\label{e3}
u = (M_s-0.5\times \mathds{1})\odot s+s
\end{equation}
where $\odot$ denotes element-wise multiplication and $u$ is the output.

In specific, we first subtract $0.5\times \mathds{1}$ from the attention maps $M_s$, with $\mathds{1}\in \mathcal{R}^{T \times C\times H \times W}$ denoting an all-one matrix, to change the range of values in $M_s$ into $[-0.5, 0.5]$. Then we element-wisely multiply the resulting attention maps with input features $s$ to dynamically emphasize the informative regions and suppress unnecessary areas. Here, the values in $M_s$ larger than 0 would strengthen the corresponding inputs features, otherwise they would weaken the input features. Finally, we add the modulated features with input features $s$ to emphasize or suppress certain spatial features, but avoid hurting original representations.

\begin{figure}[t]
  \centering
  \includegraphics[width=0.7\linewidth]{figures/figure4.pdf}
  \caption{Illustration for our temporal self-emphasizing module (TSEM).}
  \label{fig4}
  \end{figure}

\subsection{Temporal Self-Emphasizing Module}
We argue that not all frames in a video contribute equally to recognition, where some frames are more discriminative than others. For example, frames in which the signer moves his/her arms to express a sign are usually more important than those transitional frames or idle frames with meaningless contents. However, the feature extractor only employs 2D spatial convolutions to capture spatial features for each frame, equally treating frames without considering their temporal correlations. We propose a temporal self-emphasizing module (TSEM) to adaptively emphasize discriminative frames and suppress redundant ones.

As shown in fig.~\ref{fig4}, input features $u \in \mathcal{R}^{T \times C\times H \times W}$ first undergo a global average pooling layer to eliminate the spatial dimension, i.e., $H$ and $W$. Then these features pass through a convolution with kernel size of 1 to reduce channels by $r$ times into $u_r \in \mathcal{R}^{T \times C/r}$ as:
\begin{equation}
  \label{e4}
  u_r = {\rm Conv}_{K=1}({\rm AvgPool}(u))
  \end{equation}
where $K$ denotes the kernel size. To better exploit local temporal movements to identify the discriminative frames, we leverage the temporal difference operator to incorporate motion information between adjacent frames to make decisions better. Specially, we calculate the difference between two adjacent frames for $u_r$ as approximate motion information, and then concatenate it with appearance features $u_r$ as :
\begin{equation}
\label{e5}
u_m = {\rm Concat}([u_r, u_r(t+1)-u_r])
\end{equation}

Next, we send $u_m$ into a 1D temporal convolution with kernel size of $P_t$ to capture the short-term temporal information. As the size of $u_m$ is rather small, we here employ a normal temporal convolution instead of a multi-scale architecture. The features then undergo a convolution with kernel size of 1 to project channels back into $C$, and pass through a sigmoid activation function to generate attention maps $M_t \in \mathcal{R}^{T \times C}$ as:
\begin{equation}
  \label{e5}
  M_t = {\rm Sigmoid}({\rm Conv}_{K=1}(u_m))
  \end{equation}

Finally, we employ $M_t $ to emphasize the discriminative features for input $u$ in a residual way as :
\begin{equation}
  \label{e6}
  o = (M_t-0.5\times \mathds{1})\odot u+u
  \end{equation}
where $\odot$ denotes element-wise multiplication and $o$ is the output.

\section{Experiments}
\subsection{Experimental Setup}
\subsubsection{Datasets.} \textbf{PHOENIX14}~\cite{koller2015continuous} and \textbf{PHOENIX14-T}~\cite{camgoz2018neural} are both recorded from a German weather forecast broadcast before a clean background with a resolution of 210 $\times$ 260. They contain 6841/8247 sentences with a vocabulary of 1295/1085 signs, divided into 5672/7096 training samples, 540/519 development (Dev) samples and 629/642 testing (Test) samples.

\textbf{CSL-Daily}~\cite{zhou2021improving} is recorded indoor with 20654 sentences, divided into 18401 training samples, 1077 development (Dev) samples and 1176 testing (Test) samples.

\textbf{CSL}~\cite{huang2018video} is collected in the laboratory environment by fifty signers with a vocabulary size of 178 with 100 sentences. It contains 25000 videos, divided into training and testing sets by a ratio of 8:2.

\subsubsection{Training details.} We adopt ResNet18~\cite{he2016deep} as the 2D CNN with ImageNet~\cite{deng2009imagenet} pretrained weights. We place SSEM and TSEM before the second convolution in each block. The 1D CNN consists of a sequence of \{K5, P2, K5, P2\} layers where $K$ and $P$ denotes a 1D convolutional layer and a pooling layer with kernel size of 5 and 2, respectively. We then adopt a two-layer BiLSTM with 1024 hidden states and a fully connected layer for prediction. We train our model for 80 epochs with initial learning rate 0.0001 decayed by 5 after 40 and 60 epochs. Adam optimizer is adopted with weight decay 0.001 and batch size 2. All frames are first resized to 256$\times$256 and then randomly cropped to 224$\times$224, with 50\% horizontal flip and $\pm$20\% random temporal scaling during training. During inference, a central 224$\times$224 crop is simply selected. We use VE and VA losses from VAC~\cite{Min_2021_ICCV} for extra supervision.

\subsubsection{Evaluation Metric.} We use Word Error Rate (WER) as the evaluation metric, which is defined as the minimal summation of the \textbf{sub}stitution, \textbf{ins}ertion, and \textbf{del}etion operations to convert the predicted sentence to the reference sentence, as:
\begin{equation}
\label{e11}
\rm WER = \frac{ \#sub+\#ins+\#del}{\#reference}.
\end{equation}
Note that the \textbf{lower} WER, the \textbf{better} accuracy.

\begin{table}[t]
  \centering
  \begin{tabular}{cccc}
  \hline
  Configurations & FLOPs & Dev(\%) & Test(\%)\\
  \hline
  - & 3.64G& 21.2 & 22.3\\
  $K_t$=9, $K_s$=3, $N$=\textbf{1} & +0.4M  & 20.5  & 22.0 \\
  $K_t$=9, $K_s$=3, $N$=\textbf{2} & +0.6M& 20.2 & 21.8 \\
  $K_t$=9, $K_s$=3, $N$=\textbf{3} & +0.8M & \textbf{19.9}  & \textbf{21.4} \\
  $K_t$=9, $K_s$=3, $N$=\textbf{4} & +1.0M & 20.2  & 21.7 \\
  $K_t$=\textbf{7}, $K_s$=3, $N$=3 & +0.7M & 20.2  & 21.6 \\
  $K_t$=\textbf{11}, $K_s$=3, $N$=3 & +1.0M & 20.3  & 21.8 \\
  $K_t$=9, $K_s$=\textbf{7}, $N$=1 & +2.9M & 20.5 & 22.0 \\
  \hline
  \end{tabular}
  \caption{Ablations for the multi-scale architecture of SSEM on the PHOENIX14 dataset.}
  \label{tab1}
  \end{table}

\iffalse
\begin{table}[t]
    \centering
    \begin{tabular}{ccc}
    \hline
    Configurations & Dev(\%) & Test(\%)\\
    \hline
    - & 21.2 & 22.3\\
    Stage 1  & 20.6  & 21.9 \\
    Stage 1-2 & 20.4 & 21.7 \\
    Stage 1-3 & 20.1  & 21.6 \\
    Stage 1-4 & \textbf{19.9}  & \textbf{21.4} \\
    \hline
    \end{tabular}
    \caption{Ablations for the numbers of SSEMs on the PHOENIX14 dataset.}
    \label{tab2}
  \end{table}
\fi

  \begin{table}[t]
    \centering
    \begin{tabular}{ccc}
    \hline
    Configurations & Dev(\%) & Test(\%)\\
    \hline
    - & 21.2 & 22.3\\
    $M_s \odot s$  & 22.3  & 23.4 \\
    $M_s \odot s + s$ & 20.6 & 21.7 \\
    $(M_s -0.5\times \mathds{1}) \odot s$ & 20.2  & 21.5 \\
    $(M_s -0.5\times \mathds{1}) \odot s +s $ & \textbf{19.9}  & \textbf{21.4}\\
    \hline
    \end{tabular}
    \caption{Ablations for the implementations of SSEM to augment input features on the PHOENIX14 dataset.}
    \label{tab3}
    \end{table}

\begin{table}[t]
  \centering
  \begin{tabular}{ccc}
  \hline
  Configurations & Dev(\%) & Test(\%)\\
  \hline
  - & 19.9 & 21.4 \\
  \hline
  $u_r$ & 19.8  & 21.2 \\
  ${\rm Concat}([u_r, u_r(t+1)-u_r])$ & \textbf{19.5}  & \textbf{21.0} \\
  \hline
  $P_t$ = 7 & 19.6  & 21.2 \\
  $P_t$ = 9 & \textbf{19.5} & \textbf{21.0} \\
  $P_t$ = 11 & 19.7  & 21.3 \\
  \hline
  \end{tabular}
  \caption{Ablations for TSEM on the PHOENIX14 dataset.}
  \label{tab4}
  \end{table}

\begin{table}[t]
  \centering
  \begin{tabular}{ccc}
  \hline
  Configurations & Dev(\%) & Test(\%)\\
  \hline
  - & 21.2 & 22.3\\
  SSEM  & 19.9  & 21.4\\
  TSEM & 20.5 & 21.7 \\
  SSEM + TSEM & 19.8  & 21.4 \\
  TSEM + SSEM & 19.6  & 21.2 \\
  Parallelled & \textbf{19.5}  & \textbf{21.0} \\
  \hline
  \end{tabular}
  \caption{Ablations for the effectiveness of SSEM and TSEM on the PHOENIX14 dataset.}
  \label{tab5}
  \end{table}

\begin{table}[t]
  \centering
  \setlength\tabcolsep{3pt}
  \begin{tabular}{lcc}
  \hline
  Methods & Dev(\%) & Test(\%)\\
  \hline
  - & 21.2 & 22.3\\
  w/ SENet~\cite{hu2018squeeze}  & 20.7  & 21.6 \\
  w/ CBAM~\cite{woo2018cbam} & 20.5 & 21.3 \\
  \hline
  CNN+HMM+LSTM~\cite{koller2019weakly} & 26.0  & 26.0 \\
  STMC~\cite{zhou2020spatial} & 21.1  & 20.7 \\
  C$^2$SLR~\cite{zuo2022c2slr} & 20.5  & 20.4 \\
  \hline
  SEN & \textbf{19.5} & \textbf{21.0} \\
  \hline
  \end{tabular}
  \caption{Comparison with other methods of channel attention or hand and face features on the PHOENIX14 dataset.}
  \label{tab6}
  \end{table}

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{figures/spatial_figure.pdf}
  \caption{Visualizations of class activation maps by Grad-CAM~\cite{selvaraju2017grad}. Top: raw frames; Middle: class activation maps of our baseline; Bottom: class activation maps of our SEN. Our baseline usually focuses on nowhere or only attends to a single hand or face. Our SEN could generally focus on the human body (light yellow areas) and pays special attention to informative regions like hands and face (dark red areas).}
  \label{fig5}
  \end{figure}

\subsection{Ablation Study}
We perform ablation studies on the PHOENIX14 dataset and report on both development (Dev) and testing (Test) sets.

\subsubsection{Effects of the multi-scale architecture of SSEM.}
Tab.~\ref{tab1} ablates the implementations for the multi-scale architecture of SSEM. Our baseline achieves 21.2\% and 22.3\% WER on the Dev and Test Set. When fixing $K_t$=9, $K_s$=3 and varying the number of branches to expand spatial receptive fields, it's observed larger $N$ consistently brings better performance. When $N$ reaches 3, it brings no more performance gain. We set $N$ as 3 by default and test the effects of $K_t$. One can see that either increasing $K_t$ to 11 or decreasing $K_t$ to 7 achieves worse performance. We thus adopt $K_t$ as 9 by default. Notably, one can find SSEM brings few extra computations compared to our baseline. For example, the best-performing SSEM with $K_t$=9, $K_s$=3 and $N$=3 only owns 0.8M (\textless 0.1\%) extra FLOPs, which can be neglected compared to 364G FLOPs of our baseline model. Finally, we compare our proposed multi-scale architecture with a normal implementation of more computations. The receptive field of SSEM with $K_t$=9, $K_s$=3 and $N$=3 is identical to a normal convolution with $K_t$=9 and $K_s$=7. As shown in the bottom of tab.~\ref{tab1}, a normal convolution not only brings more computations than SSEM, but also performs worse, verifying the effectiveness of our architecture.

\subsubsection{Implementations of SSEM to augment inputs features.} Tab.~\ref{tab3} ablates the implementations of SSEM to augment original features. It's first observed directly multiplying the attention maps $M_s$ with input features $s$ severely degrades performance, attributed to destroying input features distributions. Implemented in a residual way by adding $s$, $M_s \odot s + s$ could notably relieve such phenomenon and achieves +0.6\% \& +0.6\% on the Dev and Test Sets. Further, we first subtract $0.5\times \mathds{1}$ from the attention maps $M_s$ to emphasize or suppress certain positions, and then element-wisely multiply it with $s$. This implementation bring +1.0\% \& +0.8\% performance boost. Finally, we update this implementation in a residual way by adding input features $s$ as $(M_s -0.5\times \mathds{1}) \odot s +s $, achieving notable performance boost by +1.3\% \& +0.9\%.

\subsubsection{Study on TSEM.} Tab.~\ref{tab4} ablates the configurations for TSEM. We here adopt SSEM as our baseline and ablate the configurations for TSEM. It's first noticed that combining motion information by concatenating $u_r(t+1)-u_r$ with $u_r$ slightly outperforms only using $u_r$ to capture short-term temporal dependencies, verifying the effectiveness of local motion information. Next, when varying $P_t$, it's observed $P_t$=9 achieves the best performance among $P_t$=[7,9,11], which is adopted by default in the following.

\subsubsection{Study on the effectiveness of SSEM and TSEM.} Tab.~\ref{tab5} studies how to combine SSEM with TSEM. We first notice that only using SSEM or TSEM could already bring a notable performance boost, by +1.3\& +0.9\% and +0.7 \& +0.6\% on the Dev and Test Sets, respectively. When further combining SSEM with TSEM by sequentially placing SSEM before TSEM (SSEM+TSEM), placing TSEM before SSEM (TSEM+SSEM) or paralleling TSEM and TSEM, it's observed SSEM+TSEM performs best with +1.7\% \& +1.3\% performance boost on the Dev and Test Sets, respectively, adopted as the default setting.

\subsubsection{Comparison with other methods.} We compare our SEN with related well-known channel attention methods like SENet~\cite{hu2018squeeze} and CBAM~\cite{woo2018cbam}, and previous CSLR methods equipped with hand and face features by extra pose-estimation networks or pre-extracted heatmaps. In the upper part of tab.~\ref{tab6}, one can see SEN largely outperforms these channel attention methods, for its superior ability to emphasize informative hand and face features. In the bottom part of tab.~\ref{tab6}, it's observed SEN greatly surpasses previous CSLR methods equipped with hand and face features, even though they employ extra heavy networks or expensive supervision. These results verify the effectiveness of our SEN in leveraging hand and face features.

\subsection{Visualizations}
\subsubsection{Visualization for SSEM.}
We sample a few frames for expressing a gloss and plot the class activation maps for our baseline and SEN with Grad-CAM~\cite{selvaraju2017grad} in fig.~\ref{fig5}. The activation maps generated by our baseline usually focus on nowhere or only attend to a single hand or face, failing to fully focus on the informative regions (e.g., hands and face). Instead, our SEN could generally focus on the human body (light yellow areas), and pays special attention to those discriminative regions like hands and face (dark red areas). These visualizations show that without additional expensive supervision, our SEN could still effectively leverage the informative spatial features in a self-supervised way.

\begin{table*}[t]
  \centering
  \setlength\tabcolsep{3pt}
  \begin{tabular}{ccccccccc}
  \hline
  \multirow{3}{*}{Methods} &\multirow{3}{*}{Backbone} & \multicolumn{4}{c}{PHOENIX14} & \multicolumn{2}{c}{PHOENIX14-T} \\
  & &\multicolumn{2}{c}{Dev(\%)} & \multicolumn{2}{c}{Test(\%)} &  \multirow{2}{*}{Dev(\%)} & \multirow{2}{*}{Test(\%)}\\
  & &del/ins & WER & del/ins& WER & & \\
  \hline

  Align-iOpt~\cite{pu2019iterative}& 3D-ResNet &12.6/2 & 37.1& 13.0/2.5 & 36.7 & -&-\\
  Re-Sign~\cite{koller2017re}& GoogLeNet&- & 27.1 &- &26.8 &- &-\\
  SFL~\cite{niu2020stochastic}& ResNet18 & 7.9/6.5 & 26.2 & 7.5/6.3& 26.8 & 25.1&26.1\\

  FCN~\cite{cheng2020fully}& Custom & - & 23.7 & -& 23.9 & 23.3& 25.1\\
  CMA~\cite{pu2020boosting} & GoogLeNet & 7.3/2.7 & 21.3 & 7.3/2.4 & 21.9  & -&-\\
  VAC~\cite{Min_2021_ICCV}& ResNet18 & 7.9/2.5 & 21.2 &8.4/2.6 & 22.3 &- &-\\
  SMKD~\cite{hao2021self}& ResNet18 &6.8/2.5 &20.8 &6.3/2.3 & 21.0 & 20.8 & 22.4\\
  \hline
  SLT$^*$~\cite{camgoz2018neural}& GoogLeNet  & - & - & - & - & 24.5 & 24.6\\
  CNN+LSTM+HMM$^*$~\cite{koller2019weakly}& GoogLeNet  & - &26.0 & - & 26.0 & 22.1 & 24.1 \\
  DNF$^*$~\cite{cui2019deep}& GoogLeNet  & 7.3/3.3 &23.1& 6.7/3.3 & 22.9 & - & -\\
  STMC$^*$~\cite{zhou2020spatial}& VGG11 & 7.7/3.4 &21.1 & 7.4/2.6 & 20.7 & 19.6 & 21.0\\
  C$^2$SLR$^*$~\cite{zuo2022c2slr} & ResNet18 & - & 20.5 &- & 20.4 & 20.2 & 20.4  \\
  \hline
  Baseline & ResNet18 & 7.9/2.5 & 21.2 &8.4/2.6 & 22.3 & 21.1 & 22.8\\
  \textbf{SEN (Ours)} & ResNet18 & 5.8/2.6  &\textbf{19.5} &   7.3/4.0 & \textbf{21.0}  & \textbf{19.3} & \textbf{20.7} \\
  \hline
  \end{tabular}
  \caption{Comparison with state-of-the-art methods on the PHOENIX14 and PHOENIX14-T datasets. $*$ indicates extra clues such as face or hand features are included by additional networks or pre-extracted heatmaps.}
  \label{tab7}
\end{table*}

\subsubsection{Visualization for TSEM.} We visualize the temporal attention maps of TSEM in fig~\ref{fig6}. We sample several frames corresponding to an output gloss 'nord' as an example. The darker color, the higher weight. One can find that TSEM tends to allocate higher weights for frames with rapid movements (the latter two frames in the first line; the middle three frames in the second line). TSEM assigns lower weights for static frames with few body movements. Such observation is consistent with our habits, as humans always pay more attention to those moving objects in the visual field to capture key movements. Those frames can also be considered conveying more important pattern for expressing a sign.

\begin{figure}[t]
  \centering
  \includegraphics[width=\linewidth]{figures/temporal_figure.pdf}
  \caption{Visualizations of temporal attention maps for TSEM. One can find that TSEM highlight frames with rapid movements and suppress those static frames.}
  \label{fig6}
  \end{figure}

\subsection{Comparison with State-of-the-Art Methods}
\textbf{PHOENIX14} and \textbf{PHOENIX14-T}. Tab.~\ref{tab7} shows a comprehensive comparison between our SEN and other state-of-the-art methods. We notice that with few extra computations, SEN could outperform other state-of-the-art methods upon both datasets. Especially, SEN outperforms previous CSLR methods equipped with hand and faces acquired by heavy pose-estimation networks or pre-extracted heatmaps (notated with *), without additional expensive supervision.

\textbf{CSL-Daily}. CSL-Daily is a recently released large-scale dataset with the largest vocabulary size (2k) among commonly-used CSLR datasets, covering daily contents. Tab.~\ref{tab8} shows that our SEN achieves new state-of-the-art accuracy upon this challenging dataset with large progresses, which generalizes well upon real-world scenarios.

\textbf{CSL}. As shown in tab.~\ref{tab9}, our SEN could achieve extreme superior accuracy (0.8\% WER) upon this well-examined dataset, outperforming existing CSLR methods.

\begin{table}[t]
  \centering
  \setlength\tabcolsep{2pt}
  \begin{tabular}{cccc}
  \hline
  Methods&  Dev(\%) & Test(\%)\\
  \hline

  LS-HAN~\cite{huang2018video}  & 39.0  & 39.4\\
  TIN-Iterative~\cite{cui2019deep}  & 32.8  & 32.4\\
  Joint-SLRT~\cite{camgoz2020sign}  & 33.1  & 32.0 \\
  FCN~\cite{cheng2020fully} & 33.2  & 32.5 \\
  BN-TIN~\cite{zhou2021improving} & 33.6  & 33.1 \\
  \hline
  Baseline & 32.8 & 32.3\\
  \textbf{SEN(Ours)} & \textbf{31.1} & \textbf{30.7} \\
  \hline
  \end{tabular}
  \caption{Comparison with state-of-the-art methods on the CSL-Daily dataset~\cite{zhou2021improving}.}
  \label{tab8}
  \end{table}

\begin{table}[t]
  \centering
  \setlength\tabcolsep{2pt}
  \begin{tabular}{cc}
    \hline
    Methods&  WER(\%)\\
    \hline

    SubUNet~\cite{cihan2017subunets}   & 11.0\\
    SF-Net~\cite{yang2019sf} & 3.8 \\
    FCN~\cite{cheng2020fully}   & 3.0 \\
    STMC~\cite{zhou2020spatial}  & 2.1 \\
    VAC~\cite{Min_2021_ICCV} & 1.6 \\
    C$^2$SLR~\cite{zuo2022c2slr} & 0.9 \\
    \hline
    Baseline  & 3.5\\
    \textbf{SEN(Ours)} & \textbf{0.8} \\
    \hline
    \end{tabular}
    \caption{Comparison with state-of-the-art methods on the CSL dataset~\cite{huang2018video}.}
    \label{tab9}

  \end{table}

\section{Conclusion}
This paper proposes a self-motivated architecture, coined as SEN, to adaptively emphasize informative spatial and temporal features. Without extra expensive supervision, SEN outperforms existing CSLR methods upon four CSLR datasets. Visualizations confirm the effectiveness of SEN in leveraging discriminative hand and face features.

redpajama_set_name: RedPajamaArXiv


text:
Q: Accidentally deleted the glusterfs-dynamic Service for my PVC; how do I recreate it? Attempting to recreate all my assets from a fresh openshift (except for my PVC), I deleted everything ($ oc delete all --all; oc delete configmap --all; oc delete secret -l namespace=visor). I did this so I could be certain my 'oc process -f template' did a complete job.
This deleted the glusterfs-dynamic Services that I didn't realize were required to mount PVCs (persistent volume claims).
Solution 1: Recreate the Services

*

*I recreated the services so that they look just like similar glusterfs-dynamic services, but that's not enough; even if the IP address matches, the PVC are still not mountable ('endpoints "glusterfs-dynamic-xxx" not found')

Solution 2: Copy data from old PVC to new PVC

*

*To copy, I need to be able to access the PVC from a pod--I can't mount the PVC...

My attempt at recreating the service:
      - apiVersion: v1
        kind: Service
        metadata:
          labels:
            gluster.kubernetes.io/provisioned-for-pvc: prom-a-pvc
            namespace: visor
          name: glusterfs-dynamic-615a9bfa-57d9-11e9-b511-001a4a195f6a
          namespace: visor
        spec:
          ports:
            - port: 1
              protocol: TCP
              targetPort: 1
          sessionAffinity: None
          type: ClusterIP

I want to be able to mount my PVCs.
But instead I get this error:
MountVolume.NewMounter initialization failed for volume "pvc-89647bcb-6df4-11e9-bd79-001a4a195f6a" : endpoints "glusterfs-dynamic-89647bcb-6df4-11e9-bd79-001a4a195f6a" not found

A: Figured it out. Woot. There was also an "endpoints" asset that got deleted that I had to recreate:
    - apiVersion: v1
      kind: Service
      metadata:
        labels:
          gluster.kubernetes.io/provisioned-for-pvc: prom-a-pvc
          namespace: visor
        name: glusterfs-dynamic-615a9bfa-57d9-11e9-b511-001a4a195f6a
        namespace: visor
      spec:
        ports:
          - port: 1
            protocol: TCP
            targetPort: 1
        sessionAffinity: None
        type: ClusterIP

    - apiVersion: v1
      kind: Endpoints
      metadata:
        labels:
          gluster.kubernetes.io/provisioned-for-pvc: prom-a-pvc
        name: glusterfs-dynamic-615a9bfa-57d9-11e9-b511-001a4a195f6a
        namespace: visor
      subsets:
      - addresses:
        - ip: 10.25.6.231
        - ip: 10.25.6.232
        - ip: 10.27.6.241
        - ip: 10.27.6.242
        - ip: 10.5.6.221
        - ip: 10.5.6.222
        ports:
        - port: 1
          protocol: TCP

I used theses commands to see what I needed to include in my yaml:
$ oc get endpoints
$ oc edit endpoints glusterfs-dynamic-820de1e7-6df6-11e9-bd79-001a4a195f6a
Links:

*

*I'm not alone: https://github.com/heketi/heketi/issues/757

*Where I learned the endpoints asset: https://lists.openshift.redhat.com/openshift-archives/users/2019-April/msg00005.html

redpajama_set_name: RedPajamaStackExchange


The Aria Collection throw blankets are light and perfect for layering. Perfect for snuggling up in on the sofa in front of the fire, but would work equally well on the end of the bed. All are throw blankets are wet finished, tumble dried and pre-shrunk.


Nothing in an organisation needs to remain fixed when an improvement can be made, regardless of the scale of the task.
Making changes to a prevailing organisational culture, operating models or dominant styles of leadership are not always easy, but they can dramatically raise engagement levels, improve performance, reduce costs and improve service.
When seeking change, the place to start is with people.
In-depth evaluation processes that measure the lasting impact of our work together and demonstrate return on investment.