text:
West Notes: Ekman-Larsson, Johnson, Vegas, San Diego Gulls
January 17, 2021 at 3:04 pm CDT | by Holger Stolzenberg 1 Comment
While there was some concern that he might miss significant time, Arizona Coyotes head coach Rick Tocchet said that defenseman Oliver Ekman-Larsson, who took a big hit from San Jose's Evander Kane (video here), Saturday, is considered to be day-to-day with a lower-body injury, although he is expected to skip the road trip to Las Vegas, according to azcsports' Jose Romero. That likely means he will miss at least two games.
Ekman-Larsson was forced to leave the game and only played 17:11 Saturday, but still managed to pick up three assists. No word on who will be recalled to replace him, but Coyotes' insider Craig Morgan believes it will either be Kyle Capobianco or Jordan Gross.
The Colorado Avalanche got some good news as veteran defenseman Erik Johnson returned to the Avalanche after missing training camp, while dealing with a positive COVID-19 test and quarantining," according to The Athletic's Peter Baugh. He is now practicing with the team, although he may need a few more days to get back into game shape. "I had some symptoms, but in the grand scheme of things it wasn't that big a deal compared to what some people are dealing with. I just consider myself lucky to recover," Johnson said.
The Vegas Golden Knights, who played the first two games of the season with just five defensemen could be ready to make a change. The team's new practice lines suggest the team could be ready to use Nicolas Hague as their sixth defenseman, despite the team's success. The Athletic's Jesse Granger writes that head coach Peter DeBoer likes what he's seen from his five-man defense, but also said he's worried about the workload that might create on them. Defenseman Alex Pietrangelo played 29:26 Saturday, a lot to ask out of any blueliner.
The San Diego Gulls, the AHL affiliate of the Anaheim Ducks, announced they have postponed Sunday's exhibition game against the Ontario Reign out of an abundance of caution due to COVID-19 protocols. This was supposed to be the second exhibition game. The first game, originally scheduled for Saturday, was postponed as well. This game was pushed back for the same reasons. The team will make a decision at a later date on when to play these game.
AHL| Anaheim Ducks| Arizona Coyotes| Colorado Avalanche| Erik Johnson| Injury| Oliver Ekman-Larsson| Vegas Golden Knights
COVID Notes: Kapanen, Granlund, Gulls-Reign
January 16, 2021 at 6:27 pm CDT | by Zach Leach 2 Comments
Pittsburgh Penguins off-season re-acquisition Kasperi Kapanen has not yet been on the ice with his teammates, but his return is imminent. Kapanen was removed from the NHL's COVID Protocol on Friday but was still held out of practice on Saturday. However, Mike DeFabo of the Pittsburgh Post Gazette reports that Kapanen took his fourth COVID test today and a fourth negative result will make it his final test. He will be able to re-join the team immediately, with head coach Mike Sullivan stating that they will have him in uniform as quickly as they can once that test result is received. Kapanen, 24, recorded 80 points combined over the past two seasons with the Toronto Maple Leafs and is expected to skate with Sidney Crosby and Jake Guentzel on the Penguins' top line.
Mikael Granlund remains on the NHL's COVID Protocol list for the Nashville Predators, but progress is being made toward getting him on the ice. Granlund's return to Nashville was initially delayed due to immigration issues that caused him to miss all of training camp. According to The Athletic's Adam Vingan, he has now arrived in Nashville and has begun the mandated quarantine following commercial travel. At no point has Granlund tested positive for Coronavirus and he should be eligible to return to the lineup as soon as his quarantine ends. Head coach John Hynes has stated that he expects Granlund to return to a top-six role for the Predators after he recorded 30 points in 63 games last season, more than half of which came after Hynes took over.
The San Diego Gulls and Ontario Reign, AHL affiliates of the Anaheim Ducks and L.A. Kings respectively, had intended to begin their preseason tonight with an exhibition game in Irvine, California. However, that game has been canceled due to precautions related to Coronavirus and in adherence with local health protocols. The two teams are scheduled to play on Sunday as well, but no determination has been made on that game yet.
AHL| Anaheim Ducks| Coronavirus| John Hynes| Kasperi Kapanen| Los Angeles Kings| Mikael Granlund| Mike Sullivan| Nashville Predators| NHL| Pittsburgh Penguins| Schedule| Sidney Crosby| Toronto Maple Leafs
Ben Hutton Signs With Anaheim Ducks
January 15, 2021 at 1:34 pm CDT | by Gavin Lee Leave a Comment
The Anaheim Ducks have officially signed Ben Hutton to a one-year contract for the rest of the season. The deal is worth $950K according to Elliotte Friedman of Sportsnet. It was clear that a deal was coming when Hutton appeared on the COVID Protocol Related Absences list yesterday. It's not clear when he will be removed from that list so that he's available to play, but the Ducks have finally announced the deal.
Hutton, 27, was one of the players that felt an offseason squeeze, forced to sign a professional tryout despite a relatively impressive NHL resume. While he likely shouldn't be deployed on a top pairing, Hutton has nearly 350 games of experience and has averaged just under 20 minutes a night over his five-year career. In 2019-20 he played in 65 games for the Los Angeles Kings and registered 16 points, while putting up strong possession numbers once again.
The Ducks relied heavily on their top two pairs last night to open the season, with Cam Fowler, Kevin Shattenkirk, Josh Manson, and Hampus Lindholm all logging at least 20 minutes. If Hutton can add a little depth to the bottom-pair rotation perhaps they won't have to put so much responsibility on those top names, at least not every game.
One other thing to remember with Hutton in Anaheim is that he could be another trade deadline piece for them to flip. The one-year, low-money deal will be attractive to contenders partway through the year, especially if he's able to prove he can still play at a high level.
Anaheim Ducks| Ben Hutton| Elliotte Friedman
Ben Hutton To Join Anaheim Ducks On PTO
After losing some of their depth when the Detroit Red Wings claimed defenseman Christian Djoos on waivers Saturday and the injury to Brendan Guhle, Sportsnet's Elliotte Friedman reports that the Anaheim Ducks are expected to sign unrestricted free agent Ben Hutton to a professional tryout agreement.
The 27-year-old blueliner, who spent last year with the Los Angeles Kings, has been a full-time player in the league for five years, four with the Vancouver Canucks. However, the Canucks opted not to offer Hutton a qualifying offer in 2019, prompting him to sign with the Kings and now leaving him on the free agent market. While hardly a top-four paring defenseman, Hutton provides solid depth as a third-pairing defenseman with both power play and penalty kill experience.
Regardless, Hutton has remained on the market through the offseason and training camp. Rumors that New Jersey and Boston were both interested in bringing the defenseman on board, but neither team pulled the trigger. Now Hutton will challenge for a spot on the Ducks roster.
Anaheim is set at the top four with Hampus Lindholm and Kevin Shattenkirk on the first pairing and Josh Manson and Cam Fowler on the second pairing. The third pairing is wide open with Jacob Larsson the leading candidate to take one of the final spots, while Hutton will likely fight it out with Jani Hakanpaa, Kodie Curran, Simon Benoit and Josh Mahura. Hutton also could eventually be put on waivers to be put onto the taxi squad, like the Ducks had hoped to do with Djoos. Considering no one was interested in signing Hutton to a PTO before, it would make sense that no one would claim him then.
Anaheim Ducks| Ben Hutton| Los Angeles Kings| Vancouver Canucks
Red Wings Claim Christian Djoos Off Waivers
January 9, 2021 at 11:14 am CDT | by Brian La Rose 5 Comments
The first waiver claim of the year is in as Sportsnet's Elliotte Friedman reports (Twitter link) that the Red Wings have claimed defenseman Christian Djoos off waivers from Anaheim.
The 26-year-old split last season between Washington and Anaheim after he was dealt to the Ducks just before the trade deadline in exchange for winger Daniel Sprong. Djoos suited up just twice for the Capitals but was a regular in Anaheim's lineup after the trade, playing in nine games while logging more than 20 minutes a night. He spent most of the season prior to the trade in the minors, putting up five goals and 27 assists in just 42 games with AHL Hershey.
Djoos signed a one-year, $1MM deal with Anaheim during the stoppage for the pandemic, a deal that actually saw his salary drop by $250K. Had he cleared waivers, his entire cap hit would have been removed off the Ducks' books in the minors but it's now gone entirely with him out of the organization.
Instead, he'll now join a new-look Detroit back end that also now has veteran Marc Staal plus free agent acquisitions Troy Stecher and Jonathon Merrill in the group. Djoos will likely battle Merrill and returnee Alex Biega for playing time and could now have the opportunity to play a more regular role after spending most of last year in the minors.
Anaheim Ducks| Christian Djoos| Detroit Red Wings| Transactions| Waivers
Brendan Guhle Out Six Weeks
January 8, 2021 at 12:01 pm CDT | by Gavin Lee Leave a Comment
Though it may get lost in the shuffle as the league deals with COVID-19 outbreaks, the Anaheim Ducks had some unfortunate news of their own today. Brendan Guhle has been ruled out for the next six weeks with a sprained MCL, essentially ending the training camp battle he was in for a roster spot.
The 23-year-old Guhle came to Anaheim in a trade nearly two years ago that saw Brandon Montour go to the Buffalo Sabres. Splitting time between the NHL and AHL, he performed adequately and looked to have an inside track on a roster spot (even if it was perhaps as the seventh defenseman) this season. The Ducks even put Christian Djoos on waivers today, indicating that he'll likely start on the taxi squad or in the AHL.
Now, Guhle faces a tough rehab and the idea of overtaking a teammate mid-season if he wants to secure playing time. Six weeks puts him back on the ice deep into February, when a good chunk of the season has already passed. That's a tough break in an already shortened schedule and could very well mean that Guhle will not eclipse the career-high of 30 games played he set last season.
Anaheim Ducks| Brendan Guhle
Large Group Of Players Placed On Waivers
January 8, 2021 at 11:18 am CDT | by Gavin Lee 8 Comments
With just a few days before the start of the regular season, a huge number of players have been placed on waivers. The full list includes:
Anaheim Ducks:
Anthony Stolarz
Andy Welinski
Christian Djoos
Andrew Poturalski
Chase De Leo
Vinni Lettieri
Sam Carrick
Andrew Agozzino
Carolina Hurricanes:
Antoine Bibeau
Steven Lorentz
Spencer Smallman
Jeremy Bracco
Gustav Forsling
Drew Shore
Max McCormick
Colorado Avalanche:
Jacob MacDonald
Dan Renouf
Kyle Burroughs
Mike Vecchione
T.J. Tynan
Miikka Salomaki
Jayson Megna
Sheldon Dries
Toronto Maple Leafs:
Nic Petan
These massive waiver placements are no different than the normal training camp cuts that would occur in late-September in a normal year. Should they clear, these players will be eligible to report to the taxi squad or AHL. There are a few names that stand out from the crowd though.
Backes, a veteran of 950 NHL games, still carries a $6MM cap hit on the final season of the five-year, $30MM contract he signed with the Boston Bruins in 2016. As Pierre LeBrun of The Athletic reports, this is not a move by the Ducks to try and rid themselves of Backes, but create some additional flexibility. At any rate, his contract basically makes him waiver-proof as no other team would want to take it on at this point in the season.
Bracco, a former top prospect that scored 79 points in 75 games for the Toronto Marlies of the AHL as recently as 2018-19, has seen his development stall and finds himself on the outside looking in for the Hurricanes once again. Djoos, once an up-and-coming defenseman in the Washington Capitals system, is now 26 and available to the whole league, should they want to take a chance.
Anaheim Ducks| Andrew Agozzino| Andrew Poturalski| Andy Welinski| Anthony Stolarz| Antoine Bibeau| Carolina Hurricanes| Chase De Leo| Christian Djoos| Colorado Avalanche| David Backes| Drew Shore| Gustav Forsling| Jayson Megna| Max McCormick| Miikka Salomaki| Nic Petan| Toronto Maple Leafs| Waivers
Ducks Sign Anthony Stolarz To Two-Year Extension
January 7, 2021 at 8:20 pm CDT | by Brian La Rose 5 Comments
Thursday: Anaheim officially announced the two-year extension for Stolarz.
Wednesday: The Ducks have locked up their insurance goalie and fulfilled an expansion draft requirement in the process. CapFriendly reports (Twitter link) that they have signed netminder Anthony Stolarz to a two-year contract extension worth $950K per year. It is a one-way pact in both seasons.
The 25-year-old joined Anaheim last offseason after signing as a Group VI unrestricted free agent from Philadelphia. He ultimately had a similar role as he did with the Flyers as he was their third-string option and spent the majority of the season with AHL San Diego as a result, compiling a 2.66 GAA with a .922 SV% in 39 games. He did get into one game with the Ducks, allowing two goals on 36 shots in a losing effort to St. Louis back in March.
Stolarz has 26 career NHL games under his belt so it's quite possible that he'll be in the mix for the number two spot behind John Gibson with Ryan Miller going year-to-year at this stage of his career. However, the primary impetus for this contract would appear to be the looming Seattle expansion draft. The only signed goalie that was eligible for exposure (each team must leave one unprotected) was Gibson and clearly, they intended to protect him. Giving Stolarz a new deal paves the way for them to unprotect him and ensure Gibson stays put and for agreeing to do so, Stolarz gets a small raise and a couple of years of job security, even if he winds up being as the third-string option throughout.
Anaheim Ducks| Anthony Stolarz| Transactions
Snapshots: Bratt, WJC Prospects, Kerfoot
January 5, 2021 at 4:55 pm CDT | by Gavin Lee 6 Comments
The New Jersey Devils could be without Jesper Bratt to start the season, as the restricted free agent is still unsigned. Elliotte Friedman of Sportsnet reports that the 22-year-old forward is still in Sweden, noting that it would be tough even now to get in on time with visa complications and quarantine. Even those things would need a contract to be worked out first, something that may not be all that close. Friedman writes that though the two sides are still communicating there is "not a ton of talk, and there's a bit of a gap at this time."
Bratt, who was a sixth-round pick just a few years ago, has been one of the bright spots on a Devils team that has struggled the last few seasons. Though his play has certainly not been consistent, he still put up 16 goals and 32 points in just 60 games last season and figures to play a top-six role on the team this year once he signs. Of course, every day he misses leaves the door open for other players to impress, including newcomer Andreas Johnsson who has been skating next to Jack Hughes and Kyle Palmieri so far.
Speaking of missing time, several prospects at the World Junior Championship will waste none at all after their championship game this evening. Bob McKenzie of TSN reports that Trevor Zegras, Jamie Drysdale, Quinton Byfield, Alex Turcotte, Arthur Kaliyev, and Tobias Bjornfot will all be on a chartered flight back to Southern California to join the Anaheim Ducks and Los Angeles Kings respective training camps. Because they are coming out of the bubble and will not be flying commercial, McKenzie reports that these players will likely not have to quarantine for seven days (though they will still have to adhere to NHL testing protocols). The same process will take Dylan Cozens and Jack Quinn to Buffalo Sabres camp.
Injuries are already popping up around the league and in Toronto the Maple Leafs could be without Alexander Kerfoot the next few days. Head coach Sheldon Keefe told reporters including Kristen Shilton of TSN that Kerfoot is day-to-day after suffering an injury today in practice. The Maple Leafs had split their training groups quite distinctly, meaning if Kerfoot is held out it would open the door for one of the expected non-roster players to get a chance.
Alex Turcotte| Alexander Kerfoot| Anaheim Ducks| Arthur Kaliyev| Bob McKenzie| Elliotte Friedman| Injury| Jamie Drysdale| Jesper Bratt| Los Angeles Kings| New Jersey Devils| Prospects| Snapshots| Toronto Maple Leafs
NHL Will Not Require Blanket Quarantine Period For AHL Recalls
January 4, 2021 at 7:14 pm CDT | by Zach Leach 5 Comments
Alongside the news of three teams opting out, four teams temporarily relocating, and realigned divisions for the coming season, more information continues to emerge following today's AHL Board of Governors meeting. Sportsnet's Chris Johnston reports that NHL clubs and their AHL affiliates have been informed that there will be no blanket quarantine period for player recalls and reassignments this season. Instead, quarantine measures will be evaluated on a case-by-case basis considering the totality of the circumstances. This will include team protocols, travel logistics, and accordance with local COVID-19 health guidelines.
As Johnston notes, this will make AHL recalls much easier for those teams whose affiliates share a city or even a state or province. Short, safe travel ability and uniform local policies will allow for much shorter quarantine periods. Teams in this situation may even ask their affiliate to maintain the same NHL-level of day-to-day quarantine protocols to make recalls even easier, perhaps even without any quarantine. The Toronto Maple Leafs, Winnipeg Jets, Vegas Golden Knights, and San Jose Sharks (if and when the team returns home from Arizona) all share a city with their AHL affiliate, as do the New Jersey Devils temporarily. The Anaheim Ducks, Arizona Coyotes, Boston Bruins (temporarily), Buffalo Sabres, Chicago Blackhawks, Colorado Avalanche, Columbus Blue Jackets, Detroit Red Wings, Los Angeles Kings, Ottawa Senators, Montreal Canadiens, Philadelphia Flyers, Pittsburgh Penguins all have their AHL affiliates within state or provincial lines as well.
For those teams with some distance between themselves and their minor league clubs, recalls could remain difficult. Especially for those Canadian teams whose affiliates remain in the U.S. – the Calgary Flames, Edmonton Oilers, and Vancouver Canucks – quarantine logistics will be a struggle. Johnston points out that for these teams and the American clubs with affiliates elsewhere in the country, travel will be a major obstacle. The one blanket policy for all NHL and AHL players this season is that a seven-day quarantine period is required following a commercial flight. This could also stand to effect any team on a long-term road trip that is desperate enough to make a recall.
However, while this policy will help a great number of teams, it is important to remember that taxi squads were established for this season to reduce the reliance on AHL recalls, at least as a frequent measure. Regardless of each NHL team's location relative to their AHL affiliate, most teams will largely use their six-man taxi squad for emergency substitutions and will have options in the meantime should they decide to recall a player who must quarantine.
AHL| Anaheim Ducks| Arizona Coyotes| Boston Bruins| Buffalo Sabres| Calgary Flames| Chicago Blackhawks| Colorado Avalanche| Columbus Blue Jackets| Detroit Red Wings| Edmonton Oilers| Los Angeles Kings| Montreal Canadiens| New Jersey Devils| NHL| Ottawa Senators| Philadelphia Flyers| Pittsburgh Penguins| Players| RIP| San Jose Sharks| Toronto Maple Leafs| Vancouver Canucks| Vegas Golden Knights| Winnipeg Jets
NHL Pulls Current Batch of "Tracking" Pucks
Minnesota Wild Acquire Ian Cole
Snapshots: Hurricanes, Babcock, Avalanche
Montreal Canadiens Announce New ECHL Affiliate
Calgary Flames Sign Michael Stone, Place On Waivers
Morning Notes: Penguins, Stützle, Fines


61 Crusader Marine PracticeX ([lLiTl] CareBearX) 5624 54% 129-109 2758 Master team games!
464 PracticeX 5544 51% 45-43 1425 Master team games!
566 PracticeX 5370 50% 4-4 56 Master team games!
616 Kachinsky PracticeX (IIIIIIIIIIII) 5408 0% 0-1 0 Master team games!
1083 Crusader Marine PracticeX ([lLiTl] CareBearX) 5151 48% 12-13 271 Master team games!
1281 Crusader Marine PracticeX ([lLiTl] CareBearX) 5087 60% 3-2 112 Master team games!


HPPSC SET Recruitment 2019: Himachal Pradesh Public Service Commission is inviting application for the Assistant Professors post. The HPPSC SET Recruitment is conducting the eligibility test for the aspirants for recruiting Assistant Professors. Likewise, the start date to apply for HP SET 2019 is 21st December 2018. Also, aspirants can apply for the HPPSC Recruitment still 10th January 2019. Also, HP SET exam is the online eligibility test for appearing HPPSC Recruitment 2019. To know further information about the HPPSC SET Recruitment, aspirants can refer the below table. So, aspirants should apply for HP SET 2019 before the last date. No other application form receiving after the last date is considered for HPPSC SET.
Aspirants like to apply for HPPSC SET Recruitment should check the official notification, before applying for the HP SET. Because the only eligible aspirants can apply for the post. Likewise, aspirants with the desire qualification can utilize this wonderful opportunity. Further, HPPSC SET Recruitment 2019 will provide a hand full of pay with excellent career growth. By following below all steps, aspirants can apply for HPPSC SET Recruitment to hold Himachal Pradesh government Jobs. Also, the aspirants must have a minimum of Masters qualification. With that, applicants can also should appear for the exam. So, the Exam for HP SET are in 2 papers.
How To Apply Online For HPPSC SET Recruitment?
The interested and eligible aspirants can apply for the HPPSC SET Recruitment. Also, aspirants can follow below all steps to apply for the post.
All the aspirants can apply for the post through online mode.
Likewise, applicants should apply for the HP SET before the last date.
Further, aspirants should enter the valid mobile number and email ID for future references.
Then, aspirants can get timely information about the HPPSC SET Recruitment.
So, candidates should follow all the instruction provided in the official notification for appearing HPPSC SET Recruitment.


Shop for the New Orleans Saints '47 NFL Breakaway Knit from lids.com and support the New Orleans Saints & NFL in style withlids.com Hats! Stand by your team on the coldest days with the NFL Breakaway Knit Cap by '47 Brand. The knit hat shows team spirit with bold stripes, a pompom in team colors and a raised embroidery mascot.


On his regular Thursday Good Morning on 7th Feb 2019, John Whittle welcomes into the studio Ann Bellamy of Community Action on Dementia. She's the coordinator of the Dance Memory Club for people living with dementia and their families and carers.
She talks about why music and dance can allow people with dementia to access memories and express themselves, and the amazing benefits the dancing group can have – including for family carers, for whom isolation can be a major problem.
The Dance Memory Club runs at the Cúram Centre, Tuesdays from 2.30 – 4 pm.
To find out more, contact Ann Bellamy on 086 145 1335.


text:
\section{Introduction}
In language modelling (LM), we learn distributions over sequences of words, subwords or characters, where the latter two can allow an open-vocabulary generation~\cite{Sutskever:2011:GTR:3104482.3104610}. We rely on subword segmentation as a widespread approach to generate rare subword units \cite{sennrich-etal-2016-neural}. However, the lack of a representative corpus, in terms of the word vocabulary, constrains the unsupervised segmentation (e.g. with scarce monolingual texts \cite{joshi-etal-2020-state}).
As an alternative, we could use character-level modelling, since it also has access to subword information \cite{kim-etal-2016-character}, but we face long-term dependency issues and require longer training time to converge.

In this context,
we focus on syllables, which are based on speech units: ``A syl-la-ble con-tains a sin-gle vow-el u-nit''. These are more linguistically-based units than characters, and behave as a mapping function to reduce the length of the sequence with a larger ``alphabet'' or syllabary. Their extraction can be rule-based and corpus-independent, but data-driven methods or hyphenation using dictionaries can approximate them as well (see \S\ref{sec:analysis-seg}).

Previous work on syllable-aware neural LM~}%{\textsc{lm}~ failed to beat characters in a closed-vocabulary generation at word-level \cite{assylbekov-etal-2017-syllable};
however, we propose to assess syllables under three new settings.
First, we analysed an open-vocabulary scenario with syllables by disregarding additional functions in the input layer (e.g. convolutional filters to hierarchically compose the representations \cite{botha-blunsom-2014-compositional}). Second, we extended the scope from 6 to 20 languages
to cover different levels of orthographic depth, which is the degree of grapheme-phoneme correspondence \cite{borgwaldt-etal-2005-onset} and a factor that can increase complexity to syllabification. English is a language with deep orthography (weak correspondence) whereas Finnish is transparent \cite{ziegler-etal-2010-orthographic}. Third, we distinguished rule-based syllabification with hyphenation tools, but also validated their proximity for LM.

Therefore, we revisit LM~}%{\textsc{lm}~ for open-vocabulary generation with syllables using pure recurrent neural networks~\cite{merity2018regularizing} for a more diverse set of languages, and compare their performance against characters and other subword units.

\section{Open-vocabulary language modelling with a comparable perplexity}

\paragraph{Language modelling} Given an input of generic sequence units (such as words, subwords or characters), denoted as $\mathbf{s}=s_1,s_2,...s_n$, a language model computes the probability of $\mathbf{s}$ as:
\begin{equation}
p(\mathbf{s})=\prod_{i=1}^{|\mathbf{s}|} p\left(s_{i} | s_{1}, s_{2}, \ldots, s_{i-1}\right)
\end{equation}

We then can use a recurrent neural network or RNN (e.g. a LSTM variation \cite{merity2018regularizing}), trained at each time step $t$, for calculating the probability of the input $s_{t+1}$, given $s_t$:
\begin{equation}
p\left(s_{t+1}|s_{\leq t}\right)=g\left(\operatorname{RNN}\left(\mathbf{w}_{t}, \mathbf{h}_{t-1}\right)\right)
\end{equation}

where $w_t$ is the learned embedding of the sequence unit $s_t$, $h_{t-1}$ is the hidden state of the RNN for the previous time step, and $g$ is a softmax function for the vocabulary space of the segmentation.

We thereafter compute the loss function $\mathcal{L}_{\mathrm{LM}}(\mathbf{s})$ for the neural LM~}%{\textsc{lm}~ as the cross entropy of the model for the sequence $\mathbf{s}$ in a time step t:
\begin{equation}
\mathcal{L}_{\mathrm{LM}}(\mathbf{s})=-\sum_{j=1}^{|V|} b_{t, j} \times \log \left(p({s}_{t, j}\right))
\end{equation}

where $|V|$ is the size of vocabulary, $ p({s}_{t, j})$ is the probability distribution over the vocabulary at each time-step $t$ and $b_{t, j}$ is an indicator if $j$ is the true token for the sequence $\mathbf{s}$.

\paragraph{Character-level perplexity} For a fair comparison across all granularities, we evaluate all results with character-level perplexity:
\begin{equation}
\operatorname{ppl}^{c} = \exp{(\mathcal{L}_{\mathrm{LM}}(\mathbf{s})\cdot\frac{|\mathbf{s}^{\text{seg}}|+1}{|\mathbf{s}^{c}|+1})}
\end{equation}

where $\mathcal{L}_{\mathrm{LM}}(\mathbf{s})$ is the cross entropy of a string $\mathbf{s}$ computed by the neural LM~}%{\textsc{lm}~, and $|\mathbf{s}^{\text{seg}}|$ and $|\mathbf{s}^{c}|$ refer to the length of $\mathbf{s}$ in the chosen segmentation and character-level units, respectively \cite{miekel2019charppl}. The extra unit considers the end of the sequence.

\paragraph{Open-vocabulary output} We generate the same input unit (e.g. characters or syllables) as an open-vocabulary LM~}%{\textsc{lm}~ task, where there is no prediction of an ``unknown'' or out-of-vocabulary word-level token \cite{Sutskever:2011:GTR:3104482.3104610}.
We thereby differ from previous work \cite{assylbekov-etal-2017-syllable}, and refrain from composing the syllable representations into words to evaluate only word-level perplexity.

\section{Experimental setup}

\paragraph{Languages and datasets}
Corpora are listed in Table \ref{tab:data}. We do not use the English Penn Treebank~\cite{marcus-etal-1993-building} as in \citet{assylbekov-etal-2017-syllable}, given that it is not suitable for open-vocabulary assessment. We then chose \mbox{WikiText-2-raw~\cite[en\textsubscript{wt2};][]{merity2016pointer}}, which contains around two million word-level tokens extracted from Wikipedia articles in English. Furthermore,
we employ 20 Universal Dependencies \cite[UD;][]{nivre-etal-2020-universal} treebanks, similarly to \citet{blevins-zettlemoyer-2019-better}.\footnote{The languages are chosen given the availability of an open-source syllabification or hyphenation tool. We prefer to use the UD treebanks, instead of other well-known datasets for language modelling (e.g. Multilingual Wikipedia Corpus \cite{kawakami-etal-2017-learning}), because they provide morphological annotation used for the study.}

\begin{table}
    \centering
    \setlength\tabcolsep{3pt}
    \resizebox{\linewidth}{!}{%
    \begin{tabular}{cl|rrr|rrr}
   &        &  \multicolumn{3}{c|}{\# tokens} & \multicolumn{3}{c}{\# types} \\ \hline
L. & Corpus & Word & Syl & Char & Word & Syl & Char \\ \hline
bg & UD-BTB & 125k & 386k & 710k & 25,067 & 6,035 & 132 \\
ca & UD-AnCora & 436k & 1,123k & 2,341k & 31,226 & 12,211 & 108 \\
cs & UD-PDT & 1,158k & 3,546k & 6,868k & 127,211 & 26,451 & 136 \\
da & UD-DDT & 81k & 215k & 442k & 16,300 & 7,362 & 96 \\
de & UD-GSD & 260k & 735k & 1,637k & 49,561 & 16,549 & 238 \\
en* & UD-EWT & 210k & 488k & 1,061k & 20,291 & 13,962 & 100 \\
en* & wikitext-2-raw & 2,089k & 4,894k & 10,902k & 33,279 & 20,173 & 398 \\
es* & UD-GSD & 376k & 1,060k & 2,043k & 46,623 & 13,676 & 292 \\
fi* & UD-TDT & 165k & 595k & 1,224k & 49,045 & 8,168 & 234 \\
fr & UD-GSD & 360k & 837k & 1,959k & 42,520 & 24,418 & 293 \\
it & UD-SET & 263k & 762k & 1,504k & 28,685 & 7,990 & 121 \\
hr & UD-ISDT & 154k & 484k & 930k & 33,365 & 7,374 & 106 \\
pt & UD-LVTB & 192k & 551k & 1,040k & 27,130 & 8,267 & 111 \\
lv & UD-Alpino & 113k & 349k & 690k & 28,677 & 7,014 & 116 \\
nl & UD-LFG & 187k & 488k & 1,074k & 26,704 & 10,553 & 96 \\
pl & UD-Bosque & 102k & 293k & 589k & 28,864 & 7,978 & 104 \\
ro & UD-RRT & 183k & 549k & 1,056k & 31,402 & 8,763 & 139 \\
ru* & UD-SynTagRus & 867k & 2,707k & 5,411k & 108,833 & 16,961 & 153 \\
sk & UD-SNK & 80k & 232k & 437k & 21,137 & 6,444 & 122 \\
tr* & UD-IMST & 38k & 126k & 242k & 13,966 & 2,595 & 84 \\
uk & UD-IU & 88k & 289k & 501k & 26,567 & 5,505 & 174 \\    \hline
    \end{tabular}
    }
    \caption{Total number of tokens and token types in the training set for words, syllables and characters (other segmentation data is in Appendix \ref{app:datasets}). We highlight (*) the languages with extracted rule-based syllables, and we report the statistics with hyphenation for the rest.}
    \label{tab:data}
\end{table}

\paragraph{Syllable segmentation} For splitting syllables in different languages, we used rule-based syllabification tools for English, Spanish, Russian, Finnish and Turkish, and a dictionary-based hyphenation tool for all of them except for Finnish and Turkish. All the tools are listed in Appendix \ref{app:segmentation}.

\paragraph{Segmentation baselines} Besides the annotated morphemes in the UD treebanks, we consider Polyglot (\url{polyglot-nlp.com}), which includes models for unsupervised morpheme segmentation trained with Morfessor \cite{virpioja-2013-morfessor}. Moreover, we employ an unsupervised subword segmentation baseline of Byte Pair Encoding~\cite[BPE;][]{sennrich-etal-2016-neural}\footnote{We use: \url{https://github.com/huggingface/tokenizers}} with different vocabulary sizes from 2,500 to 10,000 tokens, with 2,500 steps. We also fix the parameter to the syllabary size%
. Appendix \ref{app:segmentation} includes details about the segmentation format.

\paragraph{Model and training} Following other open-vocabulary LM~}%{\textsc{lm}~ studies~\cite{mielke-eisner-2019-spell, mielke-etal-2019-kind}, we use a low-compute version of an LSTM neural network, named Average SGD Weight-Dropped~\cite{merity2018regularizing}, developed in PyTorch\footnote{\url{https://github.com/salesforce/awd-lstm-lm}}. Appendix \ref{app:model} includes more details.

\section{Segmentation analysis}
\label{sec:analysis-seg}

\paragraph{Syllabification versus Hyphenation} For English, Spanish and Russian, we confirmed that hyphenation is a reasonable proxy for syllabification in LM. Table \ref{tab:syl-vs-hyphen} shows that the overlapping of hyphenation pieces concerning rule-based syllables is more than 70\%.
Furthermore, we note that the perplexity gap is not significant. The small sample is still representative because English has a deep orthography in contrast with Spanish or Russian, and Russian use a different script (Cyrillic instead of Latin). In terms of morphology, all of them are fusional, but highly agglutinative languages like Turkish or Finnish are not feasible candidates for developing a Hunspell-like dictionary, which is the source of most hyphenation tools.

\begin{table}[h!]
    \centering

\resizebox{0.8\linewidth}{!}{%
\begin{tabular}{l|cc|r}
                      & $(H \cap S)/S$ & $(S \cap H)/H$ & $\Delta\operatorname{ppl}^c$\textsubscript{S-H} \\ \hline
en\textsubscript{wt2} & 70.07\%        & 73.84\%            & -0.0001  \\
en\textsubscript{UD}  & 78.13\%        & 74.68\%        & -0.0186  \\
es                    & 76.37\%        & 61.95\%        & 0.0373   \\
ru                    & 73.58\%        & 56.13\%        & 0.0901   \\ \hline
\end{tabular}
}
\caption{Token types overlapping of syllabification (S) versus hyphenation (H) and their perplexity gap.}
\label{tab:syl-vs-hyphen}
\end{table}

\paragraph{Vocabulary growth} In Figure \ref{fig:vocab-size}, we show the vocabulary growth rate of unsupervised morphemes (with Morfessor)\footnote{We do not show the growth of the annotated morpheme pieces, because they approximate the word-level vocabulary given the presence of lemmas.} in contrast with syllables extracted by rules (S) or hyphenation (H). We do not observe a significant difference between syllabification and hyphenation, which reinforce their functional proximity for our study. We also observe that in all the UD datasets we do not have more than 10k Morfessor-based pieces, which is the reason for the upper boundary on our BPE baselines. However, for Czech, German, English and French, we observe that the syllabary size significantly surpass the number of Morfessor-based pieces. Potential explanations are the orthographic depth and the degree of syllabic complexity for those languages \cite{borleffs-etal-2017-measuring}: a low letter-sound correspondence and the difficulty to determine the syllable boundaries can induce a larger syllabary.

\begin{figure}[h!]
    \centering
\includegraphics[width=\linewidth,clip]{figs/vocabsize-all-langs.pdf}
\includegraphics[width=0.8\linewidth,clip]{figs/vocabsize-legend.pdf}
\caption{Vocabulary growth: accumulative number of token types (y-axis) per samples in the training data (x-axis) for all the UD datasets. Y-axis range is fixed.}
\label{fig:vocab-size}
\end{figure}

\paragraph{Overlapping of syllables with subwords} In Figure \ref{fig:over-morph}, we show the stacked area plot of the overlapping ratio of syllables, Morfessor-based pieces and different BPE settings concerning the annotated morphemes. We note that the proportion of syllable overlapping is relatively low in contrast with unsupervised morphemes or BPE with larger vocabulary size. The outcome is expected, as the annotated morpheme contains lemmas or full words with long sequences of characters, which are adopted by BPE with more merge-operations.

Analogously, Figure \ref{fig:over-poly} shows the overlapping of syllables and BPE concerning unsupervised morphemes. We observe that the intersection ratio of syllables has increased, and approximate the values of BPE with larger vocabulary size.

In both scenarios, it is worthy to note that the BPE setting with the largest overlapping with annotated or unsupervised morphemes has a vocabulary size that equates the syllabary. Future work can assess whether the number of unique syllables can support the tuning of the BPE vocabulary size.

Finally, in Figure \ref{fig:over-bpe}, we observe that syllables overlap the most with BPE pieces that uses a small vocabulary size of 2,500. From 5,000 to 10,000, the overlapping ratio shows a downtrend for most of the datasets. However, in 9 out of 20 datasets, the largest BPE setting (fixed with the syllabary size) shows an increase of the overlapping.

\begin{figure*}[t!]
\begin{center}
\centering

\begin{subfigure}[t]{0.31\linewidth}
\caption{Overlapping w.r.t. Morph.}
\label{fig:over-morph}
\includegraphics[width=\linewidth,clip]{figs/overlaping-morph.png}
\end{subfigure}
\begin{subfigure}[t]{0.31\linewidth}
\caption{Overlapping w.r.t. Poly.}
\label{fig:over-poly}
\includegraphics[width=\linewidth,clip]{figs/overlaping-poly.png}
\end{subfigure}
\begin{subfigure}[t]{0.32\linewidth}
\caption{Overlapping w.r.t. BPE}
\label{fig:over-bpe}
\includegraphics[width=\linewidth,clip]{figs/overlaping-bpe.png}
\end{subfigure}

\caption{Overlapping analysis of syllables (Syl-S: Rules, Syl-H: Hyphenation) with respect to annotated morphemes (Morph.), unsupervised morphemes (with Morfessor on Poly.) and BPE with different vocabulary sizes in thousands (* indicates the same vocabulary size as the syllabary). The operation $A \cap / B$ is equivalent to $(A \cap B)/B$, where $A$ and $B$ are number of token types (unique subword pieces). Overlapping ratios in plots (a-b) are normalised from 0 to 100\%, whereas in (c) is only stacked.}
\label{fig:convergence}
\end{center}
\end{figure*}

\section{Open-vocabulary LM results}

Table \ref{tab:perplexity} shows the $\operatorname{ppl}^c$ values for the different levels of segmentation we considered in the study, where we did not tune the neural LM~}%{\textsc{lm}~ for an specific setting. We observe that syllables always result in better perplexities that other granularities, even for deep orthography languages such as English or French.
The results obtained by the BPE baselines are relatively poor as well, and they could not beat characters in any dataset.
We could search for an optimal parameter for the BPE algorithm; however, the advantage of the syllables is that we do not need to tune an hyper-parameter to extract a different set of subword pieces.

As a significant outcome, we note that syllables did not fail to beat characters, at least in an open-vocabulary LM~}%{\textsc{lm}~ task, which extends the results provided by \citet{assylbekov-etal-2017-syllable}. Moreover, in Figure \ref{fig:ppl-diff} in the Appendix, we can observe a strong linear relationship of the syllable type/token ratio with the gain of $\operatorname{ppl}^c$ obtained by syllables concerning characters. In other words, if our dataset possesses a rich syllabary, we are fairly approximating the amount of word-level tokens, which reduces the $\operatorname{ppl}^c$ gain.

Beating characters implies a gain in time processing as well, given the shorter sequences of the syllable pieces. Figure \ref{fig:convergence} shows details about how many epochs each segmentation requires to converge during training, as well as how much time requires all the training. Syllable-level LM~}%{\textsc{lm}~ trains faster than characters, and even when they are slower (seconds per epoch) than using BPE-pieces or unsupervised morphemes, they obtain a better $\operatorname{ppl}^c$ score.

\begin{table}
    \centering
    \setlength\tabcolsep{3pt}
    \resizebox{\linewidth}{!}{%
    \begin{tabular}{l|ccccc}
 & Char. & Morph. & Poly. & Syl. & BPE\textsubscript{best} \\ \hline
bg & 3.56 \small{$\pm$0.03} & 4.09 \small{$\pm$0.05} & 4.69 \small{$\pm$0.01} & \textbf{2.87} \small{$\pm$0.0} & 5.19 \small{$\pm$0.01} \\
ca & 2.84 \small{$\pm$0.0} & 3.11 \small{$\pm$0.02} & 3.26 \small{$\pm$0.01} & \textbf{2.21} \small{$\pm$0.0} & 3.31 \small{$\pm$0.0} \\
cs & 3.32 \small{$\pm$0.0} & 3.11 \small{$\pm$0.01} & 4.18 \small{$\pm$0.01} & \textbf{2.66} \small{$\pm$0.0} & 4.24 \small{$\pm$0.0} \\
da & 4.25 \small{$\pm$0.01} & 4.42 \small{$\pm$0.04} & 5.6 \small{$\pm$0.0} & \textbf{3.1} \small{$\pm$0.01} & 6.21 \small{$\pm$0.03} \\
de & 3.5 \small{$\pm$0.04} & 3.36 \small{$\pm$0.08} & 3.79 \small{$\pm$0.0} & \textbf{2.48} \small{$\pm$0.0} & 3.86 \small{$\pm$0.02} \\
en\textsubscript{UD}* & 4.11 \small{$\pm$0.01} & 4.39 \small{$\pm$0.08} & 5.67 \small{$\pm$0.01} & \textbf{2.82} \small{$\pm$0.07} & 5.65 \small{$\pm$0.04} \\
en\textsubscript{wt2}* & 2.48 \small{$\pm$0.0} & - & 2.8 \small{$\pm$0.0} & \textbf{1.96} \small{$\pm$0.0} & 2.91 \small{$\pm$0.0} \\
es* & 3.16 \small{$\pm$0.01} & 3.71 \small{$\pm$0.04} & 3.95 \small{$\pm$0.01} & \textbf{2.51} \small{$\pm$0.0} & 3.98 \small{$\pm$0.0} \\
fi* & 3.77 \small{$\pm$0.01} & 4.05 \small{$\pm$0.12} & 4.76 \small{$\pm$0.01} & \textbf{3.1} \small{$\pm$0.0} & 5.27 \small{$\pm$0.01} \\
fr & 3.09 \small{$\pm$0.01} & 3.67 \small{$\pm$0.02} & 3.82 \small{$\pm$0.01} & \textbf{2.3} \small{$\pm$0.01} & 3.87 \small{$\pm$0.01} \\
hr & 3.52 \small{$\pm$0.02} & 3.92 \small{$\pm$0.01} & 4.34 \small{$\pm$0.0} & \textbf{2.8} \small{$\pm$0.0} & 4.52 \small{$\pm$0.02} \\
it & 2.8 \small{$\pm$0.0} & 3.19 \small{$\pm$0.0} & 3.43 \small{$\pm$0.01} & \textbf{2.27} \small{$\pm$0.01} & 3.61 \small{$\pm$0.0} \\
lv & 4.55 \small{$\pm$0.02} & 5.31 \small{$\pm$0.0} & 6.82 \small{$\pm$0.02} & \textbf{3.59} \small{$\pm$0.0} & 7.19 \small{$\pm$0.0} \\
nl & 3.83 \small{$\pm$0.05} & 3.69 \small{$\pm$0.1} & 4.44 \small{$\pm$0.01} & \textbf{2.76} \small{$\pm$0.01} & 4.83 \small{$\pm$0.01} \\
pl & 4.03 \small{$\pm$0.01} & 4.77 \small{$\pm$0.22} & 5.96 \small{$\pm$0.04} & \textbf{3.19} \small{$\pm$0.0} & 5.99 \small{$\pm$0.0} \\
pt & 3.31 \small{$\pm$0.01} & 3.46 \small{$\pm$0.03} & 4.03 \small{$\pm$0.01} & \textbf{2.56} \small{$\pm$0.0} & 4.24 \small{$\pm$0.01} \\
ro & 3.4 \small{$\pm$0.02} & 3.89 \small{$\pm$0.04} & 4.25 \small{$\pm$0.01} & \textbf{2.72} \small{$\pm$0.0} & 4.71 \small{$\pm$0.01} \\
ru* & 3.28 \small{$\pm$0.0} & 2.93 \small{$\pm$0.01} & 4.05 \small{$\pm$0.0} & \textbf{2.69} \small{$\pm$0.01} & 4.04 \small{$\pm$0.0} \\
sk & 6.16 \small{$\pm$0.05} & 5.1 \small{$\pm$0.07} & 7.61 \small{$\pm$0.08} & \textbf{4.62} \small{$\pm$0.01} & 10.51 \small{$\pm$0.03} \\
tr* & 4.16 \small{$\pm$0.05} & 4.86 \small{$\pm$0.05} & 6.41 \small{$\pm$0.07} & \textbf{3.66} \small{$\pm$0.03} & 6.98 \small{$\pm$0.1} \\
uk & 4.92 \small{$\pm$0.02} & 6.45 \small{$\pm$0.11} & 8.11 \small{$\pm$0.03} & \textbf{4.24} \small{$\pm$0.02} & 9.23 \small{$\pm$0.02} \\  \hline

    \end{tabular}
    }
    \caption{Character-level perplexity ($\downarrow$) in test. We show the mean and standard deviation for three runs with different seeds. ``Syl.'' presents the syllabification-based result if it is available (*), or the hyphenation otherwise.}
    \label{tab:perplexity}
\end{table}

\begin{figure}[]
\begin{center}
\centering

\begin{subfigure}[t]{0.49\linewidth}
\label{fig:wiki-ppl}
\includegraphics[width=\linewidth,clip]{figs/en-wikitext-2-raw.pdf}
\end{subfigure}
\begin{subfigure}[t]{0.49\linewidth}
\label{fig:wiki-time}
\includegraphics[width=\linewidth,clip]{figs/en-wikitext-2-raw_time.pdf}
\end{subfigure}

\caption{Left: validation perplexity for the English wikitext-2 per epoch. Right: training time (in seconds) per epoch for the same dataset. Similar plots for other datasets are included in the Appendix.}
\label{fig:convergence}
\end{center}
\end{figure}

\section{Related work}

In subword-aware LM, \citet{vania-lopez-2017-characters} investigated if we can capture morphology using characters, character n-grams, BPE pieces or morphemes with a closed-vocabulary. Whereas for open-vocabulary generation, \citet{blevins-zettlemoyer-2019-better} incorporates morphological supervision with a multi-task objective, and \citet{kawakami-etal-2017-learning,mielke-eisner-2019-spell} have focused in improving the neural architecture to jointly use the representation of characters and words in a hybrid open-vocabulary setting.

The closest study to ours is from \citet{mikolov-etal-2012-subword}, where they performed subword-grained prediction with different settings, and used syllables as a proxy to split words with low frequency, reduce the vocabulary and compress the model size. However, they only focused on English. Besides, syllable-aware LM~}%{\textsc{lm}~ was addressed by \citet{assylbekov-etal-2017-syllable} for English, German, French, Czech, Spanish and Russian, and by \citet{yu-etal-2017-syllable} for Korean. However, in both cases, the syllable units have been composed with convolutional filters into word-level representations to assess a closed-vocabulary setting. As far as we know, we propose the first syllable-level open-vocabulary LM~}%{\textsc{lm}~ study that analyses up to 20 languages and compare their results against morphemes.

\section{Limitations and opportunities}

Syllables only cannot offer a universal solution to the subword segmentation problem for all the languages, as the syllabification tools are language-dependent. Besides, the analysis should be extended to different scripts and morphological typology\footnote{Only Bulgarian, Russian and Ukranian use the Cyrillic script instead of Latin; and only Finnish and Turkish have highly agglutinative morphology instead of fusional. Polysynthetic languages can also be potential beneficiaries with syllable-level language modelling}. Furthermore, we do not encode any semantics in the syllable-vector space, with a few exceptions like in Korean \cite{choi-etal-2017-syllable}.

Nevertheless, our results confirm that syllables are reliable for LM, and building a syllable splitter might require less effort than annotating morphemes to train a robust supervised tool.\footnote{For instance, the syllabification tool that we used for English is based on five general rules published in: \url{https://www.howmanysyllables.com/divideintosyllables}. Their implementation should take less effort than annotating a UD treebank or building a Finite-State-Transducer for morphological analysis.}
Moreover, as we discussed at the end of \S\ref{sec:analysis-seg}, we could further assess whether the syllables can support the hyper-parameter tuning or impact the internal procedure of unsupervised segmentation methods like BPE or Morfessor, which are corpus-dependent.

\section{Conclusion}
We proved that syllables are valuable for an open-vocabulary LM~}%{\textsc{lm}~ task, where they behave positively even for languages with deep orthography, and they overcome character-level models that require longer time to train. Syllables do not have an embedded meaning in most of the languages; however, the required effort for their segmentation could be advantageous against morphological-aware or unsupervised-driven methods. Given our analysis, we could consider working on syllable-driven subword segmentation, neural machine translation and hybrid-LM~}%{\textsc{lm}~ (with characters and words).

\section*{Acknowledgments}
\lettrine[image=true, lines=2, findent=1ex, nindent=0ex, loversize=.15]{figs/eu-logo.png}%
{T}he first author is supported by funding from the European Union's Horizon 2020 research and innovation programme under grant agreements No 825299 (GoURMET) and the EPSRC fellowship grant EP/S001271/1 (MTStretch), whereas the second author is supported by a research grant of the ``Consejo Nacional de Ciencia, Tecnolog\'ia e Innovaci\'on Tecnol\'ogica'' (CONCYTEC, Peru) under the contract 183-2018-FONDECYT-BM-IADT-MU.

We express our thanks to Shay B. Cohen, Clara Vania and Adam L\'opez, whose provided us feedback in a very early stage of the study, and to Alexandra Birch, Barry Haddow, Fernando Alva-Manchego, John E. Miller and Marco Sobrevilla-Cabezudo, whose reviewed different draft versions. Finally, we acknowledge the support of NVIDIA Corporation with the donation of a Titan Xp GPU used for the study.

redpajama_set_name: RedPajamaArXiv


text:
Ausubel ist der Nachname von
 David Paul Ausubel (* 1918, † 2008), US-amerikanischer Pädagoge, Lerntheoretiker und Hochschullehrer
 Jesse H. Ausubel, US-amerikanischer Umweltwissenschaftler

redpajama_set_name: RedPajamaWikipedia