text:
Baystate Announces Leadership Appointments, New Name for Wing
By BusinessWest Staff August 28, 2014 560 No comment
SPRINGFIELD — With the change of Wing Memorial Hospital's parent company from UMass Memorial Health Care to Baystate Health expected to take place on Sept. 1, Baystate Health announced leadership appointments for its Eastern Region and a new name for the Palmer hospital: Baystate Wing Hospital.
As of Sept. 1, Dr. Charles Cavagnaro III, now president and CEO of Wing Memorial Hospital, will be appointed president of Baystate Health's Eastern Region, which is comprised of Baystate Mary Lane Hospital, Baystate Wing Hospital, and the employed medical practices and medical centers in that region. As he assumes this leadership position, Cavagnaro has appointed Dr. Shafeeq Ahmed chief operating officer of Baystate Health's Eastern Region. Ahmed will also continue in his role as chief medical officer of Baystate Mary Lane Hospital and Baystate Medical Practices Eastern Region.
As president of the region, Cavagnaro will provide strategic, executive, and operational leadership for the two hospitals there, which provide inpatient medical, surgical, gynecological, and behavioral-health services; emergency services; as well as a range of primary-care and other outpatient services for about 80,000 people in that region. He will report to Dennis Chalke, senior vice president of Baystate Health Community Hospitals and senior vice president and chief financial officer of Baystate Health.
Cavagnaro has served as president and CEO of Wing Memorial Hospital and Medical Centers for the past 15 years. Before that, he was Wing's vice president of Medical Affairs. In 2013, he served as interim president of UMass Memorial Medical Center in Worcester. Amid a challenging financial environment, Cavagnaro and his team have led Wing to serve a greater number of patients, add hospital beds, and increase the depth and breadth of services in its health centers and outpatient practices.
The hallmark of his leadership is a focus on quality and safety that led to an 'A' safety rating for Wing from Leapfrog Group and Top Performer status from the Joint Commission on core accountability measures. Cavagnaro is a board-certified primary-care internist in Belchertown and sees patients on a weekly basis. He is a Six Sigma and Lean Green Belt in healthcare quality-improvement processes.
Cavagnaro completed his residency in internal medicine at UMass Medical Center and is a graduate of Cornell University Medicine College in New York City. He is a member of the American College of Physician Executives and the American College of Healthcare Executives, and a fellow of the American College of Physicians. He was a founding member of the American Academy of Hospice Physicians, which is now known as the American Academy of Hospice and Palliative Medicine. He has served as a medical examiner for District IV of Hampshire County, and as a member of the board of directors of the Pioneer Valley Chapter of the Red Cross. He presently serves on the American Hospital Assoc. Regional Policy Board for the New England Region.
Ahmed was recently named one of '100 Hospital and Health System Chief Medical Officers to Know' by Becker's Hospital Review. Prior to his arrival at Baystate Mary Lane Hospital, Ahmed had served as president of the medical staff, chief of Ob/Gyn, and a member of the hospital board of directors at the Naval Hospital in Cherry Point in North Carolina. Ahmed is a board-certified obstetrician/gynecologist at Baystate Medical Practices – Mary Lane Ob/Gyn in Ware, and sees patients on a weekly basis.
Over the next several months, Cavagnaro and Ahmed will be working together as part of a larger, broadly experienced team to develop the integration strategy for Baystate Health's Eastern Region.
The name Baystate Wing Hospital aspires to honor the hospital's more-than-100-year history, and recognize the vision and contributions of the Wing family and the countless others who have contributed their time, talents, and money to making Wing the strong community health provider it is today, while also celebrating Wing's new affiliation with Baystate Health.
Tags: Health Care
Former HMC President Pays Back $860,000 for Overcompensation
Springfield College President Dr. Mary-Beth Cooper to Address Rotary
SBA Massachusetts Announces 2015 Lender Awards
By BusinessWest Staff November 20, 2015
Organ Concert Slated for Nov. 6 at Historic Old First Church
Leap Joins Easthampton Savings as Commercial Lender
By BusinessWest Staff June 3, 2014

redpajama_set_name: RedPajamaCommonCrawl


My Story: Choose Your Own Path hack, My Story: Choose Your Own Path cheat, My Story: Choose Your Own Path iOS hack, My Story: Choose Your Own Path android hack, My Story: Choose Your Own Path generator, My Story: Choose Your Own Path online cheat.
Small Diamonds Package $4.99 FREE!!
Extra Small Tickets Package $1.99 FREE!!
Medium Diamonds Package $9.99 FREE!!
Medium Tickets Package $4.99 FREE!!
Extra Small Diamonds Package $1.99 FREE!!
Large Diamonds Package $19.99 FREE!!
Large Tickets Package $9.99 FREE!!


Warri's generous prophet has done it again! this time Fufeyin restores a man's manhood (video) Within NigeriaThe God of Prophet Jeremiah Omoto Fufeyin has done it again as Mr. Chris Muabu's manhood erection was restored to normalcy.
Warri women write Buhari, Osinbajo over $6m, N1b Ugborodo fund VanguardThe fate of $6 million and N1 billion provided for the construction of Ugborodo New Town in Warri Kingdom, Delta State, is being questioned by Itsekiri leaders.
Bovi, D'Banj, others thrill fans in Warri – The Guardian Nigeria News Nigerian BulletinBovi Ugboma [/caption0'Bovi Man on Fire', a comedy show by Bovi Ugboma, will hold tomorrow, April 18, at Brownhill ultra-modern hall in Warri, Delta State.
Bovi, D'Banj, others thrill fans in Warri GuardianBovi Ugboma [/caption0'Bovi Man on Fire', a comedy show by Bovi Ugboma, will hold tomorrow, April 18, at Brownhill ultra-modern hall in Warri, Delta State.
FG Suspends Free Train Ride On Itakpe-Warri Route Plus TV AfricaThe Nigerian government has ordered the immediate suspension of the free train ride for passengers on the Itakpe-Warri route.
In Warri, water transportation thrives without life jackets ― Investigations NIGERIAN TRIBUNEInvestigations have revealed that water transportation from Warri in Delta State to many islands like Kurutie and Okerenkoko amongst others.
Shipowner Decries High Insecurity At Warri, Bonny Anchorages Leadership NewspaperPresident of the Nigeria Shipowners Association (NISA), Aminu Umar has raised the alarm over high rate of insecurity at the Warri and Bonny anchorage.
Gyan, Akushe win Warri/Effurun Peace Marathon | The Guardian Nigeria News - Nigeria and World NewsSport GuardianEmmanuel Gyan and Rose Akushe yesterday emerged winners of the 2019 Warri/Effurun Peace Marathon held in Warri, Delta State.
Gyan, Akushe win Warri/Effurun Peace Marathon GuardianEmmanuel Gyan and Rose Akushe yesterday emerged winners of the 2019 Warri/Effurun Peace Marathon held in Warri, Delta State.
Here are the Highlights from The Business Women Connect (TBWC) in Warri BellaNaijaThe second edition of The Business Women Connect which held on the 23rd of March at the Brownhill events center Warri, Delta State.
Delta Government Approves Extension Of Landing Jetty In Warri The Nigerian VoiceTo enhance security of lives and property, the Delta State is goIng further to ensure adequate security in the state. The Delta State government has approved th.


text:
\section{Analysis}

\noindent \textbf{Sensitivity to Random Seed.} To better demonstrate that training with super tickets effectively reduces model variance, we evaluate models' sensitivity to changes in random seeds during single task fine-tuning and multi-task downstream fine-tuning. In particular, we investigate fitting small tasks with highly over-parametrized models (variance is often large in these models, see Section~\ref{sec:st_exp} and~\ref{sec:mt_exp}). As shown in Table~\ref{tb:seed_study}, SuperT$\textsubscript{LARGE}$ and Ticket-Share$\textsubscript{LARGE}$ induce much smaller standard deviation in validation results. Experimental details and further analyses are deferred to Appendix~\ref{app:ana-sensitivity}.

\begin{table}[htb!]
\centering
\small
\begin{tabular}{@{\hskip3pt}l@{\hskip3pt}|C|C|C|C|C}
\toprule
& \textbf{RTE} & \textbf{MRPC} & \textbf{CoLA} & \textbf{STS-B} & \textbf{SST-2} \\ \midrule
ST-DNN\textsubscript{LARGE}  & 1.17 & 0.61 & 1.32 & 0.16 & 0.17 \\
SuperT\textsubscript{LARGE}  & 0.72 & 0.20 & 0.97 & 0.07 & 0.16 \\ \midrule
MT-DNN\textsubscript{LARGE}   & 1.43 & 0.78 & 1.14  & 0.15 & 0.18  \\
Ticket Share\textsubscript{LARGE}   & 0.99 & 0.67 & 0.81 & 0.08 & 0.16  \\ \bottomrule
\end{tabular}
\caption{Standard deviation of tasks in GLUE (dev) over $5$ different random seeds.}
\label{tb:seed_study}
\end{table}

\noindent \textbf{Tickets Importance Across Tasks.} We analyze the importance score of each ticket computed in different GLUE tasks. For each ticket, we compute the importance score averaged over tasks as the \textit{Ticket Importance}, and the proportion of the task-specific importance score out of the sum of all tasks' scores as the \textit{Task Share}, as illustrated in Figure~\ref{fig:head_task_analysis}.

We observe that many tickets exhibit almost equal \textit{Task Share}s for over $5$ out of $8$ tasks (Figure~\ref{fig:head_task_analysis}(a)(b)). While these tickets contribute to the knowledge sharing in the majority of tasks, they are considered non-expressive for tasks such as SST-2 (see Figure \ref{fig:head_task_analysis}(a)(c)(d)). This explains why SST-2 benefits little from tickets sharing. Furthermore, a small number of tickets are dominated by a single task, e.g., CoLA (Figure~\ref{fig:head_task_analysis}(c)), or dominated jointly by two tasks, e.g., CoLA and STS-B (Figure~\ref{fig:head_task_analysis}(d)). This suggests that some tickets only learn task-specific knowledge, and the two tasks may share certain task-specific knowledge.

\section{Appendix}
\label{sec:appendix}

\subsection{Single Task Experiments}
\subsubsection{Data}
\label{app:st-data}
\textbf{GLUE.} GLUE is a collection of nine NLU tasks. The benchmark includes question answering~\cite{squad1}, linguistic acceptability (CoLA, \citealt{cola2018}), sentiment analysis (SST, \citealt{sst2013}), text similarity (STS-B, \citealt{sts-b2017}), paraphrase detection (MRPC, \citealt{mrpc2005}), and natural language inference (RTE \& MNLI, \citealt{rte1,rte2,rte3,rte5,mnli2018}) tasks. Details of the GLUE benchmark, including tasks, statistics, and evaluation metrics, are summarized in Table~\ref{tab:glue}.

\subsubsection{Training}
\label{app:st-training}
We use Adamax as the optimizer. A linear learning rate decay schedule with warm-up over $0.1$ is used. We apply a gradient norm clipping of $1$. We set the dropout rate of all task specific layers as $0.1$, except $0.3$ for MNLI and $0.05$ for CoLA. All the texts were tokenized using wordpieces, and were chopped to spans no longer than $512$ tokens. All experiments are conducted on Nvidia V100 GPUs.

\subsubsection{Evaluation Results Statistics}
\label{app:st-val-stats}

We conduct $5$ sets of experiments on different random seeds. Each set of experiment consists of fine-tuning, pruning, and rewinding at $8$ sparsity levels. For results on GLUE dev set (Table~\ref{tb:st-results}), we report the average score of super tickets rewinding results over $5$ sets of experiments. The standard deviation of the results is shown in Table~\ref{tb:st-val-stats}. The statistics of the percent of weight remaining in the selected super tickets are shown in Table~\ref{tb:st-compress-stats}.

\begin{table*}[htb!]
\centering
\small
\begin{tabular}{@{\hskip3pt}l@{\hskip3pt}|C|C|C|C|C|C|C|C} \toprule
& \textbf{RTE} & \textbf{MRPC} & \textbf{CoLA} & \textbf{STS-B} & \textbf{SST-2} & \textbf{QNLI} & \textbf{QQP} & \textbf{MNLI} \\ \midrule
SuperT\textsubscript{BASE}  & 0.91 & 0.74 & 1.51 & 0.49 & 0.50 & 0.10 & 0.08 & 0.04\\
SuperT\textsubscript{LARGE} & 0.72 & 0.20 & 0.97 & 0.07 & 0.16 & 0.07 & 0.11 & 0.02 \\ \bottomrule
\end{tabular}
\caption{Standard deviation of the evaluation results on GLUE development set over $5$ different random seeds.}
\label{tb:st-val-stats}
\end{table*}

\begin{table*}[htb!]
\centering
\small
\begin{tabular}{@{\hskip3pt}l@{\hskip3pt}|C|C|C|C|C|C|C|C} \toprule
& \textbf{RTE} & \textbf{MRPC} & \textbf{CoLA} & \textbf{STS-B} & \textbf{SST-2} & \textbf{QNLI} & \textbf{QQP} & \textbf{MNLI} \\ \midrule
SuperT\textsubscript{BASE} (Mean)  & 0.83 & 0.86 & 0.89 & 0.86 & 0.93 & 0.93 & 0.93 & 0.93\\
SuperT\textsubscript{BASE} (Std Dev)  & 0.07 & 0.08 & 0.04 & 0.06 & 0.07 & 0.00 & 0.00 & 0.00\\
SuperT\textsubscript{LARGE} (Mean) & 0.82 & 0.66 & 0.84 & 0.77 & 0.79 & 0.90 & 0.84 & 0.92 \\
SuperT\textsubscript{LARGE} (Std Dev) & 0.04 & 0.04 & 0.00 & 0.10 & 0.05 & 0.03 & 0.00 & 0.00 \\
\bottomrule
\end{tabular}
\caption{Statistics of the percent of weight remaining of the selected super tickets over $5$ different random seeds.}
\label{tb:st-compress-stats}
\end{table*}

For results on GLUE test set (Table~\ref{tb:st-results-test}), as the evaluation server sets an limit on submission times, we only evaluate the test prediction under a single random seed that gives the best task-average validation results.

\subsection{Multi-task Learning Experiments}
\label{app:mtl-training}
\subsubsection{Multi-task Model Training}
We adopt the MT-DNN code base and adopt the exact optimization settings in \citet{liu2020mtmtdnn}. We use Adamax as our optimizer with a learning rate of $5\times 10^{-5}$ and a batch size of $32$. We train for a maximum number of epochs of $5$ with early stopping. A linear learning rate decay schedule with warm-up over $0.1$ was used. The dropout rate of all the task specific layers is set to be $0.1$, except $0.3$ for MNLI and $0.05$ for CoLa. We clipped the gradient norm within $1$. All the texts were tokenized using wordpieces, and were chopped to spans no longer than $512$ tokens.

Worth mentioning, the task-specific super tickets used in Ticket Share are all selected during the case where a matched learning rate (i.e., $5\times 10^{-5}$) is used in single task fine-tuning. We empirically find that, rewinding the super tickets selected under a matched optimization settings usually outperforms those selected under a mismatched settings (i.e. using two different learning rates in single-task fine-tuning and rewinding/multi-task learning). This agrees with previous observation in literature of Lottery Ticket Hypothesis, which shows that unstructured winning tickets are not only related to its weight initialization, but also model optimization path.

\subsubsection{Multi-task Model Downstream Fine-tuning}
We follow the exact optimization setting as in Section~\ref{sec:st_exp_training} and in Section~\ref{app:st-training}, except we choose learning rate in $\{1\times10^{-5}, 2\times10^{-5}, 5\times10^{-5}, 1\times10^{-4}, 2\times10^{-4}\}$, and choose the dropout rate of all task specific layers in $\{0.05, 0.1, 0.2, 0.3\}$.

\subsection{Domain Adaptation Experiments}
\subsubsection{Data}
\label{app:da-data}

\noindent \textbf{SNLI.} is one of the most widely used entailment dataset for NLI.

\noindent \textbf{SciTail} involves assessing whether a given premise entails a given hypothesis. In contrast to other entailment datasets, the hypotheses in SciTail is created from science questions. These sentences are linguistically challenging. The corresponding answer candidates and premises come from relevant web sentences. The lexical similarity of premise and hypothesis is often high, making SciTail particularly challenging.

Details of the SNLI and SciTail, including tasks, statistics, and evaluation metrics, are summarized in Table~\ref{tab:glue}.

\subsubsection{Training}
\label{app:da-training}
For single task model domain adaptation from MNLI to SNLI/SciTail, we follow the exact optimization setting as in Section~\ref{sec:st_exp_training} and in Section~\ref{app:st-training}, except we choose the learning rate in $\{5\times10^{-5}, 1\times10^{-4}, 5\times10^{-4}\}$.

\subsection{Sensitivity Analysis}
\label{app:ana-sensitivity}
\subsubsection{Randomness Analysis}
For single task experiments in Table~\ref{tb:seed_study}, we vary the random seeds only and keep all other hyper-parameters fixed. We present the standard deviation of the validation results over $5$ trails rewinding experiments. For multi-task downstream fine-tuning experiments,  we present the standard deviation of the validation results over $5$ trails, each result averaged over learning rates in $\{5\times 10^{-5}, 1\times 10^{-4}, 2\times 10^{-4}\}$. This is because the downstream fine-tuning performance is more sensitive to hyper-parameters.

\subsubsection{Hyper-parameter Analysis}
We further analyze the sensitivity of Ticket Share$\textsubscript{LARGE}$ model to changes in hyper-parameters in downstream fine-tuning in some GLUE tasks. We vary the learning rate in $\{5\times 10^{-5}, 1\times 10^{-4}, 2\times 10^{-4}\}$ and keep all other hyper-parameter fixed. Table~\ref{tb:lr_study} shows the standard deviation of the validation results over different learning rates, each result averaged over $5$ different random seeds. As can be seen, Task Share$\textsubscript{LARGE}$ exhibits stronger robustness to changes in learning rate in downstream fine-tuning.

\begin{table}[htb!]
\centering
\small
\begin{tabular}{@{\hskip3pt}l@{\hskip3pt}|C|C|C|C|C}
\toprule
& \textbf{RTE} & \textbf{MRPC} & \textbf{CoLA} & \textbf{STS-B} & \textbf{SST-2} \\ \midrule
MT-DNN\textsubscript{LARGE}   & 1.26 & 0.86 & 1.05 & 0.42 & 0.26 \\
Ticket Share\textsubscript{LARGE}  & 0.44 & 0.58 & 0.61 & 0.36 & 0.25 \\
\bottomrule
\end{tabular}
\caption{Standard deviation of some tasks in GLUE (dev) over $3$ different learning rates.}
\label{tb:lr_study}
\end{table}

\subsection{Phase Transition on GLUE Tasks}
\label{app:phase-transition}
Figure~\ref{app:phase-transition-fig} shows the phase transition plots on winning tickets on GLUE tasks absent from Figure~\ref{exp:phase_transtion}. All experimental settings conform to Figure~\ref{exp:phase_transtion}.

\begin{figure}[!htb]
    \centering
    \hspace{-0.3in}\includegraphics[width=0.9\linewidth]{Figures/phase_transition_appendix_2.pdf}
	\caption{Single task fine-tuning evaluation results of the winning tickets on the GLUE development set under various sparsity levels.}
	\label{app:phase-transition-fig}
\end{figure}

\begin{table*}[htb]
	\begin{center}
		\begin{tabular}{@{ }l|l|c|c|c|c|c@{ }}
			\toprule
			\bf Corpus &Task& \#Train & \#Dev & \#Test   & \#Label &Metrics\\ \midrule
			\multicolumn{6}{@{\hskip1pt}r@{\hskip1pt}}{Single-Sentence Classification (GLUE)} \\ \hline
			CoLA & Acceptability&8.5k & 1k & 1k & 2 & Matthews corr\\ \hline
			SST & Sentiment&67k & 872 & 1.8k & 2 & Accuracy\\ \midrule
			\multicolumn{6}{@{\hskip1pt}r@{\hskip1pt}}{Pairwise Text Classification (GLUE)} \\ \hline
			MNLI & NLI& 393k& 20k & 20k& 3 & Accuracy\\ \hline
            RTE & NLI &2.5k & 276 & 3k & 2 & Accuracy \\ \hline

			QQP & Paraphrase&364k & 40k & 391k& 2 & Accuracy/F1\\ \hline
            MRPC & Paraphrase &3.7k & 408 & 1.7k& 2&Accuracy/F1\\ \hline
			QNLI & QA/NLI& 108k &5.7k&5.7k&2& Accuracy\\ \midrule
			\multicolumn{6}{@{\hskip1pt}r@{\hskip1pt}}{Text Similarity (GLUE)} \\ \hline
			STS-B & Similarity &7k &1.5k& 1.4k &1 & Pearson/Spearman corr\\ \bottomrule
			\multicolumn{6}{@{\hskip1pt}r@{\hskip1pt}}{Pairwise Text Classification} \\ \hline
			SNLI & NLI& 549k &9.8k&9.8k&3& Accuracy\\ \hline
			SciTail & NLI& 23.5k &1.3k&2.1k&2& Accuracy\\ \bottomrule
		\end{tabular}
	\end{center}
	\caption{Summary of the GLUE benchmark, SNLI and SciTail.}
	\label{tab:glue}
\end{table*}

\section{Background}
\label{sec:related}
We briefly introduce the Transformer architecture and the Lottery Ticket Hypothesis.

\subsection{Transformer Architecture}
The Transformer \citep{vaswani2017attention} encoder is composed of a stack of identical Transformer layers. Each layer consists of a multi-head attention module (MHA) followed by a feed-forward module (FFN), with a residual connection around each. The vanilla single-head attention operates as
\begin{align*}
    \text{Att}(Q, K, V) = \text{Softmax}\left(\frac{QK^{\top}}{\sqrt{d}}\right)V,
\end{align*}
where $Q, K, V \in \RR^{l \times d}$ are $d$-dimensional vector representations of $l$ words in sequences of queries, keys and values. In MHA, the $h$-th attention head is parameterized by $W_h^Q, W_h^K, W_h^V \in \RR^{d \times d_h}$ as
\begin{align*}
    \text{H}_h (\boldsymbol{q}, \boldsymbol{x}, W_h^{\{Q,K,V\}}) = \text{Att}(\boldsymbol{q}W_h^Q,\boldsymbol{x}W_h^K,\boldsymbol{x}W_h^V),
\end{align*}
where $\boldsymbol{q}\in \RR^{l \times d}$ and $\boldsymbol{x}\in \RR^{l \times d}$ are the query and key/value vectors. In MHA, $H$ independently parameterized attention heads are applied in parallel, and the outputs are aggregated by $W_h^{O}\in \RR^{d_h \times d}$:
\begin{align*}
    \text{MHA}(\boldsymbol{q}, \boldsymbol{x})\hspace{-0.05in} = \hspace{-0.06in} \sum_h^{H} \text{H}_h (\boldsymbol{q}, \boldsymbol{x}, W_h^{\{Q,K,V\}}) W_h^{O}.
\end{align*}
Each FFN module contains a two-layer fully connected network. Given the input embedding $\boldsymbol{z}$, we let FFN($\boldsymbol{z}$) denote the output of a FFN module.

\subsection{Structured and Unstructured LTHs}

LTH \citep{frankle2018lottery} has been widely explored in various applications of deep learning \citep{brix2020successfully,movva2020dissecting, girish2020lottery}. Most of existing results focus on finding unstructured winning tickets via iterative magnitude pruning and rewinding in randomly initialized networks \citep{frankle2019stabilizing, renda2020comparing}, where each ticket is a single parameter. Recent works further investigate learning dynamics of the tickets \citep{zhou2019deconstructing, frankle2020early} and efficient methods to identify them \citep{you2019drawing,savarese2019winning}. Besides training from scratch, researchers also explore the existence of winning tickets under transfer learning regimes for over-parametrized pre-trained models across various tasks and datasets \citep{morcos2019one, yu2019playing, desai2019evaluating, chen2020lotteryb}. For example, \citet{chen2020lotterya, prasanna2020bert} have shown the existence of winning tickets when fine-tuning BERT on downstream tasks.

There is also a surge of research exploring whether certain structures, e.g., channels in convolutional layers and attention heads in Transformers, exhibit properties of the lottery tickets. Compared to unstructured tickets, training with structured tickets is memory efficient \citep{cao2019efficient}. \citet{liu2018rethinking, prasanna2020bert} suggest that there is no clear evidence that structured winning tickets exist in randomly initialized or pre-trained weights. \citet{prasanna2020bert} observe that, in highly compressed BERT (e.g., the percent of weight remaining is around $50\%$), all tickets perform equally well. However, \citet{prasanna2020bert} have not investigated the cases where the percent of weight remaining is over $50\%$.
\section{Discussion}
\label{sec:discussion}

\noindent \textbf{Structured Lottery Tickets.} LTH hypothesizes that a subset of unstructured parameters can be trained to match the full model's performance. Instead, we question whether a subset of structured weight matrices, e.g., FFN layers and attention heads, can also be trained to match the full model's performance. This question is more practically important than the unstructured one: training and inference on structured matrices are better optimized for hardware acceleration. Our results give a positive answer to this question, while previous works show that the structured tickets do not exist in highly compressed models \citep{prasanna2020bert}.

\noindent \textbf{Searching Better Generalized Super Tickets.} We select winning tickets according to the sensitivity of the model outputs with respect to the mask variables of each structure \citep{michel2019sixteen, prasanna2020bert}, as this measure is closely tied to the structure's expressive power (Section~\ref{sec:method_st}). In addition, we conduct an one-shot pruning for computational simplicity. We leave other importance measures and pruning schedules, which may help identifying better generalized super tickets, for future works \citep{voita2019analyzing, behnke-heafield-2020-losing, wang2019structured, fan2019reducing, zhou2020scheduled, sajjad2020poor}.

\noindent \textbf{Searching Super Tickets Efficiently.} Determining the compression ratio of the super tickets requires rewinding models at multiple sparsity levels. To leverage super tickets in practice, a potential direction of research is to find heuristics to determine this ratio prior or early-on in training. We leave this for future works.

\section{Conclusion}
\label{sec:conclusion}

We study the behaviors of the structured lottery tickets in pre-trained BERT. We observe that the generalization performance of the winning tickets exhibits a phase transition phenomenon, suggesting pruning can improve generalization when models are lightly compressed. Based on the observation, we further propose a tickets sharing strategy to improve multi-task fine-tuning. Our analysis paves the way for understanding the connection between model compression and generalization.

\section*{Broader Impact}

This paper studies the behavior of the structured lottery tickets in pre-trained language models. Our investigation neither introduces any social/ethical bias to the model nor amplifies any bias in the data. We do not foresee any direct social consequences or ethical issues. Furthermore, our proposed method improves performance through model compression, rendering it energy efficient.
\section{Single Task Experiments}
\label{sec:st_exp}

\subsection{Data}
General Language Understanding Evaluation (GLUE,~\citet{wang2018glue}) is a standard benchmark for evaluating model generalization performance. It contains nine NLU tasks, including question answering, sentiment analysis, text similarity and textual entailment. Details about the benchmark are deferred to Appendix~\ref{app:st-data}.

\newcolumntype{C}{@{\hskip3pt}c@{\hskip3pt}}
\begin{table*}[htb!]
\centering
\small
\begin{tabular}{@{\hskip3pt}l@{\hskip3pt}|C|C|C|C|C|C|C|C|C|C}
\toprule
& \textbf{RTE} & \textbf{MRPC} & \textbf{CoLA} & \textbf{SST} & \textbf{STS-B} & \textbf{QNLI} & \textbf{QQP} & \textbf{MNLI-m/mm} & \textbf{Average} & \textbf{Average} \\
& Acc & Acc/F1 & Mcc & Acc & P/S Corr & Acc & Acc/F1 & Acc & Score & Compression \\ \midrule
ST-DNN\textsubscript{BASE}   & 69.2 & 86.2/90.4 & 57.8 & 92.9 & 89.7/89.2 & 91.2 & 90.9/88.0 & \textbf{84.5}/84.4 & 82.8 & 100\%\\ \midrule
SuperT\textsubscript{BASE}   & \textbf{72.5} & \textbf{87.5/91.1} & \textbf{58.8} & \textbf{93.4} & \textbf{89.8/89.4} & \textbf{91.3} & \textbf{91.3/88.3} & \textbf{84.5/84.5} & \textbf{83.7} & 86.8\%\\ \midrule \midrule
ST-DNN\textsubscript{LARGE}  & 72.1 & 85.2/89.5 & 62.1 & 93.3 & 89.9/89.6 & 92.2 & 91.3/88.4 & 86.2/86.1 & 84.1 & 100\%\\ \midrule
SuperT\textsubscript{LARGE}   & \textbf{74.1} & \textbf{88.0/91.4} & \textbf{64.3} & \textbf{93.9} & \textbf{89.9/89.7} & \textbf{92.4} & \textbf{91.4/88.5} & \textbf{86.5/86.2} & \textbf{85.1} & 81.7\%\\
\bottomrule
\end{tabular}
\caption{Single task fine-tuning evaluation results on the GLUE development set. \textit{ST-DNN} and \textit{SuperT} results are the averaged score over $5$ trails with different random seeds.}
\label{tb:st-results}
\end{table*}

\newcolumntype{C}{@{\hskip3pt}c@{\hskip3pt}}
\begin{table*}[htb!]
\centering
\small
\begin{tabular}{@{\hskip3pt}l@{\hskip3pt}|C|C|C|C|C|C|C|C|C|C}
\toprule
& \textbf{RTE} & \textbf{MRPC} & \textbf{CoLA} & \textbf{SST} & \textbf{STS-B} & \textbf{QNLI} & \textbf{QQP} & \textbf{MNLI-m/mm} & \textbf{Average} & \textbf{Average} \\
& Acc & F1 & Mcc & Acc & S Corr & Acc & F1 & Acc & Score & Compression \\ \midrule
ST-DNN\textsubscript{BASE}    & 66.4 & 88.9   & 52.1 & 93.5 & 85.8    & \textbf{90.5} &  71.2   & \textbf{84.6}/83.4 & 79.6 & 100\%\\ \midrule
SuperT\textsubscript{BASE}    & \textbf{69.6} & \textbf{89.4} & \textbf{54.3} & \textbf{94.1} & \textbf{86.2} & \textbf{90.5} & \textbf{71.3} & \textbf{84.6/83.8} & \textbf{80.4} & 86.8\%\\
\bottomrule
\end{tabular}
\caption{Single task fine-tuning test set results scored using the GLUE evaluation server\footnote{\url{https://gluebenchmark.com/leaderboard}}. Results of \textit{ST-DNN\textsubscript{BASE}} are from \citet{devlin2018bert}.}
\label{tb:st-results-test}
\end{table*}

\subsection{Models \& Training}
\label{sec:st_exp_training}
We fine-tune a pre-trained BERT model with task-specific data to obtain a single task model. We append a task-specific fully-connected layer to BERT as in \citet{devlin2018bert}.

\noindent $\bullet$ $\textbf{ST-DNN\textsubscript{BASE/LARGE}}$ is initialized with BERT-base/large followed by a task-specific layer. \\
\noindent $\bullet$ $\textbf{SuperT\textsubscript{BASE/LARGE}}$ is initialized with the chosen set of super tickets in BERT-base/large followed by a task-specific layer. Specifically, we prune BERT-base/large in unit of $10\%$ heads and $10\%$ feed-forward layers (FFN) at $8$ different sparsity levels (10\% heads and 10\% FFN, 20\% heads and 20\% FFN, etc). Among them, the one with the best rewinding validation result is chosen as the set of super tickets. We randomly sample $10\%$ GLUE development set for tickets selection.

Our implementation is based on the MT-DNN code base\footnote{https://github.com/namisan/mt-dnn}. We use Adamax \citep{kingma2014adam} as our optimizer. We tune the learning rate in $\{5\times10^{-5}, 1\times10^{-4}, 2\times 10^{-4}\}$ and batch size in $\{8, 16, 32\}$. We train for a maximum of $6$ epochs with early-stopping. All training details are summarized in Appendix~\ref{app:st-training}.

\subsection{Generalization of the Super Tickets}
We conduct $5$ trails of pruning and rewinding experiments using different random seeds. Table~\ref{tb:st-results} and~\ref{tb:st-results-test} show the averaged evaluation results on the GLUE development and test sets, respectively. We remark that the gain of SuperT$\textsubscript{BASE/LARGE}$ over ST-DNN$\textsubscript{BASE/LARGE}$ is statistically significant. All the results\footnote{Except for STS-B (SuperT\textsubscript{BASE},  Table~\ref{tb:st-results}), where the p-value is $0.37$.} have passed a paired student t-test with p-values less than $0.05$. More validation statistics are summarized in Appendix~\ref{app:st-val-stats}.

Our results can be summarized as follows.

1) In all the tasks, SuperT consistently achieves better generalization than ST-DNN. The task-averaged improvement is around $0.9$ over ST-DNN$\textsubscript{BASE}$ and $1.0$ over ST-DNN$\textsubscript{LARGE}$.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=1.0\linewidth]{Figures/size_2.pdf}
	\caption{Single task fine-tuning validation results in different GLUE tasks. \textit{Upper}: Performance Gain. \textit{Lower}: Percent of weight remaining.}
	\label{exp:st-results}
\end{figure}

2) Performance gain of the super tickets is more significant in small tasks. For example, in Table~\ref{tb:st-results}, we obtain $3.3$ points gain on RTE (2.5k data), but only $0.4/0.3$ on QQP (364k data) in the SuperT\textsubscript{BASE} experiments.
Furthermore, from Figure~\ref{exp:st-results}, note that the super tickets are more heavily compressed in small tasks, e.g., for SuperT\textsubscript{BASE}, $83\%$ weights remaining for RTE, but $93\%$ for QQP.
These observations suggest that for small tasks, model variance is large, and removing non-expressive tickets reduces variance and improves generalization. For large tasks, model variance is low, and all tickets are expressive to some extent.

3) Performance of the super tickets is related to model size. Switching from SuperT$\textsubscript{BASE}$ to SuperT$\textsubscript{LARGE}$, the percent of weights remaining shrinks uniformly across tasks, yet the generalization gains persist (Figure~\ref{exp:st-results}). This suggests that in large models, more non-expressive tickets can be pruned without performance degradation.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.98\linewidth]{Figures/phase_exp_4.pdf}
	\caption{Single task fine-tuning evaluation results of the winning (blue), the random (orange), and the losing (green) tickets on the GLUE development set under various sparsity levels.}
	\label{exp:phase_transtion}
\end{figure}
\begin{figure}[!htb]
    \includegraphics[width=0.98\linewidth, height=0.44\linewidth]{Figures/phase_split_3.pdf}
    \caption{Phase transition under different randomly sampled training subsets. Note that the settings are the same as Figure~\ref{exp:phase_transtion} (bottom left), except the data size.}
    \label{exp:phase_split}
\end{figure}

\subsection{Phase Transition}

Phase transitions are shown in Figure~\ref{exp:phase_transtion}. We plot the evaluation results of the winning, the random, and the losing tickets under $8$ sparsity levels using BERT-base and BERT-large.
The winning tickets contain structures with the highest importance scores. The losing tickets are selected reversely, i.e., the structures with the lowest importance scores are selected, and high-importance structures are pruned. The random tickets are sampled uniformly across the network. We plot the averaged scores over $5$ trails using different random seeds\footnote{Except for MNLI, where we plot $3$ trails as the there are less variance among trails.}. Phase transitions of all the GLUE tasks are in Appendix~\ref{app:phase-transition}.

We summarize our observations:

\begin{table*}[htb!]
\centering
\small
\begin{tabular}{@{\hskip3pt}l@{\hskip3pt}|C|C|C|C|C|C|C|C|C|C} \toprule
& \textbf{RTE} & \textbf{MRPC} & \textbf{CoLA} & \textbf{SST} & \textbf{STS-B} & \textbf{QNLI} & \textbf{QQP} & \textbf{MNLI-m/mm} & \textbf{Average} & \textbf{Average} \\
& Acc & Acc/F1 & Mcc & Acc & P/S Corr & Acc & Acc/F1 & Acc & Score & Compression \\ \midrule
MT-DNN\textsubscript{BASE}    & 79.0 & 80.6/86.2 & 54.0 & 92.2 & 86.2/86.4 & 90.5 & 90.6/87.4 & 84.6/84.2 & 82.4 & 100\%\\
 + ST Fine-tuning   & 79.1 & 86.8/89.2 & 59.5 & \textbf{93.6} & 90.6/90.4 & 91.0 & \textbf{91.6}/88.6 & \textbf{85.3/85.0} & 84.6 & 100\%\\ \midrule
Ticket-Share\textsubscript{BASE}   & 81.2 & 87.0/90.5 & 52.0 & 92.7 & 87.7/87.5 & 91.0 & 90.7/87.5 & 84.5/84.1 & 83.3 & 92.9\%\\
 + ST Fine-tuning   & \textbf{83.0} & \textbf{89.2/91.6} & \textbf{59.7} & 93.5 & \textbf{91.1/91.0} & \textbf{91.9}  & \textbf{91.6/88.7}  & 85.0/\textbf{85.0}  &  \textbf{85.6} & 92.9\%\\ \midrule \midrule
MT-DNN\textsubscript{LARGE}   & 83.0 & 85.2/89.4 & 56.2 & 93.5 & 87.2/86.9 & 92.2 & 91.2/88.1 & 86.5/86.0 & 84.4 & 100\%\\
 + ST Fine-tuning   & 83.4 & 87.5/91.0 & 63.5 & \textbf{94.3} & 90.7/90.6 & 92.9 & \textbf{91.9/89.2} & \textbf{87.1}/86.7 & 86.4 & 100\% \\ \midrule
Ticket-Share\textsubscript{LARGE}    & 80.5 & 88.4/91.5 & 61.8 & 93.2 & 89.2/89.1 & 92.1 & 91.3/88.4 & 86.7/86.0 & 85.4 & 83.3\%\\
 + ST Fine-tuning   & \textbf{84.5} & \textbf{90.2}/\textbf{92.9} & \textbf{65.0} & 94.1 & \textbf{91.3}/\textbf{91.1} & \textbf{93.0} & \textbf{91.9}/89.1 & 87.0/\textbf{86.8} & \textbf{87.1} & 83.3\%\\ \bottomrule
\end{tabular}
\caption{Multi-task Learning evaluation results on the GLUE development set. Results of \textit{MT-DNN\textsubscript{BASE/LARGE}} with and without ST Fine-tuning are from \citet{liu2020mtmtdnn}.}
\label{tb:mt-results}
\end{table*}

1) The winning tickets are indeed the ``winners''. In Phase I and early Phase II, the winning tickets perform better than the full model and the random tickets. This demonstrates the existence of structured winning tickets in lightly compressed BERT models, which \citet{prasanna2020bert} overlook.

2) Phase transition is pronounced over different tasks and models. Accuracy of the winning tickets increases up till a certain compression ratio (Phase I); Passing the threshold, the accuracy decreases (Phase II), until its value intersects with that of the random tickets (Phase III). Note that Phase III agrees with the observations in \citet{prasanna2020bert}.
Accuracy of the random tickets decreases in each phase. This suggests that model bias increases steadily, since tickets with both low and high expressive power are discarded.
Accuracy of the losing tickets drops significantly even in Phase I, suggesting that model bias increases drastically as highly expressive tickets are pruned.

3) Phase transition is more pronounced in large models and small tasks.
For example, in Figure~\ref{exp:phase_transtion}, the phase transition is more noticeable in BERT-large than in BERT-base, and is more pronounced in RTE (2.5k) and MRPC (3.7k) than in SST (67k) and MNLI (393k).
The phenomenon becomes more significant for the same task when we only use a part of the data, e.g., Figure~\ref{exp:phase_split} vs. Figure~\ref{exp:phase_transtion} (bottom left).

\section{Multi-task Learning Experiments}
\label{sec:mt_exp}

\subsection{Model \& Training}

We adopt the MT-DNN architecture proposed in \citet{liu2020mtmtdnn}. The MT-DNN model consists of a set of task-shared layers followed by a set of task-specific layers. The task-shared layers take in the input sequence embedding, and generate shared semantic representations by optimizing multi-task objectives. Our implementation is based on the MT-DNN code base. We follow the same training settings in \citet{liu2020mtmtdnn} for multi-task learning, and in Section~\ref{sec:st_exp_training} for downstream fine-tuning. More details are summarized in Appendix~\ref{app:mtl-training}.

\noindent $\bullet$ $\textbf{MT-DNN\textsubscript{BASE/LARGE}.}$ An MT-DNN model refined through multi-task learning, with task-shared layers initialized by pre-trained BERT-base/large.\\
\noindent $\bullet$ $\textbf{MT-DNN\textsubscript{BASE/LARGE} + ST Fine-tuning.}$ A single task model obtained by further fine-tuning MT-DNN on an individual downstream task.\\
\noindent $\bullet$ $\textbf{Ticket-Share\textsubscript{BASE/LARGE}.}$ An MT-DNN model refined through the ticket sharing strategy, with task-shared layers initialized by the union of the super tickets in pre-trained BERT-base/large. \\
\noindent $\bullet$ $\textbf{Ticket-Share\textsubscript{BASE/LARGE} + ST Fine-tuning.}$ A fine-tuned single-task \textbf{Ticket-Share} model.

\subsection{Experimental Results}

Table~\ref{tb:mt-results} summarizes experimental results. The fine-tuning results are averaged over $5$ trails using different random seeds. We have several observations:

1) Ticket-Share$\textsubscript{BASE}$ and Ticket-Share$\textsubscript{LARGE}$ achieve $0.9$ and $1.0$ gain in task-average score over MT-DNN$\textsubscript{BASE}$ and MT-DNN$\textsubscript{LARGE}$, respectively.
In some small tasks (RTE, MRPC), Ticket-Share achieves better or on par results compared to MT-DNN+Fine-tuning. This suggests that by balancing the bias and variance for different tasks, the multi-task model's variance is reduced.
In large tasks (QQP, QNLI and MNLI), Ticket-Share behaves equally well with the full model. This is because task-shared information is kept during pruning and still benefits multi-task learning.

2) Ticket-Share$\textsubscript{BASE}$+Fine-tuning and Ticket-Share$\textsubscript{LARGE}$+Fine-tuning achieve $1.0$ and $0.7$ gains in task-average score over MT-DNN$\textsubscript{BASE}$+Fine-tuning and MT-DNN$\textsubscript{LARGE}$+Fine-tuning, respectively. This suggests that reducing the variance in the multi-task model benefits fine-tuning downstream tasks.

\begin{table}[htb!]
\small
\begin{tabular}{lllll}
\toprule
\multicolumn{1}{l|}{Model}                              & 0.1\% & 1\%  & 10\%  & 100\%  \\ \midrule
\multicolumn{5}{c}{\textbf{SNLI} (Dev Acc\%)}                                 \\
\multicolumn{1}{l}{\# Training Data}                     & 549   & 5493 & 54k & 549k \\ \midrule
\multicolumn{1}{l|}{$\text{MNLI-ST-DNN}\textsubscript{BASE}$}    &  82.1  &  85.1    & 88.4      & 90.7   \\
\multicolumn{1}{l|}{$\text{MNLI-SuperT}\textsubscript{BASE}$}    &  \textbf{82.9}  &  \textbf{85.5}    & \textbf{88.8}      & \textbf{91.4}   \\\midrule
\multicolumn{1}{l|}{$\text{MT-DNN}\textsubscript{BASE}$}         & 82.1   & 85.2     & 88.4      & 91.1   \\
\multicolumn{1}{l|}{$\text{Ticket-Share}\textsubscript{BASE}$}   & \textbf{83.3}   & \textbf{85.8}     & \textbf{88.9}      & \textbf{91.5}   \\ \midrule
\multicolumn{5}{c}{\textbf{SciTail} (Dev Acc\%)}                                 \\
\multicolumn{1}{l}{\# Training Data}                    &  23  & 235 & 23k & 235k \\ \midrule
\multicolumn{1}{l|}{$\text{MNLI-ST-DNN}\textsubscript{BASE}$}   &  80.6 & 88.8  & 92.0   &  95.7  \\
\multicolumn{1}{l|}{$\text{MNLI-SuperT}\textsubscript{BASE}$}   &  \textbf{82.9} & \textbf{89.8}  & \textbf{92.8}   &  \textbf{96.2}      \\ \midrule
\multicolumn{1}{l|}{$\text{MT-DNN}\textsubscript{BASE}$}        & 81.9  & 88.3 & 91.1    & 95.7   \\
\multicolumn{1}{l|}{$\text{Ticket-Share}\textsubscript{BASE}$}  &  \textbf{83.1}  &  \textbf{90.1} &  \textbf{93.5}  &   \textbf{96.5}     \\ \bottomrule
\end{tabular}
\caption{Domain adaptation evaluation results on SNLI and SciTail development set. Results of $\textit{MT-DNN\textsubscript{BASE}}$ are from \citet{liu2020mtmtdnn}.}
\label{tb:ood-adapt-results}
\end{table}

\begin{figure*}[!htb]
    \centering
    \includegraphics[width=1.0\textwidth]{Figures/head_2.png}
	\caption{Illustration of tickets importance across tasks. Each ticket is represented by a pie chart. The size of a pie indicates the \textit{Ticket Importance}, where a larger pie suggests the ticket exhibits higher expressivity. Each task is represented by a color. The share of a color indicates the \textit{Task Share}, where a even share suggests the ticket exhibits equal expressivity in all tasks.}
	\label{fig:head_task_analysis}
\end{figure*}

\section{Domain Adaptation}

To demonstrate that super tickets can quickly generalize to new tasks/domains, we conduct few-shot domain adaptation on out-of-domain NLI datasets.

\subsection{Data \& Training}
We briefly introduce the target domain datasets. The data and training details are summarized in Appendix~\ref{app:da-data} and~\ref{app:da-training}, respectively.

\noindent $\textbf{SNLI}.$ The Stanford Natural Language Inference dataset \citep{bowman2015large} is one of the most widely used entailment dataset for NLI.
It contains $570$k sentence pairs, where the premises are drawn from the captions of the Flickr30 corpus and hypotheses are manually annotated.

\noindent $\textbf{SciTail}$ is a textual entailment dataset derived from a science question answering (SciQ) dataset \citep{khot2018scitail}. The hypotheses are created from science questions, rendering SciTail challenging.

\subsection{Experimental Results}

We consider domain adaptation on both single task and multi-task super tickets. Specifically, we adapt $\text{SuperT}_\text{BASE}$ and $\text{ST-DNN}_\text{BASE}$ from MNLI to SNLI/SciTail, and adapt the shared embeddings generated by $\text{Ticket-Share}_\text{BASE}$ and by $\text{MT-DNN}_\text{BASE}$ to SNLI/SciTail. We adapt these models to $0.1\%, 1\%, 10\%$ and $100\%$ SNLI/SciTail training sets\footnote{We use the subsets released in MT-DNN code base.}, and evaluate the transferred models on SNLI/SciTail development sets. Table~\ref{tb:ood-adapt-results} shows the domain adaptation evaluation results. As we can see, SuperT and Ticket-Share can better adapt to SNLI/SciTail than ST-DNN and MT-DNN, especially under the few shot setting.

\section{Introduction}
\label{sec:intro}

The Lottery Ticket Hypothesis (LTH, \citet{frankle2018lottery}) suggests that an over-parameterized network consists of ``lottery tickets'', and training a certain collection of them (i.e., a subnetwork) can 1) match the performance of the full model; and 2) outperform randomly sampled subnetworks of the same size (i.e., ``random tickets''). The existence of such a collection of tickets, which is usually referred to as ``winning tickets'', indicates the potential of training a smaller network to achieve the full model's performance. LTH has been widely explored in across various fields of deep learning \citep{frankle2019stabilizing,zhou2019deconstructing,you2019drawing,brix2020successfully,movva2020dissecting, girish2020lottery}.

\begin{figure*}[!t]
    \centering
    \includegraphics[width=1.0\textwidth]{Figures/phase_comb.pdf}
	\caption{Illustrations of the phase transition phenomenon. \textit{Left}: Generalization performance of the fine-tuned subnetworks (the same as Figure~\ref{exp:phase_transtion} in Section~\ref{sec:st_exp}). \textit{Middle and Right}: An interpretation of bias-variance trade-off.}
	\label{fig:phase_illustration}
\end{figure*}

Aside from training from scratch, such winning tickets have demonstrated their abilities to transfer across tasks and datasets \citep{morcos2019one, yu2019playing, desai2019evaluating, chen2020lotteryb}. In natural language processing, \citet{chen2020lotterya, prasanna2020bert} have shown existence of the winning tickets in pre-trained language models. These tickets can be identified when fine-tuning the pre-trained models on downstream tasks. As the pre-trained models are usually extremely over-parameterized (e.g., BERT~\citet{devlin2018bert}, GPT-3~\citet{brown2020language}, T5~\citet{raffel2019exploring}), previous works mainly focus on searching for a highly compressed subnetwork that matches the performance of the full model. However, behavior of the winning tickets in lightly compressed subnetworks is largely overlooked.

In this paper, we study the behavior of the winning tickets in pre-trained language models, with a particular focus on lightly compressed subnetworks. We observe that generalization performance of the winning tickets selected at appropriate compression ratios can not only match, but also exceed that of the full model. In particular, we observe a \textit{phase transition} phenomenon (Figure~\ref{fig:phase_illustration}): The test accuracy improves as the compression ratio grows until a certain threshold (Phase I); Passing the threshold, the accuracy deteriorates, yet is still better than that of the random tickets (Phase II). In Phase III, where the model is highly compressed, training collapses. We refer to the set of winning tickets selected on that threshold as ``super tickets''.

We interpret the phase transition in the context of trade-offs between model bias and variance \citep[Chapter 7]{friedman2001elements}.
It is well understood that an expressive model induces a small bias, and a large model induces a large variance. We classify the tickets into three categories: non-expressive tickets, lightly expressive tickets, and highly expressive tickets.
The full model has a strong expressive power due to over-parameterization, so that its bias is small. Yet its variance is relatively large.
In Phase I, by removing non-expressive tickets, variance of the selected subnetwork reduces, while model bias remains unchanged and the expressive power sustains. Accordingly, generalization performance improves.
We enter Phase II by further increasing the compression ratio. Here lightly expressive tickets are pruned. Consequently, model variance continues to decrease. However, model bias increases and overturns the benefit of the reduced variance.
Lastly for Phase III, in the highly compressed~region, model bias becomes notoriously large and reduction of the variance pales. As a result, training breaks down and generalization performance drops significantly.

We conduct systematic experiments and analyses to understand the phase transition. Our experiments on multiple natural language understanding (NLU) tasks in the GLUE \citep{wang2018glue} benchmark show that the super tickets can be used to improve single task fine-tuning by $0.9$ points over BERT-base \citep{devlin2018bert} and $1.0$ points over BERT-large, in terms of task-average score.
Moreover, our experiments show that the phase transition phenomenon is task and model dependent. It becomes more pronounced as a larger model is used to fit a task with less training data. In such a case, the set of super tickets forms a compressed network that exhibits a large performance gain.

The existence of super tickets suggests potential benefits to applications, such as Multi-task Learning (MTL). In MTL, different tasks require different capacities to achieve a balance between model bias and variance. However, existing methods do not specifically balance the bias and variance to accommodate each task. In fact, the fine-tuning performance on tasks with a small dataset is very sensitive to randomness. This suggests that model variance in these tasks are high due to over-parameterization. To reduce such variance, we propose a \textit{tickets sharing} strategy. Specifically, for each task, we select a set of super tickets during single task fine-tuning. Then, we adaptively share these super tickets across tasks.

Our experiments show that tickets sharing improves MTL by $0.9$ points over MT-DNN$\textsubscript{BASE}$ \citep{liu2019multi} and $1.0$ points over MT-DNN$\textsubscript{LARGE}$, in terms of task-average score. Tickets sharing further benefits downstream fine-tuning of the multi-task model, and achieves a gain of $1.0$ task-average score. In addition, the multi-task model obtained by such a sharing strategy exhibits lower sensitivity to randomness in downstream fine-tuning tasks, suggesting a reduction in variance.

We summarize our contributions as follows:

$\bullet$ Our result is the first to identify the phase transition phenomenon in pruning large neural language models.

$\bullet$ Our result is the first to show that pruning can improve the generalization when the models are lightly compressed, which has been overlooked by previous works. Our analysis paves the way for understanding the connection between model compression and generalization.

$\bullet$ Motivated by our observed phase transition, we further propose a new pruning approach for multi-task fine-tuning of neural language models.

\section{Multi-task Learning with Tickets Sharing}
\label{sec:method_mt}

In multi-task learning, the shared model is highly over-parameterized to ensure a sufficient capacity for fitting individual tasks. Thus, the multi-task model inevitably exhibits task-dependent redundancy when being adapted to individual tasks. Such redundancy induces a large model variance.

We propose to mitigate the aforementioned model redundancy by identifying task-specific super tickets to accommodate each task's need. Specifically, when viewing an individual task in isolation, the super tickets can tailor the multi-task model to strike an appealing balance between the model bias and variance (recall from Section \ref{sec:method_st} that super tickets retain sufficient expressive power, yet keep the model variance low). Therefore, we expect that deploying super tickets can effectively tame the model redundancy for individual tasks.

Given the super tickets identified by each task, we exploit the multi-task information to reinforce fine-tuning. Specifically, we propose a \textit{tickets sharing} algorithm to update the parameters of the multi-task model: For a certain network structure (e.g., an attention head), if it is identified as super tickets by multiple tasks, then its weights are jointly updated by these tasks; if it is only selected by one specific task, then its weights are updated by that task only; otherwise, its weights are completely pruned. See Figure~\ref{fig:sparse_sharing} for an illustration.

\begin{figure}[!htb]
    \centering
    \includegraphics[width=0.9\linewidth]{Figures/sparse_sharing.png}
	\caption{Illustration of tickets sharing.}
	\label{fig:sparse_sharing}
\end{figure}

In more detail, we denote the weight parameters in the multi-task model as $\theta$. Suppose there are $N$ tasks. For each task $i \in \{1, \dots, N\}$, we denote $\Omega^i = \{\xi^i_{h, \ell}\}_{h=1, \ell=1}^{H, L} \bigcup \{\nu^i_{\ell}\}_{\ell=1}^{L}$ as the collection of the mask variables, where $\ell$ is the layer index and $h$ is the head index. Then the parameters to be updated in task $i$ are denoted as $\theta^i = M(\theta, \Omega^i)$, where $M(\cdot, \Omega^i)$ masks the pruned parameters according to $\Omega^i$. We use stochastic gradient descent-type algorithms to update $\theta^i$. Note that the task-shared and task-specific parameters are encoded by the mask variable $\Omega^i$. The detailed algorithm is given in Algorithm \ref{algo:main}.

Tickets sharing has two major difference compared to \textit{Sparse Sharing} \citep{sun2020learning}: 1) \citet{sun2020learning} share winning tickets, while our strategy focuses on super tickets, which can better generalize and strike a sensible balance between model bias and variance. 2) In tickets sharing, tickets are structured and chosen from pre-trained weight parameters. It does not require {\it Multi-task Warmup}, which is indispensable in \citet{sun2020learning} to stabilize the sharing among unstructured tickets selected from randomly initialized weight parameters.

\begin{algorithm}[!htb]
	\caption{Tickets Sharing}
	\label{algo:main}
	\begin{algorithmic}[1]
		\INPUT Pre-trained base model parameters $\theta$. Number of tasks $N$. Mask variables $\{\Omega^i\}_{i=1}^{N}$. Loss functions $\{\mathcal{L}^i\}_{i=1}^{N}$. Dataset $D = \bigcup_{i=1}^N D_i$. Number of epochs $T_{\max}$.
		\For{ $i$ in $N$}
		\State Initialize the super tickets for task $i$: $\theta^i = M(\theta, \Omega^i)$.
		\EndFor
		\For{epoch in $1,\dots,T_{\max}$}
		\State Shuffle dataset $D$.
    		\For{ a minibatch $b_i$ of task $i$ in $D$}
    		\State Compute Loss $\mathcal{L}^i(\theta^i)$.
    		\State Compute gradient $\nabla_\theta \mathcal{L}^i(\theta^i)$.
    		\State Update $\theta^i$ using SGD-type algorithm.
		\EndFor
		\EndFor
	\end{algorithmic}
\end{algorithm}
\section{Finding Super Tickets}
\label{sec:method_st}

We identify winning tickets in BERT through structured pruning of attention heads and feed-forward layers. Specifically, in each Transformer layer, we associate mask variables $\xi_h$ to each attention head and $\nu$ to the FFN~\cite{prasanna2020bert}:
\begin{gather*}
    \text{MHA}(Q, \boldsymbol{x}) = \sum_h^{H} \xi_h \text{H}_h (Q, \boldsymbol{x}, W_h^{\{Q,K,V\}}) W_h^{O}, \\
    \text{FFN}(\boldsymbol{z}) = \nu \text{FFN}(\boldsymbol{z}).
\end{gather*}
Here, we set $\xi_h, \nu \in \{0,1\}$, and a $0$ value indicates that the corresponding structure is pruned.

We adopt importance score \citep{michel2019sixteen} as a gauge for pruning. In particular, the importance score is defined as the expected sensitivity of the model outputs with respect to the mask variables. Specifically, in each Transformer layer,
\begin{gather*}
    I_{\text{MHA}}^h = \mathop{\EE}_{\boldsymbol{x}\sim \mathcal{D}_x}\left|\frac{\partial \mathcal{L}(\boldsymbol{x})}{\partial \xi_h}\right|, \\
    I_{\text{FFN}} = \mathop{\EE}_{\boldsymbol{x}\sim \mathcal{D}_x}\left|\frac{\partial \mathcal{L}(\boldsymbol{x})}{\partial \nu}\right|,
\end{gather*}
where $\mathcal{L}$ is a loss function and $\mathcal{D}_x$ is the data distribution. In practice, we compute the average over the training set. We apply a layer-wise $\ell_2$ normalization on the importance scores of the attention heads \citep{molchanov2016pruning, michel2019sixteen}.

The importance score is closely tied to expressive power. A low importance score indicates that the corresponding structure only has a small contribution towards the output. Such a structure has low expressive power. On the contrary, a large importance score implies high expressive power.

We compute the importance scores for all the mask variables in a single backward pass at the end of fine-tuning. We perform one-shot pruning of the same percent of heads and feed-forward layers with the lowest importance scores. We conduct pruning multiple times to obtain subnetworks, or winning tickets, at different compression ratios.

We adopt the weight rewinding technique in \citet{renda2020comparing}: We reset the parameters of the winning tickets to their values in the pre-trained weights, and subsequently fine-tune the subnetwork with the original learning rate schedule. The super tickets are selected as the winning tickets with the best rewinding validation performance.

redpajama_set_name: RedPajamaArXiv


Ready for some deliciously fragrant news? Good, us too. It's official: your whole house can smell like wedding cake now thanks to Gibson & Dehn's line of candles and diffusers in Vanilla Chiffon scent.
Assuming you opted for the most popular wedding cake flavor of all time, vanilla, and not a trendy cheese wedding cake, filling your home with the scent of wedded bliss is possibly the most uplifting idea ever. The Gibson & Dehn website describes the scent as, "Vanilla bean and a hint of toasted almond folded into ribbons of whipped sweet cream." Sounds divine, right?
If you're feeling REALLY over the top, you can lay down some cute wedding cake hints for your guests by placing these lit candles around your bridal and engagement parties. Something will smell and taste very, very familiar by the time they arrive at your reception. Even better, there's real science that suggests the scent of vanilla is a major mood enhancer.


Dad's of Disability: A Worthwhile Project — SPORK!
Social work and some of its core principles i.e. social justice and advocacy are close to my heart. SJS appreciates when an individual in the community approaches us regarding an idea or issue that clearly meshes with these values. The interview below is with such an individual, Gary Dietz, who wants to further educate society about father's experiences with their children who experience disabilities. His project has a unique approach in that it will be a book with a collection of personal essays and poems all about the male caregiver's perspective.
Why you have chosen to write this book and why the focus specifically on fathers?
I am a father of a 13 year old boy who has an interstitial deletion of the lower arm of chromosome 13. That means that some of his genetic material is missing between some of his other chromosomes. There's no fancy name for this deletion like there are for more common rare genetic issues. Read that again, it is indeed what I meant — some rare genetic issues are more common than others and the ones that are ultra-uncommon don't usually have names.


Formation of Company / Limited Liability Partnership.
Merger & Acquisition advisory and assistance.
Advising and assisting on IPO clearances, Merchant Banking issues and Underwriting services.
Filing of documents / information to Registrar of Companies (ROC matter).
Preparing of Minutes / Resolutions and other records under the corporate and allied laws.


The name of this post refers to the coast path that we walked on this day – Apprezzami l'Asino. The apocryphal story being that the path is so narrow, if two people were walking their donkeys in opposite directions it would be impossible to get past each other. So they would appraise the value of the donkeys and any goods they were carrying and then the lowest value one would be pushed off the path. With compensation to be paid by the owner of the surviving donkey. I can't imagine this really happening, can you? But it makes for a good story and advance warning about the narrow path.
More about that later. The day started with us waking up at the Lago Sirino sosta to reasonable weather. It had been pretty gloomy when we arrived so it was nice to see some sunshine amongst the swirling clouds. This sosta may be the nicest one we have stayed at in Italy. A well constructed set of half a dozen terraced pitches with electrical supply and water points between every couple of spots. Electricity had a cost of €2 for six hours but otherwise the sosta was free and the facilities clean and all in working order. There was a view over the valley (when we could see it), and from our side window we could see the lake. The road to get here had been a little deformed but it wasn't busy so we could take it easy. We had a quick stroll around the lake before we left, the hamlet looked affluent and well maintained. I can imagine it being a popular spot for a weekend day trip when the weather warms up but today everything was quiet.
After our brief perambulation we set off towards the coast. We were heading for Sapri, a coastal town behind a wide bay on this most southerly part of the Campania coast. We had a couple of parking spots we had eyed up on google. Initially we parked up along the seafront, where the parking was free in low season, but the waves were crashing over the sea wall and the traffic was quite noisy so we moved for the night to park by the football pitch.
We set off to walk the donkey path which started from the harbour. There was no parking here so we had to walk along the seafront and then up the road for a bit until we could get onto the path. We later found that there was parking a couple of km along the path which could be accessed from the SS18.
The initial section of the pathway was well constructed, wide and pretty level. There were informative signs at various spots along the path and bins and cycle racks. It was obviously a popular place for a stroll, and when we were on our way back the small carpark was full of cars, but during the day we had it to ourselves. On the map the path looks very close to the road, however the slope is steep enough that you don't feel it intrudes very often, just the unfortunate rubbish strewn under the laybys where people on the road above had ditched their food and drink containers.
We walked along this easy section of path enjoying the sight of the waves crashing against the rocks and the dart and rustle of the many lizards that were basking in the sunshine. The views south showed headland after headland disappearing into the distance, the stony cliffs contrasting beautifully with the cerulean sea. It was possible to venture closer to the coast at points to watch the waves crashing against the spiny limestone rocks.
After about 3km we reached a promontory with the ruins of a tower. From this point the path suddenly narrowed, it looked precarious where we could see it between the scrub and rocks, but it was worse from a distance than it was to walk. It is a maintained path and there were plenty of places where it had been shored up with stone walls to keep the earth in place and stop it slipping down the cliff face. At one point the path crosses the river that forms the boundary with the region of Basilicata, the river channel was completely dry, like so many rivers in limestone areas it probably only runs when there is significant rainfall. The views continued to delight, the coast becoming more craggy and indented as we progressed.
We didn't manage to reach the end of the path (in fact I wasn't sure where the path did end) because we had to turn around to get back before dusk. We agreed that this was our favourite coast path walk so far. This part of Italy was really ticking the boxes for us and we were looking forward to heading further south.


One of the newest majors at Marietta College, the Land & Energy Management degree helps prepare students to work with oil and gas exploration and extraction companies, government agencies, utility companies, conservation programs, law firms, engineering firms and financial institutions.
"The program takes advantage of our expertise on the energy side and integrates strong core liberal arts components in ethics, energy and business law, communication and negotiation." — Dr. Bob Chase, Chair of Petroleum Engineering & Geology.
One of the top programs in the state of Ohio, the Education major at Marietta College was recognized by the Ohio Board of Regents for ability to prepare students for careers in the classroom. In fact, Marietta boasts a 100-percent placement rate for Education graduates.
The Graphic Design major is for students who want to hone their visual communications skills. After studying various studio art mediums, those concepts are then melded with digital formats. Graduates work in a variety of fields, from print and web media to advertising.
The English Department at Marietta College features an exceptionally talented faculty who help develop your critical thinking and writing skills. Graduates of this program work in a variety of professions, including technical writing, software engineering, public relations and law.
Students interested in Psychology have the option of earning a four-year undergraduate degree or a five-year Bachelor of Arts/Master of Arts Degree in Psychology. The Psychology faculty provide thorough hands-on psychological research opportunities to students, who can also take advantage of the new Behavioral Research Lab and Marietta's Center for Families and Children.
You can earn two degrees from two prestigious institutions with Marietta College's 3-2 Engineering Binary Program. You'll spend your first three years at Marietta College before transferring to Columbia University or Case Western Reserve University. By the end of your undergraduate academic education, you will have earned a Bachelor of Science or Arts from Marietta, and a Bachelor of Science in Engineering from one of the two affiliate universities.
Marietta College's Accounting majors benefit from a talented faculty who are devoted to sharing their knowledge with students so they are well prepared to excel in either the Certified Public Accounting examination or the Certified Management Accounting examination. Marietta's Accounting graduates work in the nation's "Big Four" audit firms, as well as in other leading industries.
Cody Meglio '11 graduated Phi Beta Kappa and salutatorian of his class with Bachelor of Arts degrees in Accounting, Economics and Finance. Within months of graduating, he earned one of the 10 highest scores in the nation on the Uniform CPA Examination. At that time, he was the youngest CPA in the state of Virginia.
No one does History like a liberal arts college, particularly Marietta College. Students who study History at Marietta benefit from talented professors who are available for collaborative research projects and who can prepare students to succeed in either graduate or law school or in a career.
Katherine Tedesco '14 completed an internship as a research intern with the Naval History and Heritage and Heritage Command — Underwater archaeology Branch in the Washington, D.C. Navy Yard.
"After working there, I learned I really like exhibition development. I wasn't able to physically create a display, but I witnessed several displays being formed and presented to the public. I also realized that I want to work in a military museum again. Now If I get to work at the Smithsonian, I won't be upset, but there is something special about military museums and how they connect a veteran to a time in their life or how it helps the survivors of a veteran get a perspective that they couldn't have otherwise." — Katherine Tedesco '14.
Students pursuing a Physics degree at Marietta College benefit from talented professors who provide a curriculum that prepares students for graduate school or for careers in physics-related fields. Our Physics majors benefit from internships in national and international research labs, and have a keen understanding of which field of physics suits their professional interests by the time they graduate from Marietta.
Alexandra Jurgens '15 completed an internship with the Stanford Linear Accelerator Center National Accelerator Laboratory, part of the National Science Foundation's Research Experience for Undergraduates (REU). She appreciates the close-knit feel of the Physics Department.
Students who major in Health Science are prepared to work in a variety of health care professions. Graduates of the program are well equipped for advanced degree training, including fields in physician assistant, physical therapy, nursing and occupational therapy.
If you are still on the fence as to where you'd like to focus your studies, don't worry. Marietta College is a liberal arts college, which means you will be exposed to a variety of subjects throughout your four years here, which means you'll get a taste of what many majors are like. Who knows, maybe you'll choose more than one major, or pick up a couple of minors in the process! Starting your career at Marietta without a major will not set you back, and in fact, will give you a chance to figure out what you want to do … and still be able to graduate in four years. Here is a complete list of our undergraduate degrees.


text:
Q: Algebra, finding the elements of the field and solving irreducible polynomials I'm trying to do this problem from a practice final but there are no solutions.
I honestly am pretty stumped.
My thought was since it has 7 elements, then the degree of the polynomial must be one and we must be working with the field $F_7$
1a) I have no idea what that means, would the squares and cubes just be $0^2, 1^2, 2^2, 3^2 ... , 7^2$ and $0^3, ... ,  7^3$?
1b) My thought is, the field F is 7, so L must be a field with a polynomial of degree 2, but can it just be any polynomial? Don't know how to solve the polynomials $x^2-3$ and $x^2+x-1$

A: The squares in $\mathbb{F}_7$ are indeed
$$
0^2=0,\quad
1^2=1,\quad
2^2=4,\quad
3^2=2,\quad
4^2=2,\quad
5^2=4,\quad
6^2=1,
$$
so you get $\{0,1,2,4\}$.
The cubes are
$$
0^3=0,\quad
1^3=1,\quad
2^3=1,\quad
3^3=6,\quad
4^3=1,\quad
5^3=6,\quad
6^3=6,
$$
so you get $\{0,1,6\}$.
In order to find a field with $7^2$ elements, you just need to find an irreducible degree two polynomial in $\mathbb{F}_7[x]$. For instance $x^2-3$.
The field $\mathbb{L}=\mathbb{F}_7(\sqrt{3})$, where we add a root of $x^2-3$ is, up to isomorphism, the unique field with $49$ elements. In particular, also $x^2+x-1$ must have roots in it. A root must be of the form $a+b\sqrt{3}$:
$$
(a+b\sqrt{3})^2+(a+b\sqrt{3})-1=0
$$
becomes
$$
a^2+3b^2+a-1=0,\qquad 2ab+b=0
$$
so either $b=0$ or $a=-1/2=3$. Since $x^2+x-1=0$ has no roots in $\mathbb{F}_7$, the condition $b=0$ must be discarded and we get
$$
3b^2=-a^2-a+1=-9-3+1=3
$$
so $b=\pm1$. Thus the roots are $3+\sqrt{3}$ and $3-\sqrt{3}$.
You get a field with $343$ elements by adding the roots of a cubic irreducible polynomial, for instance $x^3-2$ (it has no roots, so it's irreducible because it has degree $3$).
There is no root of $x^2-3$ in this field, because this field has no subfield with $49$ elements: a field with $7^m$ elements has a subfield with $7^n$ elements if and only if $n\mid m$.

redpajama_set_name: RedPajamaStackExchange


Spoliation issues are some of the most daunting (and costly) that retailers may face in lawsuits. Klinedinst attorney Fred Heiser tackles the issue, and why retailers need to pay special attention to ensuring they do not run afoul of spoliation issues during litigation.
Most retailers know not to destroy evidence that is relevant to litigation. Yet preserving the right evidence can be a difficult issue to navigate. And a retailer's failure to recognize what evidence to keep, or how to properly do it, can lead to spoliation exposure and penalties.
Spoliation is the destruction or failure to preserve evidence necessary to contemplated or pending litigation. For retailers, this can include evidence like incident reports, video surveillance, or electronically stored information (i.e. e-mails, computer files, USB drives). Sometimes spoliation is intentional. But, more often, it occurs by accident. Either way, courts can impose weighty penalties against a retailer for failing to meet its obligation to maintain the right evidence and information once there is notice of a potential claim.
Although spoliation penalties vary by jurisdiction, they generally range from monetary sanctions to an instruction at trial that allows a jury to assume that missing evidence would have been unfavorable to a retailer that lost it. In the most egregious cases, courts will impose "death penalty" sanctions, striking a defendant's answer and nullifying its defenses, regardless of their merit.
Recognizing when litigation is reasonably foreseeable can be tricky. It occurs when a party should have known that evidence may be relevant to future litigation. Many times, this occurs well before an opposing party files a claim. Retailers should, therefore, be on the lookout for attorney representation letters, evidence preservation requests, or any other practical indication that litigation may be on the horizon. When in doubt, reach out to legal counsel for help.
Once a duty to preserve arises, retailers must maintain all documents and tangible things relevant, or potentially relevant, to a plaintiff's claims. This too can be tricky. It requires an examination of the facts and parties specific to each situation. Err on the side of caution. Retain liberally. This should not be left to chance. Again, when in doubt, work with legal counsel to determine the scope of any necessary evidence retention.
A proactive approach to evidence retention plays a major role in limiting spoliation exposure. To do this, retailers should: (1) implement a clear, concise preservation policy; (2) make sure that their employees understand and follow this policy; and (3) continue to regularly train all employees about the retention policy.
How many copies of evidence to keep.
A policy like this walks employees through what to do when evidence needs to be gathered, it helps ensure that the right evidence is kept, and it makes it more likely that retailers (or their legal counsel) can quickly locate and evaluate this evidence when it is needed months or years later.
Remember: a retention policy loses its impact when employees neglect to follow it, either because they ignore it or fail to recognize what it is. Thus, investing in employee training about a retention policy is critical. This usually comes down to explaining to employees what the policy is and highlighting for them why it is so important. Often, it is beneficial to partner with legal counsel to strategically develop this policy and properly train employees about it. If done right, retailers will be on top of their evidence retention and consistently minimize their spoliation risks.


I am a TalkTalk customer, or at least I was until the 23rd July this year.
I went to use the phone as normal and there was no dial tone, the line was completely dead? "Errm... OK" I thought, so I rang TalkTalk using my mobile phone to report the fault. Bad move, because it took 45 minutes to get through to someone and even then they could barely speak a word of English. I was told it wasn't a fault and my line had been cut off. I asked customer services why. The customer services advisor was as confused as me as to why my line had been cut off. All my payments were on time and my account was up-to-date.
He asked if I had recently moved house or signed up to another provider etc., to which I replied "no". The advisor then said that the account looked like it had been taken over by BT and suggested that I take this up with them. So in total confusion I rang BT and explained my ordeal. They told me to speak to BT wholesale to see why they have taken over my telephone line. They then said I couldn't ring them because I am a residential customer and need to report the fault to TalkTalk and get them to contact wholesale... Well that was enough of that for one day!
The next day I rang back TalkTalk and this time it took one hour to get through to a customer service advisor. I went through the same inconvenient set of questions he must have read from a script and I responded with the same answers as before. I was told they would call be back within 24 hours. Of course they didn't so I rang them again and went through the whole charade again! I rang them five days consecutively with the same questions. And then the inevitable happed...I cracked.. "where the bloody hell is the manager, I want to speak to him NOW!!!" They told me I couldn't speak a head office manager and that I could only contact them by post due to be being a residential customer. I said" How amazing, a telecommunication company that doesn't have a telephone in their head office. Thank God they have a post box.. I JUST WANT MY LINE ON now... It has cost my £25.00 in calling you so far with no one telling me what the hell has happened".
The response I got from that was silence, and then the TalkTalk customer service advisor said he would send me an email to why this has happened and we will get your line back on as soon as possible. I replied ""How the flaming heck do you think I'm going to read an email... link myself directly to the local exchange via brain waves, I'm not from the Matrix you know!" So that was that for another day!
You can probably see my frustration here. I've recently tried to sign up to SKY broadband and telephone, however, I have actually discovered my line has been completed deleted and I'll need to go back to the post office and obtain another number to start again. By the way, TalkTalk have still not called me back and obviously no email... I am at a complete loss to what has happened and have had no apology from TalkTalk.
Can Talk Talk customer service get any worse?


BEST WEDDING PLANNING BOOKS! A very proper choice.
Taking the words of wedding experts with a grain of salt is the only way to survive. That said, there are some great wedding planning books out there full of sage advice. This is probably my favorite. Emily Post's Etiquette 18th Edition This classic guide includes not only information on weddings, but on a myriad … Read moreBEST WEDDING PLANNING BOOKS! A very proper choice.
Arianna can get herself all in a Huff, but Emily will always be the Post with the Most.
Emily would actually have quite liked Arianna. She's spunky, and, we imagine, never dull at a dinner party. The same description applies to Peggy Post, the current Post holding the family post. Peggy has started answering etiquette questions in the New York Times, spicing up the online weddings section. We will be sharing some of … Read moreArianna can get herself all in a Huff, but Emily will always be the Post with the Most.


text:
Welcome to the Resource Blog!
Here you will be able to find communications from the RIM team as well as helpful resources and articles.
RIM Newsletter
Rothman Investment Management 2022 Update
Book Review: Reducing the Risk of Black Swans
January 10, 2023 Book Review
Reducing The Risk of Black Swans looks at how to go beyond just a stock/bond portfolio to reduce risk of major loss by taking exposure to other, uncorrelated risks. Investors would do well to gain exposure to time-tested risk-factors that are relatively uncorrelated to stocks such as value, small companies and momentum.
Which is the better charitable giving technique - QCD or appreciated stock donation?
January 9, 2023 Articles
While both QCDs and donating appreciated assets are helpful, there are numerous considerations in determining which technique to use.
Congress Gifts Secure 2.0
This holiday season Congress has given us SECURE 2.0. With no time to spare to avoid a government shutdown, they passed the $1.7 trillion Consolidated Appropriations Act of 2023 and sent it off the President for signature. Tucked inside the more than 4000 pages of legislation, you can find SECURE 2.0. While this "Son of SECURE" is not exactly the game changer for IRAs that its "Dad" (the original SECURE Act) was, there are still many changes that will affect IRA owners and beneficiaries.
2022 Q3 Market and Economic Update
October 29, 2022 Market Update
A strong summer rally gave some hope that the bear market was over, and we could all get back to making copious amounts of money. The month of September put an end to that hope.


text:
use futures_util::future::{FutureExt as _, TryFutureExt as _};

use grpcio::{
    CertificateRequestType, ChannelBuilder, ChannelCredentialsBuilder, EnvBuilder, RpcContext,
    ServerBuilder, ServerCredentials, ServerCredentialsBuilder, ServerCredentialsFetcher,
    UnarySink,
};
use grpcio_proto::example::helloworld::*;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
use tests_and_examples::util::{read_cert_pair, read_single_crt};

#[derive(Clone)]
struct GreeterService;

impl Greeter for GreeterService {
    fn say_hello(&mut self, ctx: RpcContext<'_>, req: HelloRequest, sink: UnarySink<HelloReply>) {
        let msg = format!("Hello {}", req.get_name());
        let mut resp = HelloReply::default();
        resp.set_message(msg);
        let f = sink
            .success(resp)
            .map_err(move |e| panic!("failed to reply {:?}", e))
            .map(|_| ());
        ctx.spawn(f)
    }
}

struct DataReload {
    switch: Arc<AtomicBool>,
}

impl ServerCredentialsFetcher for DataReload {
    fn fetch(&self) -> Result<Option<ServerCredentialsBuilder>, Box<dyn std::error::Error>> {
        if self.switch.load(Ordering::Relaxed) {
            // The CN field in the certificate of server1 is "*.test.google.fr".
            let root = read_single_crt("ca")?;
            let (server1_crt, server1_key) = read_cert_pair("server1")?;
            let new_cred = ServerCredentialsBuilder::new()
                .add_cert(server1_crt.into(), server1_key.into())
                .root_cert(root, CertificateRequestType::DontRequestClientCertificate);
            Ok(Some(new_cred))
        } else {
            // The CN field in the certificate of server0 is "*.test.google.com.au".
            let root = read_single_crt("ca")?;
            let (server0_crt, server0_key) = read_cert_pair("server0")?;
            let new_cred = ServerCredentialsBuilder::new()
                .add_cert(server0_crt.into(), server0_key.into())
                .root_cert(root, CertificateRequestType::DontRequestClientCertificate);
            Ok(Some(new_cred))
        }
    }
}

struct DataReloadFail {
    initial: AtomicBool,
}

impl ServerCredentialsFetcher for DataReloadFail {
    fn fetch(&self) -> Result<Option<ServerCredentialsBuilder>, Box<dyn std::error::Error>> {
        if self.initial.load(Ordering::Relaxed) {
            // Should return io::Error here.
            Err(Box::new(std::io::Error::new(std::io::ErrorKind::Other, "")))
        } else {
            let root = read_single_crt("ca")?;
            let (server1_crt, server1_key) = read_cert_pair("server1")?;
            let new_cred = ServerCredentialsBuilder::new()
                .add_cert(server1_crt.into(), server1_key.into())
                .root_cert(root, CertificateRequestType::DontRequestClientCertificate);
            self.initial.store(true, Ordering::Relaxed);
            Ok(Some(new_cred))
        }
    }
}

#[test]
fn test_reload_new() {
    let env = Arc::new(EnvBuilder::new().build());
    let service = create_greeter(GreeterService);
    let switch = Arc::new(AtomicBool::new(false));
    let server_creds = ServerCredentials::with_fetcher(
        Box::new(DataReload {
            switch: switch.clone(),
        }),
        CertificateRequestType::DontRequestClientCertificate,
    );
    let mut server = ServerBuilder::new(env.clone())
        .register_service(service)
        .build()
        .unwrap();
    let port = server
        .add_listening_port("127.0.0.1:0", server_creds)
        .unwrap();
    server.start();

    // To connect the server whose CN is "*.test.google.com.au".
    let creds = ChannelCredentialsBuilder::new()
        .root_cert(read_single_crt("ca").unwrap().into())
        .build();
    let ch = ChannelBuilder::new(env.clone())
        .override_ssl_target("rust.test.google.com.au")
        .set_credentials(creds)
        .connect(&format!("127.0.0.1:{port}"));
    let client1 = GreeterClient::new(ch);
    let mut req = HelloRequest::default();
    req.set_name("world".to_owned());
    let reply = client1.say_hello(&req).expect("rpc");
    assert_eq!(reply.get_message(), "Hello world");

    // To connect the server whose CN is "*.test.google.fr".
    switch.store(true, Ordering::Relaxed);
    let creds = ChannelCredentialsBuilder::new()
        .root_cert(read_single_crt("ca").unwrap().into())
        .build();
    let ch = ChannelBuilder::new(env.clone())
        .override_ssl_target("rust.test.google.fr")
        .set_credentials(creds)
        .connect(&format!("127.0.0.1:{}", port.clone()));
    let client2 = GreeterClient::new(ch);
    let mut req = HelloRequest::default();
    req.set_name("world".to_owned());
    let reply = client2.say_hello(&req).expect("rpc");
    assert_eq!(reply.get_message(), "Hello world");

    // Existing connection is still going to work.
    let mut req = HelloRequest::default();
    req.set_name("world".to_owned());
    let reply = client1.say_hello(&req).expect("rpc");
    assert_eq!(reply.get_message(), "Hello world");
}

#[test]
fn test_reload_fail() {
    let env = Arc::new(EnvBuilder::new().build());
    let service = create_greeter(GreeterService);
    let server_creds = ServerCredentials::with_fetcher(
        Box::new(DataReloadFail {
            initial: AtomicBool::new(false),
        }),
        CertificateRequestType::DontRequestClientCertificate,
    );
    let mut server = ServerBuilder::new(env.clone())
        .register_service(service)
        .build()
        .unwrap();
    let port = server
        .add_listening_port("127.0.0.1:0", server_creds)
        .unwrap();
    server.start();

    let creds = ChannelCredentialsBuilder::new()
        .root_cert(read_single_crt("ca").unwrap().into())
        .build();
    let ch = ChannelBuilder::new(env)
        .override_ssl_target("rust.test.google.fr")
        .set_credentials(creds)
        .connect(&format!("127.0.0.1:{port}"));
    let client = GreeterClient::new(ch);

    for _ in 0..10 {
        let mut req = HelloRequest::default();
        req.set_name("world".to_owned());
        let reply = client.say_hello(&req).expect("rpc");
        assert_eq!(reply.get_message(), "Hello world");
    }
}

redpajama_set_name: RedPajamaGithub


text:
Daryl Fishwick
Known for The Full Monty (1997), Hetty Wainthropp Investigates (1996), Bad Girls (1999), The Bad Mother's Handbook (2007)
Married 2 Malcolm (2000)
The Full Monty (1997)
Social Worker Social Worker See fewer
Prey (2015)
(TV Series) - Linda Nutter (2 episodes, 2015) Linda Nutter (2 episodes, 2015) See fewer
Episode #2.3 (Dec 23, 2015) Season 2, Episode 3 - Linda Nutter Linda Nutter See fewer
(TV Series) - Sarah White / Pamela Brown (2 episodes, 2007) Sarah White / Pamela Brown (2 episodes, 2007) See fewer
Ink (Nov 5, 2012) Season 14, Episode 125 - Sarah White Sarah White See fewer
An Inconsolable Loss (Dec 18, 2007) Season 9, Episode 155 - Pamela Brown Pamela Brown See fewer
Leaving (2012)
(TV Mini-series) - Estelle (1 episode, 2012) Estelle (1 episode, 2012) See fewer
Episode #1.1 (Sep 10, 2012) Season 1, Episode 1 - Estelle Estelle See fewer
Hollyoaks (2012)
(TV Series) - Margaret Hayton (9 episodes, 2012) Margaret Hayton (9 episodes, 2012) See fewer
Episode #1.3279 (Feb 16, 2012) Season 1, Episode 3,279 - Margaret Hayton Margaret Hayton See fewer
One Night (2012)
(TV Mini-series) - Vicky (1 episode, 2012) Vicky (1 episode, 2012) See fewer
Alfie (2012) Season 1, Episode 4 - Vicky Vicky See fewer
Exile (2011)
(TV Mini-series) - Wendy (3 episodes, 2011) Wendy (3 episodes, 2011) See fewer
Episode #1.3 (May 3, 2011) Season 1, Episode 3 - Wendy Wendy See fewer
Accused (2010)
(TV Series) - Emma's Mum (1 episode, 2010) Emma's Mum (1 episode, 2010) See fewer
Liam's Story (Dec 6, 2010) Season 1, Episode 4 - Emma's Mum Emma's Mum See fewer
A Passionate Woman (2010)
(TV Mini-series) - Ingrid (1 episode, 2010) Ingrid (1 episode, 2010) See fewer
Episode #1.2 (Apr 18, 2010) Season 1, Episode 2 - Ingrid Ingrid See fewer
Unforgiven (2009)
(TV Mini-series) - Carol Crossland (2 episodes, 2009) Carol Crossland (2 episodes, 2009) See fewer
Episode #1.2 (Jan 19, 2009) Season 1, Episode 2 - Carol Crossland Carol Crossland See fewer
Florence Nightingale (2008)
(TV Movie) - Music Hall Florence Music Hall Florence See fewer
The Royal Today (2008)
(TV Series) - Jeanette Patterson (1 episode, 2008) Jeanette Patterson (1 episode, 2008) See fewer
Episode #1.26 (Feb 11, 2008) Season 1, Episode 26 - Jeanette Patterson Jeanette Patterson See fewer
The Bad Mother's Handbook (2007)
(TV Movie) - Mrs. Carlisle Mrs. Carlisle See fewer
Lilies (2007)
(TV Mini-series) - Matron (1 episode, 2007) Matron (1 episode, 2007) See fewer
The Tallyman (Feb 2, 2007) Season 1, Episode 4 - Matron Matron See fewer
Strictly Confidential (2006)
(TV Series) - Maggie Radcliffe (1 episode, 2006) Maggie Radcliffe (1 episode, 2006) See fewer
Episode #1.5 (Dec 14, 2006) Season 1, Episode 5 - Maggie Radcliffe Maggie Radcliffe See fewer
The Innocence Project (2006)
(TV Series) - Della Doyle (1 episode, 2006) Della Doyle (1 episode, 2006) See fewer
Episode #1.3 (Nov 23, 2006) Season 1, Episode 3 - Della Doyle Della Doyle See fewer
Mysterious Creatures (2006)
(TV Movie) - Nurse Eileen Hume Nurse Eileen Hume See fewer
The Chase (2006)
(TV Series) - Petra Simpson (2 episodes, 2006) Petra Simpson (2 episodes, 2006) See fewer
Episode #1.8 (Sep 3, 2006) Season 1, Episode 8 - Petra Simpson Petra Simpson See fewer
Episode #1.5 (Aug 13, 2006) Season 1, Episode 5 - Petra Simpson Petra Simpson See fewer
Emmerdale Farm (2002–2006)
(TV Series) - Caroline Kershaw / Caroline Addyman (9 episodes, 2002) Caroline Kershaw / Caroline Addyman (9 episodes, 2002) See fewer
Episode #1.4318 (Mar 24, 2006) Season 1, Episode 4,318 - Caroline Kershaw Caroline Kershaw See fewer
Shameless (2005)
(TV Series) - Maureen Hale (1 episode, 2005) Maureen Hale (1 episode, 2005) See fewer
Episode #2.7 (Feb 15, 2005) Season 2, Episode 7 - Maureen Hale (as Darryl Fishwick) Maureen Hale (as Darryl Fishwick) See fewer
The Courtroom (2004)
(TV Series) - Clerk / Court Clerk / Clerk of the Court (7 episodes, 2004) Clerk / Court Clerk / Clerk of the Court (7 episodes, 2004) See fewer
Fists of Furry (Nov 17, 2004) Season 1, Episode 30 - Clerk Clerk See fewer
Clocking Off (2003)
(TV Series) - Faith Elgin (1 episode, 2003) Faith Elgin (1 episode, 2003) See fewer
Pat and Roy's Story (Mar 9, 2003) Season 4, Episode 2 - Faith Elgin Faith Elgin See fewer
Harold Shipman: Doctor Death (2002)
(TV Movie) - WPC at Kathleen Adanski's House WPC at Kathleen Adanski's House See fewer
Holby City (2001)
(TV Series) - Nurse Geraldine Maxwell (1 episode, 2001) Nurse Geraldine Maxwell (1 episode, 2001) See fewer
A Change Is Gonna Come (Apr 10, 2001) Season 3, Episode 22 - Nurse Geraldine Maxwell Nurse Geraldine Maxwell See fewer
Coronation Street (1988–2000)
(TV Series) - Jean Stewart / Lorraine Kinsey / Mandy Turner (4 episodes, 1988) Jean Stewart / Lorraine Kinsey / Mandy Turner (4 episodes, 1988) See fewer
Episode #1.4927 (Nov 12, 2000) Season 1, Episode 4,927 - Jean Stewart Jean Stewart See fewer
Dangerfield (1999)
(TV Series) - Jill Austin (1 episode, 1999) Jill Austin (1 episode, 1999) See fewer
Hidden Desires (Sep 24, 1999) Season 6, Episode 4 - Jill Austin Jill Austin See fewer
Bad Girls (1999)
(TV Series) - Mrs Hicks (2 episodes, 1999) Mrs Hicks (2 episodes, 1999) See fewer
Tangled Web (Jun 29, 1999) Season 1, Episode 5 - Mrs Hicks Mrs Hicks See fewer
The Victim (Jun 22, 1999) Season 1, Episode 4 - Mrs Hicks Mrs Hicks See fewer
(TV Series) - Sally (1 episode, 1998) Sally (1 episode, 1998) See fewer
Episode #1.5 (Dec 13, 1998) Season 1, Episode 5 - Sally Sally See fewer
The Life and Crimes of William Palmer (1998)
(TV Mini-series) - Elizabeth (2 episodes, 1998) Elizabeth (2 episodes, 1998) See fewer
Episode #1.2 (Mar 10, 1998) Season 1, Episode 2 - Elizabeth Elizabeth See fewer
Episode #1.1 (Nov 12, 1998) Season 1, Episode 1 - Elizabeth Elizabeth See fewer
Retrace (1996–1998)
(TV Series) - Miss Watkins (8 episodes, 1996) Miss Watkins (8 episodes, 1996) See fewer
Episode #2.4 (Aug 27, 1998) Season 2, Episode 4 - Miss Watkins Miss Watkins See fewer
The Missing Postman (1997)
(TV Movie) - Barmaid Barmaid See fewer
Hetty Wainthropp Investigates (1996)
(TV Series) - Motel Receptionist (1 episode, 1996) Motel Receptionist (1 episode, 1996) See fewer
Safe as Houses (Feb 7, 1996) Season 1, Episode 6 - Motel Receptionist Motel Receptionist See fewer
Children's Ward (1994)
(TV Series) - Mrs. Mason (1 episode, 1994) Mrs. Mason (1 episode, 1994) See fewer
Episode #7.1 (Oct 11, 1994) Season 7, Episode 1 - Mrs. Mason Mrs. Mason See fewer
Cracker (1993)
(TV Series) - Mrs Forbes (1 episode, 1993) Mrs Forbes (1 episode, 1993) See fewer
The Mad Woman in the Attic: Part 2 (Oct 4, 1993) Season 1, Episode 2 - Mrs Forbes Mrs Forbes See fewer
Coasting (1990)
(TV Series) - Interviewer (1 episode, 1990) Interviewer (1 episode, 1990) See fewer
Offshore (Nov 2, 1990) Season 1, Episode 2 - Interviewer Interviewer See fewer
Screen Two (1990)
(TV Series) - Cafe Waitress (1 episode, 1990) Cafe Waitress (1 episode, 1990) See fewer
The Man from the Pru (Jan 21, 1990) Season 6, Episode 2 - Cafe Waitress (as Darryl Fishwick) Cafe Waitress (as Darryl Fishwick) See fewer


text:
Brandon S. Nexsen
Brandon was born and raised in Norfolk, Virginia. He attended Wake Forest University and graduated with a Bachelor of Arts in History. While at Wake Forest he was a member of the Sigma Alpha Epsilon fraternity and the Phi Alpha Theta history honors society.
Prior to law school, Brandon worked for the Norfolk law firm of Decker, Cardon, Thomas, Weintraub, & Neskis. He attended the University of Richmond School of Law and graduated with his Juris Doctor. During law school, Brandon worked for the Virginia Department of Social Services and the Henrico County Attorney's Office.
Brandon is a Partner in our firm, joining WMMLegal in 2013.
Brandon spends much of his time in the court room – enjoying splitting his practice between criminal and civil litigation. Brandon represents individuals in all criminal arenas, including but not limited to DUI (DWI), drug charges, larceny, possession and other major felonies. He is certified by the Virginia Indigent Defense Commission for felony, misdemeanor, and juvenile criminal cases. Additionally, Brandon is a certified Guardian ad Litem.
He also represents individuals and small businesses looking for resolution in the court room through his civil litigation practice. Brandon takes pride in building long term relationships with his small business clients.
Out of the office Brandon spends his time cooking and playing tennis and golf. He lives in Henrico with wife Eleanor, two daughters, and their dog Walter.
– Wake Forest University, Bachelor of Arts
– University of Richmond, Juris Doctor
– Chesterfield Chamber of Commerce
– Midlothian Business Alliance
– Sigma Alpha Epsilon
– Phi Alpha Theta
– Top 40 Under 40 for Criminal Defense by the National Trial Lawyers
The Corrections by Jonathan Franzen
– Wake Forest Demon Deacons
– Texas Rangers
– Washington Redskins
– Minnesota Vikings


On 23 January 2014, the ALTERRE Mali team inaugurated two Jatropha oil production units in the Yorosso and Koury municipalities, in south-eastern Mali.
With Malian Ministry of Energy representatives presiding over the inauguration, it proved very successful and was greeted with enthusiasm by the technical and financial partners, local authorities and community, who turned out in force to visit the units.
The cotton production area of southeast Mali is facing a rural economic and industrial crisis following the crash affecting cotton, the region's main cash crop. At the same time, unstable hydrocarbon prices have unfortunate repercussions on the cost of energy services using that resource. Rural household budgets are seriously affected.
Consequently, since 2007, GERES has been looking into the feasibility of setting up local biofuel supply chains with a twofold objective: generating a new source of income for local communities and improving their access to energy. This supply chain relies on the production and consumption of Pure Vegetable Oil (PVO) made from Jatropha Curcas, an oilseed plant which grows locally.
As a result, GERES has launched a pilot "action research" project, as a consortium venture with IRAM, in 2 countries: Mali and Benin. The aim is to construct a "replicable scheme" of access to sustainable energy for rural communities in West Africa.
Vous êtes ici : GERES » Inaugurations in Mali : partners hail and encourage the ALTERRE farmer biofuels project!


Lean six sigma is gaining credibility and should be taken seriously by banks who want their process to reach optimal productivity, according to one chief strategy officer.
Bank officers report savings of between 20 percent and 40 percent in their re-engineered loan processing, with 70 percent not out of the question.
Banks can figure out whether they need the improvement methods by recording the time and effort expended on processing every credit. If processing time varies widely from loan to loan, improvement is needed.
Lean Six Sigma would look at each delay and methodically approach the process of "manufacturing" credit to remove variability and waste. If the process suffers from vacations, backlogs or bottlenecks, Nichols wrote, the bank might create centralized "work cell" with cross-trained personnel that can fill in for other members. Loan files might be rotated to other areas or branches.
Many banks spend a lot of time gathering, analyzing and presenting information that doesn't pertain to credit assessments, according to Nichols. Weeding out the waste and restructuring such processes can save a bank 35 percent in processing time.